Vyatta VC ISO Build Procedures

    This document describes how to build a Vyatta VC ISO image. The most
    recent version of this document is the top-level README file from 
    the "latest" branch of the "build-iso" git source repository.


Build Host Setup

    The current Vyatta ISO build requires a Debian "squeeze"-based system.
    As of now, "squeeze" is the Debian "stable" release. Therefore, to
    install a minimal squeeze system, download a squeeze installer "netinst"
    or "businesscard" ISO image from the following Debian page:

      http://www.debian.org/releases/squeeze/debian-installer/

    The supported architectures currently include "i386" and "amd64".
    During the installation, choose "Standard system utilities" when
    selecting the software to be installed (note that this is the minimal
    requirements and one can certainly install more packages).

    The Vyatta ISO build requires a file system with at least 10GB free
    space.  
    Allocate a file system of appropriate size, when installing, especially
    if installing a virtual machine based environment.

    For development the build file system should have at least 20G free.

    After the installation and reboot, your "apt" sources.list file should
    be configured to use the Debian mirror that you selected during the
    installation. In order to build the Vyatta packages and the ISO,
    install the following packages using "apt-get" or "aptitude".

      ssh build-essential sudo bzip2 curl autoconf git devscripts \
      debhelper autotools-dev automake libtool bison flex lintian \
      libglib2.0-dev libapt-pkg-dev libboost-filesystem1.42-dev \
      libncurses5-dev libdb-dev libssl-dev cdbs libmozjs-dev \
      libreadline5-dev libpam0g-dev libcap-dev libsnmp-dev gawk unzip \
      kernel-package libatm1-dev git-buildpackage libnfnetlink-dev \
      libnetfilter-conntrack-dev libattr1-dev rsync libxml2-dev \
      libedit-dev libpcap0.8-dev libpci-dev lsb-release quilt ruby \
      genisoimage liblzo2-dev unifont libpopt-dev libgmp3-dev \
      libcurl4-openssl-dev libopensc2-dev libldap2-dev libkrb5-dev \
      hardening-wrapper libgcrypt11-dev libpcre3-dev libprelude-dev \
      libgnutls-dev libperl-dev python-all-dev python-setuptools \
      live-helper syslinux libsort-versions-perl libexpat1-dev \
      libfile-sync-perl gcc-multilib libfreetype6-dev libusb-dev \
      libdevmapper-dev libmysqlclient-dev autogen libdumbnet-dev

    Now install the Vyatta package repository GPG key.

      wget -O vyatta-pubkey.gpg \
        http://packages.vyatta.com/vyatta-dev/vyatta-pubkey.gpg
      sudo apt-key add vyatta-pubkey.gpg

Tools update:
    You will also need to download and install a newer "squashfs-tools"
    package from the squeeze-backports repository. Vyatta uses extended
    file attributes, and older (pre 4.2) versions do not handle attributes.

    Debian instructions on how to setup squeeze-backports.
    Add backports to your sources.list
    1. Add this line
       deb http://backports.debian.org/debian-backports squeeze-backports main
    to your sources.list (or add a new file with the ".list" extension to /etc/apt/sources.list.d/)
    2. Run apt-get update

    After that do:
      sudo apt-get -t squeeze-backports install squashfs-tools

Additional libraries:
    You will also need to download and install libnetfilter-conntrack3_1.0.0 and
    libnetfilter-conntrack-dev_1.0.0 packages:

        PKGS="libmnl0_1.0.3-5_amd64.deb libnetfilter-conntrack3_1.0.0-1+vyatta15_amd64.deb libnetfilter-conntrack-dev_1.0.0-1+vyatta15_amd64.deb"
        for p in $PKGS; do
          wget http://packages.vyatta.com/vyatta-dev/pacifica/unstable/pool/main/$p
        done
        dpkg -i $PKGS

Getting "build-iso" Source

    The Vyatta ISO builder consists of scripts and config files
    maintained as a git repository named "build-iso". First, clone
    the build-iso git repository.

	git clone http://git.vyatta.com/build-iso.git

    This will create the source tree in the directory "build-iso".
    (At this point, you might want to switch to reading
    ./build-iso/README, which is the most recent version
    of the file which you are reading now:-)

    Our git repositories have branches, which usually correspond to
    different releases. To switch to a particular branch (let's call
    it BRANCH), do the following.

	cd build-iso
	git checkout --track -b BRANCH origin/BRANCH


Build ISO

    To build a Vyatta ISO, do the following in the "build-iso" directory
    (which has been set up as described in the previous section).

	export PATH=/sbin:/usr/sbin:$PATH
	autoreconf -i
	./configure
        sudo make iso

    The end results is the following livecd ISO image.

	livecd/binary.iso

    Note that the above build process uses the pre-built binary packages
    from the Vyatta "unstable" package repository, which contains the
    latest development packages updated in real time (i.e., they are the
    exact same packages used in Vyatta internal development build).

    If you want to make modifications to one or more packages, you will
    need to get the source of the submodule(s) that you want to change,
    make the changes, build the submodules, and then build a new ISO to
    include the new packages. This is discussed in the next section.


Build Host Setup for Building Submodules

    Before building any submodules, you will need to install the following
    Vyatta-specific packages on the build host.

    From "Debian lenny" install the following packages:
      libnet0 libnet0-dev libsablot0 libsablot0-dev

    The Debian lenny packages are available from any Debian mirrors. For
    example, the amd64 "libnet0" package can be found at the following URL:

      http://packages.vyatta.com/debian/pool/main/libn/libnet0/libnet0_1.0.2a-7_amd64.deb

    From "Debian wheezy" install the following packages (similarly, they are
    available from Debian mirrors):
      libmnl0 libmnl-dev

    From Vyatta "unstable" package repository install the following packages:
      iptables iptables-dev linux-vyatta-kbuild
      libsnmp15 libsnmp-dev libsnmp-perl libsnmp-base
      libcap-dev libcap2 libvyatta-cfg-dev libvyatta-cfg1
      libdaq0 libdaq-dev
     
    The location of the Vyatta "unstable" package repository depends on the
    source branch. For example, for the "pacifica" branch, the amd64
    "iptables" package in the "unstable" package repository can be downloaded
    from the following URL (note that the unstable package repository is
    updated in real time, so latest version may be different from the version
    below):

      http://packages.vyatta.com/vyatta-dev/pacifica/unstable/pool/main/iptables_1.4.10+vyatta+5_amd64.deb

    After downloading the packages listed above, they can be installed
    using "dpkg".

    Then put the following in "/etc/apt/preferences" to "pin" the version of
    the libcap-dev and libcap2 package (which ensures that the Vyatta package
    will be used instead of the one from Debian).

      cat >| /etc/apt/preferences.d/libcap-dev <<EOF
Package: libcap-dev
Pin: version 2.*
Pin-Priority: 1001
EOF

      cat >| /etc/apt/preferences.d/libcap2 <<EOF
Package: libcap2
Pin: version 2.*
Pin-Priority: 1001
EOF
    

Getting Submodules Source and Building

    Note that the "build-iso" git repository contains a number of
    submodules in the "pkgs/" subdirectory. To get the source for a
    submodule, do the following in the "build-iso" directory (replace
    "SUBMODULE" with the name of the module, e.g., "vyatta-cfg-quagga").

      git submodule init
      git submodule update pkgs/SUBMODULE

    The above will clone the source of the submodule into the
    "pkgs/SUBMODULE" directory. Note that the submodule needs to be on
    the same branch as "build-iso", so do the following (replace
    "BRANCH" with the branch name you used for build-iso).

      cd pkgs/SUBMODULE
      git checkout --track -b BRANCH origin/BRANCH

    In most cases, you will only update (i.e., clone) the submodule(s)
    that you are going to change. If you want to clone all submodules,
    do the following.

      git submodule update

    Remember to create and checkout the same working branch for all
    submodules.


Build submodules

    When you do a "make", the build system will build the binary packages
    for all cloned submodules and include them in the ISO (pre-built
    packages from the Vyatta "unstable" package repository will be used
    for the submodules that are not cloned). If you want to work with
    specific submodules, you can use the commands listed below.

    Using make targets

      There are a number of top-level make targets for working with
      submodules. The commands below use submodule "vyatta-cfg" as an
      example.

      To build ".deb" packages for the vyatta-cfg submodule (equivalent to
      "debuild -i -b -uc -us -nc"):

        make vyatta-cfg

      To clean the build for the vyatta-cfg submodule (equivalent to
      "debuild clean"):

        make clean-vyatta-cfg

      To do a "release" for the vyatta-cfg submodule (equivalent to
      "pkg-release -p", i.e., auto-generate everything and no confirmation):

        make release-vyatta-cfg

    Using submod-mk script

      The submod-mk script in tools/ can also be used to work with submodules.
      To build binary packages for cloned submodule(s),

        tools/submod-mk { SUBMODULE... }

      To build binary and source packages for submodule(s),

        tools/submod-mk --source { SUBMODULE... }

      To build signed binary and source packages for submodule(s),

        tools/submod-mk --signed-source { SUBMODULE... }

      To clean files generated during submodule build,

        tools/submod-clean SUBMODULE...

      To clean files generated during all submodule builds,

        tools/submod-clean

      To also clean the submodule build stage(s),

        tools/submod-clean -d [ SUBMODULE... ]

      To clean the source tree of submodules to skip subsequent builds,

        tools/submod-clean -g [ SUBMODULE... ]

      This retains the local bare git repository for quicker updates.

Changes

    After making, testing, and committing your changes to your local
    repositories, publish them to the shared repository with,

	git push git.vyatta.com:/git/SUBMODULE.git

    This will, of course, require an account on git.vyatta.com, which
    is only available to authorized vyatta developers.
    Then update and publish the super-module index to include the
    submodule changes,

	cd ../..
	git update-index pkgs/SUBMODULE
	git commit -m 'update sub-module SUBMODULE'
	git push git.vyatta.com:/git/build-iso.git

Updates using git

    Update the super-module.

	git pull
	git submodule init

    To update all modules, you can use:

	tools/submod-git pull

    Or for the paranoid

	tools/submod-git fetch
	git whatchanged HEAD..origin/master
	git merge

    To see which checked-out submodules have upstream modifications,

	git status		# search for changes in pkgs/

    Get the "from" and "to" commit ids (SHAs) with,

	git diff pkgs/SUBMODULE

    This will produce output something like,

	diff --git a/pkgs/vyatta-cfg-quagga b/pkgs/vyatta-cfg-quagga
	index 78124a9..fca1739 160000
	--- a/pkgs/vyatta-cfg-quagga
	+++ b/pkgs/vyatta-cfg-quagga
	@@ -1 +1 @@
	-Subproject commit 78124a9754efe0bda1d9c827c593c5e0635fec3c
	+Subproject commit fca1739f1985c2ba8d6316b1882365f5e26245cd

    To review what changed between these SHAs,

	cd pkgs/SUBMODULE
	git whatchanged FROM_SHA TO_SHA

    To merge these submodule changes,

	git merge
	cd ../..
	git submodule update

    The "git status" for this submodule should now be clean. Iterate
    through the other updated submodules, then rebuild the ISO.