Backend Web Farm

The web application we’ll build with Fanery is going to run inside a restricted and replicable environment.

Restricted environment

Schroot is a tool that offers many additional functionalities not found in chroot.

Alternatives like LXC or Docker are also valid but we are already running inside a Xen/KVM virtualized environment with more isolation garanties; adding another virtualization layer won’t be beneficial in term of performance or security.

Replicable environment

Our goal here is to build a restricted environment that provides the conditions for:

  • quick and easy replication to several different private virtual machines.
  • quick and easy switching between different versions of the same application.
  • graceful code reload.

Schroot bootstrap script

First let’s create a simple shell script that aid and speed-up the creation of schroot bootstrapped minimal Debian Wheezy environments.

  1. Install required packages.

    aptitude -t wheezy-backports install debootstrap schroot bzip2 cpio xz-utils
    
  2. Change schroot default bind setup.

    Do not share /home and /tmp partitions with main system.

    sed -i -e '/\(^\/home\)/ s//#\1/' -e '/\(^\/tmp\)/ s//#\1/' /etc/schroot/default/fstab
    
  3. Build /sbin/bootstrap-schroot script.

    cat > /sbin/bootstrap-schroot <<EOF
    #!/bin/sh
    
    [ \$# -lt 4 ] && {
        echo "Usage: \$0 SYSTEM VGNAME LVNAME LVSIZE [LVTYPE]"
        exit 1
    }
    
    SYSTEM=\$1
    VGNAME=\$2
    LVNAME=\$3
    LVSIZE=\$4
    LVTYPE=\${5:-xfs}
    
    lvcreate -n \$LVNAME -L\$LVSIZE \$VGNAME && \\
        mkfs.\$LVTYPE -f /dev/\$VGNAME/\$LVNAME && \\
        mount -t \$LVTYPE /dev/\$VGNAME/\$LVNAME /mnt && \\
        debootstrap --include apt-utils,aptitude,locales \\
                    --exclude tasksel,tasksel-data,nano,isc-dhcp-client \\
                    --variant=minbase wheezy /mnt http://ftp.debian.org/debian || \\
        exit 1
    
    grep -v ^mount /sbin/unlock-filesystem > /mnt/sbin/unlock-filesystem
    grep -v ^mount /sbin/lock-filesystem > /mnt/sbin/lock-filesystem
    echo "# schroot nssdatabases
    chattr -i /etc /etc/passwd /etc/shadow /etc/group /etc/gshadow /etc/services /etc/protocols /etc/networks /etc/hosts
    " >> /mnt/sbin/lock-filesystem
    chmod 500 /mnt/sbin/{lock,unlock}-filesystem
    cp -a /sbin/system-upgrade /mnt/sbin/
    cp -a /etc/apt/preferences /mnt/etc/apt/
    grep -v testing /etc/apt/sources.list > /mnt/etc/apt/sources.list
    cp -a /usr/src/libsodium/libsodium*/libsodium*.deb /mnt/usr/src/
    cp -a /etc/skel /mnt/home/operador
    chown -R operador.operador /mnt/home/operador
    
    umount /mnt
    
    cat >> /etc/schroot/schroot.conf <<EOC
    
    [\$SYSTEM]
    type=lvm-snapshot
    users=operador
    root-users=operador
    source-root-users=operador
    device=/dev/\$VGNAME/\$LVNAME
    lvm-snapshot-options=--size 2G
    EOC
    
    schroot -c source:\$SYSTEM -u root --directory /root -- sh -c "
        dpkg-reconfigure locales
        dpkg-reconfigure tzdata
        aptitude update
        aptitude full-upgrade
        aptitude -f install
        apt-get autoremove
        apt-get autoclean
        apt-get clean
        rm -f /etc/ssh_host_*
        rm -rf /var/tmp /tmp/*
        ln -s /tmp /var/tmp
        rm -f /var/log/wtmp /var/log/btmp
        /sbin/lock-filesystem
        history -c"
    EOF
    
    chmod 500 /sbin/bootstrap-schroot
    

Build Fanery compressed archive

Now it’s time to build our fanery node base compressed archive, a single xzipped files that we’ll be able to replicate as many times as required.

  1. Bootstrap Wheezy minbase.

    /sbin/bootstrap-schroot wheezy-fanery VG-NAME LV-NAME 1G
    
  2. Enter schroot in write mode.

    schroot -c source:wheezy-fanery -u root
    
  3. Install packages required to build Fanery dependencies.

    aptitude install build-essential pkg-config graphviz-dev uuid-dev libffi-dev libev-dev python-dev
    dpkg -i /usr/src/libsodium*.deb
    ldconfig -v
    
  4. Install Python virtualenv and wrappers.

    aptitude -t wheezy-backports install python-setuptools git
    easy_install pip
    pip install setuptools --no-use-wheel --upgrade
    pip install virtualenv virtualenvwrapper
    
  5. Setup virtualenvwrappers.

    su - operador
    mkdir ~/.virtualenvs
    cat >> .bashrc <<EOF
    
    VIRTUALENVWRAPPER_PYTHON=$(which python2.7)
    export WORKON_HOME=\$HOME/.virtualenvs
    export PIP_VIRTUALENV_BASE=\$WORKON_HOME
    export PIP_RESPECT_VIRTUALENV=true
    source /usr/local/bin/virtualenvwrapper.sh
    EOF
    . .bashrc
    
  6. Create project virtualenv.

    mkvirtualenv MyProject
    pip install fanery gunicorn rainbow-saddle
    python .virtualenvs/MyProject/lib/python*/site-packages/fanery/tests/test_term.py
    python .virtualenvs/MyProject/lib/python*/site-packages/fanery/tests/test_service.py
    pip install git+https://bitbucket.org/USER-NAME/MyProject.git@v0.0.1
    deactivate
    
  7. Build project start script.

    mkdir ~/bin
    cat > ~/bin/project <<EOF
    #!/bin/sh
    
    [ \$# -lt 2 ] && {
        echo "Usage: \$0 {start|stop|restart|reload} PROJECT"
        echo "           upgrade PROJECT GITREPO"
        exit 1
    }
    
    VIRTUALENVWRAPPER_PYTHON=$(which python2.7)
    export WORKON_HOME=\$HOME/.virtualenvs
    export PIP_VIRTUALENV_BASE=\$WORKON_HOME
    export PIP_RESPECT_VIRTUALENV=true
    source /usr/local/bin/virtualenvwrapper.sh
    
    ACTION=\$1
    shift
    PROJECT=\$1
    shift
    GITREPO=\$1
    PIDFILE=/var/run/\${PROJECT}.pid
    
    case "\${ACTION}" in
        start)
            workon \${PROJECT} && {
                CORES=\$(grep ^processor /proc/cpuinfo | wc -l)
                WORKERS=\$((\${CORES} * 2 + 1))
                rainbow-saddle --pid \${PIDFILE} gunicorn -w \${WORKERS} $@
            }
            ;;
        stop)
            kill -TERM \$(cat \${PIDFILE})
            ;;
        restart|reload)
            kill -HUP \$(cat \${PIDFILE})
            ;;
        upgrade)
            workon \${PROJECT} && \\
                pip install \${GITREPO} --upgrade && \\
                kill -HUP \$(cat \${PIDFILE})
            }
            ;;
    esac
    
    exit \$?
    EOF
    
    chmod 500 ~/bin/project
    
  8. Cleanup.

    exit
    aptitude -f install
    apt-get autoremove
    apt-get autoclean
    apt-get clean
    rm -rf /tmp/*
    rm -f /var/log/wtmp /var/log/btmp
    /sbin/lock-system
    history -c
    
  9. Exit schroot and create compressed archive.

    exit
    mount -t auto /dev/VG-NAME/LV-NAME /mnt
    cd /mnt
    mkdir /var/lib/schroot/archives
    find . -depth -print \
        | cpio --create --quiet --format=crc \
        | xz -6 -Csha256 \
        > /var/lib/schroot/archives/MyProject-node_$(date +%F_%T).cpio.xz
    cd /var/lib/schroot/archives
    umount /mnt
    ls -lh
    
  10. Build schroot node setup script.

    cat > /var/lib/schroot/archives/build-schroot <<EOF
    #!/bin/sh
    
    XZFILE=\$1
    SYSTEM=\$2
    VGNAME=\$3
    LVNAME=\$4
    LVSIZE=\$5
    LVTYPE=\${6:-xfs}
    
    [ \$# -lt 5 ] && {
        echo "Usage: \$0 XZFILE SYSTEM VGNAME LVNAME LVSIZE [LVTYPE]"
        exit 1
    }
    
    lvcreate -n \$LVNAME -L\$LVSIZE \$VGNAME && \\
        mkfs.\$LVTYPE -f /dev/\$VGNAME/\$LVNAME && \\
        mount -t \$LVTYPE /dev/\$VGNAME/\$LVNAME /mnt && \\
        cd /mnt && xzcat \$XZFILE | cpio -d --extract || exit 1
    
    cd /var/lib/schroot/archives
    umount /mnt
    
    /sbin/unlock-filesystem
    
    aptitude update
    aptitude full-upgrade
    aptitude -t wheezy-backports install schroot tar xz-utils
    apt-get autoremove
    apt-get autoclean
    apt-get clean
    
    sed -i -e '/\(^\/home\)/ s//#\1/' -e '/\(^\/tmp\)/ s//#\1/' /etc/schroot/default/fstab
    
    cat >> /etc/schroot/schroot.conf <<EOC
    
    [\$SYSTEM]
    type=lvm-snapshot
    users=operador
    root-users=operador
    source-root-users=operador
    device=/dev/\$VGNAME/\$LVNAME
    lvm-snapshot-options=--size 2G
    EOC
    
    /sbin/lock-filesystem
    EOF
    
    chmod 500 /var/lib/schroot/archives/build-schroot
    

Web farm node setup

We are now ready to replicate our project node in a few simple steps to whatever Debian based hardened virtual machine we selected as member of our Backend Web Farm.

  1. Preparation.

    aptitude -t wheezy-backports install schroot cpio xz-utils
    cd /var/lib/schroot/
    scp -r sshadmin@host:/var/lib/schroot/archives .
    
  2. Build schrooted node from compressed archive.

    cd /var/lib/schroot/archives
    ./build-schroot MyProject-node_VERSION.tar.xz MyProject VG-NAME LV-NAME 10G
    

WebApp management

Once in place our Web application is easily managed via schroot sessions.

  1. Start MyProject daemon.

    schroot -b -n MyProject -c MyProject -u operador
    schroot -r -c MyProject -- /home/operador/bin/project start MyProject myproject:app --log-level debug
    
  2. Project version upgrade and graceful code reload.

    schroot -c source:MyProject -u operador -- /home/operador/bin/project upgrade MyProject git+https://bitbucket.org/USER-NAME/MyProject.git@v0.0.2
    

Read schroot-faq(7) man page for more details about schroot sessions.