Hardened System

Hardened Systems are environments that focus on high security and reliability at all levels they can control.

Let’s start our journey building the basement that will support our elastic Software solution.

Physical environment

The market is rich in alternatives, looking for private cloud virtual machines helps keeping the budget low without sacrifing to much freedom. While choosing you may consider observing the following guidelines:

  • Stick to Tier 3/4 companies that offers multiple datacenter options, in multiple geo-locations/continents.
  • Make sure to get support for backup/snapshots, unmetered private LAN between VMs, HTTP(S) load balancing and truthful statistics about bandwidth usage.
  • Be sure that you get full root access and they let you install customized kernels, or even better, they let you start a fresh minimal install from official netinstall ISO.
  • Use NetCraft data and investigate deeply the Web, looking for satified and disatisfied customers.
  • Prefers proven FLOSS virtualization technologies like KVM or XEN.


From now on installation and configuration steps are presented as shell commands and scripts for the purpose of automation and easy of management.

Intermediate knowledge of Debian GNU/Linux and Shell language is required.

Operating System

The objective here is walking through the hardening process of Debian GNU/Linux 7 (Wheezy).


As starting point this documentation suppose that a fresh new install has been successfully completed following this criterias:

  1. Shadow password: enabled.

  2. Root login: disabled.

  3. Disk partitioning:

    PRIMARY     1GB     ext4            /boot
    PRIMARY     32GB    LVM     vg0
    PRIMARY     *       LVM     vg1
    LogVolume   1GB     xfs     vg0     /
    LogVolume   8GB     xfs     vg0     /usr
    LogVolume   8GB     xfs     vg0     /var
    LogVolume   8GB     xfs     vg0     /var/log
    LogVolume   2GB     xfs     vg0     /tmp
    LogVolume   4GB     swap    vg0
    LogVolume   *       xfs     vg0     /home

    Partitioning is actually a matter of taste, feel free to perform a different style of formatting.

  4. Additional packages: none.

  5. Set man suid: false.

System upgrade

  1. Setup APT mirrors list.

    cat > /etc/apt/sources.list <<EOF
    deb http://ftp.debian.org/debian/ wheezy main
    deb http://security.debian.org/ wheezy/updates main
    deb http://ftp.debian.org/debian/ wheezy-updates main
    deb http://ftp.debian.org/debian/ wheezy-proposed-updates main
    deb http://ftp.debian.org/debian/ wheezy-backports main
    # iptables SYNPROXY target
    deb http://ftp.debian.org/debian/ testing main
    cat >> /etc/apt/preferences << EOF
    Package: *
    Pin: release a=stable
    Pin-Priority: 700
    Package: *
    Pin: release a=wheezy-backports
    Pin-Priority: 650
    Package: *
    Pin: release a=testing
    Pin-Priority: 600
  2. System upgrade.

    aptitude update
    aptitude full-upgrade
  3. Adminitration tools and usefull utils that may not be present after default install.

    aptitude -t wheezy-backports install bzip2 less tmux lsof htop nmon dnsutils iputils-ping telnet-ssl vim-nox wget curl git openssh-client openssh-server sysstat iotop dstat bmon acct strace

Kernel upgrade

  1. Kernel >= 3.12 and iptables >= 1.4.21 are required for SYNPROXY support.

    aptitude -t wheezy-backports install linux-image-amd64 linux-headers-amd64
  2. Reboot with new Kernel.


Firewall setup

  1. Install required packages.

    aptitude -t testing install xtables-addons-dkms iptables netsniff-ng
    aptitude -t wheezy-backports install ca-certificates ethtool sed wget awk ipset ipcalc geoip-database-contrib libtext-csv-xs-perl unzip
  2. Download advanced iptables script with integrated sysctl tuning and hardening.

    wget -c -O /sbin/firewall https://bitbucket.org/mcaramma/linux-setup/raw/master/firewall-synproxy
    chmod 500 /sbin/firewall
  3. Download blocklists, build geoip database and load iptables rules (will take a few minutes).

    /sbin/firewall force-load

The proposed firewall script may seams intimidating at first but it’s actually well organized and self explanatory, please take the time to study its internals and enjoy the simplicity; it shows howto:

  • Blacklist Anonymous Proxies and Satellite Providers + top sources of internet attacks.
  • Blacklist bogon/hijacked/infected/abusive hosts.
  • Discard invalid/unwanted packets.
  • Tarpit/slow-down spammers.
  • Delude PortScan attempts.
  • Prevent ssh brute-force.
  • Mitigate DDoS attacks.
  • Tune and harden TCP/IP Kernel Stack.

It’s also important to stress that the script try hard to create a minimal amount of iptables rules for the job.


DNS attacks can easily turn our hardening efforts useless, dnscrypt is one way to mitigate most common DNS security threats.

  1. Preparation.

    aptitude -t wheezy-backports install build-essential checkinstall wget bzip2
  2. Build libsodium deb package.

    cd /usr/src
    mkdir libsodium && cd libsodium
    wget -c https://github.com/jedisct1/libsodium/releases/download/1.0.0/libsodium-1.0.0.tar.gz
    tar -zxvf libsodium-1.0.0.tar.gz
    cd libsodium-1.0.0
    checkinstall --nodoc
    ldconfig -v
  3. Build dnscrypt-proxy deb package.

    cd /usr/src
    mkdir dnscrypt && cd dnscrypt
    wget -c https://github.com/jedisct1/dnscrypt-proxy/releases/download/1.4.1/dnscrypt-proxy-1.4.1.tar.bz2
    tar -jxvf dnscrypt-proxy-1.4.1.tar.bz2
    cd dnscrypt-proxy-1.4.1
    checkinstall --nodoc
  4. Add dnscrypt user & init script.

    adduser --system --quiet --shell /bin/false --group --disabled-password --disabled-login --home /var/run dnscrypt
    wget -O /etc/default/dnscrypt-proxy -c https://raw.githubusercontent.com/jedisct1/dnscrypt-proxy/master/packages/debian/dnscrypt-proxy.default
    wget -O /etc/init.d/dnscrypt-proxy -c https://raw.githubusercontent.com/jedisct1/dnscrypt-proxy/master/packages/debian/dnscrypt-proxy.init
    sed -i -e '/\(\/usr\)\(\/sbin\/dnscrypt-proxy\)/ s//\1\/local\2/g' /etc/init.d/dnscrypt-proxy
    sed -i -e 's/127\.0\.0\.2/127\.0\.0\.1/g' /etc/default/dnscrypt-proxy
    chmod 550 /etc/init.d/dnscrypt-proxy
    update-rc.d dnscrypt-proxy defaults
  5. Set resolv configuration.

    cat > /etc/resolv.conf <<EOF
    search $(hostname -d)
  6. Test dnscrypt-proxy name resolution.

    /etc/init.d/dnscrypt-proxy start
    nslookup bitbucket.com

Miscelaneous hardening and help scripts

  1. Disable core dumps.

    echo 'fs.suid_dumpable = 0' >> /etc/sysctl.conf
    echo '* hard core 0' >> /etc/security/limits.conf
    echo 'ulimit -S -c 0 > /dev/null 2>&1' >> /etc/profile
  2. Decrease per process/thread stack size

    echo '* soft stack 1024' >> /etc/security/limits.conf
    echo '* hard stack 2048' >> /etc/security/limits.conf
  3. Disable CTRL+ALT+DEL reboot secuence.

    sed -i -e '/^\(ca:.*:ctrlaltdel:.*\)/ s//#\1/' /etc/inittab
  4. Disable crontab for non-root users.

    echo ALL > /etc/cron.deny
    echo root > /etc/cron.allow
    chown root.root /etc/cron.{allow,deny}
    chmod 444 /etc/cron.{allow,deny}
  5. Stop using root.

    # Disable local root login
    aptitude -t wheezy-backports install sudo
    sudo passwd -l root
    # Add user operador
    useradd -p "*" -U -m operador -G sudo
    passwd operador
    chage -M 60 -m 7 -W 7 operador
    # Add user sshadmin (remote login only via SSH key ... remember to copy your key with ssh-copy-id)
    useradd -p "*" -U -m sshadmin
    passwd sshadmin
    # Disable remote root login
    sed -i -e '/^PermitRootLogin .*/ s//PermitRootLogin no\nAllowUsers sshadmin/' /etc/ssh/sshd_config
    sed -i -e '/^#\(Banner .*\)/ s//\1/' /etc/ssh/sshd_config
    service ssh restart
    # Set motd/issue.net banner text
    cat > /etc/motd <<EOF
    Unauthorized access to this machine is prohibited
    Press <Ctrl-D> if you are not an authorized user
    cat /etc/motd > /etc/issue.net
    chown root.root /etc/{motd,issue.net}
    chmod 444 /etc/{motd,issue.net}
  6. Build /sbin/lock-filesystem script.

    cat > /sbin/lock-filesystem <<EOF
    [ -d /var/tmp ] && {
        rm -rf /var/tmp
        ln -s /tmp /var/tmp
    chattr -R +i /boot /usr /bin /sbin /lib* /root /vmlinuz* /initrd* /etc 2> /dev/null
    chattr -R -i /etc/adjtime /etc/blkid.tab /etc/mtab /etc/network/run /etc/udev/rules.d 2> /dev/null
    mount -o nosuid,noexec,nodev,remount /home
    mount -o nosuid,noexec,nodev,remount /tmp
    mount -o ro,nodev,remount /boot
    mount -o ro,nodev,remount /usr
    mount -o nodev,remount /
    chmod 500 /sbin/lock-filesystem
  7. Build /sbin/unlock-filesystem script.

    cat > /sbin/unlock-filesystem <<EOF
    mount -o exec,remount /tmp
    mount -o rw,remount /boot
    mount -o rw,remount /usr
    chattr -R -i /boot /usr /bin /sbin /lib* /root /vmlinuz* /initrd* /etc 2> /dev/null
    chmod 500 /sbin/unlock-filesystem
  8. Build /sbin/system-upgrade script.

    cat > /sbin/system-upgrade <<EOF
    aptitude update && \\
        /sbin/unlock-filesystem && \\
        aptitude \${1:-safe}-upgrade && \\
        aptitude -f install && \\
        apt-get autoremove && \\
        apt-get autoclean && \\
        apt-get clean && \\
    chmod 500 /sbin/system-upgrade
  9. Build /sbin/lock-system script.

    cat > /sbin/lock-system <<EOF
    /sbin/firewall force-load
    chmod 500 /sbin/lock-system
  10. Activate /sbin/lock-system after boot.

    sed -i -e 's/^\(exit 0\)/\/root\/lock-system\n\n\1/' /etc/rc.local
  11. Remove unnecesary packages.

    aptitude purge at nano tasksel tasksel-data task-english

    No other remotely accessible network service should stand active except ssh.

  12. Final clean-up.

    aptitude -f install
    apt-get autoremove
    apt-get autoclean
    apt-get clean
    rm -rf /tmp/*
    rm -f /var/log/wtmp /var/log/btmp
    history -c

Remark note

The proposed process is just the beginning, the first step to system hardening; a lot more can be done to strengh the security of a GNU/Linux system, like using a custom grsecurity patched kernel.

From now on every private virtual machine explained is implied to have gone through all the previously described hardening steps.