Entropy on Cloud VPS

On-Demand Random Numbers: /dev/random

Linux servers expose a source of randomness via two devices found at: /dev/random and /dev/urandom You can grab some random numbers from /dev/urandom yourself, try running the following:

od -vAn -N4 -tu4 < /dev/urandom

This command is reading from urandom and converting the random bytestream into characters for the terminal. The arguments to the od command just specify the format of the input and tell it to stop reading after 4 bytes, check it out on explainshell for a more detailed explanation.

A steady supply of unpredictable random numbers is a critical requirement for the sercurity and performance of a server. They are often used as seeds to initialize other random number generators (RNG), which are in turn used for many cryptographic purpsoes like generating SSH and PGP keys.

If you're running a web server with TLS or SSL enabled they are also likely used during the handshakes that take place between your web server and a visitor to your site at the begining of a session.
Random numbers are also the basis for many other security related functions of applications like producing csrf tokens and encrypting sensitive data.

This post details how to ensure your linux host, especially if it's a virtual machine, starts with an adequate pool of entropy and continues to maintain it. If the random numbers feeding the processes above become not so random, bad things can happen: your web server can block and wait until entropy improves (apache1), or worse, it can silently compromise the integrety of keys you generate with tools like OpenSSL by making them predictable (as it did on Debian described here and here).

What is Entropy?

One way to think of the entropy pool and /dev/[u]random on your server is as a deck of playing cards. Picking cards off the top of the deck is like reading from /dev/[u]random and will provide a stream of random selections. /dev/[u]random provides bytes that can be converted to numbers or characters. Depending on how well the deck has been shuffled it will be very hard to predict which card you will see next. "Entropy" is a way to quantify how well-shuffled the deck currently is.

If code reads from /dev/random when the server's entropy is low the call will block until entropy has improved and that could take a while, especially for a headless virtual server with low traffic. Code making use of /dev/random ideally values the result being unpredictable more than always getting a result fast. This can be a big problem if the code needs to be perfomant though, like serving a web page to a visitor.

When speed is prioritized above unpredictability garuntees you can read from /dev/urandom. If you try and read from it when system entropy is low it will still give a result immediately but is not garunteed to be as unpredictable. There is debate, but a general concensus seems to be that as long as you seed the pool with quality entropy early enough it's perfectly fine to pull from the non-blocking /dev/urandom, even during times of low system entropy.

Ensuring Minimum Entropy

Linux provides a way to estimate how much entropy your server currently has available:

cat /proc/sys/kernel/random/entropy_avail

The units of it's output are in bits. If this prints out a number above 1000 bits you're generally looking good.

Normally a linux host will keep the entropy pool high by using keyboard and mouse input, just like when one of those password managers asks you to move the mouse around and type on the keyboard before producing a random password for you. They also get entropy from other hardware, like hard drives, CPU thermal sensors, and network I/O as well as other sources.

Virtual machines have issues with the above sources of random input because they lack keyboard and mouse interrupts and may have some or all of the hardware virtualized, making it more predictable. This is especially problematic if you are hosting a web app with TLS everywhere and it doesn't receive much traffic that would help the machine generate additional entropy.

Bottom-line

There's a general concensus that if your RNG is seeded well, you should read from the non-blocking /dev/urandom. Ensuring the kernel is seeded with quality entropy when your virtual machine is started is quite easy, see below. For software on your server that uses /dev/random you can keep the entropy-pool full by installing haveged. While installing haveged on ec2 instances should be okay, other cloud services or hypervisors may have virtualized some of haveged's dependencies (rdtsc) and you should look into something like virtio-rng which passes entropy from the physical machine running the VM to the VM, but it may not be available for all hypervisors/cloud providers.

Fixing Initial Seeds

Passing in quality random data during instance creation.

Method 1, via ansible script

- name: Provision a new dash instance
  hosts: localhost
  connection: local
  gather_facts: False

  vars:
    ssh_known_hosts_file: "{{ lookup('env','HOME') + '/.ssh/known_hosts' }}"
    # Set seed from local machine urandom on OSX
    SEED: "{{ lookup('pipe','head -c 512 /dev/urandom | openssl base64 -e -A') }}"

  tasks:
  - name: Provision EC2 dash instance
    ec2:
      key_name: my_AWS_key
      group_id: sg-0f000000
      instance_type: t2.micro
      region: us-west-2
      image: ami-8936e0e9  # ubuntu 16
      wait: yes
      wait_timeout: 500
      count: 1
      user_data: |
                 #cloud-config
                 random_seed:
                     file: /dev/urandom
                     data: {{ SEED }}
                     encoding: b64
      instance_tags:
        temp: true
      vpc_subnet_id: subnet-e0000000
      assign_public_ip: yes
    register: created_instance

The SEED variable in the scipt above is set with a command specific to OSX. If you're running the ansible script on another platform change it to one of the below:

# Set seed from local machine urandom on Linux
SEED: "{{ lookup('pipe','head -c 512 /dev/urandom | base64 -w 0') }}"

# Set seed from local machine cross-platform via Python (Windows included)
SEED: "{{ lookup('pipe','python -c \"import os,base64;print(base64.b64encode(os.urandom(8)))\"') }}"
# OR
SEED: "{{ lookup('pipe','python3 -c \"import os,base64,sys;sys.stdout.buffer.write(base64.b64encode(os.urandom(8)))\"') }}"

Method 2, Push some randomness to the server after its already running

From linux

dd if=/dev/urandom bs=20 count=1 | ssh TheEC2MachineName 'cat > /dev/random'

From OSX

head -c 512 /dev/urandom | openssl base64 -e -A | ssh TheEC2MachineName 'cat > /dev/random'

From Windows (with python 2.7.X installed)

python -c "import os,base64;print(base64.b64encode(os.urandom(8)))" | ssh TheEC2MachineName 'cat > /dev/random'

Note that you'll want to do this as soon as possible, and definitely before generating any additional keys or seeding important random number generators on the server instance. Also, the instance will have already generated it's host keys via cloud-init, making this the sub-optimal method.

Installing haveged

These are instructions for installing on Ubuntu LTS 16, if you need instructions for another OS there are plenty available on the web. At the time I am writing this I didn't find any for Ubuntu 16, so:

via shell

apt-get update
apt-get install haveged apparmor-utils
sed -i.bak 's/^DAEMON_ARGS=.*/DAEMON_ARGS="-w 4096"/' /etc/default/haveged
aa-complain /usr/sbin/haveged
service haveged start
update-rc.d haveged defaults

After updating and installing haveged the config file is edited with sed to raise the minimum amount of entropy haveged should try and maintain. Next, a fix to app-armor that would otherwise prevent the haveged service from starting by setting app-armor to 'warn' for haveged only. Then the service is started and added to the list of services that start when the system starts.

via ansible

- name: Configure new dash instance
  hosts: created_instance
  remote_user: ubuntu
  strategy: debug
  gather_facts: True

  tasks:
    - name: Install required ubuntu packages
      become: True
      apt: name="{{item}}" state=installed update_cache=yes cache_valid_time=3600
      with_items:
        - haveged
        - rng-tools
        - apparmor-utils
    - name: "Configure haveged to ensure entropy"
      become: True
      #command: sed -i.bak 's/^DAEMON_ARGS=.*/DAEMON_ARGS="-w 4096"/' /etc/default/haveged
      replace:
        dest=/etc/default/haveged
        regexp='^DAEMON_ARGS=.*'
        replace='DAEMON_ARGS="-w 4096"'
      notify:
        - Restart server
        - Wait for server to restart

    # trigger restart handlers before running next task
    - meta: flush_handlers

    - name: "Change apparmor profile of haveged to 'complain'"
      become: True
      command: aa-complain /usr/sbin/haveged

    - name: "Start haveged service"
      become: True
      service: name=haveged state=started

    - name: "Start haveged when server starts"
      become: True
      command: update-rc.d haveged defaults

  handlers:

    - name: Restart server
      become: True
      shell: "sleep 2 && shutdown -r now"
      async: 1
      poll: 0
      ignore_errors: true

    - name: Wait for server to restart
      become: false
      local_action:
        module: wait_for
          host="{{ inventory_hostname }}"
          port=22
          delay=30
          timeout=500
          state=started

References

Why do HTTPS connections to my server sometimes take up to 30 seconds to establish a connection?