Deploying SSL keys securely with Ansible

Last summer, Stu wrote a piece about Ansible – a beatiful and simple automation tool with easy to read configuration files. When I started at Red Badger I joined Stu on a project that already had an Ansible setup. Even though I've never really used it before, Ansible was quite easy to pick up and make the necessary changes. But then came the need to deploy SSL certificates and their respective private keys.

The basic setup is easy enough: you need to upload a few files and add a bit of web server configuration (Nginx in this case). The problem is the provisioning configuration is kept in the git repository next to the source code, including all the necessary files and templates, specifically the SSL keys. Yes, the private keys. Not a very secure solution, since a whole lot of people have read access to the repository.

We could use passphrase protected, triple DES encrypted keys, but then someone would have to input the passphrase every time Nginx was started and all the work setting up a continuous delivery pipeline would be lost.

The goal was storing the certificates and keys in the git repository encrypted, upload them, still encrypted, over a secure channel to the server, place them into a tightly restricted directory and only then decrypt them, all that without extra tools.

I found a few Ansible SSL setups online, but none of them did what I wanted. Most just ignored the problem, some didn't, but used tools like git-encrypt to encrypt the keys. So I set out to create my own setup, with blackjack and hoo... security.

OpenSSL to the rescue

As I said, I wanted to use no extra encryption tools, because I'd have to deal with the extra setup and make sure they are actually secure and trustworthy. Fortunately, OpenSSL itself lets you take a passphrase protected key and generate an unprotected one and vice versa.

If you have a key you want to protect with a passphrase, you'd do it like this:

$ openssl rsa -des -in insecure.key -out secure.key

Reverse of that (a non-interactive version) would be

$ openssl rsa -in secure.key -out insecure.key -passin pass:{your_pass_phrase_here}

Great, that should work. We'll just protect the key up front, and then run the second command from Ansible after upload. Right? Well, there is still one big problem: the passphrase. Encrypting the keys is pointless, if you keep the passphrase in clear text in an Ansible playbook right next to the encrypted file.

Fortunately, Ansible has a neat trick up it's sleeve for just this situation: vars_prompt. Ansible lets you use variables inside playbooks and templates to abstract out the bits of configuration that are actually specific to your setup. The variables come from different sources, one of which can be vars_prompt – a special option in a playbook that prompts the user for input before the playbook is run. You can read all about it in Ansible's documentation.

Great! That allows us to store the passphrase in a secure place – the user's head. It means extracting the SSL setup into a separate playbook, so everything else can still be run without knowing the passphrase, but that is just a small price to pay.

The code

The playbook is quite simple

---
- hosts: webservers
  user: ""
  roles:
    - role: https
  vars_prompt:
    - name: ssl_passphrase
      prompt: "Enter SSL Certificate Passphrase"
      private: false

Line by line it reads: apply to hosts in a group webservers, using a user defined in the inventory file, assign these machines a role https and then the aforementioned vars_prompt, which will prompt the user to "Enter SSL Certificate Passphrase" and make the result accessible as ssl_passphrase.

The role has two task files: nginx.yml and ssl.yml. Here is the nginx.yml:

---
- name: Add SSL virtual hosts
  template: src=nginx-ssl-vhost.conf dest=/etc/nginx/sites-available/{{item.hostname}}_ssl
  with_items: ssl_virtual_hosts
  sudo: yes
  notify: restart nginx
  tags: nginx

- name: Enable SSL virtual hosts
  file: state=link
        src=/etc/nginx/sites-available/{{item.hostname}}_ssl
        path=/etc/nginx/sites-enabled/{{item.hostname}}_ssl
        owner=nginx
  with_items: ssl_virtual_hosts
  sudo: yes
  notify: restart nginx
  tags: nginx

First, it uses a template to create a virtual host configuration for each virtualhost in a list ssl_virtual_hosts defined in a group_vars file. Then it enables the created virtualhosts by creating a symlink (which is the way Nginx handles enabling and disabling virtualhosts on Ubuntu).

The relevant bit of the vars file looks something like this:

---
ssl_virtual_hosts:
  - hostname: secure.example.com
    port: 3000
    certificate: /etc/ssl/certs/secure.example.com.pem
    key: /etc/ssl/private/secure.example.com.key

and the configuration template contains the following:

server {
  listen 443;
  server_name {{item.hostname}};

  ssl on;
  ssl_certificate {{item.certificate}};
  ssl_certificate_key {{item.key}};

  location / {
    proxy_pass http://127.0.0.1:{{item.port}};
    proxy_set_header X-Real-IP $remote_addr;
  }
}

The virtual host we've defined is a simple proxy to a (node.js) server running on port 3000.

The ssl.yml is a bit more interesting:

---
- name: ssl-certs group
  group: name=ssl-cert state=present
  sudo: yes
  tags: ssl

- name: Make sure nginx user is in ssl-cert
  user: name=nginx groups=www-data,ssl-cert
  sudo: yes
  tags: nginx

- name: ssl certs dir
  file: path=/etc/ssl/certs mode=755 state=directory owner=root
  sudo: yes
  tags: ssl

- name: ssl private dir
  file: path=/etc/ssl/private mode=700 state=directory owner=root
  sudo: yes
  tags: ssl

- name: copy the certificate
  copy: src={{item.certificate_src}} dest={{item.certificate_dest}} mode=644 group=ssl-cert
  with_items: ssl_certificates
  sudo: yes
  tags: ssl
  notify: restart nginx

- name: copy the key
  copy: src={{item.key_src}} dest={{item.key_dest}} mode=640 group=ssl-cert
  with_items: ssl_certificates
  sudo: yes
  tags: ssl

- name: strip ssl keys
  command: openssl rsa -in {{item.key_dest}} -out {{item.key_stripped}} -passin pass:{{item.key_password}} creates={{item.key_stripped}}
  sudo: yes
  with_items: ssl_certificates
  tags: ssl
  notify: restart nginx

First, we create a group with access to SSL certificates, add the nginx user to the group and make sure permissions are set up correctly on the /etc/ssl/private and /etc/ssl/certs directories. Then we upload the certificate and encrypted key and finally, we strip the key using the command introduced above. For all the steps we use variables defined in a group_vars file:

---
ssl_certificates:
  - certificate_src: secure.example.com.pem
    certificate_dest: /etc/ssl/certs/secure.example.com.pem
    key_src: secure.example.com.protected.key
    key_dest: /etc/ssl/private/secure.example.com.protected.key
    key_stripped: /etc/ssl/private/secure_example.com.key
    key_password: ""

Notice especially the last line, which sets the key_password for this certificate to the prompt variable defined in the playbook.

When you run the playbook by issuing

$ ansible-playbook -i inventory deploy_ssl.yml

Ansible first asks for the SSL password and then sets everything up securely. And that's it!

What first seemed like a problem that will come down to a choice between security and automation ended up being a pretty simple setup that achieves both and all thanks to Ansible.

Taking it one step further, you could set up a key storage server from which the SSL key would get downloaded directly to the production server over SFTP (using a password from a user prompt to access the key storage), preventing a possible brute force attack on the encrypted key stored in git.

I really hope this helps someone strugling with a similar problem and at the same time shows how fun, simple and great Ansible is. It is definitely worth learning. Once you do, automate everything and get better sleep knowing your servers are expendable and your setup is secure.