Analog

I’ve recently switched most of my home and away infrastructure from Linux to various flavours of BSD. This blog post documents an attempt at deploying multiple static websites in the fastest way possible, on OpenBSD, with Ansible.

It covers automating multiple LetsEncrypt-enabled static websites, configured with onion services for access through the Tor network, using OpenBSD’s brilliant httpd. As a bonus, the content of these websites is automatically generated from cat pictures.

The requirements for this are a public IP address, ability to make DNS records, and of course… some static websites to deploy.

Basics

At the heart of the Ansible configuration for this project is the group of websites to deploy, which are defined as variables in the hosts file:

[www_sites]
site1.example.net
site1.example.com
site2.example.net
site2.example.com

Throughtout the deployment, this group is iterated over.

Generate the websites

First, to deploy multiple websites, one must have multiple websites available - I made a few, out of:

• pictures of several cat friends
• a short bio for each cat friend
• a Jinja2 HTML template for the html index

My website vars group looks like this:

[www_sites]
lion.kitty.institute
leppy.kitty.institute
fox.kitty.institute
jinxy.kitty.institute
kitty.kitty.institute

Directories for each site are iteratively created with an Ansible task:

- name: create vhost directories
  file:
    path: "/var/www/vhosts/{{ item }}"
    state: directory
    owner: www
  with_items: "{{ groups['www_sites'] }}"

The website content, in our case a generated index.html file, is also created by an Ansible task. With each iteration, a variable named ‘vhost’, which corresponds to the name of each site, is passed to the index.html.j2 template:

- name: create mock site content
  template:
    src: "index.html.j2"
    dest: "/var/www/vhosts/{{ item }}/index.html"
    owner: www
  with_items: "{{ groups['www_sites'] }}"
  vars: 
    vhost: "{{ item }}" 

The template is simple html, and uses the ‘vhost’ variable to load the correct bio and picture files. The bios and pictures files are stored in the main Ansible directory, to be used by the template:

<h1>{{ vhost }}</h1>
<hr>
<p>"{{ lookup('file','kitty_bios/' + vhost) }}"</p>
<img src= "{{  vhost }}.jpg" width="500 px"/>

The resulting index files are installed in their own directory, ready to be served.

Deploy the websites

To serve the new websites, a configuration file for httpd must be created. You’ve guessed it - this is another template: httpd.conf.j2.

The example configuration file shipped with OpenBSD is a wonderful reference here, and we want to keep the part that redirects port 80 to port 443 and allows LetsEncrypt verification for ACME challenges, for all the sites (and, aditionally, for your host):

server "{{ inventory_hostname }}" {
        listen on * port 80
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                block return 302 "https://$HTTP_HOST$REQUEST_URI"
        }
}

Then, a loop is added to make configuration blocks for all the servers in the www_sites group: Eventually, all the websites should listen on port 443 and are expected to have TLS certificates - these will be generated in the next step.

{% for vhost in groups['www_sites'] %}
server "{{ vhost }}" {
        listen on * tls port 443
        tls {
                certificate "/etc/ssl/{{ vhost }}.fullchain.pem"
                key "/etc/ssl/private/{{ vhost }}.key"
        }
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                root "/vhosts/{{ vhost }}"
        }
}
{% endfor %}

Letsencrypt the websites

Ok, time to fill /etc/ssl with certificates for these websites. As a cautionary note, httpd must be started before you attempt to request the certificates. This was…the hardest issue to debug in the entire setup.

So, throw in there:

- name: enable and start httpd
  service:
    name: httpd
    enabled: yes
    state: started

For the ACME setup, a configuration file must first be created and installed. Yes, there’s a template for that.

- name: install acme-client.conf
  template:
    src: "acme-client.conf"
    dest: "/etc/acme-client.conf"

The template is based on the default example configuration included with OpenBSD’s acme-client, wrapped in a for loop:

authority letsencrypt {
        api url "https://acme-v02.api.letsencrypt.org/directory"
        account key "/etc/acme/letsencrypt-privkey.pem"
}

{% for d in groups['www_sites'] %}
domain "{{ d }}" {
        domain key "/etc/ssl/private/{{ d }}.key"
        domain full chain certificate "/etc/ssl/{{ d }}.fullchain.pem"
        sign with letsencrypt
}
{% endfor %}

For the ACME run, ensure the names match up those used earlier for the httpd configuration file:

- name: initial acme-client run
  command: "/usr/sbin/acme-client {{ item }}"
  args:
    creates: "/etc/ssl/{{ item }}.fullchain.pem"
  with_items: "{{ groups['www_sites'] }}"
  notify:
  - reload_httpd

A task to renew certs indefinitely via cron jobs finishes the job nicely:

- name: renew certificates via root crontab
  cron:
    name: "acme-client renew {{ item }}"
    minute: "0"
    job: "sleep $((RANDOM \\% 2048)) && acme-client {{ item }} && rcctl reload httpd"
    user: root
  with_items: "{{ groups['www_sites'] }}"

To test, go to your website, which should both redirect to https, and should have a valid cert. In this case, https://fox.kitty.institute.

Onion the websites

The cats now must be made available to those under oppresive regimes or avoiding censorship. An onion address does just that.

Tor must be installed, and the appropriate configuration to enable onion services must be enabled. The torrc file is another template - which basically instructs Tor to create an onion service for each site and store the onion service files in a separate site folder:

Log notice syslog
RunAsDaemon 1
DataDirectory /var/tor
User _tor

{% for domain in groups['www_sites'] %}
HiddenServiceDir /var/tor/{{ domain }}/
HiddenServicePort 80 127.0.0.1:80
HiddenServicePort 443 127.0.0.1:443
{% endfor %}

To install tor, and the configuration file:

- name: install tor
  openbsd_pkg:
    name: ['tor']
    state: present

- name: install torrc
  template:
    src: "torrc"
    dest: "/etc/tor/torrc"
    owner: root
    group: wheel
    mode: 0644
  register: torrc

Note how copying the torrc template registers a variable. This is used later to reload tor if any changes were made to the configuration.

I should point out that this will generate random onion addresses. If you want a custom onion address you can mine it and replace the files in each directory manually.

Remember: the machine running the service must be able to read the private key, so to reduce attack surface the safest way to generate onion addresses is on the very same machine. Software to mine vanity onion addresses can be found here. When copying these, the owners and permissions of the original files need to be kept for tor to run.

My final onion addresses look like this:

meow2ecfmjschzktpwnaufh5m5fop3emrmm2mb62gvawovpiyk7jxdqd.onion
meow3pdf65knffidkkypdgtvohispjpzw6omcyoellwythv64vxo5dqd.onion
meow4u5lkpndb562ble3ityac2l3gm47wegtqp72mq2mfqmved7mnhad.onion
meow5w6vxagd7ipzvmb2h54quuzgwkanu2i3zjnb7qloakmgy3nmrgid.onion
meow6onx3grwas4k2i7lb6iecgyk6fkrypcnsatr2tri6kunv36zksid.onion

Try these in Tor browser!

The step above is entirely optional. However, after copying the torrc configuration file, tor must be running and set to run at startup:

- name: ensure tor is enabled and started
  service:
    name: tor
    enabled: yes
    state: started

- name: reload tor
  service:
    name: tor
    state: reloaded
  when: torrc.changed

The final step is integrating these tasks in with the rest of the configuration, and telling httpd what to do when it receives requests on the onion addresses. The onion address must be known before installing the httpd template - if they are generated by tor then fetching them manually would be a pain. Tor stores the onion address in a file called hostname in /var/tor/. A task can be set up to retrieve these with Ansible:

- name: retrieve onion hostnames
  fetch:
    src: "/var/tor/{{ item }}/hostname"
    dest: "files/onion_hostnames/{{ item }}"
    flat: yes
  with_items: "{{ groups['www_sites'] }}"

Now the files are in place ready to be retrieved by other tasks in Ansible. The httpd template created earlier can now be modified to use the onion addresses:

{% for vhost in groups['www_sites'] %}
server {{ lookup('file', 'onion_hostnames/' + vhost) }} {
        listen on * port 80
        location * {
                root "/vhosts/{{ vhost }}"
        }
}

This will set up listeners on port 80 for each onion, and serve the website content created earlier. You can enable https for these addresses, but as the server name certificate won’t match the onion name, the user will be presented with a warning. Note the new alias line included in the template:

server "{{ vhost }}" {
        listen on * tls port 443
        alias {{ lookup('file', 'onion_hostnames/' + vhost) }}
        tls {
                certificate "/etc/ssl/{{ vhost }}.fullchain.pem"
                key "/etc/ssl/private/{{ vhost }}.key"
        }
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                root "/vhosts/{{ vhost }}"
        }

Lots of things were omitted here for brevity, but the full template is on github.

To finish this off, a picture of a cat friend: