Analog

/pubs

Sat, 29 Aug 2020 00:00:00 +0000

Impact of Acknowledgements using IETF QUIC on Satellite Performance. Ana Custura, Tom Jones, Gorry Fairhurst. ASMS/SPSC 2020: 1-8

Rethinking ACKs at the Transport Layer. Ana Custura, Tom Jones, Gorry Fairhurst. FIT 2020: 731-736

Measuring mobile performance in the Tor network with OnionPerf. Ana Custura, Iain Learmonth, Gorry Fairhurst. MNM 2019: 233-238

Exploring Usable Path MTU in the Internet. Ana Custura, Gorry Fairhurst, Iain Learmonth. Traffic Measurement Analysis 2018: 1-8

Experience: Implications of Roaming in Europe. Anna Maria Mandalari, Andra Lutu, Ana Custura, Ali Safari Khatouni, Özgü Alay, Marcelo Bagnulo, Vaibhav Bajpai, Anna Brunström, Jörg Ott, Marco Mellia, Gorry Fairhurst. MobiCom 2018: 179-189

Exploring DSCP modification pathologies in the Internet. Ana Custura, Raffaello Secchi, Gorry Fairhurst. Comput. Commun. 127: 86-94 (2018)

Exploring DSCP modification pathologies in mobile edge networks. Ana Custura, Andre Venne, Gorry Fairhurst. MNM 2017: 1-6

For sale

Sat, 29 Aug 2020 00:00:00 +0000

VX-8E VHF Amateur Handheld Transciever for sale

TX on 70cm, 2M and 6M.
Built-in packet modem 1200/9600bd with APRS support
Comes with mounted GPS antenna, charger, 3100mAh battery, extra clip and original battery pack
5W power output
Modes: TX FM, RX FM/AM/WFM
Full specs: http://rigpix.com/yaesu/vx8e.htm

Asking price for radio, accessories and postage is £180. Contact ana@netstat.org.uk.

/about

Sat, 23 Sep 2017 00:00:00 +0000

I do Internet research and sometimes interesting things with computers.

My interests include network security, network measurements and transparency, privacy and digital freedom, amateur radio and electronics. I am also a fan of Free Software and contribute to Debian and Tor Project in my spare time.

You can drop me a GPG-encrypted email at ana@netstat.org.uk, my fingerprint is:

85E7 AAB8 C29D F383 4940 5DC0 6F8D E44D 59D7 DBCC.

A quick look at the QUIC software ecosystem

Tue, 16 Mar 2021 00:00:00 +0000

As far as transport protocols go, TCP is the most used today, and when it comes to TCP implementations, there are as many TCP stacks as there are networked operating systems. This includes everything from Windows PCs to Android Phones and Cisco routers. In most cases, an update to TCP requires a firmware change or OS update - and all endpoints, routers and network appliances may need to deploy a feature for it to work.

Not the case with QUIC, an encrypted transport layer protocol which runs on top of UDP. As QUIC implementations can live in userspace, development can happen separate from the OS. This means control over transport features shifts away from network vendors and operating system implementors to application service providers - enterprises who run the web, like Google and Facebook.

The IETF standards for HTTP/3 and QUIC version 1 are about to be published by the IETF, and the software ecosystem to go with them is still emerging. Many implementations are still under development, and while some provide servers and clients for the QUIC transport, there are very few production-ready out-of-the-box HTTP/3 servers.

Who’s using what?

Serving a website using HTTP3/QUIC needs:

an SSL library with TLS 1.3 and QUIC crypto support
a QUIC implementation
an HTTP/3-enabled webserver

It’s March 2021 at the time of writing. There are several open-source implementations of QUIC, in various stages of readiness. Here’s a diagram of all the QUICs, their TLS library dependencies, and who implements and uses them. Those who run their variant in production are highlighted in orange:

This diagram is not exhaustive, the IETF QUIC working group maintains its own list, including non-open-source implementations.

All the implementations rely on some sort of custom TLS library: the most popular appears to be BoringSSL, Google’s fork of OpenSSL. Other implementations take the approach of developing their own TLS 1.3 libraries (like PicoTLS), which depend on OpenSSL. It doesn’t look like gnuTLS is making an appearance.

Google have been using QUIC in production for a few years (and have switched to IETF QUIC), unsurprisingly as they originated the protocol.

Facebook allegedly use QUIC for more than 75% of their traffic.

Fastly and Cloudflare now allow their customers to enable QUIC for their websites.

Akamai have been using their own implementation since 2016.

Client-side, there is support for QUIC in major browsers, including Firefox nightly since last year - although it’s not enabled by default.

With so many implementations, perhaps you’re wondering if they all talk to each other? Well, the QUIC Working Group also runs interoperability testing of the various implementations.

What about us?

If you’re not a big corporation and want to serve your website over QUIC, then right now you have a few options:

nginx-quic - according to their README, code is experimental and still under development
h2o- also provides ‘experimental’ HTTP/3 support
apache with QUIC support- also under development
openlitespeed - appears to be the only production-ready option

To run an HTTP/3 server out of the box on any OS, first the custom TLS libraries need to make their way into their respective distribution. On top of this, webservers like Apache and NGINX also need to integrate support for QUIC/HTTP3 in a main release. Any other QUIC and HTTP/3 implementations would also need to be packaged for major server distributions.
In my experience with Debian packaging, this is only likely to happen after the software is mature enough, by which point TLS dependencies will have already been packaged.

For now, everything needs to be built from source.

You can of course host your website with Cloudflare or Fastly and simply enable QUIC - but where’s the fun in that?

My experience with nginx-quic

I’ve deployed nginx-quic on a Raspberry Pi. This was made more complicated due to hitting a bug in gcc 8.3 when building BoringSSL and needing to upgrade the Pi from buster to bullseye, the soon-to-be Debian Stable. After upgrading, BoringSSL built as per instructions in their repo.

Nginx-quic provides a guide for building and configuring the server. I configured the server as per instructions, and tested it with Firefox versions 78 and 85, and Chrome version 88 on both Linux and MacOS.

The mechanism to signal a client the presence of an HTTP/3 server is the use of the alt-svc header, which tells the client which QUIC versions are support and which port to use. In theory, the second time a resource is requested it will use HTTP/3. For my server, this worked… eventually, using version 85, after much page reloading, cache clearing and browser restarting. The logs show about 20 HTTP/1.1 attempts until HTTP/3 was finally used.

I have not managed to get Firefox version 78 and the latest Chrome to do the same, despite enabling QUIC support.

Final words

On one hand, controlling software at both endpoints of a network path (like Google controls its own servers and your browser!) means innovation in the QUIC transport can happen really fast.

But right now, each of my browsers comes with a different implementation of QUIC. If I want cURL with HTTP/3 support I’ll likely need to install yet another QUIC library. What happens when more applications use it? All this seems wasteful, if you consider all applications on an endpoint share a TCP stack.

Firefox modding with containers and SOCKS proxies

Thu, 24 Sep 2020 00:00:00 +0000

Here’s a guide to my lazy setup for running multiple Firefox tabs in the same session over different networks using the magic of SOCKS.

The use-case is that I sometimes want to access a web app or page which is only accessible via a specific network (i.e., my work network or Tor), but I most definitely don’t want the rest of my browsing traffic going through there.

The general idea is to use long-running SSH tunnels to provide one or more SOCKS5 proxies that can be used by Firefox (or your browser of choice). SOCKS is a protocol that allows applications to request connections through a proxy server. Applications, such as Firefox, must be configured to use it.

Generally, to do this manually, you’d first SSH with dynamic forwarding into a host on the desired network:

ssh -D1080 user@host

…and now a SOCKS proxy on localhost port 1080 is ready to forward connections to the remote host. Tor also provides a SOCKS proxy that can be used in much the same way by default on port 9050. This does not conflict with Tor Browser, which runs its own Tor daemon listening on port 9051, separate from the system Tor.

So, to use the proxy in a browser, the browser’s network settings should be changed to resemble something like this:

Firefox also has a checkbox for proxying DNS requests through the same connection.

It’s a good idea to proxy your DNS requests because 1) the remote DNS resolver might know names of resources you can’t access locally and 2) due to the prevalence of CDNs in the Internet, the IP addresses obtained this way will often correspond to servers physically closer to the tunnel endpoint, speeding up connections.

These settings could be saved under a separate Firefox profile that can be fired up whenever the SSH connection is active. Any browser requests will be forwarded to the network of the host you’re SSHed into.

Now, this is an easy substitute for a VPN, but still requires launching a new SSH connection and browser instance every time you want to browse via the remote network. Plus, multiple networks mean multiple profiles or multiple SSH connections which is a pain to manage.

Enter Firefox containers and autossh. The first is an extension that allows you to keep website data, cookies, and cache separate between tabs and websites by assigning them to different containers.

The second is a way to maintain an SSH tunnel indefinitely. The way to glue them together is Container Proxy, another Firefox extension that allows per-container proxy settings.

Here’s how it works:

`Autossh` and Tor

This is a wrapper around ssh to keep tunnels open indefinitely in the background. It can use any SSH option or config. For simplicity, I have the following config specified for my proxy host in ~/.ssh/config:

Host pxhost
    Hostname pxhost.example.com
    ServerAliveInterval 30
    ServerAliveCountMax 3
    DynamicForward 1080

This command will run autossh in the background, forever keeping the connection alive.

autossh -M 0 -f -N pxhost

To persist this on reboot, I use a systemd service file for Linux and a @reboot cronjob for macOS.

I also have Tor configured to run at startup, allowing me to use it alongside other connections. Tor can run as a service on distros using systemd. On macOS, I modified the .torrc file in my home directory to include RunAsDaemon 1, and just running tor with no options on the command line starts the SOCKS proxy.

Firefox Multi-Account Containers

The extension can be found in the official Firefox store. I have three containers: a Direct container for day-to-day browsing without a proxy, a Work container for accessing some infrastructure at work via an SSH connection into my work computer, and a Tor container for looking at .onion addresses or other web pages over Tor:

Tabs opened in each container are colour coded, and easy to keep track of.

An important note about the Tor container: using Tor as a proxy and not using Tor Browser does not provide anonymity, because any other browsers will leak client information allowing 3rd parties to identify users. I do this mostly for convenience and sometimes to avoid eavesdropping from my ISP. However, if you want anonymity, USE TOR BROWSER!

Container Proxy

Proxies for containers are not supported natively in the official Firefox extension. At the moment, another extension is required to make the containers use the tunnels. While not checked by Mozilla, this is open source and the code is auditable at https://github.com/bekh6ex/firefox-container-proxy.

It lets you configure and test the proxies with DuckDuckGo, and assign them to containers:

If the tunnels are set up to persist on reboot, and your Firefox profile is not entirely erased with the latest update, this is how it looks/works:

That’s it. Containers are cool. Use more containers, before Mozilla dies off.

Ansible x OpenBSD Web Deployment

Fri, 10 Apr 2020 00:00:00 +0000

I’ve recently switched most of my home and away infrastructure from Linux to various flavours of BSD. This blog post documents an attempt at deploying multiple static websites in the fastest way possible, on OpenBSD, with Ansible.

It covers automating multiple LetsEncrypt-enabled static websites, configured with onion services for access through the Tor network, using OpenBSD’s brilliant httpd. As a bonus, the content of these websites is automatically generated from cat pictures.

The requirements for this are a public IP address, ability to make DNS records, and of course… some static websites to deploy.

Basics

At the heart of the Ansible configuration for this project is the group of websites to deploy, which are defined as variables in the hosts file:

[www_sites]
site1.example.net
site1.example.com
site2.example.net
site2.example.com

Throughtout the deployment, this group is iterated over.

Generate the websites

First, to deploy multiple websites, one must have multiple websites available - I made a few, out of:

pictures of several cat friends
a short bio for each cat friend
a Jinja2 HTML template for the html index

My website vars group looks like this:

[www_sites]
lion.kitty.institute
leppy.kitty.institute
fox.kitty.institute
jinxy.kitty.institute
kitty.kitty.institute

Directories for each site are iteratively created with an Ansible task:

- name: create vhost directories
  file:
    path: "/var/www/vhosts/{{ item }}"
    state: directory
    owner: www
  with_items: "{{ groups['www_sites'] }}"

The website content, in our case a generated index.html file, is also created by an Ansible task. With each iteration, a variable named ‘vhost’, which corresponds to the name of each site, is passed to the index.html.j2 template:

- name: create mock site content
  template:
    src: "index.html.j2"
    dest: "/var/www/vhosts/{{ item }}/index.html"
    owner: www
  with_items: "{{ groups['www_sites'] }}"
  vars: 
    vhost: "{{ item }}"

The template is simple html, and uses the ‘vhost’ variable to load the correct bio and picture files. The bios and pictures files are stored in the main Ansible directory, to be used by the template:

<h1>{{ vhost }}</h1>
<hr>
<p>"{{ lookup('file','kitty_bios/' + vhost) }}"</p>
<img src= "{{  vhost }}.jpg" width="500 px"/>

The resulting index files are installed in their own directory, ready to be served.

Deploy the websites

To serve the new websites, a configuration file for httpd must be created. You’ve guessed it - this is another template: httpd.conf.j2.

The example configuration file shipped with OpenBSD is a wonderful reference here, and we want to keep the part that redirects port 80 to port 443 and allows LetsEncrypt verification for ACME challenges, for all the sites (and, aditionally, for your host):

server "{{ inventory_hostname }}" {
        listen on * port 80
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                block return 302 "https://$HTTP_HOST$REQUEST_URI"
        }
}

Then, a loop is added to make configuration blocks for all the servers in the www_sites group: Eventually, all the websites should listen on port 443 and are expected to have TLS certificates - these will be generated in the next step.

{% for vhost in groups['www_sites'] %}
server "{{ vhost }}" {
        listen on * tls port 443
        tls {
                certificate "/etc/ssl/{{ vhost }}.fullchain.pem"
                key "/etc/ssl/private/{{ vhost }}.key"
        }
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                root "/vhosts/{{ vhost }}"
        }
}
{% endfor %}

Letsencrypt the websites

Ok, time to fill /etc/ssl with certificates for these websites. As a cautionary note, httpd must be started before you attempt to request the certificates. This was…the hardest issue to debug in the entire setup.

So, throw in there:

- name: enable and start httpd
  service:
    name: httpd
    enabled: yes
    state: started

For the ACME setup, a configuration file must first be created and installed. Yes, there’s a template for that.

- name: install acme-client.conf
  template:
    src: "acme-client.conf"
    dest: "/etc/acme-client.conf"

The template is based on the default example configuration included with OpenBSD’s acme-client, wrapped in a for loop:

authority letsencrypt {
        api url "https://acme-v02.api.letsencrypt.org/directory"
        account key "/etc/acme/letsencrypt-privkey.pem"
}

{% for d in groups['www_sites'] %}
domain "{{ d }}" {
        domain key "/etc/ssl/private/{{ d }}.key"
        domain full chain certificate "/etc/ssl/{{ d }}.fullchain.pem"
        sign with letsencrypt
}
{% endfor %}

For the ACME run, ensure the names match up those used earlier for the httpd configuration file:

- name: initial acme-client run
  command: "/usr/sbin/acme-client {{ item }}"
  args:
    creates: "/etc/ssl/{{ item }}.fullchain.pem"
  with_items: "{{ groups['www_sites'] }}"
  notify:
  - reload_httpd

A task to renew certs indefinitely via cron jobs finishes the job nicely:

- name: renew certificates via root crontab
  cron:
    name: "acme-client renew {{ item }}"
    minute: "0"
    job: "sleep $((RANDOM \\% 2048)) && acme-client {{ item }} && rcctl reload httpd"
    user: root
  with_items: "{{ groups['www_sites'] }}"

To test, go to your website, which should both redirect to https, and should have a valid cert. In this case, https://fox.kitty.institute.

Onion the websites

The cats now must be made available to those under oppresive regimes or avoiding censorship. An onion address does just that.

Tor must be installed, and the appropriate configuration to enable onion services must be enabled. The torrc file is another template - which basically instructs Tor to create an onion service for each site and store the onion service files in a separate site folder:

Log notice syslog
RunAsDaemon 1
DataDirectory /var/tor
User _tor

{% for domain in groups['www_sites'] %}
HiddenServiceDir /var/tor/{{ domain }}/
HiddenServicePort 80 127.0.0.1:80
HiddenServicePort 443 127.0.0.1:443
{% endfor %}

To install tor, and the configuration file:

- name: install tor
  openbsd_pkg:
    name: ['tor']
    state: present

- name: install torrc
  template:
    src: "torrc"
    dest: "/etc/tor/torrc"
    owner: root
    group: wheel
    mode: 0644
  register: torrc

Note how copying the torrc template registers a variable. This is used later to reload tor if any changes were made to the configuration.

I should point out that this will generate random onion addresses. If you want a custom onion address you can mine it and replace the files in each directory manually.

Remember: the machine running the service must be able to read the private key, so to reduce attack surface the safest way to generate onion addresses is on the very same machine. Software to mine vanity onion addresses can be found here. When copying these, the owners and permissions of the original files need to be kept for tor to run.

My final onion addresses look like this:

meow2ecfmjschzktpwnaufh5m5fop3emrmm2mb62gvawovpiyk7jxdqd.onion
meow3pdf65knffidkkypdgtvohispjpzw6omcyoellwythv64vxo5dqd.onion
meow4u5lkpndb562ble3ityac2l3gm47wegtqp72mq2mfqmved7mnhad.onion
meow5w6vxagd7ipzvmb2h54quuzgwkanu2i3zjnb7qloakmgy3nmrgid.onion
meow6onx3grwas4k2i7lb6iecgyk6fkrypcnsatr2tri6kunv36zksid.onion

Try these in Tor browser!

The step above is entirely optional. However, after copying the torrc configuration file, tor must be running and set to run at startup:

- name: ensure tor is enabled and started
  service:
    name: tor
    enabled: yes
    state: started

- name: reload tor
  service:
    name: tor
    state: reloaded
  when: torrc.changed

The final step is integrating these tasks in with the rest of the configuration, and telling httpd what to do when it receives requests on the onion addresses. The onion address must be known before installing the httpd template - if they are generated by tor then fetching them manually would be a pain. Tor stores the onion address in a file called hostname in /var/tor/. A task can be set up to retrieve these with Ansible:

- name: retrieve onion hostnames
  fetch:
    src: "/var/tor/{{ item }}/hostname"
    dest: "files/onion_hostnames/{{ item }}"
    flat: yes
  with_items: "{{ groups['www_sites'] }}"

Now the files are in place ready to be retrieved by other tasks in Ansible. The httpd template created earlier can now be modified to use the onion addresses:

{% for vhost in groups['www_sites'] %}
server {{ lookup('file', 'onion_hostnames/' + vhost) }} {
        listen on * port 80
        location * {
                root "/vhosts/{{ vhost }}"
        }
}

This will set up listeners on port 80 for each onion, and serve the website content created earlier. You can enable https for these addresses, but as the server name certificate won’t match the onion name, the user will be presented with a warning. Note the new alias line included in the template:

server "{{ vhost }}" {
        listen on * tls port 443
        alias {{ lookup('file', 'onion_hostnames/' + vhost) }}
        tls {
                certificate "/etc/ssl/{{ vhost }}.fullchain.pem"
                key "/etc/ssl/private/{{ vhost }}.key"
        }
        location "/.well-known/acme-challenge/*" {
                root "/acme"
                request strip 2
        }
        location * {
                root "/vhosts/{{ vhost }}"
        }

Lots of things were omitted here for brevity, but the full template is on github.

To finish this off, a picture of a cat friend:

Packet Radio Fun

Thu, 02 Jan 2020 00:00:00 +0000

Packet radio has been around since the 70s, and is used extensively by radio amateurs. APRS (Automatic Packet Reporting System) allows amateurs to send real-time beacons which may include location or other types of information.

Alongside stations, the APRS infrastructure includes digipeaters and IGates. APRS packets may be picked up by local digipeaters (repeaters of digital packets), and then repeated to reach further stations. An IGate relays received packets to the Internet where a unified map of all APRS objects can be displayed.

I’ve tried to do APRS packet radio over the past several days with mixed results. Here is what I found.

Required equipment

The standard APRS frequency for sending messages is 144.800 MHz in Europe, in the 2 metre band. Any 2M radio that can receive and decode APRS messages will work. It’s the most fun using a handheld device and receive messages as you walk through a new area. Unfortunately, handheld radios that have a built-in TNC are very expensive - but there is an alternative. If all you have is a cheap 2M radio, it can be used with a compact bluetooth TNC device and an APRS application on your phone for the same capabilities.

If doing this stationary, there are more alternatives - as the TNC bit can be done in software, with applications such as Direwolf, and all you require is a sound card and any radio.

I’ve used an Yaesu VX-8DE, which is a handheld device with a built-in TNC.

Sending packets

It’s harder than you think.

Problem #1: Trains

When travelling by trains, I was only able to send packets for maybe ¹⁄₅ of the total journey time, due to not having GPS coordinates to build packets with. Turns out that the inside of a train is NOT the best environment for acquiring a GPS lock.

Even with the GPS fired up and sending a beacon every minute, only about 10% of all the packets sent were picked up by any digipeater. Turns out the inside of a train is NOT the best environment for RF propagation either. I was digipeated once approaching Leipzig, as the train was stopping:

Trains are still amazing despite this slight glitch. And using a bluetooth TNC should eliminate the GPS problem, as phones are generally better at acquiring GPS data (they can use a broader range of satellites than the proprietary Yaesu GPS antenna for one).

Problem #2: Other users

There can be too many APRS packets.

For example, Cambridge is very active APRS-wise. There were many stations around, some even within direct messaging distance. However, trying to send a single APRS message took several attempts due to collisions. Timing the message so that it sends between the regular beacons of other users helps, but rarely first try, and spamming messages to get something through only makes the problem worse in a crowded spectrum.

As good practice for those out there thinking of running stationary or unattended beacons I recommend sending updates at least three minutes apart.

Problem #3: Cloudy skies & rubber ducks

Several satellites have amateur radio capabilities, and there are apps to easily track satellite passes in your location.

Cambridge is also in a great spot for ISS passes, some with really a nice elevation above the horizon. The ISS has a digipeater on board. This extends the limited local 2M perspective to space and back anywhere on the continent. I spent too much time in muddy fields trying to get digipeated by the ISS, but with no success. However, I did receive stations from Austria (OE6PWE), Sweden (SM3XLY), Italy (IK1COA), Netherlands (PE1NTN), France (F6FUJ), Germany (DB1DT), Russia (UA1WBM) and Space itself (RS0ISS):

Sending failed possibly due to a combination of the bad rubber duck antenna of my handheld, sky conditions, maybe competing with other stations.

On my to-do list is to build a Yagi, and try again, as per this nice start-up guide.

LibreELEC Streaming Camera

Tue, 12 Feb 2019 00:00:00 +0000

This is a blogpost about a Raspberry Pi camera setup on LibreELEC. It includes a step by step tutorial.

If you have the very specific use case of needing to run LibreELEC on a Raspberry Pi to play media while also streaming a Pi camera, look no further. This setup has been running for the past year or so in my house. I’ve rebuilt the Docker container for it recently, which is the perfect excuse for a blogpost.

LibreELEC is a Linux distribution for running media center software Kodi. It will stream films, music, TV shows from your NAS to your TV and has several awesome add-ons. LibreELEC has no add-on for streaming from a camera. Perhaps because the drivers needed for USB cameras are not actually even compiled into the image for the RPi2. I’ve tried this setup using a USB camera and had to build a custom LibreELEC image.

However, there is support for streaming from RPi cameras and there is a Docker add-on, which can be used to run a simple camera streaming app in a container. The app I’ve chosen after some trial and error of which gruelling details I will spare you is mjpeg-streamer.

The container uses the current Raspbian stable image, stretch, on top of which it builds mjpeg-streamer. It then uses an entrypoint script to capture a still image per second from the Pi camera, which the software then turns into a www stream on port 8080.

You can get the container here. For a step by step tutorial on how to deploy this, read on.

Step by step tutorial

Ingredients:

Raspberry Pi v2 or v3
Micro SD card (>2GB) and SD card adapter
Compatible Raspberry Pi camera module
Monitor and keyboard
Wired connection or wireless dongle if using a RPi 2

Step 1: Download

Download the official LibreELEC image for the Raspberry Pi from the official website. Scroll to ‘Direct Downloads’, select ‘Raspberry Pi v2 and Raspberry Pi v3’ and click the filename link on the page. The filename looks like LibreELEC-RPi2.arm-8.2.5.img.gz at the moment of writing this blog post, yours will most likely be newer.

Step 2: Put image on SD card

I use an SD card adapter and the in-built card reader in my laptop. On Linux, after plugging the SD card in, command sudo dmesg should display the name newly inserted device, similar to the following:

ana@sinopia:~|⇒  sudo dmesg
[...]
[70540.504869] mmc0: new high speed SDHC card at address aaaa
[70540.585060] mmcblk0: mmc0:aaaa SB16G 14.8 GiB (ro)
[70540.590225] mmcblk0: p1

The identifier for SD cards on Linux usually looks like mmcblk. Careful to copy your image on the right device, and not on your local hard drive:

ana@cyan:~|⇒  gunzip LibreELEC-RPi2.arm-8.2.5.img.gz
ana@cyan:~|⇒  sudo dd if=LibreELEC-RPi2.arm-8.2.5.img of=/dev/mmcblk0
549+0 records in
549+0 records out
575668224 bytes (576 MB, 549 MiB) copied, 50.8254 s, 11.3 MB/s

This may take a minute or two. Once dd has finished, load up your MicroSD card in your Raspberry Pi.

Step 3: Configure LibreELEC

Plug a monitor and a keyboard in and turn on your RPi. LibreELEC starts an autoconfiguration wizard when it first boots. This will guide you through setting up location, timezone and most importantly, a network connection needed for the next step. Hit ‘Next’ and follow the wizard.

One you connect to a network, the interface will display your IP address. Make sure to take a note of it.
Under the initial ‘Sharing and Remote Access’ screen make sure you enable the SSH service.

For security purposes, we will disable SSH after setting up docker as the root password is hardcoded in the image.

Step 4: Download the Docker Add-on

From the left hand-side menu, navigate to Add-ons -> Install from repository -> LibreELEC Add-ons -> Services -> Docker and then select Install.

The interface will notify you once this has installed.

Step 5: Build the Docker container

In order to build and run the docker container, you need to ssh as root into your Pi, using the IP address from Step 3. In this example, my IP is 192.168.0.156.

ana@cyan:~|⇒  ssh root@172.22.152.253
root@172.22.152.253's password

At the password prompt, type ‘libreelec’, the LibreELEC default and hardcoded password.

Next, make a directory and download the Dockerfile and starting script.

LibreELEC:~ # mkdir mpeg
LibreELEC:~ # cd mpeg
LibreELEC:~/mpeg # wget https://raw.githubusercontent.com/ana-cc/dockerstuffs/master/mjpeg-streamer-rpi/Dockerfile
Connecting to raw.githubusercontent.com (151.101.60.133:443)
Dockerfile           100% |************************************************************************************************************************|   917   0:00:00 ETA
LibreELEC:~/mpeg # wget https://raw.githubusercontent.com/ana-cc/dockerstuffs/master/mjpeg-streamer-rpi/stream_from_pi.sh
Connecting to github.com (140.82.118.3:443)
stream_from_pi.sh    100% |************************************************************************************************************************| 55534   0:00:00 ETA

Inspect and (optionally) edit the stream script. The script uses command raspistill to capture an image per second from the Pi camera, which mjpeg-streamer then turns into a www stream on port 8080. You can vary the resolution, quality and time apart these images are taken with the -w -h and sleep values. Here is the script by default:

until raspistill --nopreview -vf -hf -w 640 -h 480 -q 5 -o /tmp/stream/pic.jpg -tl 100 -t 9999999 -th 0:0:0 & LD_LIBRARY_PATH=/usr/local/lib mjpg_streamer -i "input_file.so -f /tmp/stream -n pic.jpg" -o "output_http.so -w /usr/local/www"; do
	sleep 1
done

Finally, build the container and give it an easy to remember tag:

docker build -t mjpeg_container .

This should take a while, at the end your output should be similar to:

[...]
Step 12/13 : COPY stream_from_pi.sh /
 ---> 2299b11e7696
Removing intermediate container de4317561efe
Step 13/13 : ENTRYPOINT /bin/bash /stream_from_pi.sh
 ---> Running in deff3a4ebe15
 ---> b5f669ccd45e
Removing intermediate container deff3a4ebe15
Successfully built b5f669ccd45e

Step 6: Run the docker container

Now for the fun part: running and testing the newly-built container.

We want the Docker process to map the default streaming port to port 8081 of our RPi host (Kodi’s own web interface occupies port 8080 by default):

 docker run -d --privileged -p 8081:8080 mjpeg_container

And now the LibreELEC box will present the stream on port 8081. In a browser of choice, navigate to your IP address on port 8081, for example https//192.168.0.156:8081.

You should see a welcome page like the following:

Your stream can be viewed under the stream tab.

Step 8: Clean up

Worth adding an @reboot crontab to run the docker command if your box is subject to, uh, a lot of reboots.

Before ending your ssh session:

LibreELEC:~ # crontab -e

Make sure the file contains the following line:

@reboot docker run -d --privileged -p 8081:8080 mjpeg_container

This tells the box to run the docker container at every reboot so that you don’t have to start it manually.

Finally, disable SSH by going to Settings -> Services in your Kodi navigation.

This concludes the tutorial. Happy streaming!

Tor Relay Map

Fri, 01 Feb 2019 00:00:00 +0000

Here is a map that displays Tor relays!

The map also supports searching for the full or partial nickname, IP address fingerprint of a Tor relay. You can also search for relays in specific countries or ASes (country:us, as:3), or with specific flags (flag:exit or flag:authority).

To display all running relays, the search string is running:true. An empty search string will plot all the relays.

The Tor relay information is queried from Onionoo.

This was written a while ago for the only purpose of trying out JS and in the future, this will have other open source nodes added to it, including things like BSD mirrors, Bitcoin nodes etc. For now, the source can be found here.

There are several icons used to display the relays, their meanings can be found here. A legend is still on the TODO list.

Full-page version.

Custom Query

Zenburn Theme for the terminal

Wed, 23 Jan 2019 00:00:00 +0000

So today I glanced down to my i3 status bar, and thought man those colours are fugly. By default, these are white, fully saturated neon green and red on black:

‘5 minutes is enough to configure the colors for this, right?’

Some research and 2 hours down the rabbit-hole later, and everything from my text editor to the status bar is Zenburn-themed.

Zenburn is a low-contrast theme meant to ensure your eyes do not fall off when you’re having that mexican-standoff with whatever project you happen to be Vimming.

But most of us spend a lot of time in text editors and on the command line, and Zenburn is a theme for Vim only. This useful and beautiful theme does not have an .Xresources file listed on the official page of Zenburn things.

As I’m a fan of cohesion, this now exists, this blog post will explain how.

First, I extracted the colors from the original zenburn.vim using simple regex:

matches = re.findall(r'#(?:[0-9a-fA-F]{3}){1,2}', line)

I then used yattag, a Python library for creating HTML, and displayed all the unique colors from the file. It looks like this:

#c0bed1

#bc6c4c

#80d4aa

#dfdfbf

#4f4f4f

#2b2b2b

#bc8cbc

#d0d0a0

#e89393

#82a282

#304a3d

#444444

#8f8f8f

#2e3330

#242424

#f8f893

#41363c

#88b090

#f0f0c0

#dccdcc

#121212

#313c36

#efef8f

#bc6c9c

#2f2f2f

#464646

#353535

#ecbcbc

#71d3b4

#dcdccc

#7cac7c

#222222

#93b3a3

#5b605e

#dfdfdf

#000d18

#0f0f0f

#353a37

#ffd7a7

#dc8c6c

#dca3a3

#383838

#1f1f1f

#b2b2a0

#b6bf98

#e3ceab

#688060

#709080

#cc9393

#434343

#dfaf8f

#3a3a39

#acd0b3

#2c2e2e

#bca3a3

#efefaf

#545a4f

#343434

#2c302d

#000000

#a0afa0

#313633

#6c6c9c

#8c8cbc

#9fafaf

#efefef

#d0d0b8

#9ccc9c

#8faf9f

#3f3f3f

#ffffe0

#ffffff

#284f28

#664040

#ccd990

#c3bf9f

#dfcfaf

#e37170

#1c1c1b

#efdcbc

#ccccbc

#233323

#333333

#f18c96

#404040

#161616

#262626

#f0f0b0

#cfcfaf

#f0dfaf

#385f38

#040404

#9ece9e

#cbecd0

#434443

#dfe4cf

#3f4040

#ccdc90

#3d3535

#101010

#f0efd0

#ffcfaf

#7f9f7f

#2e2e2e

#181818

#9f9f9f

#8cd0d3

#cfbfaf

To get the colors to make a bit of sense, some sorting was in order. I spent some time reading this excellent article about why sorting colours is a pain. I then did some quick RGB to HSV conversion and sorting, and voilà:

#000000

#040404

#0f0f0f

#101010

#121212

#161616

#181818

#1f1f1f

#222222

#242424

#262626

#2b2b2b

#2e2e2e

#2f2f2f

#333333

#343434

#353535

#383838

#3f3f3f

#404040

#434343

#444444

#464646

#4f4f4f

#8f8f8f

#9f9f9f

#dfdfdf

#efefef

#ffffff

#3d3535

#bca3a3

#ecbcbc

#dca3a3

#cc9393

#e89393

#664040

#e37170

#dccdcc

#bc6c4c

#dc8c6c

#ffcfaf

#dfaf8f

#cfbfaf

#ffd7a7

#e3ceab

#efdcbc

#dfcfaf

#f0dfaf

#c3bf9f

#f0efd0

#3a3a39

#1c1c1b

#dcdccc

#ccccbc

#b2b2a0

#d0d0b8

#ffffe0

#dfdfbf

#cfcfaf

#f0f0c0

#d0d0a0

#f0f0b0

#efefaf

#efef8f

#f8f893

#ccd990

#ccdc90

#b6bf98

#dfe4cf

#545a4f

#688060

#434443

#a0afa0

#82a282

#7f9f7f

#9ece9e

#9ccc9c

#7cac7c

#233323

#385f38

#284f28

#cbecd0

#acd0b3

#88b090

#2c302d

#2e3330

#353a37

#313633

#313c36

#93b3a3

#8faf9f

#709080

#304a3d

#80d4aa

#5b605e

#71d3b4

#3f4040

#2c2e2e

#9fafaf

#8cd0d3

#000d18

#8c8cbc

#6c6c9c

#c0bed1

#bc8cbc

#bc6c9c

#41363c

#f18c96

This looks better and one can easily see what colors there are.

The next stop was terminal.sexy, which is an amazing terminal theme tool which allows you to vizualize and export your work easily. Here’s our theme:

And here’s how it looks in the terminal:

The i3 window colors, borders and status bar have now also been changed to match:

And for completeness, here’s a Firefox theme to match, because we can:

Starting the new year ready to sysadmin for hours on end in an ill-lit room, happy hacking!

A tiny Nagios plugin to check DNSSEC RRSIG expiry

Wed, 14 Nov 2018 00:00:00 +0000

I have used both Icinga and Nagios to monitor a variety of infrastructures, either at work, at home and anywhere in between. These share a variety of very useful plugins for monitoring lots and lots of standard stuff, from basic ping checks to filesystem mounts. Occasionally I had to write the odd script to monitor something ridiculously niche, like split-brain HSRP status between pairs of Cisco routers.

This one is a plugin for checking RRSIG expiry dates on DNSSEC records. There are other plugins out there that do exactly this, but I could not find anything that would not depend on a ton of other packages to do it (libnet-dns-sec-perl anyone?). So I decided to write something that has minimal footprint, ended up with a python wrapper around dig. Of course, this depends on dig (which should come with the plugin installation), as well as python. The python part is optional if you rewrite this in bash.

With regards to RRSIGs and DNSSEC:

RRSIGs = signature records for a zone which contain a cryptographic signature used for validating a DNSSEC response. When you sign a zone, the signature records expire after 30 days. There are recommendations for resigning a zone once a day. This is why the plugin default is to warn if the signature expires in less than 29 days (of course warning and critical thresholds can be changed with command line arguments -w and -c; the server to check against can also be specified with -s):

~./check_dns_rrsig.py cloudflare.com
Signature expires in 1 days

Here is the plugin!

This is very, very basic and will work if all you want is something to tell you when an RRSIG expires, without any other overhead.

Happy monitoring!

Unbound on a fresh FreeBSD install

Mon, 12 Nov 2018 00:00:00 +0000

I’ve recenty encountered a weird problem where unbound would not work on a fresh FreeBSD install on an APU3 board. Online research lead to finding a bunch of posts complaining about this, but no leads. As it turns out, my problem was the clock.

But first things first. Installing FreeBSD on an APU board is very easy, at the boot shell enter the following commands:

>set boot_serial=YES
>set comconsole_speed=115200
>set console=comconsole
#this delays the boot by 10s to allow the USB controller to come up
>kern.cam.boot_delay="10000"

Of course, remembering to drop into a shell at the end of the installation, and edit /boot/loader.conf so that we can use it over the console afterwards:

boot_serial="YES"
comconsole_speed="115200"
console="comconsole"
kern.cam.boot_delay="10000"
amdtemp_load="YES" #this loads the driver for the temperature sensor embedded in the CPU. It is equivalent to the km driver in OpenBSD. Yay sensors!

Now, the APU board does not have an on-board clock. I did not set the time/date during installation (lazy), as NTP was going to be set up anyway. So, at this stage in /etc/ntp.conf, I replaced pool 0.freebsd.pool.ntp.org iburst with the name of my local NTP server.

Reboot after successful installation, cue unbound not working. General DNS failure. As unbound is DNSSEC enabled, run:

unbound-anchor

This sets up the root trust anchor for DNSSEC validation, which was part of the solution to this issue.

But unbound was still not working. Do not underestimate actually checking the results of any commands run:

cat /var/unbound/root.key

The file contained no valid key, thus solving the mistery:

DNS lookups do not work as there is no valid key here, due to our system clock being, well, off by a few years.
And of course NTP does not sync the date… as we gave it a hostname, which needs to be looked up in DNS.

A quick ntpdate <ip-address> followed by running unbound-anchor again solves the issue.

So frustrating.

Installing Debian via serial console

Tue, 20 Mar 2018 00:00:00 +0000

I was recently tasked with installing an apu2d board, a small board with enough performance and NICs to be used for network appliances - in my case, a simple gateway for a point-to-point link.

This required me to do a Debian install first. The APU board has no video outputs and is accesed with a null modem cable. I’ve never done an install on the serial port and as soon as I selected an option from the Debian GNU/Linux installer boot menu I was greeted with the following error:

Undefined video mode number: 314 
Press <ENTER> to see video modes available, <SPACE> to continue, or wait 30 sec
Mode:  Resolution:  Type:
0 F00    80x25      CGA/MDA/HGC

Selecting a mode resulted in the system freezing: the kernel is presumably expecting a video output/keyboard input and crashes not finding any. The system needs to be told it’s being installed via serial console, by starting the kernel with some additional options.

Pressing ‘TAB’ allows you to edit the kernel parameters:

> /install.amd/vmlinuz vga=788 initrd=/install.amd/initrd.gz --- quiet

Changed the above to:

> /install.amd/vmlinuz vga=off initrd=/install.amd/initrd.gz --- quiet console=ttyS0,115200n8

…where ttyS0 is the default serial device, and 115200 is the baud rate required by the APU board. The ‘n’ stands for parity ‘none’ and the 8 is the number of data bits.

Optionally, vga can be changed to vga=off, which makes the video mode error go away, although the installer should start fine without it.

The installation process is identical to the regular one, with an oldschool looking installer:

Anyway, I wonder why the initial bootloader screen appears - is there something my bootloader does to redirect console output to the serial console, or is there a chip on board that has some video functionality? For the latter, the CPU is my first suspect, although this is a riddle for another blogpost.

IPv6 MTU issues

Sun, 08 Oct 2017 00:00:00 +0000

I have a Hurricane Electric IPv6 tunnel that runs on my home Cisco 887VA. After setting the tunnel up, I was having issues connecting to IPv6-enabled websites, which would never load consistently. If your first thought was “MTU blackholing”, then you weren’t far off. But it’s a bit more complicated than that.

For those who don’t know, MTU refers to the maximum size of packet that can be transmitted on a network for a specific protocol. To work the value out, systems use Path MTU Discovery. For IPv6, PMTUD relies on ICMPv6 type 2 - Packet too big messages arriving from hosts in the path that have smaller MTUs. When the sender receives a Packet too big message, it adjusts its own MTU size to reflect this, and packets larger than this value then get fragmented.

In real life, as ICMP/v6 is (sadly) often blocked in the Internet, you need to manually set the MTU for tunnels. In my case, this was already correctly set to account for PPPoE and 6to4 encapsulation.

Instead, the issue had to do with setting the TCP Maximum Segment Size, the largest possible size of a packet payload which does not include the IP and TCP headers.

Setting the TCP MSS ensures that the data payload of a packet will fit within a desired MTU. For IPv4, it is usually set to be 40 bytes lower than the MTU value, as the IP + TCP headers are 40 bytes in total. To set it, I was able to use the ip tcp adjust-mss option on the Cisco box. There was no equivalent command for IPv6.

Researching the issue revealed the version I was running at the time, 15.1(0), had no way of adjusting TCP segment sizes for IPv6. My tunnel was doomed to behave inconsistently unless I upgraded the firmware.

The newer versions of IOS can do it. But it’s not what you think: In versions since 15.2(4)M, the command ip tcp adjust-mss applies to both IPv4 and IPv6. Although they have… different header sizes. *sigh*

The tunnel started working after upgrading the firmware and making the change. Doing so also introduced 20 bytes of overhead for all IPv4 TCP connections.

But at least IPv6 works now.

Epilogue

It’s just gone past 8 PM.

Time to relax, to unwind, to eat some dinner and watch some Netflix. But something’s wrong. Netflix refuses to play. It’s complaining I’m using a proxy.

It appears that my London-based Hurricane Electric assigned IPv6 block is geo-located in the US.

With another sigh, I disable IPv6 in the browser, quietly weeping in my bowl of ~~mac and cheese~~ porridge.

THE END

monroe-cli 1.0.1 released!

Tue, 03 Oct 2017 00:00:00 +0000

Mobile broadbands are a big part of Internet today, and the ubiquity of mobile devices shapes the way we get online. The mobile broadband infrastructure provides its own challenges, from to heterogeneous access technologies to traffic shapers and firewalls buried under layers of NAT.

The MONROE project, Measuring Mobile Broadband Networks in Europe, provides a distributed infrastructure for experimenting on mobile broadband networks. At the time of writing, the platform comprises over 300 measurement servers, or nodes, in Sweden, Norway, UK, Greece, Spain and Italy.

All of the nodes within the MONROE platform are made available to researchers on an Experiment-as-a-Service basis, via a scheduling interface with client-side authentication, which looks like this:

The MONROE scheduler is feature rich, allowing users to customize experiments in a variety of ways. The experiments run on nodes in the platform in Docker containers.

monroe-cli provides both a library and a command-line tool for interacting with the scheduler, without the point and click required by the web front end. It supports all the features of the scheduler, such as experiment recurrence parameters, country selection, start time, and optional experiment parameters. It also allows users check node availability and view, cancel and delete experiments, as well as download experiment results.

Over the last few days I’ve worked on monroe-cli version 1.0.1. This version follows the changes in the API, and implements new features. Here are the highlights:

Support for node models and number of network interfaces

As the project has moved on to the second generation of MONROE nodes featuring apu2d boards, both the library and the CLI now support setting up experiments on either new models of nodes or nodes with a specific number of mobile operators. These options can be set as follows:

monroe create <experiment> --new --ifcount 2

Library support for low priority queue experiments (LPQ)

The library now supports low-priority queue experiments, a new feature which allows users to submit experiments without a given start time. These are deployed on-the-fly when the selected nodes are up and free for enough time to run an experiment. Usage example:

from monroe.core import *
s = Scheduler('clientcert.pem','clientkey.pem')
exp = s.new_experiment()
exp.start(-1)

In order to use this feature in the CLI, the '--start' argument should be set to -1.

Cleaner command line client with better default options

The initial command line syntax had a few unnecessary optional arguments. The default node type when creating an experiment is now ‘testing’, which is what most researchers are expected to use. Also, creating an experiment now submits it by default, instead of relying on --submit. When downloading or deleting experiments, the experiment ID is now a positional argument: monroe delete --exp <exp-id> becomes monroe delete <exp-id>.

The same applies for setting up client certificates, which can now be done using command monroe setup <client-cert.pkcs12>.

Perhaps the most important syntax change is the --script argument, which indicates the experiment’s Docker container, which is now a positional argument as well: monroe create acustura/monroe will submit a new experiment using Docker container acustura/monroe.

Documentation

Instructions on how to install the tool, and usage examples for the library and the CLI have been added to the project, hopefully making it more accessible to everyone.

Issues

The one outstanding issue is that, currently, the client certificates required to authenticate are md5 signed.

This means it is impossible to use python-requests, which relies on OpenSSL, and instead the library uses to not-so-graceful calls to wget. OpenSSL disabled support for certificates signed with md5 as of version 1.1.0, so the CLI tool only works (that I know of) on Debian machines, where wget is compiled against GnuTLS.

If you want to check, here’s how:

ldd /usr/bin/wget | egrep "(tls|ssl)"

As md5 is phased out from digital certificate signing, this is a temporary workaround. Here’s a paper from 2008 (!) that points out md5 is harmful. More recent research describes various attacks involving RSA-MD5 signatures, pointing out that keeping legacy crypto in current protocols weakens them.

If you’d like to check out monroe-cli, visit the github page. The work is funded under EU Horizon 2020, agreement number 644399 . This support does not imply endorsement.