christoph ender's


friday the 5th of april, 2024

Kobold letters


Turns out specific e-mails, called Kobold letters, may change their contents when they're forwarded, simply by putting properly coded CSS into the mail:

This attack is possible because most email clients allow CSS to be used to style HTML emails. When an email is forwarded, the position of the original email in the DOM usually changes, allowing for CSS rules to be selectively applied only when an email has been forwarded.

Currently the only defense appears to be switching off HTML in e-mails alltogether.

wednesday the 3rd of april, 2024

minimal inwx certbot handler

This is a prototype. Don't use in production. It's in a “works-for-me” state, but there's no error checking and not even any thourough testing involved.

I've been trying to implement the DNS-01 challenge with certbot and INWX. I'm doing this for a MFA / MobileTAN protected account, which limits the choice of API clients to the INWX PHP client.

Installation of this PHP API client is best done via composer:

apt-get install composer
composer require inwx/domrobot

Install certbot and certbot-external-auth plugin – no idea if “future” is required everywhere, however I got an error complaining about a missing “past” module without it.

apt install certbot
pip3 install future
pip3 install certbot-external-auth

Minimum handler for Let's Encrypt/INWX, to be placed above Composer's vendor directory:


require 'vendor/autoload.php';

function login() {
  $username = 'my-username';
  $password = 'my-password';
  $token = 'my-token';
  $domrobot = new \INWX\Domrobot();
  $result = $domrobot->setLanguage('en')
    ->useJson() ->useLive() ->setDebug(true)
    ->login($username, $password, $token);
  return $domrobot;

function getLeftAndRightSide($domain) {
  $domainParts = explode('.', $domain);
  $domainPartCount = count($domainParts);
  $leftSide = "";
  while ($i < $domainPartCount - 2) {
    if (strlen($leftSide) > 0) { $leftSide .= "."; }
    $leftSide .= $domainParts[$i];
    = $domainParts[$domainPartCount - 2] . "."
    . $domainParts[$domainPartCount - 1];
  return [ 'leftSide' => $leftSide, 'rightSide' => $rightSide ];

if ($argv[1] == "perform") {
  $validation = getenv('validation');
  $bothSides = getLeftAndRightSide(getenv('txt_domain'));
  $domrobot = login();
  $result = $domrobot->call('nameserver', 'createRecord', [
    'domain' => $bothSides['rightSide'],
    'type' => 'TXT',
    'name' => $bothSides['leftSide'],
    'content' => $validation,
    'ttl' => 300,
    'testing' => false ]);
elseif ($argv[1] == "cleanup") {
  $bothSides = getLeftAndRightSide(getenv('domain'));
  $domrobot = login();
  $validation = getenv('validation');
  $result = $domrobot->call('nameserver', 'info', [
    'domain' => $bothSides['rightSide'],
    'type' => 'TXT',
    'content' => $validation ]);
  $resData = $result['resData'];
  if (count($resData['record'] == 1)) {
    $id = $resData['record'][0]['id'];
    $result = $domrobot->call('nameserver', 'deleteRecord', [ 'id' => $id ]);

The script above can be used in certbot's “handler mode“ like this:

certbot \
 certonly \
 --text \
 --configurator certbot-external-auth:out \
 --preferred-challenges dns \
 --certbot-external-auth:out-public-ip-logging-ok \
 --certbot-external-auth:out-handler ${MY_DIR}/certbot-external-handler.php
 -d '' \
wednesday the 27th of march, 2024

removal from private docker registry


It seems there's no easy way to remove tags / images from a private docker registry. It seems the most straightforward way is something like this:

if [[ $# -lt 3 || $# -gt 4 ]]; then
  echo "Syntax: ${0} <registry-url> <name> <tag> [auth-name:auth-password]"
  exit 1

if [[ $# -eq 4 ]]; then AUTH_PARAMS="-u ${4}"; fi

 curl \
  --silent \
  --head \
  -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  -o /dev/null \
  -w '%header{Docker-Content-Digest}' \

curl \
  --silent \
  --head \

Before being able to delete anything, the registry's environment has to contain the following setting:


Once the script has been successfully run it's necessary to manually initiate the registry's garbage collection:

docker exec -it my-registry \
  bin/registry garbage-collect /etc/docker/registry/config.yml
sunday the 24th of march, 2024

accessing a private docker registry


While pushing and pulling to a the private docker registry can be done via standard methods, examining the contents isn't quite so simple.

A list of repositories can be obtainted by using curl:

curl \
  -u registryuser:password

For a given repository name, the list of tags can be accessed in the following way:

curl \
 -u registryuser:password
saturday the 23rd of march, 2024

apt: packages kept back


Some day, when applying upgrades with the apt command line interface, the tool might state that

The following packages have been kept back:
  <package-name-01> <package-name-02> …

indicating that same updates are available, but won't be installed. Why?

One of the more common causes is that an upgrade for an already installed package requires the installation of an additional package. However, by default apt will not install additional packages or remove anything during an upgrade operation. This behavior can be changed by running

apt-get --with-new-pkgs upgrade

It's also possible to run apt-get dist-upgrade instead, this kind of action might however do more than originally expected, since it tried to resolve conflicts and might remove some packages in order to reach it's goal.

monday the 18th of march, 2024

smtp: dsn versus mdn


There are two ways to get information about successful e-mail delivers: DSN and MDN.


DSN ist the “Delivery Status Notification”. Most have already encountered some of these in form of bounce messages which inform the sender that an e-mail is delayed or could not have been delivered, and by default these are the only events that are reported. However, it's also possible to request “success” delivery status notifications. These have to be explicitely requested, which means for Thunderbird one has to check the “Delivery Status Notifications” in the Options menu when composing a new message or, for a permanent solution, set mail.identity.default.dsn_always_request_on to true. In Outlook, check “Delivery receipt confirming the message was delivered to the recipient's mail server” in the “Tracking” section in the Outlook Mail options.

Mail servers may not always report successful deliveries. This may be due to missing DSN support, or deliberately disabling DSN support. In postfix, for example, DSN may be disabled by stating:

smtpd_discard_ehlo_keywords = silent-discard, dsn

In case the server encounters missing DSN support from a delivery target server, it sends a DSN notification by itself, stating that the message has been delivered to the target system. When DSN support is present on the recipient's machine, the destination server will send this message by itself, usually stating that the mail has been delivered to the target mailbox.

On the technical side, the type of requested DSNs for a message can be specified using the NOTIFY keyword as an parameter to the RCPT TO: line in the SMTP dialog. Furthermore, there's theORCPT parameter for the RCPT TO: command, this defines the address of the original mail recipient's address if, for example, it's forwarded to another address.


MDN, on the other hand, is the “Message Disposition Notification”. Contrary to DSNs, MDNs are handled by the MUA – Thunderbird, Outlook, etc, whereas DSNs are managed by the MTA – postfix, exim, etc. In order to request a MDN for a mail in Thunderbird, check the “Return Receipt” option in Thunderbird – alternatively, set mail.receipt.request_return_receipt_on to true – in Outlook, check “Read receipt confirming the recipient viewed the message” in the “Tracking” mail options.

Technically, MDNs are requested by adding a header field named Disposition-Notification-To:, followed by one or more mail addresses. The receiving MUA, however, is not obligated to send a confirmation. As section 2.1 of RFC 9098 states:

The presence of a Disposition-Notification-To header field in a message is merely a request for an MDN. The recipients' user agents are always free to silently ignore such a request.
monday the 11th of march, 2024

spf records for helo/ehlo


While running various tests for mail servers, I stumbled upon SpamAssassin's SPF_HELO_NONE warning. It incurs a negative score of 0.001, and the short description complains that “HELO does not publish an SPF Record”. And indeed, Section 2.1 of RFC 4408 states:

It is RECOMMENDED that SPF clients not only check the "MAIL FROM" identity, but also separately check the "HELO" identity by applying the check_host() function (Section 4) to the "HELO" identity as the <sender>.

So in addition to the “normal” SPF TXT-record which is published for the MAIL FROM-domain, there should be another TXT-record for the individual HELO-hostname which actually delivers the mail. provides some more details: if mail from is delivered from host, the following entries would represent a working configuration:        IN  TXT      "v=spf1 mx -all"        IN  MX   10  IN  TXT      "v=spf1 a -all"
saturday the 27th of january, 2024

smtp smuggling


At the end of 2023, news about a novel kind of attack on SMTP servers was published: SEC Consult found a way to concatenate multiple e-mails in a way that made many systems process the second mail as if it had been submitted on it's own – along with all the privileges that were granted for the submission of the first one. While the MTA itself isn't attacked, this can be used to make any affected mail server act like an open relay.

For testing, I found the SMTP Smuggling Tools – implemented in Python – and Xeams test – a Java implementation – very useful to test various servers.

Apparently due to some misunderstanding with CERT, free projects like Postfix were notified just before christmas, although others like Microsoft got the information as early as June (it still took Microsoft more than two months to fix the problem for Exchange Online), leading to a few non-amused comments on the corresponding Postfix SMTP Smuggling page.

In Postfix, mature fixes are available for versions 3.8.5, 3.7.10, 3.6.14 and 3.5.24 – see the Postfix SMTP Smuggling page for more details – and activated in using:

smtpd_forbid_bare_newline = normalize
thursday the 9th of november, 2023

migrating swap partitions


After replacing a VM's swap partition with another one, I noticed that the system took considerably longer to boot. Looking at the logs I found:

W: initramfs-tools configuration sets RESUME=UUID=AC0DB423-CE2C-44FA-B59E-242CA695BBDA
W: but no matching swap device is available.
I: The initramfs will attempt to resume from /dev/sda5
I: (UUID=eb8347d0-623b-4b9a-95f1-da527c084617)
I: Set the RESUME variable to override this.
… which is actually quite straightforward: The initramfs configuration in /etc/initramfs-tools/conf.d/resume hat to be adapted. The new UUID can be found by looking in /proc/swaps to find the swap partition currently in use, and look up it's corresponing UUID in /dev/disk/by-uuid/.
saturday the 14th of october, 2023

no identification using MAC address with dhcpv6


I've been trying quite unsuccessful to exclude certain clients from getting an IPv6 via DHCP in a network. Since I just wanted to exclude specific interfaces I've used the MAC address instead of the DUID – the “DHCP unique identifier”, see DHCPv6. As it turns out, although the option to exclude/identify clients via MAC addresses may be present in dhcpv6 servers, it can't be used reliably at all. As the official kea documentation states: “Unfortunately, the DHCPv6 protocol does not provide any completely reliable way to retrieve that information.”

friday the 13th of october, 2023

microsoft-activision takeover


Today, on friday the 13th, microsoft acquired activision, and with it the trademark and all that is left of Infocom, Inc. If the name “Infocom” is known to you at all, you might want to consider sharing “Microsoft consumes Activision; and a plea” or share the associated post on mastodon.

saturday the 7th of october, 2023

get mac address for ipv6


For IPv6, hosts use Neighbor Discovery instead of ARP for IPv4. Accordingly, one can use the ndisc6 tool to look for the MAC address in question:

root@wyvern:~# ndisc6 -1 fdb0:9cab:a32:1::1:0 enp2s0
Soliciting fdb0:9cab:a32:1::1:0 (fdb0:9cab:a32:1::1:0) on enp2s0...
Target link-layer address: 22:0A:CA:A1:5B:19
 from fe80::182a:1647:38d:31a
friday the 6th of october, 2023

kea dhcpv6 fails to bind link-local


Lately, after setting up an instance of an isc kea dhcp6 server , I noticed that after a reboot it would be inactive, although it had been started properly. Turned out that it simply couldn't bind the link-local address:

Oct 10 16:27:04 wyvern kea-dhcp6[1180]: WARN  DHCPSRV_OPEN_SOCKET_FAIL failed to open socket: Failed to open multicast socket on interface enp2s0, reason: Failed to open link-local socket on interface enp2s0: Failed to bind socket 13 to fe80::6a1d:efff:fe2d:6399/port=547: Cannot assign requested address
Oct 10 16:27:04 wyvern kea-dhcp6[1180]: INFO  DHCP6_OPEN_SOCKETS_FAILED maximum number of open service sockets attempts: 0, has been exhausted without success
Oct 10 16:27:04 wyvern kea-dhcp6[1180]: WARN  DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP traffic

I tried a good many things, especially really making sure that all the etc. were present. In the end, I just settled for the simplest but up today flawlessly working version:

ExecStartPre=/bin/sleep 5
tuesday the 19th of september, 2023

simplest hd keep-awake


I've been trying to get some long-type smartctl tests to run through uninterrupted. Since they're taking about 11 hours for a 4TB hd – yes the old, spinning ones – these were so far always interrupted by the hd going to sleep. After looking at some measures to deactive the various sleep mechanisms I found the best and simplest one:

while true; do uptime > /mnt/my-hd; sleep 60; done

Just writing a tiny bit of data to the disk in question every minute or so will keep it awake as long as required without having to alter any of the various sleep / powersave parameters or having to restore these settings after the smartctl runs were through.

tuesday the 12th of september, 2023

prompt failover with isc-kea-dhcp


After migrating to the new isc kea dhcp server - the successor to the older isc dhcp server – I've struggled a bit to get a server pair to do a proper failover when one of the servers fails. Turned out that there's a max-unacked-clients parameter, which tells the system how many dhcp clients need to have sent out dhcp requests before the failover occurs. By default, this is set to 5, so until you dont have five different clients waiting for an IP address, nothing's going to happen. I ended up simply setting this to 0, so once the timeout set in max-response-delay is met, there's always a guaranteed failover to the surviving server.

thursday the 7th of september, 2023

no local nagios dhcp check


One of my server pairs is running icinga and a dhcp server on each of them in HA mode for redundancy reasons. I've been trying to monitor the dhcp service using the nagios check_dhcp plugin. With the servers checking themselves, however, I mostly got many CRITICAL: No DHCPOFFERs were received replies. After some extended research, I finally stumbled over a thread named check_dhcp ignores DHCP replies [sf#1090549] in the associated github issues. This described the issue as follows:

Checking a DHCP server which runs on the local host currently is not possible with check_dhcp (though it might work if the DHCP server was compiled to use normal sockets, as opposed to the LPF/BPF/whatever interfaces it usually uses).

With this in mind, I ended up monitoring the dhcp cluster from another server pair. That doesn't really make me very happy and I hope I can find another solution in the future.

tuesday the 22nd of august, 2023

btrfs send and receive


brtfs snapshots are great for incremental backups – just create a snap from a working directory and keep on happily working on the original folder as you please: btrfs makes sure that only incremental changes from the snapshot to the current state will occupy space.

Replicating these backups to another volume using a simple copy operation doesn't however really work that great: as one would expect, simple copying can't reproduce the deduplication from the source device on the target space. This is where btrfs send and btrfs receive come in: You can stream a btrfs read-only subvolume including all it's metadata somewhere else:

btrfs send snap1 > snap1.bin
btrfs recevice in/ < snap1.bin

In the two lines above, the “snap1” directory along with it's btrfs-relevant metadata is written to “snap1.bin”. This can be transferred somewhere else and unpacked into some directory – “in” in this case – using btrfs receive. Once this has been completed, the incremental changes from “snap1“ to a “snap2” can be transferred using the following commands:

btrfs send -p snap1 snap2 > snap2.bin
btrfs recevice in/ < snap2.bin

This will reconstruct “snap2” based on the already present data from “snap1” and reconstruct the same deducpliation on the target space.

saturday the 19th of august, 2023

working around microsoft blacklisting


Catching up on yesterday's post: It's hard to deny that self-hosting mail for individuals or smaller companies has become a much greater challenge nowadays. Some references:

Especially regarding Microsoft, I've lately adapted a new way of rolling out mail servers: Get a new VM somewhere and immediately, before doing anything else, check the IP at Microsoft's “Smart Network Data Service” and/or telnet to one of their mail servers and try to get a mail delivered to an Exchange Online account. If the new machine's IP is blacklisted at SDNS or the test delivery results in the mail getting immediately dumped to the spam folder, just cancel the new VW and roll out a new one – this of course only makes sense when the VM is billed accordingly and costs are neglegible. I've never succeeded at having an IP unblocked by Microsoft without some human intervention on their part and they don't provide even the slightest information at all why some IP is on their blacklist – you'll just get a list of conditions you'll have to meet and an endless stream of “Not qualified for mitigation” messages. Just trying IPs until finding one that's not blocked saves a huge amount of time.

friday the 18th of august, 2023

office 365 “junks” microsoft mail


With all the fuzz these days about getting mail from stand-alone running smtp servers to be recognized as non-junk by the big platforms, it's quite funny to see that even Microsoft can't keep up: On a company's exchange account, which I've been assigned to use, microsoft now sorts its very own e-mails advertising the new teams app and other things into the “junk” folder all by itself.

wednesday the 16th of august, 2023

btrfs snapshot's exclusive space


How much space does a btrfs snapshot actually exclusively allocate? One simply has to run a btrfs fi du -s backup-* in order to see which space is shared between the snapshots and which is exclusively used by the snapshot listed:

     Total   Exclusive  Set shared  Filename
   2.80TiB     5.79GiB     2.80TiB  backup-2023-08-20-01-00-01
   2.80TiB     5.51GiB     2.80TiB  backup-2023-08-21-01-00-01
   2.81TiB       0.00B     2.81TiB  backup-2023-08-22-01-00-01

In the example above, backup-2023-08-22-01-00-01 is the latest snapshot. Since, in this case, this snapshot's source hasn't changed the snapshot doesn't allocate any space exclusively.

saturday the 12th of august, 2023

strict versus real-strict imapsync


imapsync is an extremely useful tool for the migration of imap accounts. It takes two sets of access information for imap accounts and syncs the source into the destination account.

When trying to migrate accounts with a very large number of messages, I encountered a few warnings about duplicates. The imapsync FAQ says the it's a problem with message identification – imapsync by default uses the Message-ID: and Received: headers to identify messages on both sides, which may fail when, for example, imap servers change one or more of these headers.

The easiest solution for my sync task – which was to migrate all the accounts to a fresh set of servers, which haven't been active before – was to switch to --useuid. This will make imapsync identify messages by the imap UIDs instead of the headers mentioned above.

This may not solve the problem immediately in case imapsync has been run before without having the --useuid set. One possible solution – the FAQ has way more details – is to set the --delete2 flag, which will remove any messages from the destination which aren't present in the source account.

Also, syncing hundreds of thousands of messages takes some time. Setting the --usecache parameter helps a great deal: for every transfer, imapsync creates a cache file in /tmp, which greatly speeds up subsequent imapsync runs. This will allow you to run the initial syncs during production and keep the downtime for the actual migration to a minimum.

friday the 11th of august, 2023

forcing windows to use openvpn-dns


While providing windows dial-in vpn clients with the dns servers addresses of the internal network using the dhcp-option DNS parameter, I found out that the name resolution didn't work reliably. After some research it turned out that this was due to windows just adding the provided dns addresses to the ones already present on the system, and using all of them for the actual name resolution.

Luckily, openvpn already provided a solution for these windows clients: it's enough to add the block-outside-dns option. This is sufficient to make windows resolve names using the provided internal dns addresses only.

saturday the 29th of july, 2023

defer domain-specific postfix delivery


Some time ago I had to migrate a mail server running multiple domains, whereby these domains were to be moved one after another instead of moving everyhing at once. That meant that the reception of mail had to be paused for specific domains only during the migration of the messages, update of the MX record and so on.

For postfix I was able to implement this using

   = check_recipient_access hash:/etc/postfix/access

The /etc/postfix/access file referenced above needs to contain the domain for which delivery should be deferred, along with the action defer, for example: DEFER

Also, don't forget to postmap /etc/postfix/access.

tuesday the 25th of july, 2023

proxying via ssh


One of the recurring jobs coming up when running mail servers is to get the IPs of your mail servers off various blacklists where they happen to turn up for in part completely unknown reasons. In order to get an IP from a blacklist the list owners have invented various ways to achieve this, and one I recently came across required some confirmation on their website while having my user agent coming from the blocked mail IP in question.

This is when I came to know about ssh's proxy abilities for the first time. It turned out that ssh has complete support for proxying, so the only two things I had to do was to first create the associated ssh tunnel for proxying:

ssh -D 8080 my.mail.server -N -v

Now ssh will forward all connections going to port 8080 on the local machine via the my.mail.server machine. This, of course, only works when on this machine ssh is set up. Now, the only thing left to do is to configure a proxy server in your favorite web browser, pointing to 8080 on localhost, and voilà – your browser's traffic is being forwarded through the mail server.

tuesday the 2nd of may, 2023

debian on nipogi-jk06


I've been looking for two simple budget machines to run debian with icinga nodes in HA-mode on. Usually the raspberrys I've been so far using would've been enough, but since the supply chain shortage it's been practically impossible to get new ones, except for creatively overprized ones.

Since it's now possible to get some fairly standard mini PCs for a just a little bit more money, I've got myself a set of nipogi jk06, which is a machine based on a Celeron N5100 with 8GB RAM and a 256GB M.2 SSD. This also had the advantage that using these as a desktop machine in parallel to their server duties wasn't a problem anymore, as would've been with a raspberry. Everything's been working fine except suspend/hibernation: The machine woke up when it was triggered from keyboard/mouse, but the screen stayed dark and couldn't be revived except via rebooting the entire machine.

In general, this wouldn't have been a problem with a server machine running 24/7,but I wanted to have this resolved despite of this. Installing the intel-microcode and non-free firmware package and the latest backported kernel did the trick.

apt-get install intel-microcode firmware-misc-nonfree
echo "deb bullseye-backports main" \
 >> /etc/apt/sources.list
apt update
apt -y -t bullseye-backports install linux-image-amd64 firmware-misc-nonfree

After that, in order to disable any kind of sleep or hibernation on the machine:

systemctl mask \
friday the 28th of april, 2023

fixed ipv6 assignment


While SLAAC is very conveninent to get multiple hosts configured with minimum effort for ipv6, it's often nice to have a set of shorter addresses for some hosts – it's much easier to remember fd00:0:0:10::1 than fd00:0:0:10:3047:8f88:6801:87b0.

This can be achieved by setting up your own router advertisement and dhcpv6 server. Using radvd, we're setting up our own example unique local address range fd00:0:0:10::/64 in /etc/radvd.conf:

interface eth0 {
  AdvSendAdvert on;
  AdvManagedFlag on;

  prefix fd00:0:0:10::/64 {
    AdvOnLink on;
    AdvAutonomous off;

Setting AdvManagedFlag on will make hosts setting up ipv6 on eth0 query the local dhcpv6 server for an ip address in our example prefix range. The AdvAutonomous off statement will ensure that ip addresses for the given prefix are only assigned by the dhcpv6 server, so that interfaces don't get multiple addresses in the fd00:0:0:10::/64 range due to SLAAC.

ISC's dhcpd server can be used to handle the dhcp requests. In /etc/dhcp/dhcpd6.conf, we'll set up our host “my-machine” so that it is always assigned ip fd00:0:0:10::1. The host is identified by it's DUID – the DHCP unique identifier – following dhcp6.client-id. The DUID can be obtained from your OS, although I found it more convenient to just look at dhcpd's logs when ip address are advertised. All other hosts are assigned addresses from the pool defined by the range6 statement.

subnet6 fd00:0:0:10::/64 {
  range6 fd00:0:0:10::1000 fd00:0:0:10::1fff;

  host my-machine {
    host-identifier option dhcp6.client-id 00:01:00:05:79:cb:06:ac:b0:88:17:7b:dd:bf;
    fixed-address6 fd00:0:0:10::1/64;

In case unknown clients shouldn't get any address at all, the range6 statement can be replaced by deny unknown-clients. Also, assigning the system's own fixed ip via dhcpcd might be “too slow” after booting and dhcpd might come to the conclusion that the address space it's providing dhcp for doesn't exist and complain “No subnet6 declaration for eth0 (no IPv6 addresses)”. In this case, set denyinterfaces eth0 in /etc/dhcpcd.conf and configure the interface using /etc/network/interfaces or related.

tuesday the 25th of april, 2023

multi-gateway openvpn server


Lately, I had to provide access to a private network over the internet using openvpn. For redundancy reasons, it had to be accessible via two separate gateways, so that whenever one failed, the private network would still be accessible using the alternative gateway. I'm skipping a lot of headache requirements / givens and just describe the solution core.

The main problem is that, when we're supposed to handle traffic for two separate internet gateways, we'll have to handle multiple default gateways. When a packet arrives from a remote IP, we have no way of telling which of the two gateways we'll have to send the reply to. To solve this, the openvpn gateway linux VM was connected via separate NICs to each of the gateways. I set up two openvpn server processes, each listening on one of the NICs. In order to implement two “default gateways” on a single machine, two additional route tables – “rt01” and “rt02” are created, each having their own default gateway and

echo 10 rt01 >> /etc/iproute2/rt_tables
echo 11 rt02 >> /etc/iproute2/rt_tables

ip route add default via table rt01
ip route add default via table rt02

Now we'll tell the routing policy database that all packages from our first openvpn server running on should use rt01, while the other one should use rt02:

ip rule add from table rt01 
ip rule add from table rt02

This helps working around the issue that openvpn always determines the default gateway on startup and always uses this for any outgoing communications.

wednesday the 15th of march, 2023

handling multiple ssh identities


Once you're using multiple identities for services like github or gitlab, along with multiple SSH keys for authentication with these systems, there's the need to tell SSH which of your keys should be used for a new connection. This can be achived using a combination of the IdentityFile and IdentitiesOnly statements, as in

IdentityFile ~/.ssh/id_ed25519-key02
IdentitiesOnly yes

While the former on it's own just adds another key to the set of identities that the SSH client will use for authentication, the latter ensures that only this specified key is used to connect to the given host and the other keys known to the ssh-agent should be ignored.

There's a little catch however: Once a key has been loaded by the local ssh-agent, it will be kept in the agent's memory until it is explicitely removed. This isn't a problem as long you're only initiating ssh connections from your local machine, but once you're using forwarding on a remote system the agent will use all the keys he has stored locally, not only the one you've specified using the IdentityFile for the remote machine you're using forwarding on. To resolve this, you can invoke ssh-add -D to clear the local agent's key storage.

monday the 20th of february, 2023

persistent dummy NICs


For monitoring purposes of a raspi device, which only has dynamic IP addresses assigned, I needed a virtual dummy NIC which can be assigned a static IP. In order to have this dummy0 interface avilable after boot the following configuration needs to be written to /etc/network/interfaces:

auto dummy0
iface dummy0 inet6 static
  address fd00:0:0:10::1
  netmask 64
  pre-up ip link add dummy0 type dummy
friday the 10th of february, 2023

icinga cluster check


In case all satellites from a non-master zone are going offline at once – if, for example, the only connection to the zone has gone down – there are initially no notifications since there's no entitiy left which could relay messages to the parent/master zone. This is where icinga's “cluster-zone” check joins the game:

apply Service "satellite-zone-health" {
  check_command = "cluster-zone"
  check_interval = 30s
  retry_interval = 10s
  vars.cluster_zone = "child-zone-name"
  assign where match("master*",

The cluster-zone check can be assigned to the master/parent nodes of a child zone. It will check whether the child zone can relay messages to the parent, and will complain if it doesn't.

Since agents are implemented as single endpoints in their own zone, the cluster-zone check can also be applied to agents of a zone:

apply Service "agent-health" {
  check_command = "cluster-zone"
  display_name = "agent-health-" +
  vars.cluster_zone =
  assign where == "current-zone" && host.vars.agent_endpoint && !match("master*",

The agent check is applied to all non-master/non-satellites hosts in a zone which have an agent assigned. As with the zone-based check, this check will complain when the assigned agents cannot relay messages any more.

sunday the 1st of january, 2023

sherlock holmes in the public domain


Although the copyright for the Sherlock Holmes canon had already expired almost everywhere in the world, some stories remained copyrighted in the U.S. until the end of 2022. Starting January 1st, 2023, the last stories from the casebook of Sherlock holmes entered the public domain and can now also be downloaded legally from the U.S.

The casebook of Sherlock Holmes is now available for download for everyone from

tuesday the 20th of december, 2022

write down everything


Reading through Brendan O'Leary's post “What I learned at GitLab that I don't want to forget” I was struck immediately by the very first point he brought up: “Write down everything”, simply because over the last few years I've come to realize that this turned out to be the one of the most important aspects of my work.

Quoting his post: But when it comes to processes and memory, people are very inconsistent… that's just human nature. It's only possible to acknowledge the full meaning of this statement when you're able to to look up the actual decisions, or even better, the whole path of reasoning leading to these agreements / assignments once they have been forgotten by almost anyone.

friday the 2nd of december, 2022

ipv4 address blocks for documentation


Turns out the IETF has assigned three subnets for the sole purpose of documentation. RFC 5737 says: The blocks (TEST-NET-1), (TEST-NET-2), and (TEST-NET-3) are provided for use in documentation.

saturday the 5th of november, 2022



Having to keep a large number of systems operational requires some kind of monitoring, which in turn needs to be able to connect to the monitored systems. So far I've set up connections using OpenVPN or SSH, but using wireguard turned out to provide the best of both worlds.

On the one hand, configuration is extremely easy. One simply needs to create a public/private key pair on both the client and the server:

wg genkey | tee privatekey | wg pubkey > publickey

A client's configuration file will include the client's private key and the server's public key, along with, in this example, the client's tunnel-ip and the server's public internet address

PrivateKey = +IbvH5g+ArYgwOnJfeIs1y+5DUtZ8NdpoJODdW2pfW8=
Address =

PublicKey = LGVTEOmohkN7Iog7w9g20upjL+NzFLseqI6dmEEj4Q8=
AllowedIPs =
Endpoint =
PersistentKeepalive = 20

For the server we'll put the server's private key and the client's public key in the file, along with the server's tunnel-ip, the udp port the server is listening on an the client's tunnel-ip.

Address =
ListenPort = 51820
PrivateKey = 8GV73D7/04YxOkdnvsrCSKmZ1EVImzKxM2IbIilwJ30=

PublicKey = pj/vmU+hz0Rn9uMkR33qb81YKqINZN55gwqQD7UeLDo=
AllowedIPs =

If the configuation resides in /etc/wireguard/wg01.conf the tunnel can be brought alive on both sides using:

systemctl start wg-quick@wg01

That's all. It appears that there's a separate [Peer] section required for every client, which in turn means every client needs to have a fixed private ip, contrary to OpenVPN where an IP is assigned by the server from a pre-defined subnet.

It's also notable that the AllowedIPs parameter defines which ip ranges are routed thought the subnet. That means it's extremely simply to route additional ip ranges: In case there's a private subnet on the other side of the tunnel, one can simply alter the AllowedIPs parameter in the following way on the client to access the subnet in question:

AllowedIPs =,

Wireguard will also work nice with ipv6. You'll just have to define an additional ipv6 tunnel, which can be done alongside of the ipv4 settings:

Address =, fd00:0:0:10::1/64
ListenPort = 51820
PrivateKey = 8GV73D7/04YxOkdnvsrCSKmZ1EVImzKxM2IbIilwJ30=

PublicKey = pj/vmU+hz0Rn9uMkR33qb81YKqINZN55gwqQD7UeLDo=
AllowedIPs =, fd00:0:0:10::2/128