blog
wireguard-before-ssh
It's time. After several incidents – terrapin, openssh/xz and the signalhandler/race-condition – all within the timespan of a single year, I've started rolling out servers which have their ssh port bound to a wireguard interface. Which means that without authenticating and connecting via wireguard first, the ssh port is not accessible from the public internet.
removal from spam blocklist by payment
Yesterday, the blacklist monitor at MxToolbox notified me that one of my mailserver IPs was just blacklisted by uceprotect-network. When I started to look into this, I was told that it's not my fault, but that I'm living in a bad neighborhood.
simple raspi for k8s setup
Since I've been setting up a lot of kubernetes installations on raspberry pi machines and still doing a lot of these I've been looking for a speedier way to get the hardware up and running. There's already the raspberry pi imager which helps a lot, but I've been looking for something which automizes even more steps. I ended up putting a small shell script together which allows writing the image in a single step and have everything in place to jump right in using ssh and start with the k8s setup itself.
restoring data from macOS addressbook backup
Today I noticed that one of my contact cards from my macOS's address book
was missing. Since I've got Apple's advanced data protection enabled, there's no way to
rewind the address book via icloud.com. To work around this
I'm regulary creating local backups using address book's “Choose File”
→ “Export” → “Contacts Archive”. I didn't want to simply restore
last week's backup though, I rather frist wanted to know what had
changed between last week's and today's state. I also didn't want to run
a restore operation since I've linked two external carddav accounts
into the address book and I had no idea what would happen to them.
ecdsa with traefik
In the context of the BSI's new requirements for TLS
I've been looking at a way to configure traefik
to use ECDSA certificates. Here's an example with
traefik having two certificate providers configured:
le-rsa2048 for RSA 2048-based certificates and
le-ecdsa for ECDSA certificates.
convert outlook .msg to .eml
Today I tried to export an e-mail from the Office 365 cloud using Outlook.
The mail in question had a
size of just a bit over 32 gigabytes, which meant that the OS X Version
of Outlook complained about exceeding the maximum supported size and
wasn't able to sync it at all. Since I didn't want to touch any
of the Exchange-related configuration – no idea whether that's possible
at all – I tried my luck with the Windows based version. This one was
able to export the message, but if course the resulting file was in
Microsoft's proprietary .msg format.
send dmarc report to external domains
Without extra measures, the reporting functionality in dmarc for aggregate|forensic reports is limited to sending mail to the same domain that the dmarc record describes. Sometimes however, for example in case of a null client – which only sends but never receives mail – it's desirable to have the reports send elsewhere.
git repository cleanup using rebase
Recently I had a few git repositories to
clean up. Over time, of lot of issues had evolved:
There were a few hundred commits spread over the last fifteen years,
from which a lot were auto-migrated from an ancient
cvs
repository so they didn't conform in any way to git's commit message
recommendations, I did commits using at least five different – partially
invalid – e-mail addresses, there were some empty commits resulting
from the cvs2git-conversion and only about half of the
commits were
signed.
The BSI's TLS handshake signature requirements
Germany's BSI, the Federal Office for Information Security, released a set of requirements for TLS communication some time ago which are binding for everyone desiring to work with government authorites. Some of these prove to be quite challenging for various reasons. They're called “TLS nach TR-03116-4 / Checkliste für Diensteanbieter”, the “check list for service providers”.
apt hold, skip packages during updates
A few days ago I ran into a bug resulting from a regular update from Grafana's apt repositories. Promtail, which I'm using to import syslogs into Grafana Loki, just broke down.
sender rewriting scheme
The “Sender Rewriting Scheme” is designed to handle the problem that SPF by itself breaks mail forwarding. When Server A sends a mail to machine B, which in turn forwards it to system C, server C's SPF check will fail since it concludes that machine B isn't allow to send from a server A's mail address. This might result in authentication results like this:
influx: get minium time value from all series
In order to free some diskspace from my influx databases, which are storing all the performance metrics from my icinga installations, I've been trying to remove the older, no longer relevant records, from the database. In order to get an overview over the current data, I've been trying to find out for how long I've been storing data at all. However, it appears there's no easy way to query the mininum time value for all series/measurements.
macOS: delv says “no crypto support”
delv is a tool for DNS lookup and validation. I've been trying to check the DNSSEC status for some domains, on macOS however, there's just an error message.
dane and tlsa basics
DANE –
DNS-based Authentication of Named Entities –
stores hash digests of certificates in TLSA DNS resource
records. In combination with
DNSSEC
it is possible to verify certificates without any
CA
using DNS alone, thereby eliminating man-in-the-middle and
downgrade attacks.
kobold letters
Turns out specific e-mails, called Kobold letters, may change their contents when they're forwarded, simply by putting properly coded CSS into the mail.
minimal inwx certbot handler
I've been trying to implement the ACME DNS-01 challenge with certbot and INWX. I'm doing this for a MFA / MobileTAN protected account, which limits the choice of API clients to the INWX PHP client.
removal from private docker registry
It seems there's no easy way to remove tags / images from a private docker registry. It appears the most straightforward way is to get the manifest digest and to put it into a HTTP delete request.
accessing a private docker registry
While pushing and pulling to a private docker registry can be done via built-in methods, examining the contents apparently can only be done using separate HTTP queries.
apt: packages kept back
Some day, when applying upgrades with the apt command line interface, the tool might state that some packages were kept back.
smtp: dsn versus mdn
There are two ways to get information about successful e-mail deliveries: DSN and MDN.
spf records for helo/ehlo
While running various tests for mail servers, I stumbled upon
SpamAssassin's SPF_HELO_NONE warning. It
incurs a negative score of 0.001, and the short description
complains that “HELO does not publish an SPF Record”.
smtp smuggling
At the end of 2023, news about a novel kind of attack on SMTP servers was published: SEC Consult found a way to concatenate multiple e-mails in a way that made many systems process the second mail as if it had been submitted on it's own – along with all the privileges that were granted for the submission of the first one. While the MTA itself isn't attacked, this can be used to make any affected mail server act like an open relay.
migrating swap partitions
After replacing a VM's swap partition with another one, I noticed that the system took considerably longer to boot.
no identification using MAC address with dhcpv6
I've been trying quite unsuccessful to exclude certain clients from getting an IPv6 via DHCP in a network. Since I just wanted to exclude specific interfaces I've used the MAC address instead of the DUID – the “DHCP unique identifier”, see DHCPv6. As it turns out, although the option to exclude/identify clients via MAC addresses may be present in dhcpv6 servers, it can't be used reliably at all. As the official kea documentation states: “Unfortunately, the DHCPv6 protocol does not provide any completely reliable way to retrieve that information.”
microsoft-activision takeover
Today, on friday the 13th, microsoft acquired activision, and with it the trademark and all that is left of Infocom, Inc. If the name “Infocom” is known to you at all, you might want to consider sharing “Microsoft consumes Activision; and a plea” or share the associated post on mastodon.
get mac address for ipv6
For IPv6, hosts use Neighbor Discovery instead of
ARP for IPv4. Accordingly, one can use the
ndisc6 tool to look for the MAC
address in question.
kea dhcpv6 fails to bind link-local
Lately, after setting up an instance of an isc kea dhcp6 server , I noticed that after a reboot it would be inactive, although it had been started properly. Turned out that it simply couldn't bind the link-local address.
simplest hd keep-awake
I've been trying to get some long-type smartctl tests to run through uninterrupted. Since they're taking about 11 hours for a 4TB hd – yes the old, spinning ones – these were so far always interrupted by the hd going to sleep. After looking at some measures to deactive the various sleep mechanisms I found the best and simplest one.
prompt failover with isc-kea-dhcp
After migrating to the new isc
kea dhcp server - the successor to the older
isc dhcp server –
I've struggled a bit to get a server pair to do a proper
failover when one of the servers fails. Turned out that
there's a max-unacked-clients parameter,
which tells the system how many dhcp clients need to have
sent out dhcp requests before the failover occurs. By
default, this is set to 5, so until you don't have five
different clients waiting for an IP address, nothing's
going to happen. I ended up simply setting this to
0, so once the timeout set in
max-response-delay is met, there's always
a guaranteed failover to the surviving server.
no local nagios dhcp check
One of my server pairs is running icinga and a dhcp
server on each of them in HA mode for redundancy reasons.
I've been trying to monitor the dhcp service using
the nagios
check_dhcp plugin. With the servers checking
themselves, however, I mostly got many
CRITICAL: No DHCPOFFERs were received
replies.
btrfs send and receive
brtfs snapshots are great for incremental backups – just create a snap from a working directory and keep on happily working on the original folder as you please: btrfs makes sure that only incremental changes from the snapshot to the current state will occupy space.
working around microsoft blacklisting
Catching up on yesterday's post: It's hard to deny that self-hosting mail for individuals or smaller companies has become a much greater challenge nowadays.
office 365 “junks” microsoft mail
With all the fuzz these days about getting mail from stand-alone running smtp servers to be recognized as non-junk by the big platforms, it's quite funny to see that even Microsoft can't keep up: On a company's exchange account, which I've been assigned to use, microsoft now sorts its very own e-mails advertising the new teams app and other things into the “junk” folder all by itself.
btrfs snapshot's exclusive space
How much space does a btrfs snapshot actually exclusively
allocate? One simply has to run a
btrfs fi du -s backup-*
in order to see which space is shared between the
snapshots and which is exclusively used by the snapshot
listed.
strict versus real-strict imapsync
imapsync is
an extremely useful tool for the migration of imap accounts.
While trying to migrate accounts with a very large number of
messages, I encountered a few warnings about duplicates. The
imapsync FAQ says the it's a problem with message
identification – imapsync by default uses the
Message-ID: and Received: headers
to identify messages on both sides, which may fail when,
for example, imap servers change one or more of these
headers.
forcing windows to use openvpn-dns
While providing windows dial-in vpn clients with the
dns servers addresses of the internal network using
the dhcp-option DNS parameter, I found
out that the name resolution didn't work reliably.
After some research it turned out that this was due to
windows just adding the provided dns addresses to the
ones already present on the system, and using all
of them for the actual name resolution.
defer domain-specific postfix delivery
Some time ago I had to migrate a mail server running multiple domains, whereby these domains were to be moved one after another instead of moving everyhing at once. That meant that the reception of mail had to be paused for specific domains only during the migration of the messages, update of the MX record and so on.
proxying via ssh
One of the recurring jobs coming up when running mail servers is to get the IPs of your mail servers off various blacklists where they happen to turn up for in part completely unknown reasons. In order to get an IP from a blacklist the list owners have invented various ways to achieve this, and one I recently came across required some confirmation on their website while having my user agent coming from the blocked mail IP in question.
debian on nipogi-jk06
I've been looking for two simple budget machines to run debian with icinga nodes in HA-mode on. Usually the raspberrys I've been so far using would've been enough, but since the supply chain shortage it's been practically impossible to get new ones, except for creatively overprized ones.
fixed ipv6 assignment
While
SLAAC is very conveninent to get multiple hosts
configured with minimum effort for ipv6, it's often nice to have
a set of shorter addresses for some hosts – it's much
easier to remember fd00:0:0:10::1 than
fd00:0:0:10:3047:8f88:6801:87b0.
multi-gateway openvpn server
Lately, I had to provide access to a private network over the internet using openvpn. For redundancy reasons, it had to be accessible via two separate gateways, so that whenever one failed, the private network would still be accessible using the alternative gateway. I'm skipping a lot of headache requirements / givens and just describe the solution core.
handling multiple ssh identities
Once you're using multiple identities for services like github
or gitlab, along with multiple SSH keys for authentication
with these systems, there's the need to tell SSH which of
your keys should be used for a new connection. This
can, for example, be achived using a combination of the
IdentityFile and IdentitiesOnly
statements.
persistent dummy NICs
For monitoring purposes of a raspi device, which only has dynamic IP addresses assigned, I needed a virtual dummy NIC which can be assigned a static IP.
icinga cluster check
In case all satellites from a non-master zone are going offline at once – if, for example, the only connection to the zone has gone down – there are initially no notifications since there's no entitiy left which could relay messages to the parent/master zone.
sherlock holmes in the public domain
Although the copyright for the Sherlock Holmes canon had already expired almost everywhere in the world, some stories remained copyrighted in the U.S. until the end of 2022. Starting January 1st, 2023, the last stories from the casebook of Sherlock holmes entered the public domain and can now also be downloaded legally from the U.S.
write down everything
Reading through Brendan O'Leary's post “What I learned at GitLab that I don't want to forget” I was struck immediately by the very first point he brought up: “Write down everything”, simply because over the last few years I've come to realize that this turned out to be the one of the most important aspects of my work.
ipv4 address blocks for documentation
Turns out the
IETF
has assigned three subnets for the sole purpose of documentation.
RFC 5737
says:
The blocks 192.0.2.0/24 (TEST-NET-1), 198.51.100.0/24
(TEST-NET-2), and 203.0.113.0/24 (TEST-NET-3) are provided
for use in documentation.
wireguard
Having to keep a large number of systems operational requires some kind of monitoring, which in turn needs to be able to connect to the monitored systems. So far I've set up connections using OpenVPN or SSH, but using wireguard turned out to provide the best of both worlds.