Sophos XGS Let’s Encrypt

This blog post is split into two parts. Part 1 is the journey to the solution, Part 2 is how to setup our Let’s Encrypt tool on your Sophos firewall. If you are only interested in the latter you can skip ahead.

Journey to a working prototype

For those that dont know what a Sophos XGS is: it’s a firewall product offered by Sophos and its a “replacement” product for the previous Sophos XG and Sophos UTM.

The Sophos UTM used to have excellent support as a web application firewall with Let’s Encrypt, but Sophos removed that feature in the XG and XGS version of their firewall offering. As we use Let’s Encrypt for most of our customers we set out to work around that.

If you Google for solutions you are either offered to use Sophos Factory or to use Let’s Encrypt on another host (windows/linux) that pushes the certificates to the Sophos firewall. It was not satisfying for us to run the certificate renewal process on different hosts so we decided to try to do the whole process on the firewall itself.

A XGS runs a Linux Kernel 4.14 and allows ssh login and a real shell via 5. Device Management -> 3. Advanced Shell. We figured this could be used to run the required programs. We decided to use the excellent acme.sh client to try to get a working http acme process and then see if we can use this to get a working setup.

For a working HTTP acme process you need port 80 to serve static files that are provided by acme.sh. To achieve that we tried getting the httpd that is running on the XGS to serve those files on any port for a start. This turned out to be rather tricky as the httpd config on a XGS requires ports and file locks that are already in use by the Sophos web interface. Luckily the XGS includes a python3 version, with a http module. So we got a working webserver that was serving files on a port of our choice like 8000 but not 80. Port 80 is either used by httpd when the “waf http to https redirection” is enabled or the Sophos iptables firewall rules drop the pakets somewhere between the prerouting and input filter (See Link Layer part of the Diagramm, our python webserver is running on the red application layer).

Packet flow diagramm for ip tables
Packet path that a packet takes through the ip tables firewall

Our initial reaction was to disable the http to https forwarding and add iptable rules to get the packets to our python webserver running on port 80. But we could not figure out which rules that are responsible for dropping those packets and it was quite ugly approach as it required to deactivate the http to https redirection.

After some consideration we had the idea to just redirect the port to our python webserver during the acme process, with

iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination :8000

This approach allowed us to not touch any Sophos settings and redirect the port for the time being and worked like a charm.

With having a webserver that serves our static files generated by acme.sh we were all set to extend this proof of concept to a script that automatically renews Let’s Encrypt certificates running fully on a Sophos XGS.

To have a real prototype we would need a storage path that is permanent between reboots of the XGS and mechanism to run our script every day.

The storage path problem was solved quite easily but just creating an acme folder in /config/ and reboot the firewall to test if our folder was still there, as it was after a reboot so we moved to the next problem.

Running a script everyday on a linux machine is usually done by cronjob or a systemd timer. Both mechanisms are not available on a Sophos XGS. So we tried the little more obscure command at which was also not available. So we opted to use a while loop with sleep to run every day. Which solved the timing problem, but we still needed something in the system to start our script after each reboot.

As the sophos xgs is a busy box system rc.d init scripts are supported and after mounting the root file system read/writable we where able to add a short script to it S01acme which just launches our renew.sh and does not wait for completion. After we where done we switchted the root filesystem back to read only.

With all those building blogs we where able to implement a working lets encrypt client directly on the sophos.

How to setup sophos-xgs-letsencrypt

First you need to login to your sophos via ssh and open an advanced shell.
Then you can run

sh -c "$(curl https://raw.githubusercontent.com/helsinki-systems/sophos-xgs-letsencrypt/main/setup.sh)"

This should take care of the installation and setup your letsencrypt account. For more info head over to the sophos-xgs-letsencrypt GitHub

After that you need to configure which domains and certificates that should be renewed, this is done by editing the /var/acme/config.csv (; semicolon seperated) with vim. Each row contains the name from the certificate in sophos webinterface in the first field and in the second field a list of domains to be requested for that certificate (seperated with a comma ,)
If you don’t have a current certificate name just pick one and use it later in the web interface (in our example that’s OWA)

To get the name for the first field you need to upload a certificate with the pem format to the Sophos web interface (under Certificates).

Example:

OWA;owa.helsinki-systems.de,exchange.helsinki-systems.de

After the configuration is done you need to run /var/acme/renew.sh to request the certificates for the first time.

Depending on if you are adding an already existing certificate to Let’s Encrypt, as in it already existed in your Sophos XGS, you are done now congrats. Otherwise there is another step.

If you do not have the certificate in your XGS yet you need to copy it to your computer using scp on linux or WinSCP on Windows.

scp admin@yoursophosip:/config/certificate/OWA.pem ./
scp admin@yoursophosip:/config/certificate/private/OWA.key ./

and reupload them using the sophos web interface under certificates.

Upgrading the Firewall

When you upgrade your firewall you have to rerun the /var/acme/setup.sh to readd the scripts to the sophos system.

Future features and open questions

Firmware Upgrade persistence

Currently we don’t know what happens to our folder in /var/acme after a firmware update if somebody knows the answer or is willing to test it and share their results, we would be happy to know (sophos@helsinki-systems). We ran a quick test upgrading from 19.5 to 20.0 and our old config folder was gone after the upgrade as well as our init script. So we switchted back to /var/acme

Old experiy stats in the sophos webinterface

Currently the expiry stats in the sophos webinterface are not updated after a certificate is renewed. If we find out where those information is stored this could be updated as well. Or handeld via the sophos firewall api

Redirect all other requests to https while running

Currently while the acme is running we are not redirecting the requests to https, but this could easily be added by a little python code.

Warranty

As it has been pointed out by the excellent LuCar Toni in the sophos community forum. Thinkering with your firewall could void the warranty. So be careful of what you do.

NixOS Ocean Sprint Retrospective

“It would be nice to escape the cold and rainy weather of German winter” is something a lot of Germans would say during the cold season of December. So we did. Andy and Janne (which will now be referred as “us” or “we”) got the honor of being invited to the NixOS Ocean Sprint in Lanzarote on the Canaries. It was honestly great to escape home after not leaving home for about two years due pandemic-related reasons. While we were told it was the coldest December in a long time it was million times better than what we got reported from home. Spending time in the sun in December is just great.

While this sounds like NixOS-sponsored vacation it was actually a very productive week of “intense NixOS hacking” (as the Ocean Sprint website puts it). In total there were 14 people hacking in a villa in Costa Teguise. We were able to exchange ideas, come up with new concepts and help each other out. There wasn’t a greater goal, just everyone bringing their NixOS projects to an island and working on them together with others. It was good to finally meet the people you have been working along for years now and being able to cooperate in person was really motivating and helped to do a lot of work very quickly. There were all sorts of larger projects, medium-size projects (like us working on switch-to-configuration.pl or refactoring the MariaDB packaging), and small ones like the regular reviewing/merging efforts along with discussions about helping the organization to steer into a more organized form.

All this being said, it wasn’t just work either. We were able to enjoy volcano climbing, body surfing, and more, along with the great weather and awesome people to spend the time with, just chatting. Most of us took another week of vacation to see more of the island and just to have a good time.

Solving the y2038 Problem with NixOS and XFS

Ok, the headline is kind of sensational, but this post does actually talk about how you can make your XFS store files with timestamps after 2038 and how that can be done nicely with NixOS.

The Problem

The X FileSystem, more commonly known as XFS, has historically only supported timestamps until 2038.

However, in version 5.10 of the Linux kernel, the XFS maintainers landed the bigtime feature, which allows for “inode timestamps from December 1901 to July 2486, and quota timer expirations from January 1970 to July 2486”. GRUB2 from version 2.06 also allows you to boot from such a filesystem.

Changing these feature flags does however require running a xfs_admin command while the filesystem is not mounted, because it needs to modify some on-disk metadata structures.

The Solution

Since we did not want to reboot every system into a live system and perform this manually or re-image them or whatever else one could do, we came up with the following idea: Why not just do this in the initramfs, before the filesystems are mounted?

Luckily, NixOS offers a handful of options to add custom scripts and binaries to the initramfs.

So we got to work and implemented a NixOS module, as we do.

This is what we came up with:

{ config, lib, pkgs, utils, ... }: let
  # TODO: remove "/nix" filter, once feature is declared stable by upstream
  fileSystems = builtins.filter (fs: utils.fsNeededForBoot fs && fs.fsType == "xfs" && fs.mountPoint == "/nix") config.system.build.fileSystems;
in {
  config = lib.mkIf (fileSystems != []) {
    boot.initrd = {
      extraUtilsCommands = /* sh */ ''
        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_admin
        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_info

        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_db
        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_spaceman
      '';

      extraUtilsCommandsTest = /* sh */ ''
        sed -i -e 's,^#!.*,#!'$out/bin/sh, $out/bin/xfs_admin $out/bin/xfs_info   # from nixos/modules/tasks/filesystems/xfs.nix
        export PATH=$out/bin:$PATH
        $out/bin/xfs_admin -V
        $out/bin/xfs_info -V
      '';

      postDeviceCommands = /* sh */ let
        # basically fsInfo from stage-1.nix, but without options and only xfs
        # we can't reuse fsInfo, because of how these things are substituted in
        xfsInfo = lib.concatMapStringsSep "\n" (fs: if fs.device != null then fs.device else "/dev/disk/by-label/${fs.label}") fileSystems;
      in ''
        echo '${xfsInfo}' | while read device; do
          if [ "$(xfs_info "$device" | grep -o 'bigtime=.')" = "bigtime=0" ]; then
            xfs_repair -n "$device" && xfs_admin -O bigtime=1 "$device"
          fi
        done
      '';
    };
  };
}

While this might seem scary at first glance, it is actually relatively straight-forward.

It does the following things:

  • Find the relevant filesystems. For now, this is limited to /nix, because while the XFS maintainers believe their code to be good and the on-disk format to be fine, they have still marked this feature as experimental for now (Edit: the feature is no longer marked experimental as of this commit which was released with kernel 5.15). When I last asked them, their timeline for removing this warning was “when we’ve waited long enough for nobody to find a problem”.
  • It adds the relevant binaries from xfsprogs to the initramfs. It also tests if they can actually be run.
  • It adds some shell code to the initramfs which performs the actual change on the filesystem. For each filesystem it checks if it has the feature already, if not it performs a fsck, because this change should and can only be performed on a clean filesystem and if the check succeeds it adds the feature.

After a host has a system with the configuration from this module applied to it, the only thing needed is a reboot and the filesystem will have support for the new feature. At least assuming the filesystem was clean.

XFS also gained some other potentially interesting features in recent years, like crc checks for metadata and support for copy-on-write files using reflink, both of which are enabled by default when creating a filesystem with mkfs.xfs from a recent xfsprogs version. If you have filesystems created before these features were available or enabled by default and want to enable them, the only things that need to be modified in the module are the check for which features are enabled and the xfs_admin command.

Shameless Self-Promotion

What follows is more or less just self-promotion, although it might still be relevant to your interests.

At Helsinki Systems, we run most of our infrastructure on NixOS. We decided on XFS on top of LVM2 for most of our storage needs, mainly because it provides the features we need while being very stable, relatively well integrated into the greater Linux ecosystem and without the need for out-of-tree kernel modules.

This storage stack is not particularly “sexy”. It is not ZFS, btrfs or bcachefs. It does not need to be that in our opinion. We would however like to be able to use all features it provides. For XFS that includes features like reflink copies and CRCs. For LVM2 that means thin-provisioning, raids and maybe also dm-integrity and VDO in the future.

To that end, we took over maintenance of lvm2 and xfsprogs in nixpkgs. We will also try our best to always keep them up to date in the future and inform the community of relevant changes.

In case you are looking for someone that will work on solutions as small as what we presented here or as large as packaging all of BigBlueButton for NixOS, feel free to shoot us an e-mail or give us a call. We also help with the maintenance of exim and dovecot in nixpkgs along with other packages, so we should hopefully be able to receive your e-mails 😀

Creating a minimal Windows ISO

So first off, Hello World! We’re Helsinki Systems and this is our blog about Linux, NixOS, and whatnot. But instead of starting of a stream of Linux posts, let’s start with something slightly different: Creating custom Windows ISOs. Why? Because it’s useful if you live in a world where installing Windows is something you do sometimes. Also it’s really handy to have a central way of upgrading all Windows installation files you handed out to your co-workers (we’ll discuss this in another post).

WinPE

As many people switching from Windows to Linux notice, most Linux distributions come pretty small by default and most functionality can be added later using some package manager. This means that the distribution used for the regular OS can also be used for a small installer – this is different with Windows. Most distros download the packages for the to-be-installed OS from the internet, which makes the installation images even smaller. Windows, on the other hand usually bundles all required files into the image which makes offline installation possible. However, the files don’t just lay around on the filesystem of your ISO. Instead they are packed into wim (Windows Imaging Format) files and not ready for use. This means the installer needs to use a different, smaller, system which is used for unpacking the wim files into the target system while also being tiny enough to be deployed via methods like PXE. This is WinPE (Windows Preinstallation Environment). WinPE doesn’t contain the files required for installing Windows and is therefore tiny compared to the Windows ISO you download from Microsoft. You could compare WinPE to a minimal linux image with a read-only root filesystem and just a shell (cmd; not PowerShell) with a few basic tools. Let’s explore that!

Getting the ADK

In order to explore WinPE, let’s build an ISO! The default Windows installer is huge and preconfigured, so we’ll build our own WinPE environment. For this, we need the Windows ADK (Windows Assessment and Deployment Kit) which you can get from here unless the link is dead by the time you discover this post. You also need the WinPE add-on (on the same page as the ADK). Last, but not least, if the current ADK version is still 1903, also install the WSIM (Windows System Image Manager) update which makes WSIM ready for use. Although this is not used for the purposes of this blog post, it’s a handy tool and might come into play in a future post. You should have a “Deployment and Imaging Tools Environment” shortcut in your start menu under the “Windows Kits” folder. I’m not sure if this needs to be run as administrator, I usually do this because I create my images in C:\.

Building WinPE

You should have your cmd open now with the correct environment. As mentioned, I work directly at C:\, so let’s copy all WinPE-related files there for a x64 installation image:

> cd C:\
> copype amd64 C:\WinPE_x64

Inside C:\WinPE_x64, you’ll find three directories. You’re likely not going to touch fwfiles, as it contains some files relevant to WinPE startup. mount is empty (we’ll get to this in a second), and media is what’s going into your WinPE image. It’s essentially the root folder of your ISO. But before exploring anything, let’s create an ISO image:

> MakeWinPEMedia /ISO C:\WinPE_x64 C:\WinPE_x64.iso

Burn this ISO onto a CD or just boot some VM with it like a normal person. On my Linux machine, I use QEMU:

$ qemu-system-x86_64 -m 2G -smp 4 -cdrom WinPE_x64.iso

After some time, you’ll be greeted with a cmd that is running wpeinit which will exit after some time and leave you with a cmd. Time to look around!

Exploring the environment

Of course, we don’t have a classic Windows shell or an explorer, so we’ll just launch notepad from the cmd and use the “Open” dialog as a file explorer (just remember to select all file-types at the bottom).

WinPE with an open notepad.

Windows finds the disk drive with the contents of the C:\WinPE_x64\media folder we used on the build machine. There is also a X:\ drive with a minimal filesystem structure. Our user is SYSTEM, so we should be able to do anything.

So before going any further, this is not a free minimal Windows you just obtained here. Looking at the Microsoft documentation, WinPE lacks a lot of features and has a hard-coded 72-hour reboot timer which prevents us from using the system for production purposes.

Now, where did we get X:\ from? At boot time, WinPE extracted media\sources\boot.wim into a ramdisk. Customizing this image file will allow us to modify the environment. As the environment is loaded into the ramdisk, all changes you make while booted into WinPE are lost after rebooting.

WinPE tooling

As we don’t have the tools we normally have like the control panel, we need something else. Microsoft gives us 3 tools: wpeinit, wpeutil, and winpeshl.

winpeshl is the tool you are likely not going to touch at all unless you’re an OEM, but I’m going to mention it for the sake of completeness. It essentially runs commands configured via a winpeshl.ini file. If the file is missing (which it is by default), it just runs the cmd you saw when booting.

wpeinit is automatically called in the cmd (this is customizable as well). When called, it initialized Plug & Play devices, parses unattend.xml settings and configures the network. If you choose to use an unattend.xml file (not discussed here), you can pass the path via the -unattend: parameter.

wpeutil is the tool to manage your running WinPE environment. You can find all command line options at the Microsoft documentation, but I’m going to mention some of the more useful options here:

  • Reboot and Shutdown do what you’d expect them to
  • SetKeyboardLayout sets the keyboard layout, but for some reason I was unable to do so, even if the command was successful
  • WaitForNetwork waits until the network is fully initialized (for example, until a DHCP lease is acquired)

Exploring the PE files of the ADK

As we saw, to modify the contents of X:\, we need to modify the corresponding wim image. To do that, we mount the image. As the NT kernel doesn’t have anything like fuse, it’s more of a “mount”. The image file is extracted and repackaged at umount time.

> Dism /Mount-Image /ImageFile:"C:\WinPE_x64\media\sources\boot.wim" /index:1 /MountDir:"C:\WinPE_x64\mount"

We “mounted” the image into the mount directory and are free to explore the files that are going to be mounted to X:\. For adding exe files and DLLs, I recommend adding them into Windows\System32 directly because the PATH variable is pointing there.

Before you start adding features, however, start writing some documentation. In case you need to create a new WinPE image (for example for x86 or with a newer Windows version), there is no way to get a list of changes.

To repackage the image, umount it. Before umounting, make sure no explorer or cmd has the mount directory as working directory!

> Dism /Unmount-Image /MountDir:"C:\WinPE_x64\mount" /commit

Replacing /commit with /discard causes the image to be umounted, but changes are lost.

startnet.cmd

startnet.cmd is a batch script located at X:\Windows\System32\startnet.cmd or C:\WinPE_x64\mount\Windows\System32\startnet.cmd. By default, it only runs wpeinit (as we saw earlier), and adding custom commands there allows for more automated installations.

Installing Windows

While having a running WinPE system is cool and all, we wanted to install Windows from there. So the first thing to do is getting a Windows installation ISO and mounting (or extracting) it:

# mount /tmp/Win10_1909_EnglishInternational_x64.iso /mnt

Serving the installation files via SMB to make them available should be as easy as:

{
  services.samba = {
    enable = true;
    enableWinbindd = false;
    enableNmbd = false;
    extraConfig = ''
      map to guest = Bad User
    '';
    shares.WInstall = {
      browseable = "yes";
      "guest ok" = "yes";
      path = "/mnt";
      public = "yes";
      locking = "no";
      "read only" = "yes";
      "acl allow execute always" = "yes";
  };
}

In case you’re using a legacy distribution, check the corresponding documentation to serve /mnt via SMB.

If you made any image modifications, rebuild the ISO as explained above. Boot it, mount the ISO, and start the installer:

> net use I: \\10.55.122.7\WInstall
> I:\setup.exe

When setup.exe finds it’s started in a WinPE environment, it behaves as the installer we know from regular Windows ISOs. That’s it, install Windows from here and you’re done.

Some tricks

These are some snippets I found and liked:

powercfg /s 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c

Running this in the WinPE environment switches to the High Performance power profile. It’s one of the first things I do in startnet.bat.

:loop
ping -n 2 127.0.0.1 > nul
net use I: \\10.55.122.7\WInstall >NUL 2>&1 || goto :loop

This retries to mount the WInstall share until it succeeds. Might be useful if the network is unstable.

wpeutil UpdateBootInfo
for /f "tokens=2* delims= " %%A in ('reg query HKLM\System\CurrentControlSet\Control /v PEFirmwareType') DO SET Firmware=%%B
if %Firmware%==0x1 echo Booted into BIOS mode
if %Firmware%==0x2 echo Booted into UEFI mode

Also useful in startnet.cmd. Shows whether the system is booted in BIOS or UEFI mode. While x64 Windows can be installed from a x86 WinPE environment, UEFI Windows cannot be installed from a BIOS-booted WinPE environment and BIOS Windows cannot be installed from UEFI-booted WinPE.

> Dism /Image:"C:\WinPE_x64\mount" /set-inputlocale:de-DE

Set the keyboard layout to german. This must be done with the wim image mounted.

> del C:\WinPE_x64\media\boot\bootfix.bin

Executing this in the ADK environment and rebuilding the ISO skips the annoying “Press any key to boot from CD” and boots into the CD directly.

What’s next?

There are two things that are still bothering me. First off, having multiple languages is theoretically possible by adding multiple ISOs into the SMB share, but having one ISO with multiple languages would save a lot of disk space.

Also, this method only works when a SMB server is available. On remote sites, having a VPN to connect to, would be useful. But that’s for another post.