Solving the y2038 Problem with NixOS and XFS

Ok, the headline is kind of sensational, but this post does actually talk about how you can make your XFS store files with timestamps after 2038 and how that can be done nicely with NixOS.

The Problem

The X FileSystem, more commonly known as XFS, has historically only supported timestamps until 2038.

However, in version 5.10 of the Linux kernel, the XFS maintainers landed the bigtime feature, which allows for “inode timestamps from December 1901 to July 2486, and quota timer expirations from January 1970 to July 2486”. GRUB2 from version 2.06 also allows you to boot from such a filesystem.

Changing these feature flags does however require running a xfs_admin command while the filesystem is not mounted, because it needs to modify some on-disk metadata structures.

The Solution

Since we did not want to reboot every system into a live system and perform this manually or re-image them or whatever else one could do, we came up with the following idea: Why not just do this in the initramfs, before the filesystems are mounted?

Luckily, NixOS offers a handful of options to add custom scripts and binaries to the initramfs.

So we got to work and implemented a NixOS module, as we do.

This is what we came up with:

{ config, lib, pkgs, utils, ... }: let
  # TODO: remove "/nix" filter, once feature is declared stable by upstream
  fileSystems = builtins.filter (fs: utils.fsNeededForBoot fs && fs.fsType == "xfs" && fs.mountPoint == "/nix") config.system.build.fileSystems;
in {
  config = lib.mkIf (fileSystems != []) {
    boot.initrd = {
      extraUtilsCommands = /* sh */ ''
        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_admin
        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_info

        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_db
        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_spaceman
      '';

      extraUtilsCommandsTest = /* sh */ ''
        sed -i -e 's,^#!.*,#!'$out/bin/sh, $out/bin/xfs_admin $out/bin/xfs_info   # from nixos/modules/tasks/filesystems/xfs.nix
        export PATH=$out/bin:$PATH
        $out/bin/xfs_admin -V
        $out/bin/xfs_info -V
      '';

      postDeviceCommands = /* sh */ let
        # basically fsInfo from stage-1.nix, but without options and only xfs
        # we can't reuse fsInfo, because of how these things are substituted in
        xfsInfo = lib.concatMapStringsSep "\n" (fs: if fs.device != null then fs.device else "/dev/disk/by-label/${fs.label}") fileSystems;
      in ''
        echo '${xfsInfo}' | while read device; do
          if [ "$(xfs_info "$device" | grep -o 'bigtime=.')" = "bigtime=0" ]; then
            xfs_repair -n "$device" && xfs_admin -O bigtime=1 "$device"
          fi
        done
      '';
    };
  };
}

While this might seem scary at first glance, it is actually relatively straight-forward.

It does the following things:

  • Find the relevant filesystems. For now, this is limited to /nix, because while the XFS maintainers believe their code to be good and the on-disk format to be fine, they have still marked this feature as experimental for now. When I last asked them, their timeline for removing this warning was “when we’ve waited long enough for nobody to find a problem”.
  • It adds the relevant binaries from xfsprogs to the initramfs. It also tests if they can actually be run.
  • It adds some shell code to the initramfs which performs the actual change on the filesystem. For each filesystem it checks if it has the feature already, if not it performs a fsck, because this change should and can only be performed on a clean filesystem and if the check succeeds it adds the feature.

After a host has a system with the configuration from this module applied to it, the only thing needed is a reboot and the filesystem will have support for the new feature. At least assuming the filesystem was clean.

XFS also gained some other potentially interesting features in recent years, like crc checks for metadata and support for copy-on-write files using reflink, both of which are enabled by default when creating a filesystem with mkfs.xfs from a recent xfsprogs version. If you have filesystems created before these features were available or enabled by default and want to enable them, the only things that need to be modified in the module are the check for which features are enabled and the xfs_admin command.

Shameless Self-Promotion

What follows is more or less just self-promotion, although it might still be relevant to your interests.

At Helsinki Systems, we run most of our infrastructure on NixOS. We decided on XFS on top of LVM2 for most of our storage needs, mainly because it provides the features we need while being very stable, relatively well integrated into the greater Linux ecosystem and without the need for out-of-tree kernel modules.

This storage stack is not particularly “sexy”. It is not ZFS, btrfs or bcachefs. It does not need to be that in our opinion. We would however like to be able to use all features it provides. For XFS that includes features like reflink copies and CRCs. For LVM2 that means thin-provisioning, raids and maybe also dm-integrity and VDO in the future.

To that end, we took over maintenance of lvm2 and xfsprogs in nixpkgs. We will also try our best to always keep them up to date in the future and inform the community of relevant changes.

In case you are looking for someone that will work on solutions as small as what we presented here or as large as packaging all of BigBlueButton for NixOS, feel free to shoot us an e-mail or give us a call. We also help with the maintenance of exim and dovecot in nixpkgs along with other packages, so we should hopefully be able to receive your e-mails 😀

Creating a minimal Windows ISO

So first off, Hello World! We’re Helsinki Systems and this is our blog about Linux, NixOS, and whatnot. But instead of starting of a stream of Linux posts, let’s start with something slightly different: Creating custom Windows ISOs. Why? Because it’s useful if you live in a world where installing Windows is something you do sometimes. Also it’s really handy to have a central way of upgrading all Windows installation files you handed out to your co-workers (we’ll discuss this in another post).

WinPE

As many people switching from Windows to Linux notice, most Linux distributions come pretty small by default and most functionality can be added later using some package manager. This means that the distribution used for the regular OS can also be used for a small installer – this is different with Windows. Most distros download the packages for the to-be-installed OS from the internet, which makes the installation images even smaller. Windows, on the other hand usually bundles all required files into the image which makes offline installation possible. However, the files don’t just lay around on the filesystem of your ISO. Instead they are packed into wim (Windows Imaging Format) files and not ready for use. This means the installer needs to use a different, smaller, system which is used for unpacking the wim files into the target system while also being tiny enough to be deployed via methods like PXE. This is WinPE (Windows Preinstallation Environment). WinPE doesn’t contain the files required for installing Windows and is therefore tiny compared to the Windows ISO you download from Microsoft. You could compare WinPE to a minimal linux image with a read-only root filesystem and just a shell (cmd; not PowerShell) with a few basic tools. Let’s explore that!

Getting the ADK

In order to explore WinPE, let’s build an ISO! The default Windows installer is huge and preconfigured, so we’ll build our own WinPE environment. For this, we need the Windows ADK (Windows Assessment and Deployment Kit) which you can get from here unless the link is dead by the time you discover this post. You also need the WinPE add-on (on the same page as the ADK). Last, but not least, if the current ADK version is still 1903, also install the WSIM (Windows System Image Manager) update which makes WSIM ready for use. Although this is not used for the purposes of this blog post, it’s a handy tool and might come into play in a future post. You should have a “Deployment and Imaging Tools Environment” shortcut in your start menu under the “Windows Kits” folder. I’m not sure if this needs to be run as administrator, I usually do this because I create my images in C:\.

Building WinPE

You should have your cmd open now with the correct environment. As mentioned, I work directly at C:\, so let’s copy all WinPE-related files there for a x64 installation image:

> cd C:\
> copype amd64 C:\WinPE_x64

Inside C:\WinPE_x64, you’ll find three directories. You’re likely not going to touch fwfiles, as it contains some files relevant to WinPE startup. mount is empty (we’ll get to this in a second), and media is what’s going into your WinPE image. It’s essentially the root folder of your ISO. But before exploring anything, let’s create an ISO image:

> MakeWinPEMedia /ISO C:\WinPE_x64 C:\WinPE_x64.iso

Burn this ISO onto a CD or just boot some VM with it like a normal person. On my Linux machine, I use QEMU:

$ qemu-system-x86_64 -m 2G -smp 4 -cdrom WinPE_x64.iso

After some time, you’ll be greeted with a cmd that is running wpeinit which will exit after some time and leave you with a cmd. Time to look around!

Exploring the environment

Of course, we don’t have a classic Windows shell or an explorer, so we’ll just launch notepad from the cmd and use the “Open” dialog as a file explorer (just remember to select all file-types at the bottom).

WinPE with an open notepad.

Windows finds the disk drive with the contents of the C:\WinPE_x64\media folder we used on the build machine. There is also a X:\ drive with a minimal filesystem structure. Our user is SYSTEM, so we should be able to do anything.

So before going any further, this is not a free minimal Windows you just obtained here. Looking at the Microsoft documentation, WinPE lacks a lot of features and has a hard-coded 72-hour reboot timer which prevents us from using the system for production purposes.

Now, where did we get X:\ from? At boot time, WinPE extracted media\sources\boot.wim into a ramdisk. Customizing this image file will allow us to modify the environment. As the environment is loaded into the ramdisk, all changes you make while booted into WinPE are lost after rebooting.

WinPE tooling

As we don’t have the tools we normally have like the control panel, we need something else. Microsoft gives us 3 tools: wpeinit, wpeutil, and winpeshl.

winpeshl is the tool you are likely not going to touch at all unless you’re an OEM, but I’m going to mention it for the sake of completeness. It essentially runs commands configured via a winpeshl.ini file. If the file is missing (which it is by default), it just runs the cmd you saw when booting.

wpeinit is automatically called in the cmd (this is customizable as well). When called, it initialized Plug & Play devices, parses unattend.xml settings and configures the network. If you choose to use an unattend.xml file (not discussed here), you can pass the path via the -unattend: parameter.

wpeutil is the tool to manage your running WinPE environment. You can find all command line options at the Microsoft documentation, but I’m going to mention some of the more useful options here:

  • Reboot and Shutdown do what you’d expect them to
  • SetKeyboardLayout sets the keyboard layout, but for some reason I was unable to do so, even if the command was successful
  • WaitForNetwork waits until the network is fully initialized (for example, until a DHCP lease is acquired)

Exploring the PE files of the ADK

As we saw, to modify the contents of X:\, we need to modify the corresponding wim image. To do that, we mount the image. As the NT kernel doesn’t have anything like fuse, it’s more of a “mount”. The image file is extracted and repackaged at umount time.

> Dism /Mount-Image /ImageFile:"C:\WinPE_x64\media\sources\boot.wim" /index:1 /MountDir:"C:\WinPE_x64\mount"

We “mounted” the image into the mount directory and are free to explore the files that are going to be mounted to X:\. For adding exe files and DLLs, I recommend adding them into Windows\System32 directly because the PATH variable is pointing there.

Before you start adding features, however, start writing some documentation. In case you need to create a new WinPE image (for example for x86 or with a newer Windows version), there is no way to get a list of changes.

To repackage the image, umount it. Before umounting, make sure no explorer or cmd has the mount directory as working directory!

> Dism /Unmount-Image /MountDir:"C:\WinPE_x64\mount" /commit

Replacing /commit with /discard causes the image to be umounted, but changes are lost.

startnet.cmd

startnet.cmd is a batch script located at X:\Windows\System32\startnet.cmd or C:\WinPE_x64\mount\Windows\System32\startnet.cmd. By default, it only runs wpeinit (as we saw earlier), and adding custom commands there allows for more automated installations.

Installing Windows

While having a running WinPE system is cool and all, we wanted to install Windows from there. So the first thing to do is getting a Windows installation ISO and mounting (or extracting) it:

# mount /tmp/Win10_1909_EnglishInternational_x64.iso /mnt

Serving the installation files via SMB to make them available should be as easy as:

{
  services.samba = {
    enable = true;
    enableWinbindd = false;
    enableNmbd = false;
    extraConfig = ''
      map to guest = Bad User
    '';
    shares.WInstall = {
      browseable = "yes";
      "guest ok" = "yes";
      path = "/mnt";
      public = "yes";
      locking = "no";
      "read only" = "yes";
      "acl allow execute always" = "yes";
  };
}

In case you’re using a legacy distribution, check the corresponding documentation to serve /mnt via SMB.

If you made any image modifications, rebuild the ISO as explained above. Boot it, mount the ISO, and start the installer:

> net use I: \\10.55.122.7\WInstall
> I:\setup.exe

When setup.exe finds it’s started in a WinPE environment, it behaves as the installer we know from regular Windows ISOs. That’s it, install Windows from here and you’re done.

Some tricks

These are some snippets I found and liked:

powercfg /s 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c

Running this in the WinPE environment switches to the High Performance power profile. It’s one of the first things I do in startnet.bat.

:loop
ping -n 2 127.0.0.1 > nul
net use I: \\10.55.122.7\WInstall >NUL 2>&1 || goto :loop

This retries to mount the WInstall share until it succeeds. Might be useful if the network is unstable.

wpeutil UpdateBootInfo
for /f "tokens=2* delims= " %%A in ('reg query HKLM\System\CurrentControlSet\Control /v PEFirmwareType') DO SET Firmware=%%B
if %Firmware%==0x1 echo Booted into BIOS mode
if %Firmware%==0x2 echo Booted into UEFI mode

Also useful in startnet.cmd. Shows whether the system is booted in BIOS or UEFI mode. While x64 Windows can be installed from a x86 WinPE environment, UEFI Windows cannot be installed from a BIOS-booted WinPE environment and BIOS Windows cannot be installed from UEFI-booted WinPE.

> Dism /Image:"C:\WinPE_x64\mount" /set-inputlocale:de-DE

Set the keyboard layout to german. This must be done with the wim image mounted.

> del C:\WinPE_x64\media\boot\bootfix.bin

Executing this in the ADK environment and rebuilding the ISO skips the annoying “Press any key to boot from CD” and boots into the CD directly.

What’s next?

There are two things that are still bothering me. First off, having multiple languages is theoretically possible by adding multiple ISOs into the SMB share, but having one ISO with multiple languages would save a lot of disk space.

Also, this method only works when a SMB server is available. On remote sites, having a VPN to connect to, would be useful. But that’s for another post.