Month: August 2021

Ok, the headline is kind of sensational, but this post does actually talk about how you can make your XFS store files with timestamps after 2038 and how that can be done nicely with NixOS.

The Problem

The X FileSystem, more commonly known as XFS, has historically only supported timestamps until 2038.

However, in version 5.10 of the Linux kernel, the XFS maintainers landed the bigtime feature, which allows for “inode timestamps from December 1901 to July 2486, and quota timer expirations from January 1970 to July 2486”. GRUB2 from version 2.06 also allows you to boot from such a filesystem.

Changing these feature flags does however require running a xfs_admin command while the filesystem is not mounted, because it needs to modify some on-disk metadata structures.

The Solution

Since we did not want to reboot every system into a live system and perform this manually or re-image them or whatever else one could do, we came up with the following idea: Why not just do this in the initramfs, before the filesystems are mounted?

Luckily, NixOS offers a handful of options to add custom scripts and binaries to the initramfs.

So we got to work and implemented a NixOS module, as we do.

This is what we came up with:

{ config, lib, pkgs, utils, ... }: let
  # TODO: remove "/nix" filter, once feature is declared stable by upstream
  fileSystems = builtins.filter (fs: utils.fsNeededForBoot fs && fs.fsType == "xfs" && fs.mountPoint == "/nix") config.system.build.fileSystems;
in {
  config = lib.mkIf (fileSystems != []) {
    boot.initrd = {
      extraUtilsCommands = /* sh */ ''
        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_admin
        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_info

        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_db
        copy_bin_and_libs ${pkgs.xfsprogs}/bin/xfs_spaceman
      '';

      extraUtilsCommandsTest = /* sh */ ''
        sed -i -e 's,^#!.*,#!'$out/bin/sh, $out/bin/xfs_admin $out/bin/xfs_info   # from nixos/modules/tasks/filesystems/xfs.nix
        export PATH=$out/bin:$PATH
        $out/bin/xfs_admin -V
        $out/bin/xfs_info -V
      '';

      postDeviceCommands = /* sh */ let
        # basically fsInfo from stage-1.nix, but without options and only xfs
        # we can't reuse fsInfo, because of how these things are substituted in
        xfsInfo = lib.concatMapStringsSep "\n" (fs: if fs.device != null then fs.device else "/dev/disk/by-label/${fs.label}") fileSystems;
      in ''
        echo '${xfsInfo}' | while read device; do
          if [ "$(xfs_info "$device" | grep -o 'bigtime=.')" = "bigtime=0" ]; then
            xfs_repair -n "$device" && xfs_admin -O bigtime=1 "$device"
          fi
        done
      '';
    };
  };
}

While this might seem scary at first glance, it is actually relatively straight-forward.

It does the following things:

• Find the relevant filesystems. For now, this is limited to /nix, because while the XFS maintainers believe their code to be good and the on-disk format to be fine, they have still marked this feature as experimental for now (Edit: the feature is no longer marked experimental as of this commit which was released with kernel 5.15). When I last asked them, their timeline for removing this warning was “when we’ve waited long enough for nobody to find a problem”.
• It adds the relevant binaries from xfsprogs to the initramfs. It also tests if they can actually be run.
• It adds some shell code to the initramfs which performs the actual change on the filesystem. For each filesystem it checks if it has the feature already, if not it performs a fsck, because this change should and can only be performed on a clean filesystem and if the check succeeds it adds the feature.

After a host has a system with the configuration from this module applied to it, the only thing needed is a reboot and the filesystem will have support for the new feature. At least assuming the filesystem was clean.

XFS also gained some other potentially interesting features in recent years, like crc checks for metadata and support for copy-on-write files using reflink, both of which are enabled by default when creating a filesystem with mkfs.xfs from a recent xfsprogs version. If you have filesystems created before these features were available or enabled by default and want to enable them, the only things that need to be modified in the module are the check for which features are enabled and the xfs_admin command.

Shameless Self-Promotion

What follows is more or less just self-promotion, although it might still be relevant to your interests.

At Helsinki Systems, we run most of our infrastructure on NixOS. We decided on XFS on top of LVM2 for most of our storage needs, mainly because it provides the features we need while being very stable, relatively well integrated into the greater Linux ecosystem and without the need for out-of-tree kernel modules.

This storage stack is not particularly “sexy”. It is not ZFS, btrfs or bcachefs. It does not need to be that in our opinion. We would however like to be able to use all features it provides. For XFS that includes features like reflink copies and CRCs. For LVM2 that means thin-provisioning, raids and maybe also dm-integrity and VDO in the future.

To that end, we took over maintenance of lvm2 and xfsprogs in nixpkgs. We will also try our best to always keep them up to date in the future and inform the community of relevant changes.

In case you are looking for someone that will work on solutions as small as what we presented here or as large as packaging all of BigBlueButton for NixOS, feel free to shoot us an e-mail or give us a call. We also help with the maintenance of exim and dovecot in nixpkgs along with other packages, so we should hopefully be able to receive your e-mails 😀