persephone

My secondary laptop.

persephone is a 13th gen Intel Framework 13 with 64GiB of RAM. It was my primary laptop when I bought it, but once I took ownership of athena after leaving the job I was using it for, I found myself using that laptop more. I'm still rather fond of the Framework though, and I think it has a lot going for it.

NixOS configuration

{
  config,
  lib,
  inputs,
  pkgs,
  utils,
  ...
}:
{
  imports = [
    "${inputs.hardware}/framework/13-inch/13th-gen-intel"
    ./hardware-configuration.nix
  ];

  networking.hostName = "persephone";
  services.openssh.enable = true;
  system.stateVersion = "23.05";

  <<nixos-config>>

  _class = "nixos";
}

The state version reflects that persephone is I think my oldest still running NixOS machine, as I bought it a few months after I started using Nix. Maybe the Raspberry Pis are slightly older, not sure.

mjm.desktop.enable = true;
mjm.desktop.niri.enable = true;
mjm.desktop.autoLogin.enable = true;

Since it's a workstation, it uses my own desktop module with all of the desktop configuration. I'm using niri on all of my workstations.

I'm also enabling auto-login, which for my machines means I'm unlocking the disk using a PIN of some kind (via the TPM in this case) and then using that same PIN to also unlock gnome-keyring after automatically logging in at the display manager.

mjm.secureboot.enable = true;

persephone uses my secureboot module to set up Lanzaboote, which enables SecureBoot and measured boot on NixOS systems. I use this to avoid needing to unlock the encrypted filesystem on boot: as long as the boot configuration hasn't been tampered with, it will be unlocked automatically.

boot.kernelPackages = pkgs.linuxPackages_latest;

Workstations generally benefit from having the latest kernel changes.

services.fwupd.enable = true;

This machine has some hardware whose firmware can be managed using fwupd, so I enable that.

services.hardware.bolt.enable = true;

This option enables managing access to Thunderbolt devices, which sometimes require approval before they can actually be used.

services.kmscon.config.font-size = 24;

persephone has a reasonably high resolution display, so I've configured the console to be a bit bigger than the default so it's actually readable.

nixpkgs.config.permittedInsecurePackages = [ "electron-39.8.10" ];

My workstations install Bitwarden Desktop to use with my Vaultwarden install. Unfortunately, Bitwarden Desktop is using an end-of-life version of Electron, so I have to allow that insecure package. This nixpkgs config key doesn't merge well across multiple modules, so I keep a single list at the host level instead.

Erase-your-darlings and preservation

mjm.state = {
  enablePreservation = true;
  persistDir = "/persist";
  directories = [
    {
      directory = "/nix";
      inInitrd = true;
    }
    "/home"
  ];
};

persephone is my only workstation that using an erase-your-darlings setup. Persistent data is stored under /persist and bind-mounted into place. Only data under /home and /var will be preserved between boots.

boot.initrd.systemd.services.rollback-root = {
  description = "Rollback Root Filesystem to Blank Snapshot";
  wantedBy = [ "initrd.target" ];
  after = [ "${utils.escapeSystemdPath "/persist-tmp"}.mount" ];
  requires = [ "${utils.escapeSystemdPath "/persist-tmp"}.mount" ];
  before = [ "sysroot.mount" ];
  unitConfig.DefaultDependencies = false;
  serviceConfig = {
    Type = "oneshot";
    ExecStart = [
      "${config.boot.bcachefs.package}/bin/bcachefs subvolume delete /persist-tmp/@root"
      "${config.boot.bcachefs.package}/bin/bcachefs subvolume create /persist-tmp/@root"
    ];
  };
};

This systemd service will delete and recreate the @root subvolume. My fileystem is mounted a little strangely to support the root being erased like this every boot. It's mounted at / with X-mount.subdir=@root and at /persist normally. The / mount is essentially a bind mount to the @root subvolume, but the bind mount can't be from the /persist mount because that would require / to already be mounted. X-mount.subdir uses namespace trickery to basically do a bind mount to a place you can't see.

That all works fine, but to manipulate the @root subvolume like this, the actual root of the filesystem needs to be mounted somewhere. At some point later in stage 1, it will be mounted at /sysroot/persist, but this service needs to run before even /sysroot is mounted. So I'll need to mount it somewhere temporarily in the initrd root. I've chosen /persist-tmp as that location.

boot.initrd.systemd.mounts = [
  {
    what = "/dev/disk/by-partlabel/persist";
    where = "/persist-tmp";
    type = "bcachefs";
    after = [ "unlock-bcachefs-${utils.escapeSystemdPath "/"}.service" ];
    requires = [ "unlock-bcachefs-${utils.escapeSystemdPath "/"}.service" ];
    options = "casefold_disabled";
  }
  <<cryptkey-mount>>
];

That /persist-tmp filesystem is mounted with a systemd mount unit in the initrd. Doing it this way, instead of in the fileSystems option, makes it easy to mount it somewhere that won't be visible after switching to stage 2. The mount is ordered to run after the service that unlocks the encryption on the filesystem.

With all of these pieces in place, the sequence of steps here is:

1. Unlock the disk

2. Mount the FS at /persist-tmp

3. Delete and recreate the @root subvolume

4. Mount the @root subvolume at /sysroot

5. Mount the FS at /sysroot/persist

boot.initrd.systemd.services.unlock-bcachefs-persist.enable = false;

The quirky mount setup I'm using produces two entries in config.fileSystems for the same device. This makes the NixOS bcachefs module generate two systemd services for unlocking it: one for each mountpoint. That's maybe something that would be good to fix, but for now, I'm just disabling one of them. It makes sense to disable the one for /persist, since the mount will already require / to be mounted first, which will then depend on the remaining unlock service, so no new dependencies are needed.

Unlocking the bcachefs disk with LUKS

I'm using bcachefs native encryption, but I also want to use some features that are only well-supported when using LUKS. I want to unlock using TPM2 with a PIN while bound to the pcrlock policy from the measured boot setup. Neither of these are possible when unlocking a bcachefs volume directly, but it's still possible to do with a little bit of indirection.

The idea is to have a small LUKS container with a filesystem inside that contains a single file, which holds the passphrase to the bcachefs volume. By unlocking the LUKS container using all the TPM2 features systemd-cryptsetup supports, the rest of the process of unlocking and mounting can proceed unattended.

So here's how that is done:

boot.initrd.luks.devices.cryptkey.device = "/.extra/global_credentials/cryptkey.cred";

This sets up the /etc/crypttab in stage 1 to unlock this small LUKS container. The unlocked contents will be able to be mounted from /dev/mapper/cryptkey. The device is a disk image is at a very particular location. The systemd-stub supports loading credentials into the initrd's filesystem when the system is booted, and Lanzaboote's stub supports the same thing. The disk image with the LUKS container lives at /boot/loader/credentials/cryptkey.cred. Any thing in that directory matching *.cred gets copied to /.extra/global_credentials/ at boot. This is how this container is made available in stage 1 without needing to mount an additional filesystem first, make a dedicated partition for it, or bundle it in the initrd at build time.

{
  what = "/dev/mapper/cryptkey";
  where = "/cryptkey";
  type = "erofs";
}

boot.initrd.supportedFilesystems.erofs = true;

A mount unit is used in stage 1 to mount the filesystem with the bcachefs passphrase to /cryptkey. Inside the LUKS container, I used an erofs, since there's no need for this filesystem to be writable.

Normally, you just make a fileSystems entry to create an entry in fstab, but I'm not doing that here because I don't want the cryptkey to be mounted under /sysroot. For one thing, it needs to get mounted before the root filesystem is even unlocked, but also it should go away completely by the time stage 2 is reached. A mount unit in stage 1 allows for that.

boot.initrd.systemd.services.unlock-bcachefs-- = {
  script = lib.mkForce ''
    ${config.boot.bcachefs.package}/bin/bcachefs unlock "/dev/disk/by-partlabel/persist"
  '';
  serviceConfig.LoadCredential = "cryptsetup.passphrase:/cryptkey/passphrase";
  unitConfig.WantsMountsFor = [ "/cryptkey" ];
};

The service NixOS provides to unlock the bcachefs disk needs some tweaks. The change to the script is hopefully a change that will be made in NixOS itself soon (perhaps I need to make it myself). As of bcachefs-tools v1.38.8, the "bcachefs unlock" command uses systemd-ask-password parameters similar to those used by systemd-cryptsetup, including credential support. Currently, NixOS makes its own systemd-ask-password calls and pipes the output to "bcachefs unlock", which doesn't give a good way to pass a credential.

The cryptsetup.passphrase credential is exactly what it sounds like, and I'm configuring the service to load it from /cryptkey, which was just configured above to be mounted. WantsMountsFor creates the necessary dependencies so that this unlock service only runs after the mount. It's important to use Wants instead of Requires in this particular case, because of the next requirement.

boot.initrd.systemd.services."systemd-cryptsetup@cryptkey" = {
  overrideStrategy = "asDropin";
  conflicts = [ "initrd-switch-root.target" ];
};

The configuration up until now is enough to get the system booting as desired: unlock the cryptkey using a TPM2 pin and the rest is unattended. But it leaves the cryptkey unlocked: a process running on the system could gain root and then surreptitiously mount the cryptkey volume somewhere and read the bcachefs passphrase and use it again sometime later. We can avoid this by forcing the systemd-cryptsetup service for it to stop before reaching stage 2. That will lock the LUKS container, requiring it to be unlocked again before it could be mounted.

This is why WantMountsFor needed to be used above rather than RequiresMountsFor. The latter would cause stopping the systemd-cryptsetup service to cascade all the way to stopping the root filesystem itself, preventing the system from booting. A weaker dependency on /cryptkey is needed, so that unlocking bcachefs waits for the passphrase to be available without having to stop later when the cryptkey is locked.

Home configuration

{
  home.sessionVariables.DIPPY_EVAL_CONCURRENCY = "4";
}

Just one small tweak for the home-manager config here. By default, my deploy tool, dippy, only evaluates two jobs at once, to avoid using too much RAM. Since persephone has quite a bit of RAM, I configure it to evaluate four jobs at once.