MicroVM profile

This machine profile is enabled for microVMs. It sets some opinionated options for how I expect my microVMs to be set up.

{
  inputs,
  lib,
  config,
  hostConfig,
  ...
}:
let
  inherit (lib) types;
  cfg = config.mjm.profiles.microvm;
in
{
  imports = [ "${inputs.microvm}/nixos-modules/microvm/options.nix" ];

  options.mjm.profiles.microvm = {
    <<options>>
  };

  config = lib.mkMerge [
    { microvm.guest.enable = lib.mkDefault cfg.enable; }
    (lib.mkIf cfg.enable {
      <<config>>
    })
  ];

  _class = "nixos";
}

The microvm.nix module assumes it's only going to be imported for microVMs, so it enables itself by default. Since I'm importing this profile unconditionally, I need to override that and only enable it if the profile is enabled.

Options

enable = lib.mkEnableOption "MicroVM profile";

As with all profiles, this one needs to be explicitly enabled.

macAddress = lib.mkOption {
  type = types.str;
};
machineId = lib.mkOption {
  type = types.str;
};

Each microVM is assigned a MAC address prefixed with "02:" for its virtual NIC, as well as a machine ID. Usually these are generated with "dippy vm new".

group = lib.mkOption {
  type = types.enum [
    "early"
    "late"
  ];
  default = "late";
};

Each microVM belongs to a group, either "early" or "late". The host uses a set of systemd targets to synchronize starting the VMs. This is particularly important on apollo, which hosts the vast majority of microVMs, and where trying to start all of them at once creates a lot of contention. The goal is to place VMs where downtime is more noticeable in the early group so that they are started sooner. Ideally, stopping the VMs will also be ordered in this way, but I haven't quite figured out how to make systemd do that with my targets yet.

Config

microvm.hypervisor = lib.mkDefault "cloud-hypervisor";

Almost all of my VMs use cloud-hypervisor: it is my hypervisor of choice for this sort of thing. QEMU is available as a fallback if I need features that cloud-hypervisor doesn't support. I don't use any others.

microvm.machineId = cfg.machineId;

Set the machine ID configured in the profile option.

microvm.registerWithMachined = true;

Register the microVM with systemd-machined when it starts. This is a little thing, but it's kind of nice to be able to run machinectl and see the list of microVMs.

assertions = [
  {
    assertion = hostConfig.mjm.networkd.macvlan.enable;
    message = "A microVM host must have `mjm.networkd.macvlan.enable` set to true, as microVMs rely on MACVTAP network interfaces.";
  }
];
microvm.interfaces = [
  {
    type = "macvtap";
    id = "vm-${config.networking.hostName}";
    macvtap.link = hostConfig.mjm.networkd.bridgeParentName;
    macvtap.mode = "bridge";
    mac = cfg.macAddress;
  }
];

I use MACVLAN bridging rather than a virtual bridge for networking my microVMs. I think it's the most straightforward way to do things, particularly if you have a NIC you can dedicate to it.

Each microVM gets a MACVTAP interface, which is named vm- on the host side. It's configured to use bridge mode, and to bind to the configured MACVLAN interface on the host. See the networkd section of the nixos/base module for more about that. An assertion ensures that the host is actually set up properly for MACVLAN.

=> nixos/base module

microvm.shares = [
  {
    tag = "ro-store";
    source = "/nix/store";
    mountPoint = "/nix/.ro-store";
    proto = "virtiofs";
  }
]
++ map (d: rec {
  proto = "virtiofs";
  inherit (d) tag;
  source = tag;
  mountPoint = d.directory;
}) config.mjm.state.directories;

My microVMs have two folders shared from the host by default. First, the host's /nix/store is mounted at /nix/.ro-store. microvm.nix treats a share of /nix/store special: if present, it will skip building a squashfs/erofs store for the VM, and will instead add a read-only bind mount from the mountpoint to /nix/store. I'm not sure why they suggest mounting it somewhere besides /nix/store and relying on the bind mount they add. It seems like I could just mount it directly to /nix/store and that would be fine. But I'm not interesting in testing that change right now.

The second folder shared is /var/lib/microvms//var which gets mounted to /var. All data that is expected to be persisted should end up under /var. Everything else ends up on the tmpfs at /. You may notice that the code is not actually handling /var specifically: it's using mjm.state.directories. I used to be more granular about what directories were persisted on my machines, but I've abandoned that approach. It's tedious to keep track of for very little win. In general, persisting all of /var is what you actually want.

I should come back here and clean up this code to just add the share for /var, but I'd also want an assertion that it's actually the only thing in mjm.state.directories.

microvm.cloud-hypervisor.platformOEMStrings = lib.mkIf (config.mjm.spire.agent.joinToken != "") [
  "io.systemd.credential:spire_agent_join_token=${config.mjm.spire.agent.joinToken}"
];
microvm.qemu.extraArgs = lib.mkIf (config.mjm.spire.agent.joinToken != "") [
  "-smbios"
  "type=11,value=io.systemd.credential:spire_agent_join_token=${config.mjm.spire.agent.joinToken}"
];

When a microVM is first provisioned, or if it's been offline long enough for its SPIRE registration to expire, it uses a join token to get its initial certificate from the SPIRE server. This is passed as a systemd credential. It's nice to pass it in from outside the VM, because it avoids needing to rebuild the VM to add the token and avoids needing to restart the VM once the token has been removed from the config after it's no longer needed.

cloud-hypervisor and QEMU have different ways of passing this information along, so I set both.

mjm.consul.enable = true;
mjm.server.enable = true;
mjm.server.enableGarbageCollection = false;
mjm.spire.agent.enable = true;

Enable the basic set of services for my servers, to avoid needing to repeat these in every individual microVM. Of course, the automatic garbage collection needs to be disabled since the microVMs use read-only Nix stores shared from their host.

mjm.home-manager.enable = false;

Disable setting up home-manager, since it overcomplicates the microVMs and would cause undesired reboots.

nix.enable = false;
system.switch.enable = false;
environment.defaultPackages = [ ];
environment.stub-ld.enable = false;
system.disableInstallerTools = true;
services.lvm.enable = false;
boot.bcache.enable = false;
programs.fish.generateCompletions = false;
xdg = {
  icons.enable = false;
  sounds.enable = false;
};

Disable various things that are enabled by default but aren't needed for a microVM. Some of these might have an impact on boot time, but more than anything they eliminate dependencies which reduces build time and gives less reasons for a microVM to change and need to restart. The build times do add up when you are building tens of systems.

console.enable = false;
systemd.services."serial-getty@ttyS0".enable = false;
systemd.services."serial-getty@hvc0".enable = false;
systemd.services."getty@tty1".enable = false;
systemd.services."autovt@".enable = false;
systemd.enableEmergencyMode = false;

The serial console on the microVMs is not something I can interact with, so there's no reason to bother starting a tty on it. And if there's no tty, there's no reason to have systemd's emergency mode either.

boot.kernelParams = [
  "panic=1"
  "boot.panic_on_fail"
  "nomodeset"
  (lib.mkIf (config.microvm.hypervisor != "qemu") "vga=0x317")
];

Since a microVM isn't interactive, there's nothing particularly useful that can be done on a panic, so rather than just sit there, the machine reboots automatically. The machine will also panic if it fails to boot, triggering the same reboot behavior.

Since there's no graphics for the machine, kernel modesettings is disabled, and the display is configured for a basic 1024x768x16 resolution. For some reason, setting the resolution this way for QEMU caused the VM to fail to boot, so that's omitted for that hypervisor.

mjm.networkd.macvlan.enable = false;

Make sure that MACVLAN support in the guest doesn't accidentally get turned on by explicitly setting it to false. This was more important in the past when I had service modules enabling it, which is no longer the case. It would be surprising for this to end up turned on by mistake, but it doesn't hurt to be sure.