UPS monitoring

{ config, lib, ... }:
let
  cfg = config.mjm.nut;
  <<ups-names>>
in
{
  options.mjm.nut = {
    <<options>>
  };

  config = lib.mkIf cfg.enable (
    lib.mkMerge [
      {
        <<config>>
      }
      (lib.mkIf (cfg.mode == "client") {
        <<client-config>>
      })
      (lib.mkIf (cfg.mode == "server") {
        <<server-config>>
      })
    ]
  );
}

This modules configures NUT: Network UPS Tools. This suite of tools allows for a machine to monitor the state of a UPS that is connected to it, and it allows it to broadcast that information to other machines on the network that are also connected to the same UPS. The machines can use this information to shutdown safely before the battery dies.

Options

enable = lib.mkEnableOption "NUT";

This module must be enabled explicitly. Some machines are not connected to a UPS at all.

mode = lib.mkOption {
  type = lib.types.enum [
    "server"
    "client"
  ];
  default = "client";
};

I run NUT in either client or server mode depending on the machine. A server has a UPS connected to it via USB. A client is plugged in to a UPS to get its power, but doesn't have direct access to data from the UPS, and therefore must get it from the server. Most machines are clients, so that is the default.

connectedUPSName = lib.mkOption {
  type = lib.types.enum upsNames;
};

upsNames = [
  "or500"
  "smart500"
];

A NUT client needs to be configured to observe the status of the particular UPS it's connected to. I have two UPSes in my rack, because one wasn't really able to keep things alive for any useful amount of time. This option indicates which of the two should be observed.

The names are defined as a separate binding because they are also used later in the config. Technically, it's possible to yank them back out of the option type, but it's really verbose.

serverHostname = lib.mkOption {
  type = lib.types.str;
  default = "arges.home.mattmoriarity.com";
};

This option is generally left as the default value, and determines the address the clients should connect to to reach the server that is connected to their UPS.

Config

Common config

This configuration is shared by both clients and servers.

mjm.services.nut-client = {
  secrets.enable = true;
};

Both clients and servers need access to the "nut-client" service's secrets, which holds the password clients will use to connect to the server.

power.ups.enable = true;

Really the only common thing that can be done with the NixOS UPS module is enable it. The remaining configuration is different for clients and servers.

Client-only config

power.ups.mode = "netclient";

Exactly what it seems: this tells the NixOS NUT module to configure services for running as a network client. In this mode, only the "upsmon" service is running.

power.ups.upsmon.monitor.${cfg.connectedUPSName} = {
  system = "${cfg.connectedUPSName}@${cfg.serverHostname}";
  user = "upsmon_secondary";
  type = "secondary";
  passwordFile = "/run/nut-client-creds.sock";
};

This configures the machine to monitor the UPS it is connected to. If I remember right, the distinction between primary and secondary users in NUT is about how it orders shutting down machines when the battery gets low. Secondary users will be notified to shutdown first, before primary users, because once the primary machine goes down, no one is there anymore to tell the secondaries to do it.

The password file is pointed at the spiffe-creds socket for the nut-client service, since the NixOS NUT module already uses systemd credentials for the password files. This will provide the password to upsmon by fetching it from Vault.

mjm.spire.creds.nut-client.aliases = {
  "upsmon.service/upsmon_password_${cfg.connectedUPSName}" = "nut-client/secondary_password";
};

The NixOS NUT module uses a particular format for the credential names for each password configured for the monitored UPSes. Therefore, an alias is configured for spiffe-creds to tell it which secret in Vault corresponds to it.

Server-only config

mjm.services.nut = {
  secrets.enable = true;
};

The server also needs secrets for the "nut" service, which holds the password the server will use monitor the UPSes.

power.ups.mode = "netserver";
power.ups.openFirewall = true;

Setting the mode to "netserver" configures the machine to run additional services to actually connect to the UPSes and provide their information to clients. Of course, if network clients will be connecting, then ports on the firewall will need to be opened.

power.ups.ups.or500 = {
  driver = "usbhid-ups";
  port = "auto";
  directives = [
    ''vendorid = "0764"''
    ''productid = "0601"''
  ];
};

This configures the first UPS connected to the server machine. It is a CyberPower OR500LCDRM1Ua, which uses the generic usbhid-ups driver.

power.ups.ups.smart500 = {
  driver = "tripplite_usb";
  port = "auto";
  directives = [
    ''vendorid = "09ae"''
    ''productid = "0001"''
  ];
};

This configures the second UPS connected to the server machine. It is a TrippLite SMART500RT1U, which uses a special TrippLite USB driver.

The keys used to define the UPSes (or500 and smart500) correspond to the allowed values for the connectedUPSName option, since they become the identifiers used to request information about a particular UPS from the server.

power.ups.upsd.listen = [ { address = "0.0.0.0"; } ];

upsd is configured to listen on any IPv4 address. I should probably expand this to include IPv6 as well.

power.ups.users = {
  upsmon = {
    upsmon = "primary";
    passwordFile = "/run/nut-creds.sock";
  };
  upsmon_secondary = {
    upsmon = "secondary";
    passwordFile = "/run/nut-client-creds.sock";
  };
};

This configures the users who are allowed to connect to upsd. There are two users here: a primary which is used only by the server, and a secondary that is used by clients. The "upsmon" option is essentially a role: it's a predefined set of actions the user is allowed to do corresponding to what a primary or secondary user in upsmon needs to be able to do.

The primary fetches its password from the spiffe-creds instance for the "nut" service, and the secondary from "nut-client".

power.ups.upsmon.monitor = lib.genAttrs upsNames (name: {
  system = "${name}@${cfg.serverHostname}";
  user = "upsmon";
  type = "primary";
  passwordFile = "/run/nut-creds.sock";
});

The server monitors all of the UPSes connected to it in its capacity as the primary. Much like the clients, it loads its password through spiffe-creds, but from the "nut" service since that's the one with the primary user's password.

mjm.spire.creds = {
  nut.aliases = {
    "upsmon.service/upsmon_password_or500" = "nut/primary_password";
    "upsmon.service/upsmon_password_smart500" = "nut/primary_password";
    "upsd.service/upsdusers_password_upsmon" = "nut/primary_password";
  };
  nut-client.aliases = {
    "upsd.service/upsdusers_password_upsmon_secondary" = "nut-client/secondary_password";
  };
};

Much like with the client, aliases are needed for spiffe-creds to produce the right secrets from the predetermined credential names the NixOS module uses. This time there are more of them, since the server is watching two UPSes and also needs to read the passwords for both users to set them up.

services.prometheus.exporters.nut = {
  enable = true;
  openFirewall = true;
  nutVariables = [
    "battery.charge"
    "battery.runtime"
    "battery.voltage"
    "battery.voltage.nominal"
    "input.voltage"
    "input.voltage.nominal"
    "ups.load"
    "ups.status"
  ];
  extraFlags = [ "--log.level=debug" ];
};

The server also runs the NUT exporter for Prometheus. This exposes the information from upsd as Prometheus metrics, so they can be used to define alerts in Grafana.

The explicit variable list is there because the "battery.runtime" variable is not requested by default, but is absolutely a metric I'm interested in.

mjm.consul.services.nut-exporter = {
  inherit (config.services.prometheus.exporters.nut) port;

  checks.up = {
    http.path = "/";
  };
};

The NUT exporter is registered as a service in Consul. I can't use my normal method of tagging Consul services to be scraped for metrics because a separate request must be made for each UPS. So instead, there is a dedicated Prometheus job set up to scrape the exporter. The Consul service allows that job to target nut-exporter.service.consul.