spiffe-creds

spiffe-creds is a tool to get secrets from Vault to services using systemd's credentials mechanism. It uses a JWT token obtained from SPIFFE to authenticate to Vault as a single service identity, and then it can provide secrets upon request to systemd services via a Unix socket.

ctx := context.Background()

<<setup-tracing>>

var cli CLI
c := kong.Parse(&cli, kong.BindTo(ctx, (*context.Context)(nil)))
if err := c.Run(); err != nil {
	slog.ErrorContext(ctx, "spiffe-creds failed", "error", err)
	os.Exit(1)
}

This is pretty common setup across my Go programs. I use Kong to make it easy to define CLI subcommands and arguments. Every subcommand has access to the parent context. If running the subcommand produces an error, then the error is logged and the command exits with a failing code.

type CLI struct {
	Serve ServeCmd `cmd:""`
	Check CheckCmd `cmd:""`
}

The CLI type for spiffe-creds has two subcommands. Their implementation will explored in more detail later.

exp, err := otlptracehttp.New(ctx)
if err != nil {
	panic(err)
}

res, err := resource.New(ctx, resource.WithFromEnv(),
	resource.WithTelemetrySDK(),
	resource.WithOS(),
	resource.WithHost(),
	resource.WithAttributes(semconv.ServiceName("spiffe-creds")))
if err != nil {
	panic(err)
}

tracerProvider := trace.NewTracerProvider(
	trace.WithBatcher(exp),
	trace.WithResource(res))
defer func() {
	if tracerProvider.Shutdown(ctx); err != nil {
		panic(err)
	}
}()
otel.SetTracerProvider(tracerProvider)

Before running the chosen subcommand, spiffe-creds sets up OpenTelemetry tracing. This will automatically send traces to the default OpenTelemetry HTTP endpoint, if something is listening. On my machines, that would be Grafana Alloy, which would send the traces to Tempo. I don't have the tracing collector set up on every machine that uses spiffe-creds: only machines that already have it running for some other reason.

var tracer = otel.Tracer("git.midna.dev/mjm/nix-config/packages/spiffe-tool/cmd/spiffe-creds")

A global tracer is created for the main package that can be used throughout the implementation of commands.

package main

import (
	"context"
	"log/slog"
	"os"

	"github.com/alecthomas/kong"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	"go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.17.0"
)

<<tracer>>

<<cli>>

func main() {
	<<main>>
}

spiffe-creds serve

The serve command runs the server that listens for incoming credentials requests from systemd.

type ServeCmd struct {
	Paths   []string          `name:"path"`
	Aliases map[string]string `name:"alias" mapsep:" " env:"SECRET_ALIASES"`

	vault *api.Client
}

The serve command must be given at least one path to a Vault KV secret. These paths form the roots of where secrets can be loaded from. For instance, a path of "prod/services/foo" will allow a service to request a credential named "foo_bar" to get the value of the "bar" key in "prod/services/foo".

To support situations where it's not possible to choose the credential name used with systemd, aliases may also be passed in. An alias is a key-value pair where the key is "/" and the value is a slash-separated path to a secret value, starting with the basename of one of the configured paths. Aliases also make it possible to use the same credential name to serve different secrets to different systemd services.

The serve command also stores a Vault API client, which is set up when the command is first run and then is used to fetch secrets upon request.

Spinning up the server

func (c *ServeCmd) Run(ctx context.Context) error {
	<<get-socket-listener>>
	<<setup-signal-context>>
	<<create-vault-client>>
	<<update-token-timer>>
	<<accept-loop>>
	<<handle-loop>>
	<<systemd-notify-ready>>
	<<wait>>
}

When run, the serve command has a number of different tasks to do, so we can go through them one-by-one.

listeners, err := activation.Listeners()
if err != nil {
	return fmt.Errorf("getting socket listeners: %w", err)
}

if len(listeners) != 1 {
	return fmt.Errorf("incorrect number of listeners (expected 1, got %d)", len(listeners))
}
l := listeners[0]
defer l.Close()

spiffe-creds is expecting to be run under systemd socket activation, so this gets the listener for the socket file descriptor that was passed in by systemd. The socket in question should be a Unix domain socket that only root can use. Otherwise, an unexpected unprivileged process would be able to request the secrets that spiffe-creds was configured to give access to.

ctx, stop := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
defer stop()

This provides a context to use going forward which will be cancelled when the program receives an interrupt or terminate signal. It's an easy way to give the various concurrent tasks that are going to be started soon a way to know when to drop what they are doing so the program can exit.

c.vault, err = newVaultClient(ctx)
if err != nil {
	return fmt.Errorf("creating vault client: %w", err)
}

slog.InfoContext(ctx, "created vault client")

Next, a new Vault API client is created and stored on the command struct.

func useSPIFFE() bool {
	return os.Getenv("SPIFFE_ENDPOINT_SOCKET") != ""
}

func newVaultClient(ctx context.Context) (*api.Client, error) {
	if useSPIFFE() {
		<<new-vault-client-with-spiffe>>
	} else {
		<<new-vault-client-without-spiffe>>
	}
}

How the client is actually created depends on if the service is running in a context that has access to the SPIFFE Workload API. In production this is always the case, but the alternative path is useful for local testing during development.

First, the SPIFFE case:

wc, err := workloadapi.New(ctx)
if err != nil {
	return nil, fmt.Errorf("creating workload api client: %w", err)
}

source, err := workloadapi.NewX509Source(ctx, workloadapi.WithClient(wc))
if err != nil {
	return nil, fmt.Errorf("creating x509 source: %w", err)
}

serverID := spiffeid.RequireFromString("spiffe://home.mattmoriarity.com/svc/vault")

vaultConfig := api.DefaultConfig()
transport := vaultConfig.HttpClient.Transport.(*http.Transport)
transport.TLSClientConfig = tlsconfig.TLSClientConfig(source, tlsconfig.AuthorizeID(serverID))

c, err := api.NewClient(vaultConfig)
if err != nil {
	return nil, fmt.Errorf("creating vault client: %w", err)
}

The workload API is going to be used for two purposes here. In this snippet, it's being used to verify the identity of the Vault server that the client will talk to. The TLSClientConfig that it creates will make sure the Vault server's certificate is a valid SPIFFE certificate, and that it corresponds to the expected ID for the Vault service. That TLSClientConfig is applied to an otherwise default Vault API client config, and then a new Vault API client is created with it.

for {
	select {
	case <-ctx.Done():
		return nil, ctx.Err()
	default:
	}

	if err := updateVaultToken(ctx, c); err != nil {
		slog.WarnContext(ctx, "error getting initial vault token", "error", err)
		time.Sleep(2 * time.Second)
		continue
	}

	return c, nil
}

The Vault client needs a token in order to be useful. A helper function called updateVaultToken will take a client and get a new token for it, so it can be used here to set the initial token.

Calling it just once isn't sufficient though. This service might be started due to an incoming request before it is actually possible to successfully login to Vault. A particularly common reason for this is that the Raspberry Pis don't have a hardware clock, so there is a delay to them getting an accurate time after booting. During this delay, the SPIFFE JWT tokens they get will not be treated as valid, because they look like they are issued in the future.

Regardless of the reason, it's important for the resilience of my systems that this service tolerate temporary failures with logging in to Vault. So creating a new Vault client will retry logging in indefinitely until it succeeds.

The implementation of updateVaultToken is as follows.

func updateVaultToken(ctx context.Context, c *api.Client) error {
	if useSPIFFE() {
		<<fetch-jwt-svid>>
		<<login-to-vault>>
		<<set-token-on-client>>
	}
	return nil
}

Updating the token only makes sense when using SPIFFE. Otherwise, the program won't have the information necessary to renew the token anyway.

jwtSource, err := workloadapi.NewJWTSource(ctx)
if err != nil {
	return fmt.Errorf("creating jwt source: %w", err)
}
defer jwtSource.Close()

svid, err := jwtSource.FetchJWTSVID(ctx, jwtsvid.Params{
	Audience: os.Getenv("VAULT_ADDR"),
})
if err != nil {
	return fmt.Errorf("fetching jwt svid: %w", err)
}

Logging in to Vault via SPIFFE, in my setup, works via Vault's JWT auth engine. Essentially, a SPIFFE JWT token can be exchanged with Vault for a Vault token. The Vault instance has entities that are aliased to SPIFFE IDs, and those entities have access to the secrets for the corresponding service.

So the first step to logging in is to get a JWT token via SPIFFE. The Vault instance's address is used as the audience of the token.

req := map[string]any{
	"role": "spiffe",
	"jwt":  svid.Marshal(),
}
resp, err := c.Logical().WriteWithContext(ctx, "auth/spiffe/login", req)
if err != nil {
	return fmt.Errorf("authorizing vault with jwt token: %w", err)
}

With JWT token in hand, the login payload can be constructed. The auth engine mount and role are both called "spiffe". Writing the small payload with the token to the login endpoint should produce a response with a valid Vault token.

token, err := resp.TokenID()
if err != nil {
	return fmt.Errorf("getting token from auth response: %w", err)
}

c.SetToken(token)

Assuming the login request was successful, the token can be retrieved from the response and set on the client.

That covers constructing a Vault client for use with SPIFFE. For testing on a dev machine, the process is a bit simpler:

homeDir, err := os.UserHomeDir()
if err != nil {
	return nil, fmt.Errorf("getting user home dir: %w", err)
}

tokenBytes, err := os.ReadFile(path.Join(homeDir, ".vault-token"))
if err != nil {
	return nil, fmt.Errorf("reading vault token from file: %w", err)
}

c, err := api.NewClient(nil)
if err != nil {
	return nil, fmt.Errorf("creating vault client: %w", err)
}
c.SetToken(strings.TrimSpace(string(tokenBytes)))
return c, nil

This reads the token from ~/.vault-token, which is where the Vault CLI tool stores the token of the logged in user. This token is just used as-is, and it is assumed to be valid. If not, that can be corrected by hand by running "vault login".

After the Vault client is created, it's time to set up a few goroutines to run loops for the duration the service is running.

go func() {
	for {
		select {
		case <-ctx.Done():
			return
		case <-time.After(3 * time.Minute):
			slog.InfoContext(ctx, "getting new vault token")
			if err := updateVaultToken(ctx, c.vault); err != nil {
				slog.WarnContext(ctx, "failed to update vault token", "error", err)
			}
		}
	}
}()

The JWT tokens issued by SPIRE only last for five minutes because they are less safe than x.509 certificates. This five minutes also applies to the Vault token that the JWT token is exchanged for. Because of this, the Vault token needs to be continually updated as long as this service is running. Every three minutes, it will attempt to update the token, and log a warning if doing so fails. Vault's API client uses a lock when setting the token, so this is safe to do from another goroutine like this.

FIXME: If this fails, it's going to wait another 3 minutes to retry, which will be after the token expires. Another retry loop in here is needed.

connCh := make(chan net.Conn)
go func() {
	for {
		conn, err := l.Accept()
		if err != nil {
			if errors.Is(err, net.ErrClosed) {
				return
			}

			slog.WarnContext(ctx, "failed to accept new connection", "error", err)
			continue
		}

		connCh <- conn
	}
}()

The purpose of this loop is to accept incoming connections and dispatch them to a channel. Typically, the channel wouldn't be needed: a goroutine could be dispatched directly from here to handle each accepted connection. The reason for the channel will be more obvious when showing the loop that receives from it.

The Accept() call will block until it either receives a connection or produces an error. In most cases, it will log the error as a warning and continue trying to accept connections. The exception is if the error is net.ErrClosed, which means the listener was closed while waiting for a connection. That's a normal situation: it will happen every time the program terminates. All that needs to be done then is return from the function and end the goroutine.

go func() {
	for {
		slog.InfoContext(ctx, "waiting for connection")
		select {
		case conn := <-connCh:
			go c.handleConn(ctx, conn)
		case <-time.After(10 * time.Minute):
			slog.InfoContext(ctx, "no new connections for a while, exiting")
			stop()
			return
		case <-ctx.Done():
			return
		}
	}
}()

This loop basically just the accept loop again, but using channels and with errors already handled. The reason the channel was needed at all is so that it can be raced against a ten minute timer. As the log message suggests, the goal of this is to let the service exit on its own when it hasn't handled any connections for a while. The service will be run using systemd's socket activation, so the socket will remain open even after the service exits, and systemd will start the service again if there are new connections.

When a connection is received, a goroutine is spawned to handle it. The implementation of that will be shown a little later. Using a goroutine here means that this loop should never block for a meaningful amount of time, which is why it's okay to use an unbuffered channel for the connections.

If ten minutes pass without a new connection coming through the channel, a message is logged, and the stop() function on the context is called, cancelling it. As you'll see soon, this will cause the program to terminate successfully.

And in the case that the context is cancelled before either of those happen, the goroutine is simply ended by returning.

daemon.SdNotify(false, daemon.SdNotifyReady)

With all three loops up and running, the service is ready to go, so it will notify systemd that that's the case. The service should be running as Type=notify, so systemd will not consider it started until this notification is received.

<-ctx.Done()

slog.Info("terminating")
return nil

Now the only thing to do is wait on the context, which prevents the program from exiting prematurely. Once the context is cancelled, it means the program should terminate. A message is logged, and the defer earlier kicks in to close the listener, stopping the accept loop. That's all there is to do.

Handling connections

func (c *ServeCmd) handleConn(ctx context.Context, conn net.Conn) {
	defer conn.Close()

	<<setup-conn-span>>
	<<get-systemd-addr>>
	<<resolve-key>>
	<<get-secret-loop>>
}

Handling a single connection is a pretty simple sequence of steps: get the systemd unit and credential name, resolve them to a particular secret, and then try to get the data for that secret and write it to the connection.

ctx, span := tracer.Start(ctx, "HandleConn",
	trace.WithAttributes(
		attribute.Stringer("net.addr.local", conn.LocalAddr()),
		attribute.Stringer("net.addr.remote", conn.RemoteAddr())))
defer span.End()

handleError := func(err error) {
	span.RecordError(err)
	span.SetStatus(codes.Error, err.Error())
	slog.ErrorContext(ctx, "error handling connection", "error", err)
}

The handling of a connection is wrapped in a tracing span, similar to how an HTTP request handler would be done. The local and remote addresses from the connection are included as attributes in the span for debugging. The remote address is of particular interest, since it contains information from systemd about what credential is being requested.

A local handleError function is also defined here to reduce some boilerplate. Errors are both logged and recorded in the tracing span.

addr, err := parseSystemdAddr(conn.RemoteAddr().String())
if err != nil {
	handleError(err)
	return
}
span.SetAttributes(attribute.String("systemd.unit", addr.Unit), attribute.String("systemd.cred.name", addr.Name))

The parseSystemdAddr helper function is called with the remote address from the connection. The address will be formatted in a particular way by systemd, as described in systemd.exec(5):

The returned socket name is formatted as NUL RANDOM "/unit/" UNIT "/" ID, i.e. a NUL byte (as required for abstract namespace socket names), followed by a random string (consisting of alphadecimal characters), followed by the literal string "/unit/", followed by the requesting unit name, followed by the literal character "/", followed by the textual credential ID requested.

The parseSystemdAddr function extracts the useful information from this into a small struct:

type systemdAddr struct {
	Unit string
	Name string
}

func parseSystemdAddr(s string) (systemdAddr, error) {
	cmps := strings.Split(s, "/")
	if len(cmps) != 4 {
		return systemdAddr{}, fmt.Errorf("address %q should have four slash-separated components", s)
	}

	if cmps[1] != "unit" {
		return systemdAddr{}, fmt.Errorf("second component of %q should be %q", s, "unit")
	}

	return systemdAddr{
		Unit: cmps[2],
		Name: cmps[3],
	}, nil
}

There's some validation that the address matches the expected format, and then the relevant components are yanked out. From there, those pieces can be used to resolve the actual secret that is being requested.

key, err := c.resolveAlias(addr)
if err != nil {
	handleError(err)
	return
}
if key == "" {
	key, err = c.resolvePath(addr)
	if err != nil {
		handleError(err)
		return
	}
}

vaultPath := path.Dir(key)
vaultKey := path.Base(key)
span.SetAttributes(attribute.String("vault.kv.path", vaultPath), attribute.String("vault.kv.key", vaultKey))

First it attempts to resolve the address as an alias. If that fails to match anything, then it will resolve it as a normal path. Both are done with methods on the ServeCmd struct, so that they have access to the paths and aliases the service was configured to use. Note that failing to find a matching alias is not an error: the key can be empty while err is nil. This is the case that falls through to resolving by path.

Both methods return the key, if resolved properly, as a path, where the dirname is the path to the secret in Vault, and the basename is the key within that secret whose value should be retrieved. Both of these values are set in the tracing span.

Resolving an alias looks like this:

func (c *ServeCmd) resolveAlias(addr systemdAddr) (string, error) {
	if keyPath, ok := c.Aliases[addr.AliasKey()]; ok {
		svcName, rest, found := strings.Cut(keyPath, "/")
		if !found {
			return "", fmt.Errorf("secret key path %q must be at least two components", keyPath)
		}
		for _, p := range c.Paths {
			if path.Base(p) == svcName {
				return path.Join(p, rest), nil
			}
		}

		return "", fmt.Errorf("credential named %q aliased to %q did not match any configured services (%v)", addr.Name, keyPath, c.Paths)
	}

	return "", nil
}

func (a systemdAddr) AliasKey() string {
	return a.Unit + "/" + a.Name
}

The systemd unit and credential name are stuck together and then used to look up a matching alias. If the alias is found, it needs to be resolved into a full path to a secret. This is similar to what resolving by path will do, but it doesn't share code because aliases don't need to use the same kind of workarounds that credential names do to avoid using slashes.

So for an alias, the key path the alias maps to is split at the first slash. The first component is the service name, and should match the basename of one of the paths configured for this instance. Since aliases are already known ahead of time, it might be good to validate these at service start: make sure every alias actually matches a defined path. When a matching path is found, it is combined with the remaining portion of the key path for the alias to make a full path for the secret.

To make this concrete, an example might help. Say Vault has a secret at "prod/services/foo", and within that secret is a key "bar" with the value we want to retrieve. A systemd unit "baz.service" wants to retrieve that secret using a credential named "quux". To accomplish this, spiffe-creds should be run with "prod/services/foo" as a path with an alias mapping "baz.service/quux" to "foo/bar".

Resolving by path directly looks like this:

func (c *ServeCmd) resolvePath(addr systemdAddr) (string, error) {
	svcName, rest, err := splitCredentialName(addr.Name)
	if err != nil {
		return "", err
	}

	for _, p := range c.Paths {
		if path.Base(p) == svcName {
			return path.Join(p, rest), nil
		}
	}

	return "", fmt.Errorf("credential named %q did not match any configured services (%v)", addr.Name, c.Paths)
}

When resolving this way (which is the more conventional and preferred way), the credential name alone is used to determine the secret being requested. The unit doesn't matter. The credential name is split up into two components: the service name and the rest of the key path. There are currently two ways to express this in the credential name, which are described below in the implementation of splitCredentialName. If the credential name is invalid, an error is returned and propagated to the caller. Otherwise, the service name is used similar to how it is for aliases to find a matching path. If the path for the service is found, it is combined with the rest of the key path to make the full path to the secret.

Using the same example as above, retrieving the "bar" key from "prod/services/foo" without an alias would mean running spiffe-creds with the same path of "prod/services/foo" and then requesting a credential named "foo.bar". If instead the "bar" key was under "prod/services/foo/managed", the correct credential name would be "foo.managed.bar".

Unlike resolveAlias, there's no way out of resolvePath without either having a secret key to request from Vault or producing an error.

func splitCredentialName(name string) (string, string, error) {
	svcName, keyPath, found := strings.Cut(name, ".")
	if found {
		keyPath = strings.ReplaceAll(keyPath, ".", "/")
		return svcName, keyPath, nil
	}

	svcName, keyPath, found = strings.Cut(name, "_")
	if found {
		keyPath = strings.ReplaceAll(keyPath, "__", "/")
		return svcName, keyPath, nil
	}

	return "", "", fmt.Errorf("credential name %q is missing service name prefix", name)
}

As mentioned above, there are two ways to write a credential name that refers to a particular secret. The new preferred one is to use dot-separated components as a substitute for slashes, which are invalid in filenames and therefore credential names. In this schema, a credential named "foo.bar.baz" would be split into "foo" and "bar/baz".

The older way is uglier and really only exists because I didn't think of the dot-separated way sooner. Here the service name is separated from the rest of the key path by an underscore. Because underscores are often used in the keys for my secrets, a double-underscore ("__") is used to separate path components. In this schema, a credential named "foo_bar__baz" would be split into "foo" and "bar/baz".

This older way will be gradually phased out and eventually removed.

Now that the path to the secret is resolved, with any errors already handled, the only thing left to do is fetch the value for the key and write it to the socket.

handleError = func(err error) {
	span.RecordError(err)
	slog.WarnContext(ctx, "error getting secret from vault", "error", err)
	time.Sleep(2 * time.Second)
}
for {
	scrt, err := c.vault.KVv2("kv").Get(ctx, vaultPath)
	if err != nil {
		handleError(err)
		continue
	}

	data, ok := scrt.Data[vaultKey]
	if !ok {
		err := fmt.Errorf("key %q is missing from secret at %q", vaultKey, vaultPath)
		handleError(err)
	} else {
		fmt.Fprintf(conn, "%v", data)
		return
	}
}

Since it is possible for Vault to be temporarily unavailable, it's important to think about how to handle that case. Delivering systemd credentials doesn't really have a good way to report errors back to the service or systemd, so if you just close the connection, you end up with services that are trying to run with empty secrets, getting stuck in weird and surprising failure modes. The best approach I've come up with so far is to retry failures, and hope that the outcome of that is more sensible. This could lead to a service being stuck for a while unable to start, but that's easier to observe than a service not having actual secret data in its credential files.

Going into the retry look, the handleError function from above is redefined to behave more correctly for handling errors in the loop. These are still recorded as events in the tracing span and logged as warnings, but they don't set the whole span to an error status. Instead, they wait two seconds before trying again. Perhaps a real exponential backoff would be better but I think this is probably fine for this.

The actual body of the loop is what you would expect. Fetch the secret at the resolved path from Vault, and look up the key in the resulting data. If that goes well, write the data to the socket and exit the loop. If it doesn't go well, handle the error as described and run the loop again.

The mechanism for serving secrets this way is pretty simple at its core. Most of the difficulty comes from navigating network errors and from mapping systemd credentials to the correct secret.

cmd_serve.go:

package main

import (
	"context"
	"errors"
	"fmt"
	"log/slog"
	"net"
	"os/signal"
	"path"
	"strings"
	"syscall"
	"time"

	"github.com/coreos/go-systemd/v22/activation"
	"github.com/coreos/go-systemd/v22/daemon"
	"github.com/hashicorp/vault/api"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/trace"
)

<<serve-cmd>>
<<serve-cmd-run>>
<<handle-conn>>
<<systemd-addr>>
<<resolve-alias>>
<<resolve-path>>
<<split-credential-name>>

vault.go:

package main

import (
	"context"
	"fmt"
	"log/slog"
	"net/http"
	"os"
	"path"
	"strings"
	"time"

	"github.com/hashicorp/vault/api"
	"github.com/spiffe/go-spiffe/v2/spiffeid"
	"github.com/spiffe/go-spiffe/v2/spiffetls/tlsconfig"
	"github.com/spiffe/go-spiffe/v2/svid/jwtsvid"
	"github.com/spiffe/go-spiffe/v2/workloadapi"
)

<<new-vault-client>>
<<update-vault-token>>