This is my experience of setting up vault-agents on my servers to handle dynamic secrets in the configurations of my services.
Table of Contents
🔗What is vault-agent?
Vault-agent1 is a kind of companion tool for vault2, which allows one to start integrating vault and its capabilties into existing systems or systems that cannot easily be changed to integrate vault natively. For my use-case, the most important feature is the option to fetch secrets from vault and use it to template out files, specifically configuration files. This makes it a lot easier to manage secrets that need to be available locally at program startup, without having to update it manually everytime it changes or exipres.
🔗Configuring the Vault Server
First we need to configure the vault server, such that we can generate the tokens for our agents and assign them the appropriate policies.
For this purpose, I first create a kind of account for each of the servers, that a vault agent will be installed on. In vault this translates to an identity and an alias for each of the servers. All of these identities are then grouped into a vault-agent group, which will actually have the shared policies assigned to it, such that there is only one place to manage all of the shared access, while also retaining the option to assign policies to individual servers, for example if one of them required extra credentials as it fulfills a unique job not shared by other servers.
To then allow the vault agents to actually authenticate with vault, I decided to simply use tokens directly instead of generating credentials using something like AppRole, but I still do not love this approach yet. For this I created a backend role in the token auth mechanism, which only allows the previously configured aliases to authenticate against it (as well as allowing orphaned tokens and renewing).
This means I can now generate a unique vault token for each of my servers and they can use it to authenticate as that server against vault to load any secrets they might need.
🔗Deploying Vault Agents
Deploying the vault agents to my servers now comes down to:
- installing the vault agent software on the server
- getting a token for the server from vault
- deploying the general config for the vault agents
🔗Deploying templated Configs
Now that vault-agents are running on my servers, it is time to actually use them for something useful. This means I need to make it easy to integrate the deployment of templated configurations for other services, which I decided to do by introducing a simple ansible-role. This role takes care of creating the correct configuration for the vault-agent to template out a file and placing it in a predefined location, where the vault-agent can pick it up. It will then also trigger the vault-agent on the server to restart / reload its configuration to pick up the newly created config and start outputting the templated configuration.
So all I need to do, to deploy a service using templated configurations, is give the vault agent user the permissions to write the config to where my service expects it, give it the template to use and providing a command to execute whenever the written out file changes, in case the service does not automatically detect changes to it.
🔗Example Uses
🔗Nomad - mTLS
My nomad cluster uses mTLS to encrypt its communication over the network, this means every node needs the CA certificate and its own certificate to communicate with other nodes. Previously this meant managing the certificates and their livecycles semi-manual, I automated all the steps, but needed to trigger the updates manually.
However now with the vault-agents, I can configure them to automatically fetch the shared CA certificate(s) and generate the certificate for the current node dynamically, including renewing it and notifying nomad accordingly. This workflow is a lot easier to maintain, as there is no human involvement to rotate the leaf-certificates, and allows for enforcing best practices, like short lifetimes for the leaf-certificates.
🔗Nomad - Consul ACLs
My nomad cluster is very tightly integrated into consul, to make use of its service-mesh capabilties and some other features. However because I have ACLs enabled on the consul cluster, every nomad client requires an ACL token to connect to consul. Managing this manually would either mean a lot of maintenance for me or using tokens with very long lifetimes, which are both not good options.
Instead I can use the consul secret-engine in vault and the vault-agents, to dynamically obtain an ACL token for each server, which contains just enough permissions for nomad and has a short lifetime to follow best-practices as much as possible (for my context).
🔗Some Issues
I want to cover some of the issues that I encountered / still have, just so I am clear on the downsides of this setup.
🔗Vault Token expiring while vault-agent is not running
I have one server that likes to randomly freeze on the OS level, which is mostly fine in my setup as all the normal workloads can just migrate and once I get to restarting the server everything just comes back. However because I use a vault token, which expires after at most one day with my current config, it frequently happens that the token for the server expires before I get around to restarting it, meaning I need to run my vault-agent deployment again to generate a new token for the server. This is not a huge issue for me in this case, but mostly annoying as I need to keep that in mind during maintenance and adds another kind of weird thing.
🔗One bad config infects the entire agent
If I accidentally deploy one bad vault-agent config, for example by writing a custom config for templating a config, then the entire vault-agent on the server is not working properly and thus affects other services as well. I have no good approach to prevent this, besides enforcing the use of the ansible-role, which could be updated to make sure that after every change the agent is still running without issues, but that is not yet implemented.