Generate mTLS certificates for Nomad using Vault

You can use Vault's PKI Secrets Engine to generate and renew dynamic X.509 certificates for your Nomad cluster nodes and Vault Agent to automatically create the appropriate certificate and key files on your nodes. With this method, each node has a unique certificate with a relatively short time-to-live (TTL). This feature, along with automatic certificate rotation, allows you to safely and securely scale your cluster while using mutual TLS (mTLS).

In this tutorial, you will secure your existing Nomad cluster with mTLS using Vault's PKI secrets engine to create both a root and intermediate CA. You will also use Vault Agent to fetch, renew, and periodically rotate your mTLS certificates on your Nomad nodes.

Prerequisites

Running this tutorial requires the following:

• A Nomad environment with Vault installed. You can use this repository to provision a sandbox environment. This tutorial will assume a cluster with one server node and three client nodes.

Prepare Vault

If you already have a Vault environment, skip ahead to Log in to Vault.

Initialize Vault server

Run the following command to initialize the Vault server and receive an unseal key and initial root token. If you are running the environment provided in this guide, the Vault server is co-located with the Nomad server. Be sure to note the unseal key and initial root token as you will need these two pieces of information.

$ vault operator init -key-shares=1 -key-threshold=1

The vault operator init command creates a single Vault unseal key for convenience. For a production environment, we recommended that you create at least five unseal key shares and securely distribute them to independent operators. The command defaults to five key shares and a key threshold of three. If you have provisioned more than one server, the others will become standby nodes and remain unsealed.

Unseal Vault

Run the command and provide your unseal key.

$ vault operator unseal

The output looks similar to the following.

Key                    Value---                    -----Seal Type              shamirInitialized            trueSealed                 falseTotal Shares           1Threshold              1Version                1.0.3Cluster Name           vault-cluster-d1b6513fCluster ID             87d6d13f-4b92-60ce-1f70-41a66412b0f1HA Enabled             trueHA Cluster             n/aHA Mode                standbyActive Node Address    <none>

Log in to Vault

Use the login command to authenticate with Vault using the initial root token from earlier.

$ vault loginToken (will be hidden): <your initial root token>

Successful output looks similar to the following.

Success! You are now authenticated. The token information displayed belowis already stored in the token helper. You do NOT need to run "vault login"again. Future Vault requests will automatically use this token....

Prepare the PKI environment

This tutorial uses a common and recommended pattern which is to have one mount act as the root CA and the other as the intermediate CA. The root CA signs only the intermediate CA CSRs from other PKI secrets engines. For a higher level of security, you can store your CA outside of Vault and use the PKI engine only as an intermediate CA.

Generate the root CA

Enable the root PKI secrets engine at the pki path.

$ vault secrets enable pkiSuccess! Enabled the pki secrets engine at: pki/

Tune the PKI secrets engine to issue certificates with a maximum time-to-live (TTL) of 87600 hours.

$ vault secrets tune -max-lease-ttl=87600h pkiSuccess! Tuned the secrets engine at: pki/

Generate the root certificate and save the certificate as CA_cert.crt.

$ vault write -field=certificate pki/root/generate/internal \    common_name="global.nomad" ttl=87600h > CA_cert.crt

Generate the intermediate CA and CSR

Enable the intermediate PKI secrets engine at the pki_int path.

$ vault secrets enable -path=pki_int pkiSuccess! Enabled the pki secrets engine at: pki_int/

Tune the PKI secrets engine at the pki_int path to issue certificates with a maximum time-to-live (TTL) of 43800 hours.

$ vault secrets tune -max-lease-ttl=43800h pki_intSuccess! Tuned the secrets engine at: pki_int/

Generate a CSR from your intermediate CA and save it as pki_intermediate.csr.

$ vault write -format=json pki_int/intermediate/generate/internal \    common_name="global.nomad Intermediate Authority" \    ttl="43800h" | jq -r '.data.csr' > pki_intermediate.csr

Sign and deploy the intermediate CA certificate

Sign the intermediate CA CSR with the root certificate and save it as intermediate.cert.pem.

$ vault write -format=json pki/root/sign-intermediate \    csr=@pki_intermediate.csr format=pem_bundle \    ttl="43800h" | jq -r '.data.certificate' > intermediate.cert.pem

Import the signed intermediate certificate into Vault.

$ vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

Set up the AppRole auth method

Vault offers many authentication methods including AWS, GCP, Azure, GitHub, LDAP, and Okta. In a production environment running in a cloud provider, we recommend using the machine's identity and the cloud provider's auth method.

The AppRole authentication method allows applications or services to authenticate with Vault-defined roles. For on-premises environments, we recommend AppRoles. In this tutorial, Nomad is using the AppRole auth method to retrieve certificates from Vault with the PKI engine.

Enable the AppRole auth method.

$ vault auth enable approleSuccess! Enabled approle auth method at: approle/

Create a policy to access the role endpoint

Create a policy file named tls-policy.hcl, add the following contents to it, and save the file. This adds only the update capability for the pki_int/issue/nomad-cluster path. Refer to the Vault policies page for additional information.

path "pki_int/issue/nomad-cluster" {  capabilities = ["update"]}

Write the policy into Vault.

$ vault policy write tls-policy tls-policy.hclSuccess! Uploaded policy: tls-policy

Create an approle

Create an approle named nomad-cluster. Consider increasing or decreasing the TTL and SecretID values to match the your environment.

$ vault write auth/approle/role/nomad-cluster \  token_type=batch \  token_policies="tls-policy" \  secret_id_ttl=10m \  token_ttl=10m \  token_max_ttl=15m \  secret_id_num_uses=10

The output looks similar to the following.

Success! Data written to: auth/approle/role/nomad-cluster

Next, get the role ID and save it to a file.

$ vault read -field="role_id" auth/approle/role/nomad-cluster/role-id > roleid

Then, get the SecretID and save it to a file. The SecretID is similar to a password in that you should keep it secret and confidential. Refer to the best practices page for additional information about working with the SecretID.

$ vault write -field="secret_id" auth/approle/role/nomad-cluster/secret-id > secretid

Create and populate the templates directory

The templates directory contains the files used by Vault Agent to render the certificates and keys on the nodes in your cluster.

Create a directory named templates in /opt/nomad.

$ sudo mkdir /opt/nomad/templates

Create a template file for each configuration below, add the contents to it, and save it with the filename given.

We recommend creating only the server and CLI files for server nodes and client and CLI files for client nodes.

{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}{{ .Data.Cert }}{{ end }}

{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}{{ .Data.Key }}{{ end }}

{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=server.global.nomad" "ttl=24h"}}{{ .Data.CA }}{{ end }}

{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=client.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}{{ .Data.Cert }}{{ end }}

{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=client.global.nomad" "ttl=24h" "alt_names=localhost" "ip_sans=127.0.0.1"}}{{ .Data.Key }}{{ end }}

{{ with pkiCert "pki_int/issue/nomad-cluster" "common_name=client.global.nomad" "ttl=24h"}}{{ .Data.CA }}{{ end }}

{{ with pkiCert "pki_int/issue/nomad-cluster" "ttl=24h"}}{{ .Data.Cert }}{{ end }}

{{ with pkiCert "pki_int/issue/nomad-cluster" "ttl=24h"}}{{ .Data.Key }}{{ end }}

Create the certificates output directory

Create the rendered-cert-files directory to contain the generated certificate files.

$ sudo mkdir /opt/nomad/rendered-cert-files

Start Vault Agent

Create a directory for the configuration named vault-agent in /opt/vault.

$ sudo mkdir /opt/vault/vault-agent

Create the Vault Agent configuration file, name it vault_agent.conf, add the following contents to it, and save the file in the /opt/vault/vault-agent directory.

Modify this configuration file to match the templates created for each specific node from the previous step. Include or remove the server and client template blocks as necessary.

pid_file = "./pidfile"vault {  address = "https://127.0.0.1:8200"  # For demonstration purposes only. In a production environment,  # tls_skip_verify value should NOT be set to true.  tls_skip_verify = true}auto_auth {  method {    type      = "approle"    config = {      role_id_file_path = "/opt/vault/vault-agent/roleid"      secret_id_file_path = "/opt/vault/vault-agent/secretid"      remove_secret_id_file_after_reading = false    }  }  sink {    type = "file"    wrap_ttl = "30m"    config = {      path = "sink_file_wrapped_1.txt"    }  }  sink {    type = "file"    config = {      path = "sink_file_unwrapped_2.txt"    }  }}listener "tcp" {  address = "127.0.0.1:8100"  tls_disable = true}template_config {  static_secret_render_interval = "10m"  exit_on_retry_failure = true  max_connections_per_host = 20}# Server templates; remove if on client nodetemplate {  source       = "/opt/nomad/templates/server.agent.crt.tpl"  destination  = "/opt/nomad/rendered-cert-files/server.agent.crt"}template {  source       = "/opt/nomad/templates/server.agent.key.tpl"  destination  = "/opt/nomad/rendered-cert-files/server.agent.key"}template {  source       = "/opt/nomad/templates/server.ca.crt.tpl"  destination  = "/opt/nomad/rendered-cert-files/server.ca.crt"}# Client templates; remove if on server nodetemplate {  source       = "/opt/nomad/templates/client.agent.crt.tpl"  destination  = "/opt/nomad/rendered-cert-files/client.agent.crt"}template {  source       = "/opt/nomad/templates/client.agent.key.tpl"  destination  = "/opt/nomad/rendered-cert-files/client.agent.key"}template {  source       = "/opt/nomad/templates/client.ca.crt.tpl"  destination  = "/opt/nomad/rendered-cert-files/client.ca.crt"}# CLI templatestemplate {  source       = "/opt/nomad/templates/cli.crt.tpl"  destination  = "/opt/nomad/rendered-cert-files/cli.crt"}template {  source       = "/opt/nomad/templates/cli.key.tpl"  destination  = "/opt/nomad/rendered-cert-files/cli.key"}

Then, start Vault Agent on each node.

$ sudo vault agent -config=/opt/vault/vault-agent/vault_agent.conf

The output shows the agent rendering the local files with the certificate data.

# ...2024-11-26T10:09:54.347-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/cli.crt.tpl" => "/opt/nomad/rendered-cert-files/cli.crt"2024-11-26T10:09:54.367-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/server.ca.crt.tpl" => "/opt/nomad/rendered-cert-files/server.ca.crt"2024-11-26T10:09:54.405-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/client.agent.key.tpl" => "/opt/nomad/rendered-cert-files/client.agent.key"2024-11-26T10:09:54.425-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/server.agent.key.tpl" => "/opt/nomad/rendered-cert-files/server.agent.key"2024-11-26T10:09:54.463-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/cli.key.tpl" => "/opt/nomad/rendered-cert-files/cli.key"2024-11-26T10:09:54.484-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/client.agent.crt.tpl" => "/opt/nomad/rendered-cert-files/client.agent.crt"2024-11-26T10:09:54.569-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/server.agent.crt.tpl" => "/opt/nomad/rendered-cert-files/server.agent.crt"2024-11-26T10:09:54.592-0500 [INFO]  agent: (runner) rendered "/opt/nomad/templates/client.ca.crt.tpl" => "/opt/nomad/rendered-cert-files/client.ca.crt"

Configure Nomad to use TLS

Add the following tls stanza to the agent configuration file of each Nomad server and client node in the cluster.

tls {  http = true  rpc  = true  rpc_upgrade_mode = true  ca_file   = "/opt/nomad/rendered-cert-files/server.ca.crt"  cert_file = "/opt/nomad/rendered-cert-files/server.agent.crt"  key_file  = "/opt/nomad/rendered-cert-files/server.agent.key"  verify_server_hostname = true  verify_https_client    = true}

tls {  http = true  rpc  = true  ca_file   = "/opt/nomad/rendered-cert-files/client.ca.crt"  cert_file = "/opt/nomad/rendered-cert-files/client.agent.crt"  key_file  = "/opt/nomad/rendered-cert-files/client.agent.key"  verify_server_hostname = true  verify_https_client    = true}

Reload the Nomad configuration on each node.

$ systemctl reload nomad

After the reload, remove the rpc_upgrade_mode = true line from each server agent configuration file and save the file. This instructs the servers to only accept TLS connections.

 tls {   http = true   rpc  = true-  rpc_upgrade_mode = true    # ...

Then, reload the Nomad configuration on each server node again. Refer to the RPC Upgrade Mode page for additional information.

$ systemctl reload nomad

Configure the Nomad CLI

The Nomad CLI defaults to HTTP instead of HTTPS. Configure the Nomad CLI to connect using TLS by providing paths to the certificate files through environment variables.

export NOMAD_CACERT="/opt/nomad/rendered-cert-files/server.ca.crt"export NOMAD_CLIENT_CERT="/opt/nomad/rendered-cert-files/cli.crt"export NOMAD_CLIENT_KEY="/opt/nomad/rendered-cert-files/cli.key"

Run the status command to confirm connectivity from the CLI.

$ nomad statusNo running jobs

Encrypt server gossip

All communications over RPC and HTTP in your Nomad cluster are now secure with mTLS.

Nomad servers also communicate with the Serf gossip protocol, which does not use TLS. To learn how to configure gossip encryption, follow the Enable Gossip Encryption for Nomad tutorial.

Next steps

In this tutorial, you configured the Vault PKI engine, installed and used Vault Agent to automatically create certificate files on cluster nodes, and enabled mTLS with those certificates to encrypt traffic between the nodes.

Continue your learning with these resources:

• AppRole pull authentication tutorial
• Vault Agent templates
• Vault PKI Secrets Engine
• Build Your Own Certificate Authority (CA)
• Learn about Vault Proxy and how it compares to Vault Agent