Cache secrets with Vault Agent

Nearly all requests to Vault must be accompanied by a valid token which means that Vault clients must first authenticate with Vault and acquire a token. Secure Introduction of Vault Clients tutorial talked about the approaches to solve this secret zero problem.

Vault Agent with AWS and Vault Agent with Kubernetes tutorials walked through the Vault Agent's Auto-Auth.

Challenge and solution

Depending on the location of your Vault clients and its secret access frequency, you may face some scaling or latency challenge. Even with Vault Performance Replication enabled, the pressure on the storage backend increases as the number of token or lease generation requests increase. Vault 1.0 introduced batch tokens as a solution to relieve some pressure on the storage backend. By design, batch tokens do not support the same level of flexibility and features as service tokens. Therefore, if you need a periodic token for example, you would need service tokens.

To increase the availability of tokens and secrets to the clients, Vault Agent introduced the Caching function.

Vault Agent Caching can cache the tokens and leased secrets proxied through the agent which includes the auto-auth token. This allows for easier access to Vault secrets for edge applications, reduces the I/O burden for basic secrets access for Vault clusters, and allows for secure local access to leased secrets for the life of a valid token.

Prerequisites

To complete this section of the tutorial, you will need:

• AWS account and associated credentials that allow for the creation of resources
• Terraform v0.15.2 or later installed

Provision the cloud resources

1. Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

   Clone the repository:

   $ git clone https://github.com/hashicorp/vault-guides

   Or download the repository:

   

   This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.

2. Set your working directory to where the /identity/vault-agent-caching/terraform-aws was downloaded.

   $ cd vault-guides/identity/vault-agent-caching/terraform-aws

   The directory contains the following files.

   $ tree.├── aws.tf├── iam.tf├── instances.tf├── kms.tf├── network.tf├── outputs.tf├── security-groups.tf├── templates│   ├── userdata-vault-client.tpl│   └── userdata-vault-server.tpl├── terraform.tfvars.example├── variables.tf└── versions.tf1 directory, 12 files

   Note

   The example Terraform in this repository is created for the demo purpose.

3. Set an AWS_ACCESS_KEY_ID environment variable to hold your AWS access key ID.

   $ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"

4. Set an AWS_SECRET_ACCESS_KEY environment variable to hold your AWS secret access key.

   $ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"

5. Create a file named terraform.tfvars and specify the key_name. (You can copy the terraform.tfvars.example file and rename it as terraform.tfvars.)

   Example:

   terraform.tfvars

   # EC2 key pair name to access EC2 instances (should already exist) on the AWS regionkey_name = "vault-test"

   Tip

   If you don't have an EC2 key pair, follow the AWS documentation to create one.

6. Perform a terraform init to pull down the necessary provider resources.

   $ terraform init

7. Run terraform apply to provision AWS resources.

   $ terraform apply

   When prompted, enter yes to proceed.

   The Terraform output will display the public IP address to SSH into your Vault server and client instances.

   ...Apply complete! Resources: 23 added, 0 changed, 0 destroyed.Outputs:endpoints =Vault Server IP (public):  54.219.129.15Vault Server IP (private): 10.0.101.50For example:   ssh -i vault-test.pem ubuntu@54.219.129.15Vault Client IP (public):  54.183.212.51Vault Client IP (private): 10.0.101.209For example:   ssh -i vault-test.pem ubuntu@54.183.212.51

Configure AWS IAM auth method

In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.

1. SSH into the Vault Server instance.

   

   When you are prompted, enter "yes" to continue.

   $ ssh -i <path_to_key> ubuntu@<public_ip_of_server>...Are you sure you want to continue connecting (yes/no)? yes

   NOTE: If you received Permissions 0664 for '<key_name>.pem' are too open error, be sure to set the file permission appropriately.

   $ chmod 600 <key_name>.pem

2. Run the vault operator init command to initialize the Vault server.

   $ vault operator init...Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNqSuccess! Vault is initialized...

   Note

   The Vault server is configured to auto-unseal with AWS Key Management Service (KMS); therefore, once the server is initialized, it gets automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS tutorial.

3. Copy the Initial Root Token value.

4. Log into Vault using the generated initial root token.

   $ vault loginToken (will be hidden):

5. Examine and then execute the /home/ubuntu/aws_auth.sh script.

   $ cat aws_auth.sh

6. Execute the script.

   $ ./aws_auth.shSuccess! Enabled the kv secrets engine at: kv/Success! Uploaded policy: myappSuccess! Enabled aws auth method at: aws/Success! Data written to: auth/aws/config/clientSuccess! Data written to: auth/aws/role/app-roleSuccess! Enabled userpass auth method at: userpass/Success! Data written to: auth/userpass/users/student

   This script enables key/value v1 secrets engine at kv, create myapp policy, enables aws auth method and create a role named app-role. Also, userpass is enabled and created a user (student) with myapp policy attached.

7. Check the myapp policy.

   $ vault policy read myapppath "kv/*" {  capabilities = ["create", "read", "update", "delete"]}path "aws/creds/*" {  capabilities = ["read", "update"]}path "sys/leases/*" {  capabilities = ["create", "update"]}path "auth/token/*" {  capabilities = ["create", "update"]}

8. Examine and then execute the /home/ubuntu/aws_secrets.sh script.

   $ cat aws_secrets.sh

9. Execute the script.

   $ ./aws_secrets.shSuccess! Enabled the aws secrets engine at: aws/Success! Data written to: aws/config/rootSuccess! Data written to: aws/config/leaseSuccess! Data written to: aws/roles/readonly

   This script enables aws secrets engine, configures it and set the generated secret's lease to 1 hour and lease max to 24 hours. It also configures readonly role which is mapped to is mapped to arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess IAM policy.

Start Vault Agent

1. Now, SSH into the Vault Client instance.

   

   When you are prompted, enter "yes" to continue.

   $ ssh -i <path_to_key> ubuntu@<public_ip_of_client>...Are you sure you want to continue connecting (yes/no)? yes

2. Explore the Vault Agent configuration file, /home/ubuntu/vault-agent.hcl

   $ cat vault-agent.hclexit_after_auth = falsepid_file = "./pidfile"auto_auth {   method "aws" {       mount_path = "auth/aws"       config = {           type = "iam"           role = "app-role"       }   }   sink "file" {       config = {           path = "/home/ubuntu/vault-token-via-agent"       }   }}cache { }api_proxy {   use_auto_auth_token = true}listener "tcp" {   address = "127.0.0.1:8200"   tls_disable = true}vault {   address = "http://<vault-server-host>:8200"}

   Tip

   Also, read the Server Side Request Forgery (SSRF) Protection section.

   Note

   Notice the api_proxy block sets the use_auto_auth_token parameter to true. Since the auto-auth sink is set to /home/ubuntu/vault-token-via-agent, in absence of client token on requests sent to the API proxy, Vault Agent will use the auto-auth token read from vault-token-via-agent. The vault block defines the Vault server location (e.g. http://10.0.101.209:8200).

3. Execute the following command to start the Vault Agent with log level set to debug.

   $ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug==> Vault server started! Log data will stream in below:==> Vault agent configuration:           Api Address 1: http://127.0.0.1:8200                     Cgo: disabled               Log Level: debug                 Version: Vault v1.3.0             Version Sha: 30f07c76e1ea0551f28f5b8f7537f0de49d27f802019-03-06T03:12:47.616Z [INFO]  sink.file: creating file sink2019-03-06T03:12:47.616Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent2019-03-06T03:12:47.617Z [DEBUG] cache: auto-auth token is allowed to be used; configuring inmem sink2019-03-06T03:12:47.617Z [INFO]  cache: starting listener: addr=127.0.0.1:82002019-03-06T03:12:47.618Z [INFO]  auth.handler: starting auth handler2019-03-06T03:12:47.618Z [INFO]  auth.handler: authenticating2019-03-06T03:12:47.618Z [INFO]  sink.server: starting sink server2019-03-06T03:12:47.883Z [INFO]  auth.handler: authentication successful, sending token to sinks2019-03-06T03:12:47.883Z [INFO]  auth.handler: starting renewal process2019-03-06T03:12:47.883Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent2019-03-06T03:12:47.883Z [DEBUG] cache.leasecache: storing auto-auth token into the cache2019-03-06T03:12:47.887Z [INFO]  auth.handler: renewed auth token

4. Open a second SSH terminal into the client machine.

5. Verify the client token stored in /home/ubuntu/vault-token-via-agent.

   $ more vault-token-via-agent

6. Check the details about the token (e.g. attached policies, TTL, etc.).

   $ VAULT_TOKEN="$(cat vault-token-via-agent)" vault token lookup

7. Set VAULT_AGENT_ADDR environment variable.

   $ export VAULT_AGENT_ADDR="http://127.0.0.1:8200"

Request dynamic secrets and tokens via agent

Cache evictions

The eviction of cache will occur when the agent fails to renew leases or tokens. This can happen when the cached lease/token hits it's maximum TTL or if the renewal results in an error. Agent also does some best-effort cache evictions by observing specific request types and response codes.

While agent observes requests and evicts cached entries automatically, you can trigger a cache eviction by invoking the /agent/v1/cache-clear endpoint.

1. If you need to manually evict a stale lease, invoke the /agent/v1/cache-clear endpoint with lease ID for which you wish to evict from the cache.

   Example:

   $ curl --request POST \       --data '{ "type": "lease", "value": "aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc" }' \       $VAULT_AGENT_ADDR/agent/v1/cache-clear

   The agent log should show the following:

   [DEBUG] cache.leasecache: received cache-clear request: type=lease namespace= value=aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc[DEBUG] cache.leasecache: canceling context of index attached to accessor[DEBUG] cache.leasecache: successfully cleared matching cache entries[DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/aws/creds/readonly[DEBUG] cache.leasecache: evicting index from cache: id=dd0f9e775... path=/v1/aws/creds/readonly method=GET

2. Let's see what happens when you revoke a token.

   $ vault token revoke s.AkdB7Rgr0t8mIexWEGHPsNfn

   Or, revoke a token via API.

   $ curl --request POST \       --data '{"token": "s.AkdB7Rgr0t8mIexWEGHPsNfn"}' \       $VAULT_AGENT_ADDR/v1/auth/token/revoke

   Examine the agent log:

   ...[INFO]  cache: received request: path=/v1/auth/token/revoke method=POST[DEBUG] cache: using auto auth token: path=/v1/auth/token/revoke method=POST[DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/revoke method=POST[INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/revoke method=POST[DEBUG] cache.leasecache: cancelling context of index attached to token[DEBUG] cache.leasecache: successfully cleared matching cache entries[DEBUG] cache.leasecache: triggered caching eviction from revocation request[DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/auth/token/create[DEBUG] cache.leasecache: evicting index from cache: id=b5715bdca771174... path=/v1/auth/token/create method=POST

   Note

   In the agent log, notice the message, "triggered caching eviction from revocation request". When a token revocation request is made via the agent, the agent evicts the cached entries associated with the revoked token.

   Similarly, if you revoked a staled AWS lease, the agent will automatically evict the cache.

3. If a situation requires you to clear all cached tokens and leases (e.g. reset after a number of testing), set the type to all.

   $ curl --request POST --data '{ "type": "all" }' \       $VAULT_AGENT_ADDR/agent/v1/cache-clear

Clean up

1. On the server SSH terminal, execute the following command to revoke all leases.

   $ vault lease revoke -prefix aws/creds/readonly

2. Execute the following commands to destroy cloud resources.

   $ terraform apply -destroy -auto-approve

3. Delete state files.

   $ rm -rf .terraform terraform.tfstate*

Server side request forgery (SSRF) protection

Add require_request_header = true in the Agent's listener configuration stanza to enable the SSRF protection.

Example:

exit_after_auth = falsepid_file = "./pidfile"auto_auth {   method "aws" {       mount_path = "auth/aws"       config = {           type = "iam"           role = "app-role"       }   }   sink "file" {       config = {           path = "/home/ubuntu/vault-token-via-agent"       }   }}cache { }api_proxy {   use_auto_auth_token = true}listener "tcp" {   address = "127.0.0.1:8200"   require_request_header = true}vault {   address = "http://<vault-server-host>:8200"}

Notice the listener block.

When the require_request_header is set to true, the Agent listener will reject all requests that do NOT have the proper X-Vault-Request: true header.

For example, the command to request AWS credentials you saw in the request dynamic secrets and tokens via agent step must look like:

$ curl -header "X-Vault-Request: true" \       $VAULT_AGENT_ADDR/v1/aws/creds/readonly

With absence of the X-Vault-Request header, the Agent will throw missing 'X-Vault-Request' header error, and the request will not be propagated to the Vault server.

Help and reference

• Blog post: Why Use the Vault Agent for Secrets Management?
• Video: Streamline Secrets Management with Vault Agent and Vault 0.11
• Secure Introduction of Vault Clients
• Vault Agent Auto-Auth
• Vault Agent Caching
• AWS Auth Method

Other Vault Agent tutorials:

• Vault Agent with AWS
• Vault Agent with Kubernetes
• Vault Agent Templates
• Vault Agent Windows Service
• Using HashiCorp Vault Agent with .NET Core