Reload secrets after expiration with Vault Agent with .NET Core

If your .NET application needs some secrets (e.g. database credentials), your organization might offer HashiCorp Vault to store and manage them for you. As a developer, you need a way to retrieve secrets from Vault for your application to use.

While you can use a C# client library to authenticate to Vault and retrieve secrets with application code, you can better scale your application's interaction with Vault by using Vault Agent. Vault Agent will handle the authentication and retrieval of secrets from Vault, enabling the scalability of the distribution and rotation of short-lived secrets.

This tutorial demonstrates how to use Vault Agent and Consul template to authenticate to Vault, retrieve database usernames and passwords, generate a configuration file, and reload an application each time the database secrets expire.

Prerequisites

This lab was tested on macOS using an x86_64 based processor. If you are running macOS on an Apple silicon-based processor, use a x86_64 based Linux virtual machine in your preferred cloud provider.

• Docker
• Docker Compose
• .NET v5.0.0+
• Consul template

Retrieve the demo application

Retrieve the configuration by cloning or downloading the hashicorp/vault-guides repository from GitHub.

Clone the repository.

$ git clone https://github.com/hashicorp/vault-guides

Or download the repository.

This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found under the secrets/dotnet-vault/ProjectApi/ directory.

Switch your working directory to secrets/dotnet-vault/.

$ cd vault-guides/secrets/dotnet-vault

You should find the ProjectApi/ sub-directory.

$ tree --dirsfirst -L 1.├── ProjectApi├── database├── README.md├── cleanup.sh├── cleanup_vault_agent.sh├── demo_setup.sh├── docker-compose-vault-agent-template.yml├── docker-compose-vault-agent-token.yml├── docker-compose.yml├── get_db_username.sh├── list_passwords.sh├── new_secret.sh├── projects-role-policy.hcl├── revoke_passwords.sh├── run_app.sh├── vault_agent_template.sh└── vault_agent_token.sh2 directories, 15 files

Vault and database setup

Your application, called project-api needs to reference a Vault deployment and Microsoft SQL Server (MSSQL). Create the dependencies by running the setup script, which will configure and populate data for both Vault and MSSQL.

$ bash demo_setup.shCreating network "dotnet-vault_vpcbr" with driver "bridge"Building dbStep 1/6 : FROM microsoft/mssql-server-linux:latest ---> 314918ddaedfStep 2/6 : RUN mkdir -p /usr/src/app ---> Using cache ---> 3b3601a28c41Step 3/6 : WORKDIR /usr/src/app ---> Using cache ---> b844aca08dcbStep 4/6 : COPY . /usr/src/app ---> Using cache ---> 661442b758f8Step 5/6 : RUN chmod +x /usr/src/app/import-data.sh ---> Using cache ---> e95b64527c1bStep 6/6 : CMD /bin/bash ./entrypoint.sh ---> Using cache ---> e4e729f2944fSuccessfully built e4e729f2944fSuccessfully tagged db:latestCreating dotnet-vault_vault_1 ... doneCreating dotnet-vault_db_1    ... doneSuccess! Enabled approle auth method at: approle/Success! Enabled the database secrets engine at: projects-api/database/Success! Enabled the kv secrets engine at: projects-api/secrets/Key              Value---              -----created_time     2020-11-17T21:54:42.8659611Zdeletion_time    n/adestroyed        falseversion          1Success! Data written to: projects-api/database/roles/projects-api-roleSuccess! Uploaded policy: projects-apiSuccess! Data written to: auth/approle/role/projects-api-role

In addition, the operations team already configured Vault with a database secrets engine that will create a new username and password on-demand for the Microsoft SQL Server database.

To examine this workflow, run get_db_username.sh to make an API call to Vault and generate a new username and password for use.

$ bash get_db_username.shKey                Value---                -----lease_id           projects-api/database/creds/projects-api-role/vMfl84jPjvh1j0tWxdmNj5G1lease_duration     1hlease_renewable    truepassword           XqUwRHYLrg-w8munBWuJusername           v-token-projects-api-role-XQmyY60e53mnkbE1RTFL-1605719519

Copy the generated username and store it in the DB_USERNAME environment variable.

Example:

$ export DB_USERNAME=v-token-projects-api-role-XQmyY60e53mnkbE1RTFL-1605719519

Similarly, copy the generated password and store it in the DB_PASSWORD environment variable.

Example:

$ export DB_PASSWORD=XqUwRHYLrg-w8munBWuJ

The MSSQL contains the HashiCorp database with a table called Projects. It contains information about HashiCorp's projects.

Connect to the MSSQL running in the dotnet-vault_db_1 container with the username stored in the DB_USERNAME and the password stored in the DB_PASSWORD environment variable.

$ docker exec -it dotnet-vault_db_1 /opt/mssql-tools/bin/sqlcmd \    -S localhost -U ${DB_USERNAME} \    -P ${DB_PASSWORD} -d HashiCorp

Now, select the Projects table.

$ SELECT * FROM Projects

Execute the GO command to execute the select command and view the table entries.

$ GOId           YearOfFirstCommit  GitHubLinkVagrant      2010               https://github.com/hashicorp/vagrantPacker       2013               https://github.com/hashicorp/packerTerraform    2014               https://github.com/hashicorp/terraformNomad        2015               https://github.com/hashicorp/nomadConsul       2013               https://github.com/hashicorp/consulVault        2015               https://github.com/hashicorp/vaultWaypoint     2020               https://github.com/hashicorp/waypointBoundary     2020               https://github.com/hashicorp/boundary(8 rows affected)

Enter exit to quit the docker exec command.

$ exit

Your operations team has given you a Vault role and secret to log into Vault using the approle auth method.

You can find the role identifier at ProjectApi/vault-agent/role-id.

$ echo $(cat ProjectApi/vault-agent/role-id)projects-api-role

You can find the secret ID at ProjectApi/vault-agent/secret-id.

$ echo $(cat ProjectApi/vault-agent/secret-id)f0119893-03b0-845d-7991-07068fc9ff32

The secret ID can only be used five times before it expires.

Configure Vault Agent to create application settings files

Rather than write code within the example application to authenticate and read secrets from Vault, you can run Vault Agent as a separate process that watches for changes to the secrets and creates a new ProjectApi/appsettings.json file.

Open the Vault Agent configuration file, ProjectApi/vault-agent/config-vault-agent-template.hcl in your preferred text editor to review. It automatically authenticates to Vault using the auto_auth configuration and a template stanza to read from a template for appsettings.json and write out the values.

pid_file = "/vault/ProjectApi/vault-agent/pidfile"vault {  address = "http://vault:8200"}auto_auth {  method {    type = "approle"    config = {      role_id_file_path                   = "/vault/ProjectApi/vault-agent/role-id"      secret_id_file_path                 = "/vault/ProjectApi/vault-agent/secret-id"      remove_secret_id_file_after_reading = false    }  }  sink {    type = "file"    config = {      path = "/vault/ProjectApi/vault-agent/token"    }  }}template {  source      = "/vault/ProjectApi/vault-agent/appsettings.ctmpl"  destination = "/vault/ProjectApi/appsettings.json"}

When Vault Agent starts, it will authenticate to Vault, and write the Vault token to its sink location which is /ProjectApi/vault-agent/token. It generates an appsettings.json file based on ProjectApi/vault-agent/appsettings.ctmpl.

Open the ProjectApi/vault-agent/appsettings.ctmpl in a text editor to review. This template file reads database username and password from Vault, and generate the appsettings.json file for the ASP.NET Core application. It contains most of the default configuration for an ASP.NET Core application but creates a ConnectionString based on the database username and password from Vault.

{  "Logging": {    "LogLevel": {      "Default": "Information",      "Microsoft": "Warning",      "Microsoft.Hosting.Lifetime": "Information"    }  },  "AllowedHosts": "*",  "ConnectionStrings": {    {{- with secret "projects-api/database/creds/projects-api-role" }}    "Database": "Server=.;Database=HashiCorp;user id={{ .Data.username }};password={{ .Data.password }}"    {{- end }}  }}

Run the script vault_agent_template.sh to start Vault Agent as a background process.

$ bash vault_agent_template.shSuccess! Data written to: projects-api/database/roles/projects-api-roleCreating dotnet-vault_vault-agent_1 ... done

Open the resulting ProjectApi/appsettings.json file. The database connection string contains the username and password generated by Vault.

$ cat ProjectApi/appsettings.json{  "Logging": {    "LogLevel": {      "Default": "Information",      "Microsoft": "Warning",      "Microsoft.Hosting.Lifetime": "Information"    }  },  "AllowedHosts": "*",  "ConnectionStrings": {    "Database": "Server=.;Database=HashiCorp;user id=v-approle-projects-api-role-0Bm50K1YI3gmEauKaLGZ-1605720749;password=s-HGuusa77OTw41gSqg9"  }}

The operations team configured Vault to rotate the database credentials after two minutes. If you wait for a few minutes and check the ProjectApi/appsettings.json again, the username and password will not be the same!

$ cat ProjectApi/appsettings.json{  "Logging": {    "LogLevel": {      "Default": "Information",      "Microsoft": "Warning",      "Microsoft.Hosting.Lifetime": "Information"    }  },  "AllowedHosts": "*",  "ConnectionStrings": {    "Database": "Server=.;Database=HashiCorp;user id=v-approle-projects-api-role-idtYLeweQw1g3EReoFk3-1605721008;password=fBN0-d2Bk7BoPJibBMvo"  }}

Vault Agent will update ProjectApi/appsettings.json each time Vault rotates the database username and password. The example application uses ASP.NET Core, which allows a live reload of the application each time ProjectApi/appsettings.json changes. This makes your database credentials to be short-lived and significantly reduces its vulnerability.

Open the ProjectApi/Program.cs file in a text editor to view how the configuration reloads on changes. The CreateHostBuilder method configures an application configuration by adding the appsettings.json file. You must configure the file to reloadOnChange: true (at line 2) in order for the application to reload based on changes to the file.

public static IHostBuilder CreateHostBuilder(string[] args) =>    Host.CreateDefaultBuilder(args)        .ConfigureAppConfiguration((hostingContext, config) =>        {          config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);          // TRUNCATED        .ConfigureWebHostDefaults(webBuilder =>        {          webBuilder.UseStartup<Startup>();        });

In addition to reloading the application upon a file change, you need to reload the database connection string in the application. In ProjectApi/Models/ProjectContext.cs, override the OnConfiguring method of the DbContext.

using Microsoft.EntityFrameworkCore;namespace ProjectApi.Models{  public class ProjectContext : DbContext  {    public ProjectContext(DbContextOptions<ProjectContext> options) : base(options)    {    }    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)    {      optionsBuilder.UseSqlServer(Startup.Configuration.GetSection("ConnectionStrings")["Database"]);    }    public DbSet<Project> Projects { get; set; }  }}

After you added code to handle a live reload of your application, run the run_app.sh script. This will restore the .NET packages, retrieve the Vault secret ID and set it as environment variable, and run the application.

$ bash run_app.sh  Determining projects to restore...  All projects are up-to-date for restore.Building...info: Microsoft.Hosting.Lifetime[0]      Now listening on: https://localhost:5001info: Microsoft.Hosting.Lifetime[0]      Now listening on: http://localhost:5000info: Microsoft.Hosting.Lifetime[0]      Application started. Press Ctrl+C to shut down.info: Microsoft.Hosting.Lifetime[0]      Hosting environment: Developmentinfo: Microsoft.Hosting.Lifetime[0]      Content root path: /Users/rosemarywang/hashicorp/vault-guides/secrets/dotnet-vault/ProjectApi

Open your browser to https://localhost:5001/api/Projects. It will return a JSON list of HashiCorp projects, the year of their first commit, and GitHub Links.

If you wait 5-10 minutes and refresh the browser with the API call, you will still be able to access the Project API. Vault Agent continues to write the new database username and password into ProjectApi/appsettings.json and the demo application retrieves the new database connection string. By using Vault Agent to create the application settings file and adding reload capabilities within application code, you maintain the availability of your application while reducing the lifetime of your database username and password.

Enter CTRL-C to exit the running application.

...TRUNCATED...^Cinfo: Microsoft.Hosting.Lifetime[0]      Application is shutting down...

Clean up the Vault Agent you deployed.

$ bash cleanup_vault_agent.sh

Add Consul template to reload applications on changes

The demo application uses ASP.NET Core, which does support live reload capability. However, some .NET applications do not support application reload. You can add Consul template to trigger an application reload when Vault updates database usernames and passwords.

Open the ProjectApi/Program.cs file in a text editor, and comment out the line for config.AddJsonFile (at line 22).

public static IHostBuilder CreateHostBuilder(string[] args) =>        Host.CreateDefaultBuilder(args)            .ConfigureAppConfiguration((hostingContext, config) =>            {              // config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);

In ProjectApi/Models/ProjectContext.cs, comment out the method override OnConfiguring (line 11 through 14).

public class ProjectContext : DbContext{public ProjectContext(DbContextOptions<ProjectContext> options) : base(options){}// protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)// {//   optionsBuilder.UseSqlServer(Startup.Configuration.GetSection("ConnectionStrings")["Database"]);// }public DbSet<Project> Projects { get; set; }}

When you use a combination of Vault Agent and Consul template, you do not need application code to handle reloading.

Open the Vault Agent configuration file, ProjectApi/vault-agent/config-vault-agent-token.hcl to review. The difference between the config-vault-agent-token.hcl and config-vault-agent-template.hcl is that the config-vault-agent-token.hcl file does not define the template block.

pid_file = "/vault/ProjectApi/vault-agent/pidfile"vault {  address = "http://vault:8200"}auto_auth {  method {    type = "approle"    config = {      role_id_file_path                   = "/vault/ProjectApi/vault-agent/role-id"      secret_id_file_path                 = "/vault/ProjectApi/vault-agent/secret-id"      remove_secret_id_file_after_reading = false    }  }  sink {    type = "file"    config = {      path = "/vault/ProjectApi/vault-agent/token"    }  }}

Open the ProjectApi/vault-agent/config-consul-template.hcl file to review. This file configures the Consul template. The configuration will use the token generated by Vault Agent and generates the ProjectApi/appsettings.json file based on the template. Consul template's configuration includes the exec stanza, which executes a dotnet run as a child process each time the template changes. When the database username and password changes, Consul template will issue a SIGTERM to the application and wait for it to exit in 15 seconds before restarting the application.

vault {  address = "http://127.0.0.1:8200"  vault_agent_token_file = "./vault-agent/token"  unwrap_token = false  renew_token = true}exec {  command = "dotnet run"  splay = "10s"  kill_signal = "SIGTERM"  kill_timeout = "15s"}template {  source      = "./vault-agent/appsettings.ctmpl"  destination = "./appsettings.json"}

In terminal, execute the vault_agent_token.sh script to start Vault Agent, write a token to a file, and start Consul template. The Project API will start.

$ bash vault_agent_token.shCreating dotnet-vault_vault-agent_1 ... doneBuilding...info: Microsoft.Hosting.Lifetime[0]      Now listening on: https://localhost:5001info: Microsoft.Hosting.Lifetime[0]      Now listening on: http://localhost:5000info: Microsoft.Hosting.Lifetime[0]      Application started. Press Ctrl+C to shut down.info: Microsoft.Hosting.Lifetime[0]      Hosting environment: Developmentinfo: Microsoft.Hosting.Lifetime[0]      Content root path: /Users/rosemarywang/hashicorp/vault-guides/secrets/dotnet-vault/ProjectApi

You can access the API in your browser at https://localhost:5001/api/Projects.

As the vault_agent_token.sh script execute, the Consul template logs record the rotation of the database credentials and restarts the application for the changes to the credentials.

$ bash vault_agent_token.sh...TRUNCATED...2020/11/18 20:47:16.332227 [WARN] vault.read(projects-api/database/creds/projects-api-role): TTL of "2m" exceeded the effective max_ttl of "36s"; TTL value is capped accordingly2020/11/18 20:47:16.332247 [WARN] vault.read(projects-api/database/creds/projects-api-role): renewer done (maybe the lease expired)info: Microsoft.Hosting.Lifetime[0]      Application is shutting down...Building...info: Microsoft.Hosting.Lifetime[0]      Now listening on: https://localhost:5001info: Microsoft.Hosting.Lifetime[0]      Now listening on: http://localhost:5000info: Microsoft.Hosting.Lifetime[0]      Application started. Press Ctrl+C to shut down.info: Microsoft.Hosting.Lifetime[0]      Hosting environment: Developmentinfo: Microsoft.Hosting.Lifetime[0]      Content root path: /Users/rosemarywang/hashicorp/vault-guides/secrets/dotnet-vault/ProjectApi

Summary

In this tutorial, you used Vault Agent and Consul template to reload the application when secrets change values in HashiCorp Vault. This pattern reduces the need to add application code to authenticate to Vault and retrieve its secrets.

Help and Reference

• The Vault Agent with AWS tutorial walks through Vault Agent using AWS auth method
• The Vault Agent with Kubernetes tutorial walks through Vault Agent using Kubernetes auth method
• Vault Agent Caching provides step-by-step instruction of using Vault Agent Caching