HashiDays One conference. Three cities. Find a city near you
Vault
Auto unseal

Auto-unseal Vault using Azure Key Vault

  • 9min
  • |
  • Terraform
  • Vault

When a Vault server is started, it starts in a sealed state and it does not know how to decrypt data. Before any operation can be performed on the Vault, it must be unsealed. Unsealing is the process of constructing the master key necessary to decrypt the data encryption key.

Unseal with Shamir's Secret Sharing

Challenge

Vault unseal operation requires a quorum of existing unseal keys split by Shamir's Secret sharing algorithm. This is done so that the "keys to the kingdom" won't fall into one person's hand. However, this process is manual and can become painful when you have many Vault clusters as there are now many different key holders with many different keys.

Solution

Vault supports opt-in automatic unsealing via cloud technologies: AliCloud KMS, AWS KMS, Azure Key Vault, Google Cloud KMS, and OCI KMS. This feature enables operators to delegate the unsealing process to trusted cloud providers to ease operations in the event of partial failure and to aid in the creation of new or ephemeral clusters.

Unseal

This tutorial demonstrates an example of using Azure Key Vault to auto-unseal a Vault server.

Prerequisites

This tutorial assumes the following:

  • Terraform installed and configured
  • You have a Microsoft Azure account
    • Tenant ID
    • Client ID
    • Client Secret
    • Subscription ID

Azure permissions

This tutorial uses Terraform to provision a Vault environment to demonstrate the auto-unseal. The following Azure resources will be created:

  • Azure Key Vault
  • Virtual network with CIDR of 10.0.0.0/16
    • Subnet with CIDR, 10.0.1.0/24
  • Public IP
  • Security group
  • A virtual machine to host a Vault server and accessible via public IP

Therefore, the service principal you use to perform the steps in this tutorial must have enough permission to provision all these resources.

Production

Create a dedicated service principal for Vault to perform auto-unseal. The Vault service principal only requires the Azure built-in Key Vault Secrets User and Key Vault Crypto User roles.

Do not use the Terraform service principal used in this tutorial as the Vault service principal in production. The Terraform service principal has been given permission to deploy and configure the additional Azure resources listed above.

Create an Azure service principal for Terraform

To run the provided Terraform configuration, you need an Azure service principal with appropriate permissions to deploy and configure Vault. The following demonstrates the creation of a service principal.

Note

Refer to Terraform Azure provider documentation to create a service principal via the Azure CLI.

  1. Launch the Microsoft Azure Portal and sign in.

  2. Select Azure Active Directory and select Properties. Azure Active Directory

  3. Copy the Tenant ID. Tenant ID

  4. In a terminal, set the variable TENANT_ID to the Tenant ID.

    $ TENANT_ID=<Tenant ID>

  5. From the side navigation, select App registrations. App registrations

  6. Select New registrations. New App registration

  7. Enter education in the Name field.

  8. Click Register.

  9. Copy the Application (client) ID. Client ID

  10. In a terminal, set the variable CLIENT_ID to the Application (client) ID.

    $ CLIENT_ID=<Client ID>

  11. From the side navigation, select Certificate & secrets.

  12. Under the Client secrets, click New client secret.

  13. Enter a description in the Description field. Azure portal

  14. Click Add.

  15. Copy the client secret value. Azure directory application secret value

  16. In a terminal, set the variable CLIENT_SECRET to the client secret value.

    $ CLIENT_SECRET=<Client secret>

  17. From the side navigation, click API permissions. Azure directory application API permissions

  18. Under Configured permissions, click Add a permission. Azure directory application add a permission

  19. Under Supported legacy APIs, click Azure Active Directory Graph. Azure directory app perms legacy

  20. Click Delegated permissions. Azure Active Directory Graph

  21. Expand User, select the check-box for User.Read.

  22. Click Application permissions. Azure directory app application permissions

  23. Expand Application, select the check-box for Application.ReadWrite.All.

  24. Expand Directory, select the check-box for Directory.ReadWrite.All.

  25. Click API permissions.

  26. Click Grant admin consent for azure to grant the permissions. Azure directory app permissions grant admin consent

  27. Navigate to the Subscriptions blade.

  28. Copy the Subscription ID. Subscription ID

  29. In a terminal, set the variable SUBSCRIPTION_ID to the Subscription ID.

    $ SUBSCRIPTION_ID=<Subscription ID>

  30. Click the name of the subscription.

  31. From the side navigation, click Access control (IAM). Azure subscription access control

  32. Click Add to expand the menu.

  33. Click Add a role assignment. Azure subscription access add role assignment

  34. Choose Owner from the Role select field.

  35. Choose User, group, or service principal from the Assign Access To select field.

  36. Enter the application name or application id in the Select field.

  37. Click the application when it is displayed. Azure subscription access role assignment to app

  38. Click Save.

    The application is created with the correct permissions and you have these identifiers and credentials:

    • Tenant ID
    • Client ID
    • Client Secret
    • Subscription ID

  39. Display the created credentials.

    $ echo -e \  "tenant_id=$TENANT_ID  subscription_id=$SUBSCRIPTION_ID  client_id=$CLIENT_ID  client_secret=$CLIENT_SECRET"

Download demo assets

Retrieve the additional configuration by cloning the hashicorp-education/learn-vault-autounseal-azure repository from GitHub.

$ git clone https://github.com/hashicorp-education/learn-vault-autounseal-azure

This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found within a sub-directory.

Go into the learn-vault-autounseal-azure directory.

$ cd learn-vault-autounseal-azure

Working directory

This tutorial assumes that the remainder of commands are executed within this directory.

Step 1: Provision the cloud resources

The terraform files reads the credentials from the terraform.tfvars and generates the resources defined in the main.tf.

Modify terraform.tfvars.example with a public SSH key, public_key, and Azure credentials: tenant_id; client_id; client_secret and subscription_id.

terraform.tfvars.example

tenant_id="0000000-0000-0000-0000000000"public_key = "ssh-rsa AAAA..."client_id="0000000-000000-0000000000"client_secret="AABBBCCCDDDDEEEFFF"subscription_id="0000000-0000-0000-0000-0000000000"

Save the file and create a copy of the file named terraform.tfvars.

$ cp terraform.tfvars.example terraform.tfvars

The main.tf file generates a new resource group with:

  • Virtual machine with Vault already installed
  • Azure Vault Key (Test-vault-xxxx)
  • A key (generated-key)

Initialize the Azure provider plugins.

$ terraform init

Create an execution plan.

$ terraform plan...Plan: 12 to add, 0 to change, 0 to destroy.

Apply the changes.

$ terraform apply -auto-approve...Outputs:ip = 13.82.62.56key_vault_name = Test-vault-1e5a88dessh-addr =    Connect to your virtual machine via SSH:    $ ssh azureuser@13.82.62.56

Step 2: Test the auto-unseal feature

Create a variable named AZURE_VAULT_IP to store the IP address of the virtual machine.

$ AZURE_VAULT_IP=$(terraform output -raw ip)

SSH into the virtual machine with the azureuser user.

$ ssh azureuser@$AZURE_VAULT_IP

Within this SSH session, check the status of the Vault server.

$ vault statusKey                      Value---                      -----Recovery Seal Type       azurekeyvaultInitialized              falseSealed                   trueTotal Recovery Shares    0Threshold                0Unseal Progress          0/0Unseal Nonce             n/aVersion                  n/aHA Enabled               true

The output displays that the Vault server is not initialized (Initialized is false).

Initialize the Vault server.

$ vault operator init

Check the status of the Vault server.

$ vault statusKey                      Value---                      -----Recovery Seal Type       shamirInitialized              trueSealed                   falseTotal Recovery Shares    5Threshold                3Version                  1.3.0Cluster Name             vault-cluster-092ba5deCluster ID               8b173565-7d74-fe5b-a199-a2b56b7019eeHA Enabled               false

The output displays that the Vault server is unsealed (Sealed is false).

Vault started as a service. The service writes its output to a log.

Display the Vault server log.

$ sudo journalctl --no-pager -u vault...==> Vault server configuration:        Azure Environment: AzurePublicCloud           Azure Key Name: generated-key         Azure Vault Name: Test-vault-a414d041                Seal Type: azurekeyvault                      Cgo: disabled               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")                Log Level: (not set)                    Mlock: supported: true, enabled: false                  Storage: file                  Version: Vault v1.3.0              Version Sha: 37a1dc9c477c1c68c022d2084550f25bf20cac33 ==> Vault server started! Log data will stream in below: [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set [INFO]  core: stored unseal keys supported, attempting fetch [INFO]  core: vault is unsealed...

The logs displays that the Azure Vault key is being fetched to unseal the Vault server. The Vault server automatically unseals every time Vault is started.

Restart the Vault server.

$ sudo systemctl restart vault

Check the status of the Vault server.

$ vault status

The output displays that the Vault server is automatically unsealed.

Display the Vault service configuration file.

$ cat /usr/lib/systemd/system/vault.service[Unit]Description=Vault AgentRequires=network-online.targetAfter=network-online.target[Service]Restart=on-failurePermissionsStartOnly=trueExecStartPre=/sbin/setcap 'cap_ipc_lock=+ep' /usr/local/bin/vaultExecStart=/usr/local/bin/vault server -config /etc/vault.d/config.hclExecReload=/bin/kill -HUPKillSignal=SIGTERMUser=azureuserGroup=azureuser[Install]WantedBy=multi-user.target

Review the Vault server configuration file.

$ cat /etc/vault.d/vault.hclstorage "file" {  path = "/opt/vault"}listener "tcp" {  address     = "0.0.0.0:8200"  tls_disable = "true"}seal "azurekeyvault" {  client_id      = "YOUR-APP-ID"  client_secret  = "YOUR-APP-PASSWORD"  tenant_id      = "YOUR-AZURE-TENANT-ID"  vault_name     = "Test-vault-XXXXXX"  key_name       = "generated-key"}ui=truedisable_mlock = true

Notice the Vault configuration file defines the azurekeyvault stanza with all parameter values properly populated: client ID, client secret, tenant ID, vault name (generated by Terraform), and Azure Key Vault key name.

Permissions

The client you define in the azurekeyvault stanza must have the Key Vault Secrets User and Key Vault Crypto User roles.

Warning

Although the listener stanza disables TLS (tls_disable = "true") for this tutorial, Vault should always be used with TLS in production to provide secure communication between clients and the Vault server. It requires a certificate file and key file on each Vault host.

Key rotation

With auto-unseal enabled, set up Azure Key Vault with key rotation using the Azure Automation Account and Vault will recognize newly rotated keys since the key metadata is stored with the encrypted data to ensure the correct key is used during decryption operations.

Step 3: Clean up

When you are done experimenting this demo follow these steps to clean up.

Destroy the resources that created by Terraform.

$ terraform destroy -auto-approve

Remove the Terraform state files.

$ rm -rf .terraform terraform.tfstate*

Help and reference

Collection Overview

Auto unseal

 Next

Auto-unseal using GCP Cloud KMS