Cache secrets with Vault Agent

Nearly all requests to Vault must be accompanied by a valid token which means that Vault clients must first authenticate with Vault and acquire a token. Secure Introduction of Vault Clients tutorial talked about the approaches to solve this secret zero problem.

Vault Agent with AWS and Vault Agent with Kubernetes tutorials walked through the Vault Agent's Auto-Auth.

Challenge and solution

Depending on the location of your Vault clients and its secret access frequency, you may face some scaling or latency challenge. Even with Vault Performance Replication enabled, the pressure on the storage backend increases as the number of token or lease generation requests increase. Vault 1.0 introduced batch tokens as a solution to relieve some pressure on the storage backend. By design, batch tokens do not support the same level of flexibility and features as service tokens. Therefore, if you need a periodic token for example, you would need service tokens.

To increase the availability of tokens and secrets to the clients, Vault Agent introduced the Caching function.

Vault Agent Caching can cache the tokens and leased secrets proxied through the agent which includes the auto-auth token. This allows for easier access to Vault secrets for edge applications, reduces the I/O burden for basic secrets access for Vault clusters, and allows for secure local access to leased secrets for the life of a valid token.

Prerequisites

To complete this section of the tutorial, you will need:

• AWS account and associated credentials that allow for the creation of resources
• Terraform v0.15.2 or later installed

Provision the cloud resources

1. Clone or download the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.

   Clone the repository:

   $ git clone https://github.com/hashicorp/vault-guides.git
   

   Copy

   Or download the repository:

   

   This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.

2. Set your working directory to where the /identity/vault-agent-caching/terraform-aws was downloaded.

   $ cd vault-guides/identity/vault-agent-caching/terraform-aws
   

   Copy

   The directory contains the following files.

   $ tree
   .
   ├── aws.tf
   ├── iam.tf
   ├── instances.tf
   ├── kms.tf
   ├── network.tf
   ├── outputs.tf
   ├── security-groups.tf
   ├── templates
   │   ├── userdata-vault-client.tpl
   │   └── userdata-vault-server.tpl
   ├── terraform.tfvars.example
   ├── variables.tf
   └── versions.tf
   
   1 directory, 12 files
   

   Copy

   Note

   The example Terraform in this repository is created for the demo purpose.

3. Set an AWS_ACCESS_KEY_ID environment variable to hold your AWS access key ID.

   $ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"
   

   Copy

4. Set an AWS_SECRET_ACCESS_KEY environment variable to hold your AWS secret access key.

   $ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"
   

   Copy

5. Create a file named terraform.tfvars and specify the key_name. (You can copy the terraform.tfvars.example file and rename it as terraform.tfvars.)

   Example:

   terraform.tfvars

   # EC2 key pair name to access EC2 instances (should already exist) on the AWS region
   key_name = "vault-test"
   

   Tip

   If you don't have an EC2 key pair, follow the AWS documentation to create one.

6. Perform a terraform init to pull down the necessary provider resources.

   $ terraform init
   

   Copy

7. Run terraform apply to provision AWS resources.

   $ terraform apply
   

   Copy

   When prompted, enter yes to proceed.

   The Terraform output will display the public IP address to SSH into your Vault server and client instances.

   ...
   Apply complete! Resources: 23 added, 0 changed, 0 destroyed.
   
   Outputs:
   
   endpoints =
   Vault Server IP (public):  54.219.129.15
   Vault Server IP (private): 10.0.101.50
   
   For example:
      ssh -i vault-test.pem ubuntu@54.219.129.15
   
   Vault Client IP (public):  54.183.212.51
   Vault Client IP (private): 10.0.101.209
   
   For example:
      ssh -i vault-test.pem ubuntu@54.183.212.51
   

Configure AWS IAM auth method

In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.

1. SSH into the Vault Server instance.

   

   When you are prompted, enter "yes" to continue.

   $ ssh -i <path_to_key> ubuntu@<public_ip_of_server>
   ...
   Are you sure you want to continue connecting (yes/no)? yes
   

   Copy

   NOTE: If you received Permissions 0664 for '<key_name>.pem' are too open error, be sure to set the file permission appropriately.

   $ chmod 600 <key_name>.pem
   

   Copy

2. Run the vault operator init command to initialize the Vault server.

   $ vault operator init
   ...
   Initial Root Token: s.20JnHBY66EKTj9zyR6SjTMNq
   
   Success! Vault is initialized
   ...
   

   Copy

   Note

   The Vault server is configured to auto-unseal with AWS Key Management Service (KMS); therefore, once the server is initialized, it gets automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS tutorial.

3. Copy the Initial Root Token value.

4. Log into Vault using the generated initial root token.

   $ vault login
   Token (will be hidden):
   

   Copy

5. Examine and then execute the /home/ubuntu/aws_auth.sh script.

   $ cat aws_auth.sh
   

   Copy

6. Execute the script.

   $ ./aws_auth.sh
   
   Success! Enabled the kv secrets engine at: kv/
   Success! Uploaded policy: myapp
   Success! Enabled aws auth method at: aws/
   Success! Data written to: auth/aws/config/client
   Success! Data written to: auth/aws/role/app-role
   Success! Enabled userpass auth method at: userpass/
   Success! Data written to: auth/userpass/users/student
   

   Copy

   This script enables key/value v1 secrets engine at kv, create myapp policy, enables aws auth method and create a role named app-role. Also, userpass is enabled and created a user (student) with myapp policy attached.

7. Check the myapp policy.

   $ vault policy read myapp
   
   path "kv/*" {
     capabilities = ["create", "read", "update", "delete"]
   }
   path "aws/creds/*" {
     capabilities = ["read", "update"]
   }
   path "sys/leases/*" {
     capabilities = ["create", "update"]
   }
   path "auth/token/*" {
     capabilities = ["create", "update"]
   }
   

   Copy

8. Examine and then execute the /home/ubuntu/aws_secrets.sh script.

   $ cat aws_secrets.sh
   

   Copy

9. Execute the script.

   $ ./aws_secrets.sh
   Success! Enabled the aws secrets engine at: aws/
   Success! Data written to: aws/config/root
   Success! Data written to: aws/config/lease
   Success! Data written to: aws/roles/readonly
   

   Copy

   This script enables aws secrets engine, configures it and set the generated secret's lease to 1 hour and lease max to 24 hours. It also configures readonly role which is mapped to is mapped to arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess IAM policy.

Start Vault Agent

1. Now, SSH into the Vault Client instance.

   

   When you are prompted, enter "yes" to continue.

   $ ssh -i <path_to_key> ubuntu@<public_ip_of_client>
   ...
   Are you sure you want to continue connecting (yes/no)? yes
   

   Copy

2. Explore the Vault Agent configuration file, /home/ubuntu/vault-agent.hcl

   $ cat vault-agent.hcl
   
   exit_after_auth = false
   pid_file = "./pidfile"
   
   auto_auth {
      method "aws" {
          mount_path = "auth/aws"
          config = {
              type = "iam"
              role = "app-role"
          }
      }
   
      sink "file" {
          config = {
              path = "/home/ubuntu/vault-token-via-agent"
          }
      }
   }
   
   cache { }
   
   api_proxy {
      use_auto_auth_token = true
   }
   
   listener "tcp" {
      address = "127.0.0.1:8200"
      tls_disable = true
   }
   
   vault {
      address = "http://<vault-server-host>:8200"
   }
   

   Copy

   Tip

   Also, read the Server Side Request Forgery (SSRF) Protection section.

   Note

   Notice the api_proxy block sets the use_auto_auth_token parameter to true. Since the auto-auth sink is set to /home/ubuntu/vault-token-via-agent, in absence of client token on requests sent to the API proxy, Vault Agent will use the auto-auth token read from vault-token-via-agent. The vault block defines the Vault server location (e.g. http://10.0.101.209:8200).

3. Execute the following command to start the Vault Agent with log level set to debug.

   $ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug
   
   ==> Vault server started! Log data will stream in below:
   
   ==> Vault agent configuration:
   
              Api Address 1: http://127.0.0.1:8200
                        Cgo: disabled
                  Log Level: debug
                    Version: Vault v1.3.0
                Version Sha: 30f07c76e1ea0551f28f5b8f7537f0de49d27f80
   
   2019-03-06T03:12:47.616Z [INFO]  sink.file: creating file sink
   2019-03-06T03:12:47.616Z [INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent
   2019-03-06T03:12:47.617Z [DEBUG] cache: auto-auth token is allowed to be used; configuring inmem sink
   2019-03-06T03:12:47.617Z [INFO]  cache: starting listener: addr=127.0.0.1:8200
   2019-03-06T03:12:47.618Z [INFO]  auth.handler: starting auth handler
   2019-03-06T03:12:47.618Z [INFO]  auth.handler: authenticating
   2019-03-06T03:12:47.618Z [INFO]  sink.server: starting sink server
   2019-03-06T03:12:47.883Z [INFO]  auth.handler: authentication successful, sending token to sinks
   2019-03-06T03:12:47.883Z [INFO]  auth.handler: starting renewal process
   2019-03-06T03:12:47.883Z [INFO]  sink.file: token written: path=/home/ubuntu/vault-token-via-agent
   2019-03-06T03:12:47.883Z [DEBUG] cache.leasecache: storing auto-auth token into the cache
   2019-03-06T03:12:47.887Z [INFO]  auth.handler: renewed auth token
   

   Copy

4. Open a second SSH terminal into the client machine.

5. Verify the client token stored in /home/ubuntu/vault-token-via-agent.

   $ more vault-token-via-agent
   

   Copy

6. Check the details about the token (e.g. attached policies, TTL, etc.).

   $ VAULT_TOKEN="$(cat vault-token-via-agent)" vault token lookup
   

   Copy

7. Set VAULT_AGENT_ADDR environment variable.

   $ export VAULT_AGENT_ADDR="http://127.0.0.1:8200"
   

   Copy

Request dynamic secrets and tokens via agent

Cache evictions

The eviction of cache will occur when the agent fails to renew leases or tokens. This can happen when the cached lease/token hits it's maximum TTL or if the renewal results in an error. Agent also does some best-effort cache evictions by observing specific request types and response codes.

While agent observes requests and evicts cached entries automatically, you can trigger a cache eviction by invoking the /agent/v1/cache-clear endpoint.

1. If you need to manually evict a stale lease, invoke the /agent/v1/cache-clear endpoint with lease ID for which you wish to evict from the cache.

   Example:

   $ curl --request POST \
          --data '{ "type": "lease", "value": "aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc" }' \
          $VAULT_AGENT_ADDR/agent/v1/cache-clear
   

   Copy

   The agent log should show the following:

   [DEBUG] cache.leasecache: received cache-clear request: type=lease namespace= value=aws/creds/readonly/8DLXo8nsgcIpx7iQPpCvWtlc
   [DEBUG] cache.leasecache: canceling context of index attached to accessor
   [DEBUG] cache.leasecache: successfully cleared matching cache entries
   [DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/aws/creds/readonly
   [DEBUG] cache.leasecache: evicting index from cache: id=dd0f9e775... path=/v1/aws/creds/readonly method=GET
   

2. Let's see what happens when you revoke a token.

   $ vault token revoke s.AkdB7Rgr0t8mIexWEGHPsNfn
   

   Copy

   Or, revoke a token via API.

   $ curl --request POST \
          --data '{"token": "s.AkdB7Rgr0t8mIexWEGHPsNfn"}' \
          $VAULT_AGENT_ADDR/v1/auth/token/revoke
   

   Copy

   Examine the agent log:

   ...
   [INFO]  cache: received request: path=/v1/auth/token/revoke method=POST
   [DEBUG] cache: using auto auth token: path=/v1/auth/token/revoke method=POST
   [DEBUG] cache.leasecache: forwarding request: path=/v1/auth/token/revoke method=POST
   [INFO]  cache.apiproxy: forwarding request: path=/v1/auth/token/revoke method=POST
   [DEBUG] cache.leasecache: cancelling context of index attached to token
   [DEBUG] cache.leasecache: successfully cleared matching cache entries
   [DEBUG] cache.leasecache: triggered caching eviction from revocation request
   [DEBUG] cache.leasecache: context cancelled; stopping renewer: path=/v1/auth/token/create
   [DEBUG] cache.leasecache: evicting index from cache: id=b5715bdca771174... path=/v1/auth/token/create method=POST
   

   Note

   In the agent log, notice the message, "triggered caching eviction from revocation request". When a token revocation request is made via the agent, the agent evicts the cached entries associated with the revoked token.

   Similarly, if you revoked a staled AWS lease, the agent will automatically evict the cache.

3. If a situation requires you to clear all cached tokens and leases (e.g. reset after a number of testing), set the type to all.

   $ curl --request POST --data '{ "type": "all" }' \
          $VAULT_AGENT_ADDR/agent/v1/cache-clear
   

   Copy

Clean up

1. On the server SSH terminal, execute the following command to revoke all leases.

   $ vault lease revoke -prefix aws/creds/readonly
   

   Copy

2. Execute the following commands to destroy cloud resources.

   $ terraform apply -destroy -auto-approve
   

   Copy

3. Delete state files.

   $ rm -rf .terraform terraform.tfstate*
   

   Copy

Server side request forgery (SSRF) protection

Add require_request_header = true in the Agent's listener configuration stanza to enable the SSRF protection.

Example:

exit_after_auth = false
pid_file = "./pidfile"

auto_auth {
   method "aws" {
       mount_path = "auth/aws"
       config = {
           type = "iam"
           role = "app-role"
       }
   }

   sink "file" {
       config = {
           path = "/home/ubuntu/vault-token-via-agent"
       }
   }
}

cache { }

api_proxy {
   use_auto_auth_token = true
}

listener "tcp" {
   address = "127.0.0.1:8200"
   require_request_header = true
}

vault {
   address = "http://<vault-server-host>:8200"
}

Notice the listener block.

When the require_request_header is set to true, the Agent listener will reject all requests that do NOT have the proper X-Vault-Request: true header.

For example, the command to request AWS credentials you saw in the request dynamic secrets and tokens via agent step must look like:

$ curl -header "X-Vault-Request: true" \
       $VAULT_AGENT_ADDR/v1/aws/creds/readonly

With absence of the X-Vault-Request header, the Agent will throw missing 'X-Vault-Request' header error, and the request will not be propagated to the Vault server.

Help and reference

• Blog post: Why Use the Vault Agent for Secrets Management?
• Video: Streamline Secrets Management with Vault Agent and Vault 0.11
• Secure Introduction of Vault Clients
• Vault Agent Auto-Auth
• Vault Agent Caching
• AWS Auth Method

Other Vault Agent tutorials:

• Vault Agent with AWS
• Vault Agent with Kubernetes
• Vault Agent Templates
• Vault Agent Windows Service
• Using HashiCorp Vault Agent with .NET Core