Retrieve secrets with Vault AWS Lambda extension

12min
|
Vault
Terraform

Challenge

AWS Lambda is a powerful tool available from Amazon Web Services. It offers Serverless functions that cost nothing when not used, and then fractions of a penny when invoked.

A common requirement for a Lambda function is to retrieve information from a secure location, and use it to access a database, file store or other secure location.

If you are using Vault for secrets management, Lambda requires access to Vault for authentication and to retrieve secrets.

Solution

Including the Vault Lambda Extension in the execution environment enables functions to retrieve secrets from Vault.

With the Vault Lambda Extension with your Lambda runtime, and setting the proper environment variables, the function will be able to retrieve secrets from Vault before the function executes. The function now has the credentials needed to implement your business logic.

Background

Lambda creates instances of serverless functions as needed, which the structure includes a few things like a runtime, environment variables and the executable/interpreter. Lambda has standard runtimes for many languages available, but if that runtime does not include a particular library or dependency it can be added a few ways. One of particular interest is the Lambda layer, which is an addition of a library, module or dependency to the standard Lambda runtime for your language. When a Lambda function is invoked, these are all loaded into the runtime process and then executed.

Lambda

Extensions are additions to the runtime process and can added as a Lambda layer to your runtime environment. If the proper environment variables are set, the Vault AWS Lambda Extension will run authenticate and securely retrieve secrets before your Lambda function invokes.

Scenario introduction

Lambda

First, using a Terraform config you will create the function with supporting IAM Roles and Policies, and review the infrastructure that is created.

You will then set up HCP Vault Dedicated to allow access from your AWS Account to Vault Dedicated via an IAM Role. After the initial Vault Dedicated setup, you will create a Vault Role and Policy that allow the execution role the Lambda function uses to access one KV Secret (kv/data/test/lambda) in your instance of Vault Dedicated.

Next, you will create a secret in Vault, and then set up AWS Authentication to enable the function you created earlier to connect to your Vault Dedicated instance.

Finally, you will observe it all working together.

Prerequisites

This tutorial requires an AWS account, Terraform, Vault CLI and the example Terraform configuration to create a demonstration environment.

AWS account - you need to be familiar with the AWS Console UI and where to find the Lambda. For this tutorial, you should create an AWS IAM User that has sufficient permissions to create resources in your account, and generate an Access Key and Secret Access Key for this lab. Please refer to IAM Users for more details.
Terraform CLI
AWS CLI
Git
HCP account with a service principal with contributor role already created.
- Some experience with the HCP interface, particularly getting the values for VAULT_TOKEN and VAULT_ADDR.

Clone example repository

Clone the Terraform configuration and Lambda function code from the following repository:

$ git clone https://github.com/hashicorp-education/learn-vault-intro-lambda-extension.git

Change into the repository directory.
```
$ cd learn-vault-intro-lambda-extension
```
The repository contains Terraform configuration to create the following resources:
- Vault Dedicated cluster
- Lambda Function
- IAM Lambda execution role

Set up for the Terraform configuration

Log into the HCP Portal.

In HCP, under Access control (IAM) choose Service Principals.
From the Service Principals page, choose the link that your service principle's name uses and go into the details page.
in the Create service principal key page, choose + Generate key.
Copy the Client ID then, in a terminal, set the HCP_CLIENT_ID environment variable to the copied value.
```
$ export HCP_CLIENT_ID=<COPIED_CLIENT_ID>
```
Switch back to the HCP Portal and copy the Client Secret then, in a terminal, set the HCP_CLIENT_SECRET environment variable to the copied value.
```
$ export HCP_CLIENT_SECRET=<COPIED_CLIENT_SECRET>
```
Terraform is now able to authenticate with HCP.

Set AWS_ACCESS_KEY_ID to store your AWS Access Key.

$ export AWS_ACCESS_KEY_ID=<<AWS Access Key here>>

Set AWS_SECRET_ACCESS_KEY to store your AWS Secret Access Key.

$ export AWS_SECRET_ACCESS_KEY=<<AWS Secret Access Key here>>

Now set the target AWS region.
```
$ export TF_VAR_aws_region=us-west-2
```

Create the infrastructure

Run terraform init to initialize the Terraform configuration.

$ terraform init
 ...snip...
 Terraform has been successfully initialized!
 ...snip...

Package the Lambda function through the build script.

$ ./build.sh
 $PWD/learn-vault-intro-lambda-extension/demo-function $PWD/learn-vault-intro-lambda-extension
 Removing old builds...

 Making new zip...
    adding: handler.py (deflated 31%)
 $PWD/learn-vault-intro-lambda-extension

Apply the changes.
```
$ terraform apply
...
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes
```
The terminal output displays the plan that it found and the resources it creates.
Enter yes to confirm and resume.
Note
Keep in mind that answering yes at this time creates actual resources with associated costs.
Verify that when the terraform apply command completes, smd you see the following.
```
 ...snip...
 Apply complete! Resources: 8 added, 0 changed, 0 destroyed.
```

Tour your creation

Take a quick tour to learn what was provisioned by Terraform in the AWS and HCP UI.

Launch your AWS Account Console and open the Lambda console. Select the vault-lambda-extension-demo-function function.

Scroll down to the Code Source tab, and examine the Python code.

import json
import os

def lambda_handler(event, context):
   # the location of secret
   secret_file = os.environ['VAULT_SECRET_FILE_DB']
   f = open(secret_file, "r")

   # put the json object into a python dictionary
   dct = json.loads(f.read())

   # print off your API Key
   print("API Key: ", dct["data"]["data"]["api-key"])

This code is straightforward, getting the location of the secret that is returned from Vault from an environment variable, reading it into a Python map and then printing it to standard output. Understanding the environment variables are critical to understanding the Vault lambda extension.

Click on the Configuration tab and click on the environment variables section on the left.

Environment Variables

The environment variables should be similar to as follows.

Variable Name	Value
VAULT_ADDR	https://XXX-XXXX-XXXX-NNNNNNNN.NNNNNNNN.NN.hashicorp.cloud:8200
VAULT_AUTH_PROVIDER	aws
VAULT_AUTH_ROLE	vault-role-for-aws-lambdarole
VAULT_NAMESPACE	admin
VAULT_SECRET_FILE_DB	/tmp/vault_secret.json
VAULT_SECRET_PATH_DB	kv/data/test/ec2

The Vault Lambda Extension is configured via these environment variables. So VAULT_AUTH_PROVIDER indicates it should use AWS Authentication, VAULT_NAMESPACE indicates the namespace that will be used, and so forth. Please see the previous link for details.

In order to use the Vault CLI locally, you need to go to the HCP UI, find your cluster to get the Vault token, and address.
Choose your Vault instance and click on Access Vault, and choose Command-line (CLI).

From there some variables need to set in your terminal.

Set CLI variables

$ export VAULT_ADDR="[YOUR_CLUSTER_ADDER_HERE]"; export VAULT_NAMESPACE="admin"

You still need to get the token. Go back to the main screen for your Vault cluster and find for the New admin token and choose Generate token. Generating it places it in your paste buffer. Copy the below and then set the admin token with:
```
$ export VAULT_TOKEN=[YOUR_TOKEN_HERE]
```
Note
This token expires in 6 hours.
For more details. please refer to the configure development hosts, paying attention to the HCP sections for details on setting environment variables.

Create a secret in Vault

Create a standard KV secret through CLI.

Check Vault status to make sure the CLI is properly set up.

$ vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.12.0+ent
Build Date               2022-10-10T19:00:46Z
Storage Type             raft
Cluster Name             vault-cluster-2127cf0f
Cluster ID               28bcf54f-94ec-2eed-1e54-2d67b58bbaaf
HA Enabled               true
HA Cluster               https://XXX.XX.XX.XXX:8201
HA Mode                  active
Active Since             2022-10-26T16:57:12.105415484Z
Raft Committed Index     413
Raft Applied Index       413
Last WAL                 173

Enable KV Secrets Engine v2.

$ vault secrets enable -version=2 kv
Success! Enabled the kv secrets engine at: kv/

Create a secret named api-key.

$ vault kv put kv/test/lambda api-key="ABCDEFG9876"
=== Secret Path ===
kv/data/test/lambda

======= Metadata =======
Key                Value
---                -----
created_time       2022-10-27T16:30:34.405574019Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Enable AWS auth method in Vault

In order for the Lambda function to be able to contact the Vault Dedicated instance you need AWS authentication set up.

List the authentication is already enabled.

$ vault auth list
Path      Type        Accessor                  Description
----      ----        --------                  -----------
token/    ns_token    auth_ns_token_dd101c2c    token based credentials

Now create a policy called vault-policy-for-aws-lambda-role, which allows read access the secret created above.

$ vault policy write vault-policy-for-aws-lambda-role - << EOF
# Grant 'read' permission to paths prefixed by 'kv/data/test/lambda'
path "kv/data/test/lambda" {
    capabilities = [ "read" ]
  }
EOF

List the available policies.

$ vault policy list
default
hcp-root
vault-policy-for-aws-lambda-role

For your lambda function to be able to connect to your instance of Vault Dedicated, go ahead and enable AWS Authentication.
```
$ vault auth enable aws
Success! Enabled aws auth method at: aws/
```

List the available auth methods.

$ vault auth list
Path      Type        Accessor                  Description
----      ----        --------                  -----------
aws/      aws         auth_aws_b3ac2184         n/a
token/    ns_token    auth_ns_token_dd101c2c    token based credentials

Configure AWS Authentication with the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY already set up.

$ vault write auth/aws/config/client secret_key=$AWS_SECRET_ACCESS_KEY access_key=$AWS_ACCESS_KEY_ID
Success! Data written to: auth/aws/config/client

Create an AWS auth method role in Vault that gives access to the IAM role vault-lambda-extension-demo-lambda-role and attach the policy vault-policy-for-aws-lambda-role to it.

Note

Update the <YOUR_ACCOUNT_ID> with your AWS Account ID in the following command before running it.

$ vault write auth/aws/role/vault-role-for-aws-lambdarole auth_type=iam bound_iam_principal_arn=arn:aws:iam::<YOUR_ACCOUNT_ID>:role/vault-lambda-extension-demo-lambda-role policies=vault-policy-for-aws-lambda-role
Success! Data written to: auth/aws/role/vault-role-for-aws-lambdarole

List the roles.

$ vault list /auth/aws/role
Keys
----
vault-role-for-aws-lambdarole

Examine the details of vault-role-for-aws-lambdarole.

$ vault read auth/aws/role/vault-role-for-aws-lambdarole
Key                               Value
---                               -----
allow_instance_migration          false
auth_type                         iam
bound_account_id                  []
bound_ami_id                      []
bound_ec2_instance_id             <nil>
bound_iam_instance_profile_arn    []
bound_iam_principal_arn           [arn:aws:iam::144830530561:role/vault-lambda-extension-demo-lambda-role]
bound_iam_principal_id            [AROA3SCJYFFCPXMX5QMOP]
bound_iam_role_arn                []
bound_region                      []
bound_subnet_id                   []
bound_vpc_id                      []
disallow_reauthentication         false
inferred_aws_region               n/a
inferred_entity_type              n/a
policies                          [vault-policy-for-aws-lambda-role]
resolve_aws_unique_ids            true
role_id                           b821fe4b-ff5c-22ac-9594-7f95858a64fc
role_tag                          n/a
token_bound_cidrs                 []
token_explicit_max_ttl            0s
token_max_ttl                     0s
token_no_default_policy           false
token_num_uses                    0
token_period                      0s
token_policies                    [vault-policy-for-aws-lambda-role]
token_ttl                         0s
token_type                        default

Test the function

Go back to the AWS Console, find vault-lambda-extension-demo-function and choose the Test tab and press Test button.

Examine the output. In addition to a summary of execution, find for text similar to the below.

2022-10-28T14:45:04.456Z [INFO]  vault-lambda-extension: Initialising
2022-10-28T14:45:04.667Z [INFO]  vault-lambda-extension: Initialised
2022-10-28T14:45:04.667Z [INFO]  vault-lambda-extension: Starting HTTP proxy server
220140c1-4d3c-4656-b9ba-a6df57a1f1d82022-10-28T14:45:04.668Z [INFO]  vault-lambda-extension: Waiting for event...
EXTENSION   Name: vault-lambda-extension    State: Ready    Events: [INVOKE,SHUTDOWN]
START RequestId: 7588b408-1ae6-4866-822b-cb19947d50b5 Version: $LATEST
2022-10-28T14:45:04.774Z [INFO]  vault-lambda-extension: Received event
2022-10-28T14:45:04.774Z [INFO]  vault-lambda-extension: Waiting for event...
API Key:  ABCDEFG9876
END RequestId: 7588b408-1ae6-4866-822b-cb19947d50b5
REPORT RequestId: 7588b408-1ae6-4866-822b-cb19947d50b5  Duration: 1.54 ms   Billed Duration: 2 ms   Memory Size: 128 MB Max Memory Used: 65 MB  Init Duration: 437.45 ms

From the line with API Key: ABCDEFG9876 you know that the extension retrieved the API_KEY value from Vault.

Clean up your infrastructure

Clean up your infrastructure provisioned by Terraform.
```
$ terraform destroy
```
The terraform destroy deletes the HCP Cluster, Lambda function and all the support structures - IAM Role, Policy, etc.
Delete the terraform state.
```
$ rm *tfstate*
```
In the AWS IAM Console, find the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID you created for this tutorial and make them inactive and delete them both.

Unset all the environment variables used in this tutorial.

$ unset AWS_SECRET_ACCESS_KEY; unset AWS_ACCESS_KEY_ID; unset HCP_CLIENT_ID; unset HCP_CLIENT_SECRET

Check the environment variables were unset:
```
$ env | grep 'AWS\|HCP'
```
If the variables were successfully unset they will not appear in the result of this command. An execution and successful unset will provide no output.

Next steps

In this tutorial you built and deployed a Lambda function using the Vault AWS Lambda Extension using Terraform. If you are interested in more detail on using Terraform to manage Vault Dedicated refer to Manage Codified Vault on HCP Vault Dedicated with Terraform That simple function retrieved a simple KV secret from Vault, and printed it to the CloudWatch logs. The extension is configured by the Lambda environment variables, and automatically authenticates and retrieves a secret before the function is invoked.

You can learn more about the Vault AWS Lambda Extension by reading the blog post Use AWS Lambda Extensions to Securely Retrieve Secrets From HashiCorp Vault and by reviewing the Vault AWS Lambda Extension code repository. Of particular interest are the environment variables used to configure the extension.

Next, you enabled AWS Authentication for Vault Dedicated, and you can learn more through the Set up AWS Auth Method for HCP Vault Dedicated tutorial and the documentation AWS Authentication.

If you are interested in a lab with a more complex example, you can try the Lambda Vault Extension tutorial. That tutorial has a more detailed scenario using a secret to then access a MYSQL Database. In addition there is a section on caching the secret retrieved from Vault.

References

Automate GitHub workflows

Build custom Vault plugins