Retrieve secrets from Vault using C# client library with .NET Core

If your .NET application needs some secrets (e.g. database credentials), your organization might offer HashiCorp Vault to store and manage them for you. As a developer, you need a way to retrieve secrets from Vault for your application to use.

You can write your own HashiCorp Vault HTTP client to read secrets from the Vault API or use a community-maintained library.

An client library allows your C# application to retrieve secrets from Vault, depending on how your operations team manages Vault.

This tutorial demonstrates how to use a Vault C# client to retrieve static and dynamic Microsoft SQL Server database credentials from Vault. The ASP.NET Core application uses Vault Sharp, a library which provides lightweight client-side support for connecting to Vault. When database credentials change, you will need to restart the example application.

Prerequisites

• Docker
• Docker Compose
• .NET v5.0.0+
• tree command line utility for visualizing directory structures.

Step 1: Retrieve the demo application

Retrieve the configuration by cloning or downloading the hashicorp-education/learn-vault-secrets-dotnet-vault repository from GitHub.

Clone the repository.

$ git clone https://github.com/hashicorp-education/learn-vault-secrets-dotnet-vault.git

Or download the repository.

Switch your working directory to learn-vault-secrets-dotnet-vault.

$ cd learn-vault-secrets-dotnet-vault

You should find the ProjectApi/ sub-directory.

$ tree --dirsfirst -L 1
.
├── ProjectApi
├── database
├── README.md
├── cleanup.sh
├── cleanup_vault_agent.sh
├── demo_setup.sh
├── docker-compose-vault-agent-template.yml
├── docker-compose-vault-agent-token.yml
├── docker-compose.yml
├── get_db_username.sh
├── list_passwords.sh
├── new_secret.sh
├── projects-role-policy.hcl
├── revoke_passwords.sh
├── run_app.sh
├── vault_agent_template.sh
└── vault_agent_token.sh

2 directories, 15 files

The demo ASP.NET Core application leverages the VaultSharp library to communicate with Vault.

Vault and database setup

Your application, called project-api needs to reference a Vault deployment and Microsoft SQL Server (MSSQL). Create the dependencies by running the setup script, which will configure and populate data for both Vault and MSSQL.

$ bash demo_setup.sh

Creating network "dotnet-vault_vpcbr" with driver "bridge"
Building db
Step 1/6 : FROM microsoft/mssql-server-linux:latest
 ---> 314918ddaedf
Step 2/6 : RUN mkdir -p /usr/src/app
 ---> Using cache
 ---> 3b3601a28c41
Step 3/6 : WORKDIR /usr/src/app
 ---> Using cache
 ---> b844aca08dcb
Step 4/6 : COPY . /usr/src/app
 ---> Using cache
 ---> 661442b758f8
Step 5/6 : RUN chmod +x /usr/src/app/import-data.sh
 ---> Using cache
 ---> e95b64527c1b
Step 6/6 : CMD /bin/bash ./entrypoint.sh
 ---> Using cache
 ---> e4e729f2944f

Successfully built e4e729f2944f
Successfully tagged db:latest
Creating dotnet-vault_vault_1 ... done
Creating dotnet-vault_db_1    ... done
Success! Enabled approle auth method at: approle/
Success! Enabled the database secrets engine at: projects-api/database/
Success! Enabled the kv secrets engine at: projects-api/secrets/
Key              Value
---              -----
created_time     2020-11-17T21:54:42.8659611Z
deletion_time    n/a
destroyed        false
version          1
Success! Data written to: projects-api/database/roles/projects-api-role
Success! Uploaded policy: projects-api
Success! Data written to: auth/approle/role/projects-api-role

The MSSQL contains the HashiCorp database with a table called Projects. It contains information about HashiCorp's projects.

Connect to the MSSQL running in the dotnet-vault_db_1 container as the user, sa with password, Testing!123.

$ docker exec -it dotnet-vault_db_1 /opt/mssql-tools/bin/sqlcmd -S localhost \
    -U sa -P 'Testing!123' -d HashiCorp

Now, select the Projects table.

$ SELECT * FROM Projects

Execute the GO command to execute the select command and view the table entries.

$ GO

Id           YearOfFirstCommit  GitHubLink

Vagrant      2010               https://github.com/hashicorp/vagrant
Packer       2013               https://github.com/hashicorp/packer
Terraform    2014               https://github.com/hashicorp/terraform
Nomad        2015               https://github.com/hashicorp/nomad
Consul       2013               https://github.com/hashicorp/consul
Vault        2015               https://github.com/hashicorp/vault
Waypoint     2020               https://github.com/hashicorp/waypoint
Boundary     2020               https://github.com/hashicorp/boundary

(8 rows affected)

Enter exit to quit the docker exec command.

$ exit

Your operations team has given you a Vault role and secret to log into Vault using the approle auth method.

Set the environment variable for VAULT_ADDR to your Vault development instance.

$ export VAULT_ADDR='http://127.0.0.1:8200'

To authenticate to Vault, use the role projects-api-role and the secret ID stored in ProjectApi/vault-agent/secret-id. Vault will return a token and you store it in the VAULT_TOKEN environment variable.

$ export VAULT_TOKEN=$(vault write --field=token auth/approle/login \
    role_id=projects-api-role \
    secret_id=$(cat ProjectApi/vault-agent/secret-id))

Check the Vault token in the VAULT_TOKEN environment variable. Setting the environment variable allows you to log into Vault manually.

$ echo $VAULT_TOKEN

s.3qG4azKpWNqiyCDG4NSwpsJi

In the example ASP.NET Core application, you will configure the application to use the static database password. Later, you will refactor to use the dynamic database username and password.

Add a configuration provider to access Vault

In ASP.NET Core, you can add a configuration provider to retrieve configuration information outside of appsettings.json. The ASP.NET Core demo application uses a configuration provider for Vault to connect to Vault and retrieve secrets on application startup.

Open the ProjectApi/CustomOptions/VaultConfiguration.cs file in your preferred text editor to view. The VaultConfigurationProvider constructor takes a set of options to connect to Vault using the approle auth method.

public VaultConfigurationProvider(VaultOptions config)
{
  _config = config;

  var vaultClientSettings = new VaultClientSettings(
      _config.Address,
      new AppRoleAuthMethodInfo(_config.Role,
                                _config.Secret)
  );

  _client = new VaultClient(vaultClientSettings);
}

The options in VaultOptions include the Vault address, the role identifer, secret, mount path for the secrets (projects-api/), and secret type (secrets or database).

public class VaultOptions
{
  public string Address { get; set; }
  public string Role { get; set; }

  public string Secret { get; set; }
  public string MountPath { get; set; }
  public string SecretType { get; set; }
}

The Vault configuration provider also overrides the Load method with a method to retrieve the database credentials from Vault based on the SecretType and store it into database:userID and database:password configuration.

public override void Load()
{
  LoadAsync().Wait();
}

public async Task LoadAsync()
{
  await GetDatabaseCredentials();
}

public async Task GetDatabaseCredentials()
{
  var userID = "";
  var password = "";

  // TRUNCATED

  Data.Add("database:userID", userID);
  Data.Add("database:password", password);
}

If you configure this for your own application, you can update the GetDatabaseCredentials with a more generic method to retrieve the secrets you need from Vault. In the demo application, you can retrieve the static database password from projects-api/secrets or dynamic database username and password from projects-api/database.

Next, the VaultExtensions class creates a configuration builder called AddVault that creates the Vault client when you build the application.

public static class VaultExtensions
{
  public static IConfigurationBuilder AddVault(this IConfigurationBuilder configuration,
  Action<VaultOptions> options)
  {
    var vaultOptions = new VaultConfigurationSource(options);
    configuration.Add(vaultOptions);
    return configuration;
  }
}

Open ProjectApi/Program.cs. The demo application references the AddVault configuration builder in CreateHostBuilder. If the Vault role is defined, the application will retrieve the Vault configuration from appsettings.json. However, to protect the Vault secret, it will read it from the VAULT_SECRET_ID environment variable.

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration((hostingContext, config) =>
        {

          config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
          config.AddEnvironmentVariables(prefix: "VAULT_");

          var builtConfig = config.Build();

          if (builtConfig.GetSection("Vault")["Role"] != null)
          {
            config.AddVault(options =>
              {
                var vaultOptions = builtConfig.GetSection("Vault");
                options.Address = vaultOptions["Address"];
                options.Role = vaultOptions["Role"];
                options.MountPath = vaultOptions["MountPath"];
                options.SecretType = vaultOptions["SecretType"];
                options.Secret = builtConfig.GetSection("VAULT_SECRET_ID").Value;
              });
          }
        })
        .ConfigureWebHostDefaults(webBuilder =>
        {
          webBuilder.UseStartup<Startup>();
        });
}

Run the application using a static database password

The demo application uses a static database password stored in Vault's key-value store to connect to the MSSQL database. You can use the key-value store to store API tokens and other static sensitive information. You manually manage static secrets, assuming responsibility for their rotation.

Open ProjectApi/appsettings.json with your terminal. The Vault.SecretType defaults to secrets, which the application uses to access that statically defined database password.

$ cat ProjectApi/appsettings.json

{
  ...TRUNCATED...
  "Vault": {
    "Address": "http://127.0.0.1:8200",
    "Role": "projects-api-role",
    "MountPath": "projects-api/",
    "SecretType": "secrets"
  }
}

Open ProjectApi/CustomOptions/VaultConfiguration.cs. The configuration provider receives the SecretType of secrets. If it receives that secret type, it will retrieve the database password you discovered in Vault at projects-api/secrets/static.

public async Task GetDatabaseCredentials()
{
  var userID = "";
  var password = "";

  if (_config.SecretType == "secrets")
  {
    Secret<SecretData> secrets = await _client.V1.Secrets.KeyValue.V2.ReadSecretAsync(
      "static", null, _config.MountPath + _config.SecretType);

    userID = "sa";
    password = secrets.Data.Data["password"].ToString();
  }

  // TRUNCATED

  Data.Add("database:userID", userID);
  Data.Add("database:password", password);
}

Verify that the database password already exists at projects-api/secrets/static with the Vault CLI.

$ vault kv get projects-api/secrets/static

====== Metadata ======
Key              Value
---              -----
created_time     2020-11-17T21:54:42.8659611Z
deletion_time    n/a
destroyed        false
version          1

====== Data ======
Key         Value
---         -----
password    Testing!123

In this example, the operations team already added a static database password to Vault's key-value store. You may be able to add passwords or API tokens to Vault yourself, depending on whether or not your Vault administrator enables that permission.

In your terminal, run the run_app.sh script. This will restore the .NET packages, retrieve the Vault secret ID and set it as environment variable, and run the application.

$ bash run_app.sh

  Determining projects to restore...
  All projects are up-to-date for restore.
Building...
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/rosemarywang/hashicorp/vault-guides/secrets/dotnet-vault/ProjectApi

Open a web browser and enter https://localhost:5001/api/Projects in the address. It returns a JSON list of HashiCorp projects, the year of their first commit, and GitHub Links.

Enter CTRL-C to exit the running application.

...TRUNCATED...
^Cinfo: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

This means that the application successfully retrieved the static password for the database and connected to it. However, if your database administrator changes this static password, you will need to update it in Vault and restart your application. The pattern outlined in the demo application demonstrates the retrieval of static secrets, such as password, API tokens, or keys that must be manually rotated and updated.

Run the application using a dynamic database username and password

Your Vault administrator may provide you with dynamic database usernames and passwords instead, which allows Vault to issue a new set of credentials based on a time-to-live parameter. When a database username and password expires, you must reload the application to retrieve new credentials from Vault.

The demo application will use a dynamic database username and password managed by Vault's database secrets engine to connect to the MSSQL database. Vault can be configured with secrets engines to manage the rotation of secrets.

Open the ProjectApi/appsettings.json in your preferred text editor, and change the Vault.SecretType defaults to database instead of secrets (at line 17).

The ProjectApi/appsettings.json file should look as follow.

{
  ...TRUNCATED...
  "Vault": {
    "Address": "http://127.0.0.1:8200",
    "Role": "projects-api-role",
    "MountPath": "projects-api/",
    "SecretType": "database"
  }
}

Open ProjectApi/CustomOptions/VaultConfiguration.cs. The configuration provider receives the SecretType of database. If it receives that secret type, it will retrieve the database username and password Vault generates at projects-api/database/creds/projects-api-role.

public async Task GetDatabaseCredentials()
{
  var userID = "";
  var password = "";

  // TRUNCATED

  if (_config.SecretType == "database")
  {
    Secret<UsernamePasswordCredentials> dynamicDatabaseCredentials =
    await _client.V1.Secrets.Database.GetCredentialsAsync(
      _config.Role,
      _config.MountPath + _config.SecretType);

    userID = dynamicDatabaseCredentials.Data.Username;
    password = dynamicDatabaseCredentials.Data.Password;
  }

  Data.Add("database:userID", userID);
  Data.Add("database:password", password);
}

The demo application accesses the Vault endpoint at projects-api/database/creds/projects-api-role to get a new database username and password. Vault has been configured to generate a new username and password that expire after two minutes. The expiration time of the secret can be updated to the time commensurate to your security policy for a database.

To examine this, use the Vault CLI to read a new set of database credentials at projects-api/database/creds/projects-api-role.

$ vault read projects-api/database/creds/projects-api-role

Key                Value
---                -----
lease_id           projects-api/database/creds/projects-api-role/cMegKcVr62qTCnFbnduDB8dZ
lease_duration     2m
lease_renewable    true
password           H1IoAtWPZq-NGzYV5Zmh
username           v-approle-projects-api-role-arDFo7XBn327cR1L4kF7-1605651978

In your terminal, run the run_app.sh script. This will restore the .NET packages, retrieve the Vault secret ID and set it as environment variable, and run the application.

$ bash run_app.sh

  Determining projects to restore...
  All projects are up-to-date for restore.
Building...
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/rosemarywang/hashicorp/vault-guides/secrets/dotnet-vault/ProjectApi

Open your browser to https://localhost:5001/api/Projects and verify that it returns a JSON list of HashiCorp projects, the year of their first commit, and GitHub Links.

Wait two minutes. After a few minutes, refresh the browser page with https://localhost:5001/api/Projects. The API will throw a SQLException that the application's database username can no longer log into the database.

Enter CTRL-C to exit the running application, and then restart the application such that the application can request a new database username and password from Vault.

$ bash run_app.sh

Open your browser to https://localhost:5001/api/Projects and the API request completes successfully again.

Re-authenticating to Vault

You can continue to reload the application to retrieve a new set of database usernames and passwords. However, you reload your application more than 5 times, your application will no longer start up and throws an error that the secret ID is invalid.

$ bash run_app.sh

...TRUNCATED...
Unhandled exception. System.AggregateException: One or more errors occurred. ({"errors":["invalid secret id"]}
)
 ---> VaultSharp.Core.VaultApiException: {"errors":["invalid secret id"]}

The operations team that set up Vault for this example limited the number of times you can use the secret to authenticate to Vault. To allow your application to start, you need to retrieve a new secret.

Use the terminal to run some automation to retrieve a new secret ID.

$ bash new_secret.sh

Restart the application and it will successfully authenticate to Vault.

$ bash run_app.sh

Next steps

In this tutorial, you used the C# client library to retrieve static and dynamic secrets from HashiCorp Vault. However, the patterns shown in this example do not fully ensure the availability of secrets or account for dynamic secrets with very short lifetimes. You must build additional logic into your application to watch for changes in dynamic secrets and reload. Furthermore, your operations team may provide additional security measures to introduce a Vault token to your application securely. The automation to handle these additional measures, such as response wrapping the secret ID, will need to be added to your application code.

If you cannot add additional logic into your application code to ensure the availability of secrets, handle live reloads for rotating secrets, or additional security measures for authenticating to Vault, refer to our tutorial on using Vault Agent and Consul Template with .NET Core applications.