Secrets in ASP.NET Core with AWS Secrets Manager – Super Simple & Secure

In this article, we will discuss securing Confidential Data and Secrets in ASP.NET Core with AWS Secrets Manager. We will also walk through various aspects and concepts of Secrets Management in your ASP.NET Core Application.

Problem Statement

Any application that you work with will definitely have some secrets or sensitive information attached to it. This includes API Keys, Mailbox Passwords, Database Connection Strings, and so on. Hard coding such sensitive information is definitely a bad idea, both from the code quality point of view as well as the security aspect.

So, what’s the next best thing in .NET applications to tackle this issue? appsettings.json. This is valid up to some extent only. AppSettings.json is meant for storing configuration-related values only, and not any sensitive information. Yes, you can definitely use this while building POC applications, MVP, but not really a good option for production applications. Also, anything you write within appsettings.json is going to be checked into your source control, which exposes your secrets to the entire world (unless you specifically choose to ignore certain appsettings.json files in advance). However, this is not really a worthy approach. So, what’s next?

AWS Secrets Manager

Secrets Manager of ASP.NET Core is meant for this exact purpose. But again, the ASP.NET Core secrets manager relies on the file system to store the required secrets. For development purposes, the ASP.NET Core feature “secrets-manager” works just about fine. For production applications, it’s better to have the secrets somewhere stored centrally. Various options include AWS Secrets Manager, Hashicorp Vault, Azure Key Vault, and so on.

If your existing application already uses the AWS Cloud Infrastructure, AWS Secrets Manager is the best option for you. It is specifically built to store and load secrets of your application very securely and also enable you to do various other operations like Secret Rotation, Replication, and so on.

Pricing

The FREE Tier in AWS offers a 30-day trial period for testing out AWS Secrets Manager. After that, every secret you store will cost you 0.40 USD per month (including replications). Note that this is also a pro-rated pricing plan. Apart from this, for every 10,000 API calls either fetch or set secrets on AWS Secrets Manager would cost you about 0.05 USD per month, which is quite affordable. What are your thoughts? You can find the detailed pricing information here.

Pre-Requisites

As usual, here is what’s required to follow along with this tutorial.

• AWS Account – Free Tier would do. Grab yours from here.
• .NET 6 or above.
• Visual Studio, I am using VS 2022 Community Edition.
• AWS Toolkit for Visual Studio.
• Ensure that your machine is authenticated to use AWS Services. I use AWS CLI Profiles to make my system access AWS. You can find the detailed guide here.

Exploring the Dashboard

First up, Let’s log in to our AWS Management Console, search for Secrets Manager, and Open it up. Here is the default landing page.

Click on Store a new secret. Just for demo purposes, we will try to store a random connection string. In the next screen, let’s select the secret type as Others, paste in the connection string as plaintext, and click on Next.

Now, we will have to give a name to our secret. You can give it any name you wish, but for easier management and identification, it’s recommended to have a particular naming convention. For instance, AWS recommends the name to be of the following format.

prod/AppBeta/MySql, where prod refers to the environment, AppBeta refers to the name of the application, and finally MySql, which denotes a connection string on a configuration related to a database.

We will name our secret as dev/HelloWorld/ConnectionString. So, this makes it so much easier to identify secrets from a long list. Optionally you can also give it a short description.

To ensure High Availability, you can also choose to opt in for secret replication. This creates read-only replicas of your secrets in other AWS regions as well. Note that there are charges associated with replications.

In the next section, you can configure automated secret rotation. We will skip this for now. It will be covered in a later part of this article. Click on Next, review your changes, and create this new secret by clicking Store.

Give it a couple of seconds, and you will be able to see that the secret has been stored.

With that done, let’s see how to retrieve this newly created secret from our ASP.NET Core Web API.

Reading from AWS Secrets Manager using the AWS .NET SDK

Open up Visual Studio, and create a new ASP.NET Core Web API Project. Note that I am using Visual Studio Community 2022 Preview with the latest .NET 8 Preview 3 SDK installed.

I will just clean up the project so that we have the bare minimum code to get started with.

Let’s Code

You have to install the following packages to set up the AWS Secrets Manager SDK.

Install-Package AWSSDK.Extensions.NETCore.Setup
Install-Package AWSSDK.SecretsManager

Once that is done, replace your Program.cs with the following.

using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonSecretsManager>();
var app = builder.Build();
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

app.MapGet("/secret", async (IAmazonSecretsManager secrets) =>
{
    var request = new GetSecretValueRequest()
    {
        SecretId = "dev/HelloWorld/ConnectionString"
    };
    var data = await secrets.GetSecretValueAsync(request);
    return Results.Ok(data.SecretString);
});

app.Run();

Firstly, in lines 7 & 8, we will be registering our AWS SDK dependencies into the ASP.NET Core DI Container. We will be specifically adding IAmazonSecretsManager into our services.

Next, we will create a Minimal Endpoint just to retrieve the Secret that we stored earlier in AWS Secrets Manager. Note that this will be an async operation, with injecting the IAmazonSecretsManager instance.

First, we will have to create a request object and pass the secretId as the name of the secret that we have added earlier. Once that is done, in line 24, we send a request to AWS using the secrets client to get the value. We finally returned the Secret String from the response.

That’s it. Let’s run our application and open up Swagger.

We will be sending a request to the /secret GET endpoint. As you can see we are able to fetch the secret that we stored in AWS Secrets Manager. This is just for demo purposes. You really would not want to return any of the secrets as a response to the client, especially the connection strings. Ideally, this should be used within the application, during the startup phase, where the application would try to connect to a database using this connection string details.

Version Stages

Another important concept of AWS Secrets Manager is Versioning. Secrets are versioned, as in, whenever you change a secret or it gets rotated, a newer version of the secret is created, along with a version id. Note that all the versions are not maintained, but only the following three.

• The current version – AWSCURRENT
• The previous version – AWSPREVIOUS
• The pending version (during rotation) – AWSPENDING

So, where do you use this? While creating a Request object, this can be set. Here AWSCurrent, AWSPrevious, and AWSPending are known as Version stages.

For example, if the following is your request, you will get back the response as the previous value of the string and not the current. If you leave this field empty, it will default back to AWSCURRENT, which as the name suggests returns the latest secret value.

var request = new GetSecretValueRequest()
{
    SecretId = "dev/HelloWorld/ConnectionString",
    VersionStage = "AWSPREVIOUS"
};

Integrating AWS Secrets with ASP.NET Core at Runtime

That is how easily you can fetch secrets from AWS Secrets Manager in your ASP.NET Core WebAPI. But, let’s see a real-time use case and implementation. Ideally, these secret values are expected to be bounded to Application Configurations, right? As in, the connection string would be expected to be part of appsettings.json (in lower environments), or environment variables in higher environments like Prod or QA.

For instance, there will be a strongly typed class that is named DatabaseSettings and has a property under it named ConnectionString. Using IOptions Patterns, at run time, we will be binding the values from appsettings.json to the Options Pattern instance.

New to Options Pattern in ASP.NET Core? Read my article to get started with configurations in ASP.NET Core applications.

So, how do we make this possible with the current implementation? I need to load all my secrets in advance, during startup, so that we do not always send a request to AWS. Here is how.

First, let’s create a new Class named DatabaseSettings.

public class DatabaseSettings
{
    public string? ConnectionString { get; set; }
}

In your appsettings.json, add the new section named DatabaseSettings, with the ConnectionString.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "DatabaseSettings": {
    "ConnectionString": "Host=localhost;Database=rootTenantDb;Username=postgres;Password=root;Include Error Detail=true"
  }
}

We will make use of the Options Pattern here. Add the following to your Program.cs.

builder.Services.AddOptions<DatabaseSettings>().BindConfiguration(nameof(DatabaseSettings));

And, we will add a new GET endpoint (from-options) that will return the connection string value from the ConfigurationProvider.

app.MapGet("/from-options", (IOptions<DatabaseSettings> options) =>
{
    return Results.Ok(options.Value.ConnectionString);
});

With that done, let’s run our Web API.

As you can see, the API returns the configurations stored in appsettings.json. This is fine for local development environments, but when you take your application to production, you should not be storing sensitive data in your appsettings.json, but rather in secure storage like AWS Secrets Manager.

To achieve this, we will have to add AWS Secrets Manager as a Configuration Provider to fetch values directly from AWS. Here is how.

Install the following package.

Install-Package Kralizek.Extensions.Configuration.AWSSecretsManager

Next, let’s add a new secret in AWS Secrets Manager. Note that this will be specifically for Production Environments only.

I added the following secret.

Host=production;Database=rootTenantProductionDb;Username=postgres;Password=supersecret;Include Error Detail=true

And named my secret as Production/Demo/DatabaseSettings__ConnectionString. It is important to have the double underscore between DatabaseSettings and ConnectionString because they are stored in a parent-child form within our Application’s Configuration Classes.

Next, add the following lines of code, to register AWS Secrets Manager as one of the Configuration Provider.

1
builder.Configuration.AddSecretsManager(configurator: config =>
2
{
3
    config.SecretFilter = record => record.Name.StartsWith($"{builder.Environment.EnvironmentName}/Demo/");
4
    config.KeyGenerator = (secret, name) => name
5
                    .Replace($"{builder.Environment.EnvironmentName}/Demo/", string.Empty)
6
                    .Replace("__", ":");
7
});

Line 3 ensures that we are not loading the secrets that are not required for our application. So we add a filter for loading secrets whose names start with EnvironmentName/Demo/. By default, the Environment name will be Development in your local instance. Later in this article, for testing, we will change it to Production via the launchsettings.json

Next, in line 4, we will be adjusting the name of the secret, so that it is how the application expects it to be. For example, Production/Demo/DatabaseSettings__ConnectionString will be transformed to DatabaseSettings:ConnectionString. This should ideally be mapped into the ASP.NET Core Configuration.

Apart from this, I have also removed the check in Program.cs where we enable Swagger only for Development Environments.

Let’s see it in action. Build and Run your ASP.NET Core Web API.

When you hit the /from-options endpoint, you will notice that you are still getting the data from appsettings.json. Why so?

This is because the current environment is set to Development, and our application tries to load Development/Demo/DatabaseSettings__ConnectionString from AWS Secrets Manager. But since such a secret doesn’t exist, it will default back to the next configuration provider, which is appsettings.json.

Now, to load from AWS Secrets Manager, we will have to change the current Environment Name to Production. You can do this by opening Launchsettings.json and setting the ASPNETCORE_ENVIRONMENT variable to Production.

{
  "$schema": "http://json.schemastore.org/launchsettings.json",
  "profiles": {
    "https": {
      "commandName": "Project",
      "dotnetRunMessages": true,
      "launchBrowser": true,
      "launchUrl": "swagger",
      "applicationUrl": "https://localhost:7001;http://localhost:5004",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Production"
      }
    }
  }
}

Save your changes, build, and re-run your ASP.NET Core Web API. Try to hit the /from-options endpoint. This time around, you can see that we are loading the value of the secret from AWS Secrets Manager seamlessly. This is how you would load settings and secrets in your production-grade application.

You can improve the efficiency of your application even more by adding this check.

if (!builder.Environment.IsDevelopment())
{
    builder.Configuration.AddSecretsManager(configurator: config =>
    {
        config.SecretFilter = record => record.Name.StartsWith($"{builder.Environment.EnvironmentName}/Demo/");
        config.KeyGenerator = (secret, name) => name
                        .Replace($"{builder.Environment.EnvironmentName}/Demo/", string.Empty)
                        .Replace("__", ":");
    });
}

This ensures that SecretsManager is loaded only if the Application Environment is not Development. You get the idea, you can change it as per your requirement.

Handling Changes in Secrets

Now that we are able to load Secrets into our Configuration, how about the scenario where the value of the secrets is changed manually or as a part of secrets rotation? How will your application load the latest values without the need to restart your servers?

I have covered this part specifically in my previous article where we discussed about Options Pattern in ASP.NET Core. You can read more about it here.

In your Minimal API, you just have to switch from IOptions to IOptionsMonitor.

app.MapGet("/from-options", (IOptionsMonitor<DatabaseSettings> options) =>
{
    return Results.Ok(options.CurrentValue.ConnectionString);
});

Additionally, you will also have to add a new PollingInterval setting in your Configuration Registration.

if (!builder.Environment.IsDevelopment())
{
    builder.Configuration.AddSecretsManager(configurator: config =>
    {
        config.SecretFilter = record => record.Name.StartsWith($"{builder.Environment.EnvironmentName}/Demo/");
        config.KeyGenerator = (secret, name) => name
                        .Replace($"{builder.Environment.EnvironmentName}/Demo/", string.Empty)
                        .Replace("__", ":");
        config.PollingInterval = TimeSpan.FromSeconds(5);
    });
}

With these changes, let’s run our application and hit our /from-options endpoint.

the first request returns back the existing data from the secrets. With the application running, I went ahead and updated the value of the secret, and changed the host to newproduction.

After about 5 seconds, the API also returned the newly updated values.

Note that there are additional costs incurred because of this. When you set the PollingInterval to 5 seconds, the App makes periodic API calls (every 5 seconds) to AWS to fetch the latest Secrets values. In real-time application, this can add up to your AWS Bills. Use this sparingly in your applications!

Secret Rotation

There is one other concept that you really would need to know, which is Secret Rotation. It is a managed service that allows your secrets to be rotated/changed periodically in an automated way. This ensures that there is no real way to brute force the secrets. Amazon allows up to 4 hours of secret rotation per secret with no additional costs. We will cover this in a separate article.

Summary

In this article, we learned about Secrets in ASP.NET Core with AWS Secrets Manager, the use cases of secret management, and various options shipped with ASP.NET Core. We got introduced to AWS Secrets Manager which is a secure way to store sensitive data of your application. Further, we explored the dashboard, and used the AWS SDK to retrieve secrets from AWS! We went on to build a practical use case where the application has to load values from appsettings.json (in the Development environment) and from AWS Secrets Manager(in the Production environment). There are several other concepts covered as well.

Make sure to share this article with your colleagues if it helps you! Helps me get more eyes on my blog as well. Thanks!