Amazon SNS and ASP.NET Core - Building Super Scalable Notification Systems for .NET Applications on AWS

In this article, we’ll bring together Amazon SNS and ASP.NET Core to build super scalable notification systems. Amazon SNS is an excellent choice if you’re looking to build a scalable, real-time notification system. With its ease of use and flexibility, SNS, or the Simple Notification Service by Amazon is a popular choice for developers looking to build notification systems for web and mobile applications. You’ll learn how to integrate Amazon SNS with your .NET Core application, set up a notification topic, and subscribe to receive notifications.

By the end of this article, you’ll have a comprehensive understanding of Amazon SNS and how to use it to build robust notification systems for your ASP.NET Core applications. So, let’s dive in and explore the power of Amazon SNS and ASP.NET Core!

What is Amazon SNS, or Simple Notification Service?

Amazon SNS or Simple Notification Service is a fully managed serverless notification (Pub/Sub) service by AWS which is super scalable and cost-effective. This service can deliver system-to-system notifications to build decoupled microservice applications that can scale, as well as can push system-to-person notifications to your end users with SMS, emails, and so on. In case of failures, the service automatically manages retries for you using dead-letter queues, which we will be studying in detail later in this article.

As requirements and the need to optimize system costs grow, it’s an important aspect to have scalable notification workflows which can handle a large volume of messages and deliver them reliably to their intended recipients. Combining Amazon SNS and ASP.NET Core allows developers to build scalable and reliable notification systems that can handle a large volume of messages and deliver them to multiple endpoints. This can be especially useful for businesses that need to send notifications to a large number of users or systems, such as real-time updates for applications, reminders, and alerts.

Amazon SNS provides a FIFO model of message processing to ensure the message order.

It’s also a highly reliable service since it has a retry mechanism out of the box. More on this later in this article.

Amazon SNS vs SQS

Amazon SNS falls into a different category of messaging in AWS. They are somewhat similar to SQS (which we discussed in the previous article), but here are the key differences.

• When a message is published to an SNS Topic, it is delivered to all subscribers using Fanout. This is a push mechanism, which means messages are delivered to subscribers automatically as soon as they are published. Whereas, SQS is a pull-based service, which means messages are retrieved from the queue by consumers when they are ready to process them.
• SQS queues are used to store messages until they are processed by a consumer. SNS doesn’t store messages.
• SNS can send messages to a wide range of endpoints, including email, SMS, mobile push notifications, HTTP endpoints, and more.
• SQS is designed for asynchronous messaging and is ideal for use cases where message delivery can be delayed, or where messages need to be processed in a specific order. SNS is designed for real-time messaging and is ideal for use cases where message delivery needs to be immediate or near-immediate.

The choice between SNS and SQS will depend on the specific requirements of your application and how you want to deliver your messages. Anyhow, both of these messaging services are quite crucial in building decoupled applications. The chances are that you would probably need to use both SNS and SQS at certain points in your system depending on its use case.

Prerequisites

Before getting started with the hands-on, ensure that you have the following items in check:

• An AWS Account. Even a Free Tier would do.
• AWS CLI Profiles configured.
• .NET 7 SDK installed on your machine.
• Visual Studio IDE.
• AWS SDKs installed.
• Working Knowledge of AWS Lambda.

It’s highly recommended that you go through my previous article about Amazon SQS before getting started with this walkthrough.

PubSub Architecture with Amazon SNS

The PubSub or the Publisher Subscriber Messaging model is a crucial mode of communication in distributed systems. Here is how the data flows with Amazon SNS in the picture.

First, the publisher, which could be a distributed system, microservice, or other AWS services will publish a message with a Subject to the Amazon SNS Topic. The SNS, which is a fully managed PubSub Messaging system of AWS will push the messages from the publisher to the subscribed clients. As mentioned, there could be various client protocols subscribed to a particular topic, such as AWS Lambda, SQS, Kinesis Firehose, Emails, and HTTP Endpoints.

Messages can be filtered so that only a particular client/group of clients will receive the intended messages. SNS also supports delivery protocols that can define how retries are handled if the subscriber is not reachable. When the maximum defined retry threshold is crossed, AWS SNS either discards the message or pushes the failed message into a dead letter queue for future retries.

We will look more into DLQ or Dead Letter Queues later in this article.

SNS Topic is a logic point of access within the SNS Communication channel.

Creating an Amazon SNS Topic

As usual, we will get started with Amazon SNS and ASP.NET Core by first getting familiarized with what the Management Dashboard of this AWS Service looks like. Login to your AWS Management Console, search for SNS, and open it up.

If you are opening it up for the first time, you should see a similar screen.

In the sidebar, you can see the following options, where you can manage topics & subscriptions.

Let’s click on Topics and try to create a new one.

As you see, these are the two available types, ie, Standard and FIFO.

FIFO can publish up to 300 message sets per second, ensuring that the order is maintained. But it only supports SQS protocol. As in, it can push messages to an SQS queue only.

As for the standard queue, it offers the highest publishing throughput, along with multiple subscription protocols like SQS, Lambda, Email, HTTP, and so on.

I have named my new Topic as “sample-topic” and selected the Standard SNS topic type.

SNS also provides Encryption, which adds a layer of security to your topic messages. Also, you can set up access policies, for which we will be keeping the default values.

For Delivery policies, we can set up the default retry count, for which the SNS keeps on retrying and publishing the messages on failures. By default, SNS will perform 3 retries, each with a delay of 20 seconds. You can change these values by unchecking the “Use default delivery policy” checkbox.

Delivery Status Logging helps you to log the statuses of each message received/sent to your subscriber.

With all this reviewed, you can proceed to create the SNS topic.

Amazon SNS and ASP.NET Core - Publishing the Message to the SNS Topic

Let’s build an ASP.NET Core Application that will act as Publisher in our demonstration and send out SNS Messages to Subscribers (which we will add in the next section). Open up Visual Studio 2022 IDE and create a new Solution. I named mine AmazonSNS.Demo and set the framework to .NET 7.

You can get the source code of the implementation from my GitHub Account. Do follow me on GitHub as well 😊

First, install the following NuGet packages so that we can use the AWS SDK packages to consume the SNS Service on our ASP.NET Core application. Open up Package Manager Console and run the following commands.

Install-Package AWSSDK.SimpleNotificationService
Install-Package AWSSDK.Extensions.NETCore.Setup

Next, you will have to register the IAmazonSimpleNotificationService into the DI Container of your ASP.NET Core application. Open up Pogram.cs and add the following piece of code.

builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddAWSService<IAmazonSimpleNotificationService>();

Optionally, you can also define the default profile and aws region in appsettings.json. I am skipping this step in my application since I have already configured my development machine to be authenticated with an AWS User Profile using AWS CLI Profile. Learn more about this from my recent youtube video (https://www.youtube.com/watch?v=oY0-1mj4oCo&t=191s). While you are at it, do not forget to subscribe to my channel as well :)

"AWS": {
  "Profile": "default",
  "Region": "ap-south-1"
}

Next, we will have to define classes, specifically for ProductCreatedNotification and the actual CreateProductRequest. Add the following classes and record to your Publisher project.

public interface INotification
{
}
public class ProductCreatedNotification : INotification
{
    public ProductCreatedNotification(int id, string? name, string? description)
    {
        Id = id;
        Name = name;
        Description = description;
    }

    public int Id { get; private set; }
    public string? Name { get; private set; }
    public string? Description { get; private set; }
}

public record CreateProductRequest(int Id, string? Name, string? Description);

Now that all our basic setup is completed, let’s add a new endpoint that will actually publish a message based on the input message that it receives. Under the Controllers folder, add a new API controller named ProductsControllers. Following is the code for adding an endpoint that will create a new product and publish the ProductCreated Notification to a new SNS Topic.

[Route("api/[controller]")]
[ApiController]
public class ProductsController : ControllerBase
{
    private readonly IAmazonSimpleNotificationService _snsClient;
    private readonly ILogger<ProductsController> _logger;
    private const string ProductTopic = "product-topic";

    public ProductsController(IAmazonSimpleNotificationService snsClient, ILogger<ProductsController> logger)
    {
        _snsClient = snsClient;
        _logger = logger;
    }

    [HttpPost]
    public async Task<IActionResult> CreateProductAsync(CreateProductRequest request, CancellationToken cancellationToken)
    {
        //perform product request validation
        //save incoming product to database

        //create message
        var message = new ProductCreatedNotification(request.Id, request.Name, request.Description);

        //check if product topic already exists
        var topicArnExists = await _snsClient.FindTopicAsync(ProductTopic);

        //extract the topic arn of the sns topic
        //if the topic is not found, create a new topic
        string topicArn = "";
        if (topicArnExists == null)
        {
            var createTopicResponse = await _snsClient.CreateTopicAsync(ProductTopic);
            topicArn = createTopicResponse.TopicArn;
        }
        else
        {
            topicArn = topicArnExists.TopicArn;
        }
        //create and publish a new message to the sns topic arn
        var publishRequest = new PublishRequest()
        {
            TopicArn = topicArn,
            Message = JsonSerializer.Serialize(message),
            Subject = "ProductCreated"
        };
        _logger.LogInformation("Publish Request with the subject : '{subject}' and message: {message}", publishRequest.Subject, publishRequest.Message);
        await _snsClient.PublishAsync(publishRequest);
        return Ok();
    }
}

In lines #5-13, we are injecting the IAmazonSimpleNotificationService and Logger services into the construction of the controller. Note that we are also defining a constant string for the required topic name. This is the topic we will be creating in AWS SNS and pushing our product-related notifications to.

Line 15 onwards is our new controller endpoint which is a POST method that will accept a request of type CreateProductRequest and return a valid status code.

Let’s assume that we have already written the code for validating this request, and creating a new Product in a database. I am skipping this to keep the current implementation simple, as we are currently only interested in sending notifications using Amazon SNS.

Line 22, we are creating a message object by passing the incoming product details like the id, name, and description of the product.

Line 25, we are explicitly checking if there is an SNS Topic already available in our AWS account with the topic name “product-topic”. If exists, it will return the Topic Metadata. Else it will return a null.

Lines #30-34, if the topic is not present, we will create a new topic with the name “product-topic”. From the response to line #32, we will extract the TopicARN. Note that we need the TopicArn to be able to publish messages on a particular topic.

Line #37, if the Topic already exists, simply extract the topic arn from the response of the FindTopicAsync method from line #25.

TopicARN is the unique resource name of AWS Resources. It is usually of the following format : arn:aws:sns:<aws-region>:<account>:<topic-name>

In lines #40-45, we are creating a new PublishRequest object with the parameters like TopicArn, the Message (which we have serialized the message object from earlier), and finally the subject, which is ProductCreated. Note that you can send different types of messages to SNS Topic. It’s up to the subscriber to react to the incoming message type, deserialize to the appropriate type and consume it as needed.

Finally, we log the request and subject and publish the request using the sns client.

That’s it. Now that we have built our Publisher, let’s discuss who can receive the message sent by this publisher. As mentioned earlier in this article, the supported subscription protocols are as follows.

• SQS
• Email
• Kinesis
• AWS Lambda
• HTTP(s) endpoints.

In this demonstration, we will specifically focus on simple subscribers such as the Email and AWS Lambda Subscriptions. I will cover the other protocols in a different article.

But before, let’s create the SNS topic using the application that we have just built. Run the application, open up Postman, and post the following body to the endpoint `https://localhost:7253/api/products\`

{
    "id":1,
    "name":"Apple iPhone 14 (128 GB)",
    "description":"Apple iPhone 14 (128 GB)"
}

An important concept to note here is that our new SNS topic once created, will have no subscribers. Any message sent to an SNS Topic with 0 subscribers will be immediately discarded. In our case, we are just hitting the endpoint to ensure that we have created our new SNS topic programmatically.

Once we get a 200 OK response on Postman, simply go to your AWS Management Console and open up Amazon SNS. If everything goes as expected, you will be able to see our new topic “product-topic” created as expected.

Next, let’s add some subscribers to this topic.

Email Subscription

The idea is that when a new message is pushed from our publisher, the Amazon SNS will send a copy of this notification to us via email. I created a dummy email account using Ethereal.

Ethereal is a super cool fake SMTP service that comes in very handy for developers while building email-based systems, I regularly use this. Once you have your email account ready, let’s switch back to AWS SNS and open up the new topic.

Under the Subscriptions tab, click on create a subscription.

Here, ensure that you have the right Topic ARN selected, set the protocol as Email-JSON, and in the endpoint field, type in the email address you would like to send the SNS Notification. In my case, I have added the Ethereal mail id.

Once you click on Create Subscription, you will receive a Confirmation mail at your provided email address. Here is what I got in my ethereal mailbox.

To confirm the subscription, simply copy the URL from the SubscribeUrl field and paste in into a new tab on your browser. This will confirm the subscription, and you will see an XML response like the one below on your browser.

You can also see that, back in your AWS SNS Management screen, the status of your new email subscription is confirmed.

This means that you have successfully added a new email subscriber to your Amazon SNS topic and anytime there is a new notification pushed by a publisher to this topic, there will be a new email generated. Let’s test this.

Open up Postman again, and send the request to the products endpoint.

I switched back to my Ethereal mailbox and was able to see the new message. As you can see in the below screenshot, the subject and message are populated as we expected. This is how simple it is to set up an Email Subscription to an Amazon SNS topic.

Lambda Subscription

Before starting this section, it’s recommended that you have good HandsOn Experience with creating and deploying AWS Lambda with the .NET Runtime. I have written a couple of beginner-friendly articles on my blog about AWS Lambda for #dotnet Developers.

Here are the links to the mentioned articles.

• AWS Lambda with .NET 6 – Getting Started with Serverless Computing
• Amazon API Gateway with .NET – AWS Lambda & DynamoDB Integrations
• Securing Amazon API Gateway with Lambda Authorizer in .NET
• Hosting ASP.NET Core Web API with AWS Lambda – Truly Serverless REST APIs

Next, to the same SNS topic, we will add a new Lambda Subscriber. We will keep everything simple.

In your Visual Studio, under the same solution we were working on earlier, let’s create a new Lambda Project using an SNS template.

In the next screen, you will be asked to select a Blueprint for your Lambda. To speed up things, let’s select the Simple SNS Function blueprint.

I am not doing many changes to the Lambda. I just modified the log message.

private async Task ProcessRecordAsync(SNSEvent.SNSRecord record, ILambdaContext context)
{
    context.Logger.LogInformation($"Processed SNS with subject: {record.Sns.Subject} and message: {record.Sns.Message}");
    // TODO: Do interesting work based on the new message
    await Task.CompletedTask;
}

This is going to log the subject and message onto cloudwatch logs. With that done, let’s deploy this lambda to AWS.

As usual, simply right-click the Lambda project and select “Publish to AWS Lambda”.

In the Lambda configuration screen, give a name for your lambda, select the correct profile and region, and click next.

In the next screen, select/create a new role that has cloudwatch access. This is very important for this demonstration. Once done, click on upload.

I forgot about this while setting up the Lambda. I had to later go to Lambda Roles in IAM and manually added Cloudwatch Access to the role. So make sure that Lambda has permission to write to a Cloudwatch Log Group.

This should start zipping your Lambda and uploading it to AWS Lambda.

Once done, you can go back to your AWS Lambda, and verify that the function is uploaded as expected.

Now that Lambda is in place, we need to add this new Lambda as a Subscriber to the SNS topic that we created earlier. For this, navigate to AWS Lambda, open up the new Lambda, and open the Configurations tab. Here, click the triggers and add a new one.

As you see below, set the trigger as SNS, and select the correct topic. And finally, save it.

You can go back to your Amazon SNS Topic and refresh the Subscriptions. The Lambda also would have been now added as a new subscriber to this SNS topic.

This means that if our ASP.NET Core Publisher endpoint is hit again, a new email will be sent, and also the Lambda will be invoked, which will log the message as we had defined while creating the Lambda.

Let’s test this now. I sent another POST request to the API locally. Open up the Lambda logs, and you will be able to see the new message that had been published.

Amazon SNS Filter Policy

The Subscribers receive all the messages that are published to the SNS topic by default. However, Amazon SNS Supports filtering the incoming messages at the Subscriber level so that only particular messages are received by a given subscriber.

For this, we will have to do a small code change in the way we publish the message. We will have to add a message attribute to our publish request.

Let’s take a hypothetical use case, where the Lambda wants to read the message that is intended for the Lambda only. For this, we can add a new message attribute to the publish request as follows.

publishRequest.MessageAttributes.Add("Scope", new MessageAttributeValue()
{
    DataType = "String",
    StringValue = "Email"
});

With that code added, go to the AWS SNS Topic and edit the Lambda Subscription.

Now, as you see, Publish is sending a message with the attribute Scope set to Email. While the Lambda Subscriber filters the messages and is looking for messages that have a filter policy with the scope set as Lambda. Ideally, when our current publisher code sends a new notification, Lambda should not be receiving the message. You can test this as well.

Retries

By default, when there is an issue with Amazon SNS sending a notification to its subscriber, it automatically retries it. Remember, in the first section of this article, we talked about the retry counts and policies while creating an Amazon SNS Topic via the AWS Management Console.

The failures usually come up when an endpoint of the subscriber is not reachable.

Open up the product-topic SNS topic, and navigate to Delivery Policies. Here you can see that we have set the retry counts to 3 with periods of 20 seconds gap.

If the Retry Count is exhausted, the SNS message is discarded and lost. In case, you want to not miss the SNS Messages, you can redirect the failed SNS Messages to a Dead Letter Queue or DLQ.

SNS Dead Letter Queues

A dead letter queue is an Amazon SQS queue that an SNS Subscription can target for messages that weren’t delivered to the subscriber. Once the retry limit is exhausted, these messages can be held in the Dead Letter Queues for further analysis or reprocessing.

Let’s quickly see how to set up a DLQ and assign it to a Subscriber. As mentioned, a DLQ is an SQS Queue only. So, open up Amazon SQS and create a new Queue.

I named my new SQS queue “product-dlq” and left everything else to the default values.

Now we need to assign this dlq to a subscriber. For example, let’s open up the Lambda Subscription and navigate to the Redrive policy.

Here, you can edit the Subscription and assign the created SQS Queue as the Dead Letter Queue.

Now, the newly connected SQS Queues, by default would not permit message delivery from the SNS Topic. To allow an Amazon SNS topic to send messages to an Amazon SQS queue, you must create an Amazon SQS queue policy.

You can open up the SQS Queue, Edit it and add the following Statement (append the new statement).

{
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:ap-south-1:821175633958:product-dlq",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:ap-south-1:821175633958:product-topic"
        }
      }
    }
  ]
}

Now this will allow the SNS topic to push the message to the Dead Letter Queue (SQS ) when there are any failures in sending the messages to the Lambda Subscription.

That’s a wrap for this article. Do let me know your suggestions and feedback. Are you already using Amazon SNS in your ASP.NET Core Projects?

Summary

In this detailed article, we covered Amazon SNS and ASP.NET Core, How the PubSub Architecture of Amazon SNS looks like, and all about Topics and Subscribers. We then created an ASP.NET Core that was able to create new topics using the SDKs provided by AWS and send out messages to the SNS Topic. We also looked into how to subscribe to a particular Amazon SNS Topic using the supported protocols such as Email and AWS Lambda. Finally, we discussed Retries and Dead Letter Queues to retain the failed messages for future analysis and reprocessing.

You can check out the source code of this implementation from here.

Make sure to share this article with your colleagues if it helped you! Helps me get more eyes on my blog as well. Thanks!

Stay Tuned. You can follow this newsletter to get notifications when I publish new articles – https://newsletter.codewithmukesh.com/subscribe. Do share this article with your colleagues and dev circles if you found this interesting. Thanks!