Pagination in Amazon DynamoDB with .NET - Improve your API Performance!

Amazon DynamoDB is the go-to serverless NoSQL Database Solution from AWS and very sought-after for building serverless applications on the cloud. Pagination is one feature that helps massively improve your application response times in case you have large volumes of data stored in your database. In this article, we will learn about implementing pagination in Amazon DynamoDB with the .NET AWS SDK!

Understanding Pagination - The Need for Paging

Pagination is a commonly used technique in web applications, especially APIs to split large lists of data into separate pages, allowing the users to navigate with ease. It helps improve the user experience by making content more accessible and reducing server response times for massive data sets.

The idea is simple. Do you really need to load all the data from the database at once? Most probably, your answer would be NO. Instead of returning all the records from a database, you need an approach to return a smaller set of data which is required. To view the subsequent set of records, one has to send another request to the API, which would then fetch the next set of records, and so on. This greatly improves the response time and costs of your servers. It doesn’t really make sense to return tens of thousands of records to the client in a single go, right?

In traditional pagination implementations, paginated results usually can provide the number of pages available, links to each of the pages, next pages, previous page, current item count of page, and so on. Things are quite different in how pagination can be implemented for DynamoDB.

Pagination with Amazon DynamoDB

Coming to DynamoDB, there is an actual cost attached to each database record you retrieve from the cloud. In order to limit the server costs, implementing pagination while interacting with DynamoDB is highly recommended.

Every time you send a query request to DynamoDB via the .NET SDK, a property named LastEvaluatedKey is returned as part of the response. This plays a pivotal role in paginating the results from DynamoDB. If the LastEvaluatedKey property has a value, it signifies that the table has data to be paginated.

There are 2 instances where DynamoDB returns you a paginated response,

1. If the result returned is more than 1 MB in size, DynamoDB attempts to limit the response size by paginating. This ensures that a lot of data is not sent over through the network, leading to slightly better performance.
2. If you explicitly specify a limit in your Query Request by setting a value to the Limit property, pagination will be applied. This is something that’s more often used. In this tutorial as well, we will be explicitly setting the limit of the query request.

Let’s build a simple .NET Web API to explore pagination in Amazon DynamoDB. But before that, here are some prerequisites.

Pre-Requisites

• An AWS Account.
• Your development machine must be authenticated to interact with AWS. Read more.
• Working knowledge of Amazon DynamoDB, and its SDK. Read more.
• An IDE. I will be using Visual Studio 2022 Community Edition.

Getting started with Paging in Amazon DynamoDB with .NET

For this demo, we will be using .NET 8 Web API, and for IDE, I will be using Visual Studio 2022 Community as always.

First up, create a new ASP.NET Core Web API project. I named mine as DDBPagination. Make sure to set the target framework to .NET 8.

Install the required AWS SDK packages as follows.

Install-Package AWSSDK.DynamoDBv2
Install-Package AWSSDK.Extensions.NETCore.Setup

Once the packages are installed, you need to register the IAmazonDynamoDB and IDynamoDBContext. Open up Program.cs and add the following.

builder.Services.AddAWSService<IAmazonDynamoDB>();
builder.Services.AddScoped<IDynamoDBContext, DynamoDBContext>();

In this demo, we will implement Pagination over a set of Product data from DynamoDB.

Next, we will create a Product class with which we work on in this demonstration. Create a new folder named Models, and add in the following class.

[DynamoDBTable("products")]
public class Product
{
    [DynamoDBHashKey("category")]
    public string? Category { get; set; }
    [DynamoDBRangeKey("id")]
    public string? Id { get; set; }
    [DynamoDBProperty("name")]
    public string? Name { get; set; }
    [DynamoDBProperty("price")]
    public float? Price { get; set; }
}

Note that we have added AWS DynamoDB-specific attributes to each of the properties of the class like DynamoDBHashKey and DynamoDBRangeKey. This is essential for the hash and range key mappings.

Since our API is to return paged data, let’s create a wrapper that can hold a list of items of type T and also should include the LastEvaluatedKey key, so that the client can use this key to navigate to the next page as and when required.

Within the same Models folder, create a new class named PagedResult.

public class PagedResult<T> where T : class
{
    public List<T> Items { get; set; } = new List<T>();
    public string? PaginationToken { get; set; }
}

As mentioned earlier, we are building a .NET 8 Web API which will have a POST endpoint that can return a paged set of data. To this endpoint, we will have to pass parameters like Product Category and the LastEvaluatedKey, which I have named as PaginationToken for better readability.

I created a separate folder named Requests and add the following ProductSearchRequest class to it.

public class ProductSearchRequest
{
    public string? Category { get; set; }
    public string? PaginationToken { get; set; }
}

Next, we will switch to Program.cs and add our Minimal POST API Endpoint as follows.

1
app.MapPost("/", async (IAmazonDynamoDB client, IDynamoDBContext context, [FromBody] ProductSearchRequest request) =>
2
{
3
    var partitionKey = "category";
4
    var query = new QueryRequest
5
    {
6
        TableName = "products",
7
        Limit = 3,
8
        KeyConditionExpression = $"{partitionKey} = :value",
9
        ExpressionAttributeValues = new Dictionary<string, AttributeValue>
10
        {
11
            {":value", new AttributeValue { S = request.Category }}
12
        }
13
    };
14
    if (!string.IsNullOrEmpty(request.PaginationToken))
15
    {
16
        query.ExclusiveStartKey = JsonSerializer.Deserialize<Dictionary<string, AttributeValue>>(Convert.FromBase64String(request.PaginationToken!));
17
    }
18

19
    var response = await client.QueryAsync(query);
20
    var lastEvaluatedKey = response.LastEvaluatedKey.Count == 0 ? null : Convert.ToBase64String(JsonSerializer.SerializeToUtf8Bytes(response.LastEvaluatedKey, new JsonSerializerOptions()
21
    {
22
        DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingDefault
23
    }));
24
    var data = response.Items
25
         .Select(Document.FromAttributeMap)
26
         .Select(context.FromDocument<Product>);
27
    var pagedResult = new PagedResult<Product>
28
    {
29
        Items = data.ToList(),
30
        PaginationToken = lastEvaluatedKey,
31
    };
32
    return Results.Ok(pagedResult);
33
});

• Firstly, this is a POST endpoint that accepts the body of type ProductSearchRequest. Also, not that we are injecting both IAmazonDynamoDB and IDynamoDBContext here.
• From our database schema (Product), we are aware that the partition key will be category. We will use this as the key in the query request that we are going to form in the next part of the code.
• Lines #4 to #13 is where we define the Query Request. Here, we set the table name, the limit of the records to be returned (I have set it to 3 just for demo purposes), and a KeyConditionExpression that takes in the category value from the incoming request. Note that the limit is the crucial parameter in this request. Since I have set the limit to 3, this request should always return me a maximum of 3 records.
• Next, if the incoming request contains a token, or LastEvaluatedKey, we will set the value to the query.ExclusiveStartKey parameter of the request.
• At Line #19, we are sending the query request via the client.
• As soon as we receive a response, we first extract the LastEvaluatedKey, if it exists.
• Then we map the response items to our custom datatype, which is Product.
• With that done, we will create a new object of type PagedResult<Product>, add in the products and the LastEvaluatedKey, and return this as the response of the API. That’s it!

Let’s now navigate to the AWS Management Console and manually create a DDB table with some initial values.

I have created a new table in DynamoDB and named it products. For the composite keys, we will specify a combination of Category and ID as the partition and sort key.

Not a lot of data, but I have inserted about 10 records that we are going to fetch from our API in a paged manner. If you remember, we have set the page limit to 3.

Let’s build and run our ASP.NET Core Web API.

Open up Swagger, and send a POST request to the / endpoint. Here is the request that I am sending. Note that we are not providing any value for the paginationToken property, but setting the category property as electronics.

{
  "category": "electronics"
}

And here is the response, with 3 records as expected. Also, note that we have a value for paginationToken now. We will use this value for our next request.

{
  "items": [
    {
      "category": "electronics",
      "id": "1",
      "name": "iphone 16",
      "price": 1399
    },
    {
      "category": "electronics",
      "id": "10",
      "name": "p6 pro",
      "price": 599
    },
    {
      "category": "electronics",
      "id": "2",
      "name": "s25 ultra",
      "price": 1299
    }
  ],
  "paginationToken": "eyJjYXRlZ29yeSI6eyJCUyI6W10sIkwiOltdLCJNIjp7fSwiTlMiOltdLCJTIjoiZWxlY3Ryb25pY3MiLCJTUyI6W119LCJpZCI6eyJCUyI6W10sIkwiOltdLCJNIjp7fSwiTlMiOltdLCJTIjoiMiIsIlNTIjpbXX19"
}

Using the paginationToken from the previous response, I have formed the following request.

{
  "category": "electronics",
  "paginationToken": "eyJjYXRlZ29yeSI6eyJCUyI6W10sIkwiOltdLCJNIjp7fSwiTlMiOltdLCJTIjoiZWxlY3Ryb25pY3MiLCJTUyI6W119LCJpZCI6eyJCUyI6W10sIkwiOltdLCJNIjp7fSwiTlMiOltdLCJTIjoiMiIsIlNTIjpbXX19"
}

As you can see, we will get the next page of results from DynamoDB.

{
  "items": [
    {
      "category": "electronics",
      "id": "3",
      "name": "x40 pro",
      "price": 899
    },
    {
      "category": "electronics",
      "id": "4",
      "name": "z15 lite",
      "price": 549
    },
    {
      "category": "electronics",
      "id": "5",
      "name": "a12 mini",
      "price": 349
    }
  ],
  "paginationToken": "eyJjYXRlZ29yeSI6eyJCUyI6W10sIkwiOltdLCJNIjp7fSwiTlMiOltdLCJTIjoiZWxlY3Ryb25pY3MiLCJTUyI6W119LCJpZCI6eyJCUyI6W10sIkwiOltdLCJNIjp7fSwiTlMiOltdLCJTIjoiNSIsIlNTIjpbXX19"
}

That’s how you can implement pagination in Amazon DynamoDB using the .NET SDK.

ScanIndexForward - Reverse Sorting

As you can see from our API response, all the returned data is sorted in ascending order of the sort key, which is the product ID. In case you want to reverse the sort, you would have to introduce another property within your QueryRequest, which is the ScanIndexForward. By default, this is set to true. If you set it to false, the API will now return the data in reversed order.

var query = new QueryRequest
{
    ScanIndexForward = false,
    TableName = "products",
    Limit = 3,
    KeyConditionExpression = $"{partitionKey} = :value",
    ExpressionAttributeValues = new Dictionary<string, AttributeValue>
    {
        {":value", new AttributeValue { S = request.Category }}
    }
};

And here is the response, as expected.

{
  "items": [
    {
      "category": "electronics",
      "id": "9",
      "name": "k10 slim",
      "price": 299
    },
    {
      "category": "electronics",
      "id": "8",
      "name": "u20 plus",
      "price": 449
    },
    {
      "category": "electronics",
      "id": "7",
      "name": "q7 max",
      "price": 799
    }
  ],
  "paginationToken": "eyJjYXRlZ29yeSI6eyJCUyI6W10sIkwiOltdLCJNIjp7fSwiTlMiOltdLCJTIjoiZWxlY3Ryb25pY3MiLCJTUyI6W119LCJpZCI6eyJCUyI6W10sIkwiOltdLCJNIjp7fSwiTlMiOltdLCJTIjoiNyIsIlNTIjpbXX19"
}

Limitations

There is no straightforward/efficient approach to calculate the total count of records in a DynamoDB table. You have to use a Scan operation to get to the count, which can be very costly at times since a Scan operation retrieves all the records from the database.

Similarly, finding page numbers is also not supported directly. You will need to fetch the total number of records, with which you will be able to calculate the page numbers.

Thus, the ideal approach while using DynamoDB as your data source would be to have an infinite scroll kind of setup on your UI, where as soon as the user starts scrolling to the bottom of the first sets of data, the next paged data is loaded. This is easily achievable using the LastEvaluatedKey property that we learned about earlier. This is more efficient and budget-friendly while using Amazon DynamoDB in your .NET applications.

Filter Expressions

I also wanted to highlight the usage of FilterExpressions in DynamoDB. With FilterExpressions defined on your Query Request, all the data is fetched from the database first, and only then it is filtered in memory to return the expected response. Because filter expressions retrieve all items before filtering, they can consume more read capacity units (RCUs) than a more targeted query or scan operation. This can result in higher costs, especially for large datasets. Thus, sparingly use FilterExpressions!

That’s it for this article! I hope you enjoyed it. We went in-depth about implementing Pagination in Amazon DynamoDB with .NET SDKs using the LastEvaluatedKey, improving API performance, and lowering the cloud bill.

Let me know in the comment section which #dotnetonaws article you need next. Cheers!