Build a Simple Event-Driven Application with Amazon SQS, Lambda, and DynamoDB – SAP-C02 Study Guide

Purpose of This Lesson

This lesson demonstrates a basic event-driven serverless workflow:

1. A message is sent to an Amazon SQS queue.
2. An AWS Lambda function is triggered by messages in the queue.
3. Lambda processes the message contents.
4. The processed data is written to an Amazon DynamoDB table.
5. Lambda execution details are recorded in Amazon CloudWatch Logs.

The pattern separates message producers from message consumers. A producer can place work on the queue without needing to invoke Lambda directly or wait for processing to complete.

Key Concepts

Event flow

The architecture is:

Producer or AWS CLI -> Amazon SQS Standard Queue -> AWS Lambda -> Amazon DynamoDB

The SQS queue acts as a durable buffer between the producer and the processing function. Lambda polls the queue through an event source mapping and invokes the function when messages are available.

This design is useful when:

• Work can be processed asynchronously.
• The producer and consumer should be loosely coupled.
• Processing demand may vary over time.
• Temporary downstream failures should not immediately cause the producer to fail.

Amazon SQS Standard Queue

A Standard Queue provides highly scalable message processing and at-least-once delivery. Messages may occasionally be delivered more than once and ordering is not guaranteed.

The consumer must therefore be designed for duplicate delivery. In this example, writing records to DynamoDB should be idempotent or use a suitable conditional-write strategy if duplicate processing would be harmful.

Important SQS behavior includes:

• Messages remain available until successfully processed or otherwise removed.
• A visibility timeout temporarily hides a message while Lambda processes it.
• If processing fails or the visibility timeout expires, the message can become visible again.
• A dead-letter queue can isolate messages that repeatedly fail.
• Batch size controls how many messages Lambda receives in one invocation.

AWS Lambda as an SQS consumer

Lambda is configured with an SQS trigger. The trigger is implemented using an event source mapping that polls the queue and invokes the function with one or more messages.

The function needs permissions to:

• Read messages from SQS.
• Delete messages after successful processing.
• Read queue attributes.
• Write logs to CloudWatch Logs.
• Write items to DynamoDB.

For production designs, prefer a least-privilege execution role that grants access only to the required queue, table, and log operations. Broad managed policies can simplify a lab but are not normally the best security choice.

DynamoDB table design

The demonstration creates a DynamoDB table with a table name and partition key. The Lambda code uses those values when inserting the message data.

The partition key is part of the table’s data model. It must match the attribute name expected by the application. A mismatch between the configured key and the Lambda code can cause failed writes even when the AWS resources themselves are healthy.

DynamoDB is appropriate for this workflow when the application needs:

• Low-latency key-value or document access.
• Serverless capacity management.
• High availability across multiple Availability Zones.
• Flexible scaling without managing database servers.

CloudWatch observability

Lambda automatically records invocation output and runtime errors in CloudWatch Logs when its execution role allows log delivery. Logs are the first place to inspect when:

• The function returns an exception.
• A DynamoDB write fails.
• The function cannot access SQS or DynamoDB.
• The message payload does not match the expected format.

For a production implementation, add metrics and alarms for Lambda errors, throttles, duration, SQS queue depth, age of the oldest message, and dead-letter queue activity.

Configuration accuracy

The sample function uses configuration values such as the DynamoDB table name, partition key, and AWS Region. Hard-coding these values is acceptable for a short demonstration, but production applications should generally use environment variables, parameter stores, or deployment-time configuration.

The configured Region must align with the resources being accessed. Cross-Region access is possible in some designs but adds latency, cost, IAM complexity, and potential data-transfer considerations.

Architecture Decision Guide

Requirement	Recommended choice	Reason or tradeoff
Asynchronous decoupling between producer and consumer	Amazon SQS	Durable buffering and independent scaling
Very high throughput with no ordering requirement	SQS Standard Queue	High scale, but at-least-once delivery and possible reordering
Strict ordering for related messages	SQS FIFO Queue	Ordered processing and deduplication features, with different throughput characteristics
Serverless message processing	Lambda with an SQS event source mapping	No server management; concurrency follows queue demand
Key-value or document persistence	DynamoDB	Managed, highly available, low-latency storage
Repeatedly failing messages	SQS dead-letter queue	Separates poison messages for investigation or reprocessing
Direct synchronous client request	API Gateway plus Lambda	Use when the caller needs an immediate response rather than queue-based processing
Guaranteed idempotent writes	DynamoDB conditional writes or idempotency keys	Protects against duplicate SQS deliveries
Fine-grained application permissions	IAM customer-managed or inline policy	Reduces the blast radius compared with broad managed policies

Exam-Relevant Takeaways

• SQS and Lambda are commonly combined for asynchronous, event-driven processing.
• Lambda does not need to be publicly exposed for SQS to invoke it.
• The SQS-to-Lambda integration polls the queue and invokes the function in batches.
• SQS Standard Queues provide at-least-once delivery, so duplicate processing is possible.
• Consumers should be idempotent, especially when writing to a database.
• The visibility timeout should be long enough for the Lambda function to finish processing. If it is too short, the same message may be delivered again while the original invocation is still running.
• Configure a dead-letter queue when failed messages require isolation and operational review.
• Lambda’s execution role must include permissions for SQS consumption, DynamoDB writes, and CloudWatch Logs.
• DynamoDB’s partition-key attribute name must match the application’s expected schema.
• CloudWatch Logs help diagnose runtime errors, authorization failures, malformed payloads, and database write failures.
• A Standard Queue does not provide ordering. Choose FIFO when ordering and deduplication are explicit requirements.
• Queue-based designs absorb bursts better than direct synchronous invocation, but they introduce processing delay and eventual consistency from the producer’s perspective.

Common Exam Traps

• Assuming SQS Standard preserves order: It does not. Use FIFO when ordering is required.
• Assuming one message always equals one Lambda invocation: Lambda can receive a batch of messages in one invocation.
• Ignoring duplicate delivery: At-least-once delivery requires idempotent processing.
• Setting visibility timeout too low: Messages can reappear before processing completes, causing duplicates.
• Using broad permissions as the final design: A lab policy may work, but production architecture should restrict actions and resources.
• Expecting SQS to invoke Lambda through a public endpoint: Lambda polls SQS using the managed integration; no public API endpoint is required.
• Forgetting partial batch failure behavior: If one message in a batch fails and failure handling is not configured appropriately, successfully processed messages may be retried. Design batch processing and failure reporting carefully.
• Confusing an SQS trigger with SNS fanout: SQS is a durable queue and consumer buffer. SNS is a pub/sub notification service; SNS can publish to multiple SQS queues for fanout.
• Using a queue when the caller needs an immediate result: SQS introduces asynchronous processing. Use a synchronous API pattern when the client must receive the result during the request.

Real-World Engineer Notes

• Use environment variables or external configuration for table names, queue URLs, Regions, and deployment-specific settings.
• Prefer infrastructure as code, such as AWS CloudFormation, AWS CDK, or Terraform, instead of manually creating resources.
• Define a dead-letter queue and monitor it with CloudWatch alarms.
• Set Lambda reserved concurrency when protecting downstream systems such as DynamoDB or an external API.
• Tune SQS batch size, Lambda timeout, and visibility timeout together. The visibility timeout should exceed the expected processing time, including retry considerations.
• Use structured logs with message identifiers, correlation IDs, and processing outcomes. Avoid logging sensitive payload data.
• Consider DynamoDB conditional writes or a processed-event table when duplicate messages could create incorrect records.
• Apply encryption requirements appropriately. SQS and DynamoDB support encryption at rest, and customer-managed AWS KMS keys may be required for specific compliance or key-control requirements.
• Deleting resources after a hands-on exercise prevents unnecessary charges and reduces account clutter. In a production environment, deletion must be controlled through change management and retention policies.

Quick Reference Summary

• Producer: Sends a JSON message to SQS, using the queue URL.
• Buffer: SQS stores messages until Lambda successfully processes them.
• Consumer: Lambda is triggered through an SQS event source mapping.
• Database: DynamoDB stores the processed product information.
• Monitoring: CloudWatch Logs capture Lambda execution details.
• Delivery model: Standard SQS is at least once and does not guarantee order.
• Reliability requirement: Make the Lambda consumer idempotent.
• Security requirement: Grant the Lambda execution role only the required SQS, DynamoDB, and logging permissions.
• Failure handling: Use visibility-timeout tuning, retries, and a dead-letter queue.

Flashcards

1. What AWS service buffers messages in this architecture?

Amazon SQS.

2. How does Lambda receive messages from SQS?

An SQS event source mapping polls the queue and invokes the Lambda function with one or more messages.

3. What delivery guarantee does an SQS Standard Queue provide?

At-least-once delivery. Duplicate messages are possible, and ordering is not guaranteed.

4. Why should an SQS consumer be idempotent?

The same message may be delivered more than once, and repeated processing should not create incorrect duplicate effects.

5. Which SQS queue type supports ordered processing?

An SQS FIFO Queue.

6. What happens when a message’s visibility timeout expires before processing succeeds?

The message becomes visible again and may be delivered for another processing attempt.

7. What database service stores the processed records in this pattern?

Amazon DynamoDB.

8. Which Lambda role capability is required for application logging?

Permission to create log groups and streams and publish log events to CloudWatch Logs.

9. What is the purpose of an SQS dead-letter queue?

To isolate messages that fail repeatedly so they can be investigated or handled separately.

10. Why must the DynamoDB partition-key name match the Lambda code?

DynamoDB item writes must use the table’s defined key schema. A mismatched key attribute causes the write to fail.

Practice Questions

Question 1

A company uses an SQS Standard Queue to trigger a Lambda function that inserts order records into DynamoDB. Occasionally, the same order is stored twice. Which change best addresses the root cause?

A. Increase the Lambda memory allocation only
B. Make the Lambda processing idempotent using an order ID and conditional write
C. Replace DynamoDB with Amazon RDS
D. Increase the SQS message retention period

Correct answer: B

SQS Standard provides at-least-once delivery, so duplicate processing is expected to be possible. An idempotency key and DynamoDB conditional write can prevent duplicate effects.

Question 2

A Lambda function processing SQS messages often receives the same message while the first invocation is still running. What is the most likely configuration problem?

A. The SQS visibility timeout is too short
B. The DynamoDB table has a partition key
C. CloudWatch Logs is enabled
D. The queue is encrypted

Correct answer: A

If the visibility timeout expires before processing finishes, SQS makes the message visible again. Set the timeout appropriately relative to the Lambda timeout and expected processing duration.

Question 3

An application must process related messages in exact order and avoid duplicate deliveries where possible. Which architecture is most appropriate?

A. SQS Standard Queue and Lambda
B. SQS FIFO Queue and Lambda
C. Amazon SNS Standard Topic and Lambda
D. Amazon EventBridge Scheduler and DynamoDB

Correct answer: B

SQS FIFO supports ordered processing and message deduplication features. The consumer should still be designed defensively because application-level retries and failures must be handled correctly.

Question 4

A Lambda function triggered by SQS fails with an AccessDenied error when inserting records into DynamoDB. The queue trigger is configured correctly. Which action should be taken first?

A. Make the DynamoDB table publicly accessible
B. Add the required DynamoDB write permission to the Lambda execution role
C. Add an API Gateway endpoint
D. Change the queue to FIFO

Correct answer: B

Lambda uses its execution role to call AWS services. The role must grant the required DynamoDB actions on the target table, in addition to permissions for SQS consumption and CloudWatch logging.

Question 5

A producer submits work in bursts, and the business does not require an immediate response. The downstream database must not be overwhelmed during spikes. Which design best meets the requirements?

A. Invoke Lambda synchronously for every request
B. Place requests on SQS and use controlled Lambda concurrency to process them
C. Expose DynamoDB directly to the clients
D. Send all requests directly to the database from the producer

Correct answer: B

SQS provides durable buffering, while Lambda concurrency controls can regulate the processing rate and protect the downstream database. The tradeoff is that processing becomes asynchronous and may introduce queueing delay.