DynamoDB Streams: Change Data Capture and Lambda Integration – SAP-C02 Study Guide

Purpose of This Lesson

DynamoDB Streams provides a change data capture mechanism for Amazon DynamoDB. It records item-level changes made to a table so downstream consumers—especially AWS Lambda—can react to inserts, updates, and deletes.

This pattern is useful when an application needs to perform asynchronous processing after a DynamoDB item changes, without placing that processing directly in the request path.

Key Concepts

What DynamoDB Streams Captures

When an item is inserted, updated, or deleted in a DynamoDB table, DynamoDB Streams can capture a corresponding stream record. A record represents the modification and can include selected versions of the item.

Typical uses include:

• Triggering an AWS Lambda function after a database change
• Maintaining derived or denormalized data
• Sending notifications or events to downstream systems
• Auditing changes for a limited period
• Starting asynchronous workflows
• Updating search indexes or caches

DynamoDB Streams is enabled at the table level. A consumer retrieves the stream records and processes them independently of the application that performed the write.

Stream Record Retention

DynamoDB Streams retains records for 24 hours. Consumers must process the records within this retention window if the data is required for downstream processing.

This makes DynamoDB Streams suitable for near-real-time event processing, but not as a long-term event archive. If durable historical retention is required, a consumer should copy the events to another service such as Amazon S3, Amazon Kinesis Data Streams, or a database designed for that purpose.

Stream View Types

The stream view type determines what information is included in each record:

Stream view type	Data captured	Typical use
KEYS_ONLY	Partition key and sort key, if present	Identify the changed item and retrieve current data separately
NEW_IMAGE	Complete item after the modification	Process the newly written state
OLD_IMAGE	Complete item before the modification	Compare or react to the previous state
NEW_AND_OLD_IMAGES	Complete item before and after the modification	Calculate differences, audit changes, or process transitions

Choosing a smaller view type reduces the amount of data delivered to consumers. Choose the view based on the event-processing requirement rather than automatically capturing both images.

Lambda Integration

AWS Lambda can consume DynamoDB Streams through an event source mapping. Lambda polls the stream and invokes the function with batches of records.

The processing flow is:

1. An application inserts, updates, or deletes an item in DynamoDB.
2. DynamoDB writes a change record to the table’s stream.
3. Lambda polls the stream through an event source mapping.
4. Lambda receives one or more records in an event batch.
5. The function processes the changes and performs downstream actions.

This is asynchronous processing. The original DynamoDB write does not need to wait for the Lambda function to complete.

Ordering and Processing Considerations

DynamoDB Streams provides a time-ordered sequence of item modifications. Consumers should still be designed for retries and duplicate processing because event-driven systems commonly provide at-least-once processing behavior.

Lambda event source mappings support controls such as batch size and starting position. For reliable applications, functions should be idempotent so that processing the same change more than once does not produce an incorrect result.

Exam-Relevant Takeaways

• DynamoDB Streams captures inserts, updates, and deletes at the item level.
• Stream records are retained for 24 hours.
• The four stream view types are KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, and NEW_AND_OLD_IMAGES.
• AWS Lambda can be triggered by DynamoDB Streams using an event source mapping.
• DynamoDB Streams is designed for change processing and integration, not long-term event storage.
• Use NEW_AND_OLD_IMAGES when processing requires both the previous and resulting item states.
• Use KEYS_ONLY when the consumer only needs to identify the modified item and can retrieve additional data separately.
• Downstream processing should account for retries and should be idempotent.
• Stream processing is asynchronous and can decouple the database write path from secondary actions.

Architecture Decision Guide

Requirement	Recommended approach	Reason
Run code after a DynamoDB item changes	DynamoDB Streams with AWS Lambda	Native event-driven integration
Identify which item changed without carrying the full item	KEYS_ONLY	Minimizes stream record contents
Process the state after a write	NEW_IMAGE	Provides the complete resulting item
Process the state before deletion or update	OLD_IMAGE	Provides the previous item contents
Compare an item’s previous and current values	NEW_AND_OLD_IMAGES	Provides both versions
Retain change events beyond 24 hours	Copy records to S3, Kinesis Data Streams, or another durable store	DynamoDB Streams is not a long-term archive
Perform slow or independent downstream work	Streams plus asynchronous Lambda processing	Keeps the application write path decoupled
Guarantee safe repeated processing	Idempotent Lambda logic and appropriate deduplication design	Consumers may retry records

Common Exam Traps

• Confusing DynamoDB Streams with Kinesis Data Streams: They are different services. DynamoDB Streams is tightly integrated with table item changes and has a 24-hour retention period.
• Assuming Streams stores events indefinitely: It does not. Events must be exported or processed before the 24-hour retention window expires.
• Selecting the wrong image type: NEW_IMAGE does not include the previous item, and OLD_IMAGE does not include the resulting item. Both are required for before-and-after comparisons.
• Assuming the stream contains only updates: Inserts, updates, and deletes can all generate records.
• Treating Lambda processing as synchronous: The DynamoDB write succeeds independently of the downstream Lambda execution.
• Ignoring retries: Lambda-based stream consumers should tolerate repeated processing.
• Using DynamoDB Streams as a general event bus: It is specifically associated with item changes in a DynamoDB table. For arbitrary application events, consider services such as Amazon EventBridge or Amazon Kinesis Data Streams.

Real-World Engineer Notes

• Select the minimum stream view type that satisfies the consumer’s requirements. Capturing full old and new images can increase event payload size and processing work.
• Make Lambda handlers idempotent. For example, use a deterministic operation identifier or conditionally write derived records so retries do not create duplicate effects.
• Monitor Lambda errors, throttling, iterator age, and batch failures. A growing iterator age indicates that the consumer is falling behind the stream.
• If downstream processing is business-critical, define how failures are handled and how records are replayed or recovered within the retention window.
• For long-term audit or analytics, forward stream records to durable storage rather than relying on the 24-hour stream buffer.
• Keep the original DynamoDB transaction focused on the authoritative write. Use the stream for secondary actions that do not need to block the request.

Quick Reference Summary

• Source: Amazon DynamoDB table
• Events: Item inserts, updates, and deletes
• Consumer: AWS Lambda or another supported stream consumer
• Retention: 24 hours
• Integration model: Asynchronous change data capture
• View types: KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, NEW_AND_OLD_IMAGES
• Best practice: Design consumers to be idempotent and monitor processing lag
• Not suitable for: Long-term event retention or arbitrary application event distribution

Flashcards

1. Q: What does DynamoDB Streams capture?

A: A time-ordered sequence of item-level changes— inserts, updates, and deletes—made to a DynamoDB table.

1. Q: How long are DynamoDB Stream records retained?

A: 24 hours.

1. Q: Which stream view type includes only the item’s key attributes?

A: KEYS_ONLY.

1. Q: Which view type provides the complete item after a modification?

A: NEW_IMAGE.

1. Q: Which view type provides the complete item before a modification?

A: OLD_IMAGE.

1. Q: Which view type is appropriate for comparing an item’s previous and current states?

A: NEW_AND_OLD_IMAGES.

1. Q: How does Lambda consume a DynamoDB Stream?

A: Through an event source mapping that polls the stream and invokes the function with batches of records.

1. Q: Is DynamoDB Streams a long-term event archive?

A: No. Records are retained for only 24 hours and should be copied elsewhere if longer retention is required.

1. Q: Why should a Lambda consumer be idempotent?

A: Stream records may be retried or processed more than once, so repeated processing must not produce incorrect results.

1. Q: What is a common use for DynamoDB Streams?

A: Triggering asynchronous downstream processing, such as updating a derived record, sending a notification, or maintaining an index.

Practice Questions

Question 1

A company stores customer orders in DynamoDB. Whenever an order changes, it must asynchronously update a separate summary table. The update logic should not increase the latency of the order-write API. Which architecture best meets the requirement?

A. Place the summary-table update directly in the application transaction before returning the response
B. Enable DynamoDB Streams and configure an AWS Lambda event source mapping
C. Schedule an Amazon EventBridge rule to scan the DynamoDB table every hour
D. Export the entire DynamoDB table to Amazon S3 after every write

Correct answer: B

Explanation: DynamoDB Streams captures item changes and Lambda can process them asynchronously. The other choices either add request latency, introduce unnecessary delay, or are inefficient for near-real-time item change processing.

Question 2

A Lambda function must calculate the difference between an item’s previous value and its new value whenever the item is updated. Which DynamoDB Stream configuration is required?

A. KEYS_ONLY
B. NEW_IMAGE
C. OLD_IMAGE
D. NEW_AND_OLD_IMAGES

Correct answer: D

Explanation: Calculating a before-and-after difference requires both the complete previous item and the complete resulting item.

Question 3

A solution uses DynamoDB Streams to feed a compliance archive. The organization must retain all change records for seven years. What should the solutions architect recommend?

A. Increase the DynamoDB Streams retention period to seven years
B. Use KEYS_ONLY and assume the records remain available indefinitely
C. Configure a consumer to write stream records to durable long-term storage
D. Disable stream expiration by increasing the DynamoDB table’s provisioned capacity

Correct answer: C

Explanation: DynamoDB Streams retains records for 24 hours. A consumer must copy the records to a durable destination such as Amazon S3 for long-term retention.

Question 4

A DynamoDB table receives frequent updates. A downstream process only needs to know which item changed and can retrieve the current item separately. Which stream view type minimizes the information included in each stream record?

A. KEYS_ONLY
B. NEW_IMAGE
C. OLD_IMAGE
D. NEW_AND_OLD_IMAGES

Correct answer: A

Explanation: KEYS_ONLY includes only the item’s key attributes, minimizing the stream record contents while still identifying the changed item.

Question 5

A Lambda function processes DynamoDB Stream records by charging a customer for each order event. Occasionally, the same stream record is delivered again after a retry. What is the most important design requirement?

A. Make the Lambda function idempotent
B. Use OLD_IMAGE instead of NEW_IMAGE
C. Increase the DynamoDB table’s read capacity
D. Disable Lambda retries

Correct answer: A

Explanation: Idempotent processing prevents duplicate charges or other incorrect side effects when a record is retried. Changing the stream image type or table read capacity does not solve duplicate side effects.