Amazon SQS Queues, FIFO, Polling, and Dead-Letter Queues – SAP-C02 Study Guide

Purpose of This Lesson

Amazon Simple Queue Service (Amazon SQS) provides asynchronous communication between application components. The primary architectural benefit is decoupling: producers can submit work without requiring consumers to process it immediately.

This improves resilience during traffic bursts, allows each tier to scale independently, and helps prevent messages from being lost when a downstream application is temporarily unable to keep up.

Key Concepts

Decoupled application integration

In a direct integration, one application calls another application synchronously or sends work directly to it. The receiving application must process requests at approximately the rate they arrive. A sudden workload spike can overwhelm the consumer and result in failed or lost work.

With SQS between the applications:

1. A producer submits messages to the queue.
2. The producer can continue handling incoming work without waiting for the consumer.
3. A consumer polls the queue for messages.
4. The consumer processes messages at the rate its capacity allows.

The queue acts as a durable buffer between the two tiers. This pattern is appropriate when the work can be processed asynchronously and does not require an immediate response.

Standard queues

Standard SQS queues provide:

• Very high and nearly unlimited throughput for common queue operations.
• At-least-once delivery, meaning a message can occasionally be delivered more than once.
• Best-effort ordering, meaning messages are generally ordered but can be delivered out of sequence.

Applications using Standard queues should make message processing idempotent. Processing the same message more than once should not create an incorrect result, such as charging a customer twice or creating duplicate records.

If ordering matters, the application can include sequence information in each message and use application logic to handle ordering, or it can use an SQS FIFO queue instead.

FIFO queues

FIFO means first in, first out. FIFO queues are designed for applications that require ordered processing and stronger duplicate-handling behavior.

Important FIFO concepts include:

• Message group ID: Messages in the same group are processed in order. Message groups also allow independent groups to be processed concurrently.
• Message deduplication ID: Used to identify duplicate messages within the deduplication interval.
• Exactly-once processing behavior: FIFO deduplication prevents supported duplicate sends from introducing duplicate messages into the queue. Consumers should still be designed defensively and idempotently.

The throughput figures commonly tested for FIFO queues are up to 300 messages per second per API operation type without batching, or up to 3,000 messages per second when batching up to 10 messages per operation. Actual limits can depend on the FIFO throughput configuration and AWS Region, so verify current service quotas when designing a production system.

Dead-letter queues

A dead-letter queue (DLQ) is not a separate SQS queue type. It is a Standard or FIFO queue configured to receive messages that repeatedly fail processing.

A source queue uses a redrive policy to specify:

• The DLQ destination.
• The maximum receive count allowed before a message is moved to the DLQ.

DLQs isolate poison messages so they do not repeatedly block normal processing. Engineers can then inspect, troubleshoot, correct, and potentially replay those messages.

For reliable ordering behavior, the source queue and its DLQ should use compatible queue types. A FIFO source should use a FIFO DLQ when preserving FIFO semantics is important.

Delay queues

A delay queue postpones the visibility of newly submitted messages for a configured period. Consumers cannot retrieve those messages until the delay expires.

Delay queues are useful when work should not begin immediately, such as postponing a retry or allowing a short settling period after an event is created.

A delay queue affects messages when they are initially added. This is distinct from the visibility timeout applied after a consumer receives a message. The visibility timeout temporarily hides an in-flight message while it is being processed; a delay queue delays the message before it becomes available to consumers.

Short polling and long polling

SQS consumers retrieve messages by polling the queue.

Short polling returns immediately. It can return an empty response even when messages exist because the request checks only a subset of SQS servers. Frequent empty requests can increase API request volume and cost.

Long polling waits for messages to arrive instead of returning immediately. It is enabled when ReceiveMessageWaitTimeSeconds is greater than zero, up to a maximum of 20 seconds.

Benefits of long polling include:

• Fewer empty responses.
• Lower request volume and potentially lower cost.
• More efficient consumers, especially when queues are often empty.

A value of 0 disables long polling and results in short polling. Long polling can be configured at the queue level or for an individual ReceiveMessage request using the WaitTimeSeconds parameter.

Exam-Relevant Takeaways

• Use SQS to decouple producers and consumers and absorb temporary workload spikes.
• SQS consumers poll queues; queues do not directly push messages to consumers.
• Standard queues provide very high throughput, best-effort ordering, and at-least-once delivery.
• FIFO queues provide ordered processing and message deduplication, but have lower throughput constraints than Standard queues.
• Use message group IDs to preserve ordering within independent FIFO message groups.
• Design Standard-queue consumers to be idempotent because duplicate delivery is possible.
• A DLQ is configured through a redrive policy and receives messages after they exceed the maximum receive count.
• A DLQ is not a third queue type: it is another Standard or FIFO queue.
• Delay queues postpone initial message visibility; they are not a substitute for a visibility timeout.
• Long polling waits for messages for up to 20 seconds and can reduce empty responses and request costs.
• SQS is appropriate for asynchronous work. It does not remove the need to account for processing time, retries, failures, and message retention.

Architecture Decision Guide

Requirement	Recommended choice	Reason
Absorb bursts between application tiers	SQS Standard	Very high throughput and asynchronous buffering
Preserve strict order for related messages	SQS FIFO with a message group ID	Ordering is maintained within each message group
Avoid duplicate processing errors	Idempotent consumer logic; consider FIFO deduplication	Standard queues can deliver a message more than once
Isolate repeatedly failing messages	DLQ configured with a redrive policy	Separates poison messages for investigation
Prevent work from starting immediately	Delay queue or per-message delay	Newly sent messages remain unavailable temporarily
Reduce empty receive responses	Long polling	Waits for messages instead of returning immediately
Maximize throughput with no ordering requirement	Standard queue	Standard queues have substantially higher throughput potential
Process messages in batches	Batch SQS API operations	Reduces API calls and improves throughput efficiency

Common Exam Traps

• Assuming Standard queues guarantee order: They provide best-effort ordering only.
• Assuming SQS delivers messages exactly once: Standard queues provide at-least-once delivery, so duplicates are possible.
• Confusing FIFO ordering with global ordering: FIFO ordering is enforced within a message group. Different groups can be processed independently.
• Treating a DLQ as a queue mode: A DLQ is a configured destination queue, not a separate SQS queue type.
• Using a Standard DLQ for a FIFO source without considering ordering: Select a compatible FIFO DLQ when FIFO behavior must be preserved.
• Confusing delay with visibility timeout: Delay prevents a new message from being initially available; visibility timeout hides a message after receipt while processing occurs.
• Assuming long polling returns immediately: Long polling waits for messages or until the configured wait period ends.
• Choosing FIFO automatically for every workload: FIFO provides ordering and deduplication features but has lower throughput constraints and may be unnecessary when order is not a requirement.
• Ignoring idempotency: Even when using FIFO, robust consumers should tolerate retries and partial failures safely.

Real-World Engineer Notes

• Size consumer capacity independently from producer capacity. Queue depth and message age are useful signals for deciding when to scale consumers.
• Monitor DLQ message counts and investigate them rather than treating the DLQ as permanent storage.
• Make processing idempotent using a business identifier, a deduplication record, or a conditional write in the target data store.
• Use long polling for consumers that frequently encounter empty queues, but ensure the client timeout is longer than the SQS wait time.
• Batch operations can improve efficiency, but the application must handle partial batch failures correctly.
• FIFO message groups can become a bottleneck if too many messages are assigned to one group. Use multiple groups when ordering is required only for related entities.
• SQS decouples availability and processing speed, but it does not make work instantaneous. Set operational expectations around queue delay and backlog recovery.

Quick Reference Summary

• Purpose: Asynchronous, decoupled communication and workload buffering.
• Standard: High throughput, best-effort ordering, at-least-once delivery.
• FIFO: Ordered processing within message groups and deduplication support.
• DLQ: Captures messages that exceed the configured receive-count threshold.
• Delay queue: Delays initial message visibility.
• Short polling: Returns immediately and may produce empty responses.
• Long polling: Waits up to 20 seconds for messages and reduces unnecessary requests.
• Core design rule: Make consumers idempotent and monitor failures, retries, queue depth, and message age.

Flashcards

1. Q: What problem does SQS primarily solve?

A: It decouples producers and consumers and buffers work during traffic spikes or temporary consumer failures.

1. Q: Does an SQS queue push messages to an application?

A: No. Consumers poll the queue for messages.

1. Q: What ordering guarantee does a Standard queue provide?

A: Best-effort ordering; messages can be delivered out of order.

1. Q: What delivery model does a Standard queue use?

A: At-least-once delivery, so duplicate delivery is possible.

1. Q: When should a FIFO queue be selected?

A: When strict ordering and FIFO deduplication behavior are required.

1. Q: What is a message group ID used for?

A: It groups related FIFO messages whose processing order must be preserved.

1. Q: What is a message deduplication ID used for?

A: It helps identify duplicate FIFO messages within the deduplication interval.

1. Q: Is a dead-letter queue a distinct SQS queue type?

A: No. It is a Standard or FIFO queue configured as the destination for repeatedly failed messages.

1. Q: What controls when a message is moved to a DLQ?

A: The source queue’s redrive policy and maximum receive count.

1. Q: What does a delay queue do?

A: It makes newly submitted messages unavailable until the configured delay expires.

1. Q: What is the maximum SQS long-polling wait time?

A: 20 seconds.

1. Q: What happens when ReceiveMessageWaitTimeSeconds is set to zero?

A: Long polling is disabled and short polling is used.

1. Q: Why should Standard-queue consumers be idempotent?

A: A message can occasionally be delivered more than once.

Practice Questions

Question 1

A web application receives a sudden surge of orders. The order-processing service cannot process all orders immediately, but orders must not be lost. The business does not require strict ordering. Which architecture is most appropriate?

A. Send each order directly from the web tier to the processing service
B. Place an SQS Standard queue between the web tier and processing service
C. Use an SQS FIFO queue exclusively because all orders are business-critical
D. Use short polling and discard messages when the consumer is busy

Correct answer: B

Explanation: An SQS Standard queue decouples the tiers and buffers the burst while consumers process messages asynchronously. FIFO is unnecessary when strict ordering is not required, and messages should not be discarded when the consumer is busy.

Question 2

A payment workflow requires related operations to be processed in order. It also needs protection against duplicate messages submitted by a producer. Which SQS design best meets the requirements?

A. Standard queue with short polling
B. Standard queue with a delay setting
C. FIFO queue using an appropriate message group ID and deduplication ID
D. Standard queue with a DLQ only

Correct answer: C

Explanation: FIFO queues provide ordered processing within a message group and support deduplication using deduplication IDs. A DLQ handles failures but does not provide ordering or deduplication by itself.

Question 3

A consumer frequently polls an almost-empty queue. The application makes many receive requests that return no messages, increasing request costs. Which change should an architect recommend?

A. Enable long polling with a wait time greater than zero
B. Configure a DLQ
C. Convert the queue to FIFO
D. Add a message group ID

Correct answer: A

Explanation: Long polling waits for messages to arrive for up to 20 seconds, reducing empty responses and unnecessary receive requests. The other options address different requirements.

Question 4

Messages in an SQS queue repeatedly fail because of malformed input. The consumer continues receiving the same messages, preventing normal work from progressing. What is the best solution?

A. Configure a redrive policy with a DLQ and an appropriate maximum receive count
B. Enable short polling
C. Increase the producer’s send rate
D. Change the queue to Standard without configuring a DLQ

Correct answer: A

Explanation: A DLQ isolates messages that exceed the configured receive count. Engineers can inspect and remediate malformed messages without allowing them to remain in the main processing flow indefinitely.

Question 5

An architect chooses an SQS Standard queue for a workload that can receive duplicate deliveries. Which application design is required?

A. The consumer must process every message synchronously before accepting another
B. The consumer must be idempotent
C. The producer must use a FIFO message group ID
D. The queue must use a 20-second delay

Correct answer: B

Explanation: Standard queues provide at-least-once delivery. Idempotent processing ensures that a duplicate message does not create duplicate business effects.