Amazon CloudWatch Logs: Collection, Retention, Encryption, and Destinations – SAP-C02 Study Guide

Purpose of This Lesson

Amazon CloudWatch Logs provides centralized collection and management for application, operating system, and custom log data. It can receive logs from AWS services, EC2 instances, and on-premises servers, then retain, encrypt, filter, process, and route that data to other AWS services.

For architecture and certification scenarios, the key decisions are how logs reach CloudWatch, how long they should be retained, how they are protected, and whether they need real-time processing or batch export.

Key Concepts

CloudWatch Logs structure

CloudWatch Logs organizes data into:

• Log groups: Containers for related logs, usually representing an application, service, or environment.
• Log streams: Sequences of log events from an individual source, such as an EC2 instance, container, or application process.
• Log events: Individual records containing a timestamp and message.

A log group can have a retention policy, encryption configuration, metric filters, subscription filters, and access policies applied to it.

Log collection sources

CloudWatch Logs can receive data from several types of sources:

• Amazon EC2 instances: Use the unified CloudWatch agent to collect application logs, system logs, and custom metrics.
• On-premises servers: The same unified CloudWatch agent can send selected logs and metrics to AWS.
• AWS Lambda: Lambda automatically integrates with CloudWatch Logs when its execution role permits log delivery.
• Other AWS services: Many AWS services can publish logs directly to CloudWatch Logs when the service supports that integration and the required permissions are configured.

The unified CloudWatch agent is especially useful when both log files and host-level metrics are required. It can collect information such as memory utilization, disk usage, application log files, and operating system logs.

Permissions are required

A service or agent must be authorized to publish logs or metrics. For example:

• An EC2 instance using the CloudWatch agent needs an IAM role with permissions to publish metrics and log events.
• A Lambda function needs permissions such as logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents, typically through its execution role.
• A principal exporting or reading logs needs the appropriate CloudWatch Logs permissions.
• A subscription destination may require a resource policy or service-specific permissions, depending on the target and configuration.

A correctly installed agent does not guarantee successful delivery if its IAM role, network access, or configuration is incorrect.

Retention and expiration

CloudWatch Logs supports configurable retention periods for log groups. This allows organizations to balance operational needs, compliance requirements, and storage cost.

When a retention period is configured, log events older than that period are removed according to the service’s retention behavior. If no expiration policy is configured, logs are retained indefinitely, which can increase cost and create unnecessary compliance exposure.

Retention should be selected based on the purpose of the logs:

• Short retention for high-volume troubleshooting logs.
• Longer retention for audit, security, or regulatory records.
• Export or archive to Amazon S3 when low-cost, long-term retention is required.

Encryption with AWS KMS

CloudWatch Logs encrypts log data at rest. A log group can also be associated with a customer managed AWS KMS key when additional control over key policies, key rotation, or cross-account governance is required.

Using a customer managed key introduces additional configuration requirements:

• The key must be available in the appropriate Region.
• KMS key policies must allow CloudWatch Logs and authorized administrators to use the key.
• Deleting or disabling the key can make encrypted log data inaccessible.

Encryption does not replace IAM authorization. Users and services still need CloudWatch Logs permissions to read, write, filter, or export data.

Exporting logs and subscription filters

CloudWatch Logs supports multiple ways to send log data elsewhere.

#### Export to Amazon S3

Logs can be exported to Amazon S3 for archival, analysis, or integration with data platforms. This is generally a batch-oriented approach rather than a real-time streaming mechanism.

Amazon S3 is often the better destination when the requirements emphasize:

• Long-term retention.
• Low storage cost.
• Data lake integration.
• Centralized archival across accounts or Regions.
• Later analysis with services such as Amazon Athena.

#### Subscription filters

Subscription filters continuously match log events and deliver matching data to a supported destination. They are appropriate when log processing must occur with low latency.

Common destinations include:

• Amazon Kinesis Data Streams: Useful when custom consumers need to process logs from a stream.
• Amazon Data Firehose: Useful for managed delivery to destinations such as Amazon S3, Amazon OpenSearch Service, or other supported endpoints.
• AWS Lambda: Useful for custom real-time processing, alert enrichment, transformation, or routing.
• Amazon OpenSearch Service: Useful for searchable operational log analytics and dashboards.

A subscription filter is different from a historical export. Use a subscription for ongoing delivery of new matching events; use export for moving existing log data in a batch-oriented workflow.

CloudWatch Logs Insights and metric filters

CloudWatch Logs can be queried with CloudWatch Logs Insights for interactive investigation. Metric filters can turn matching log patterns into CloudWatch metrics, which can then drive alarms or dashboards.

For example, an application can create a metric from occurrences of an ERROR pattern and alarm when the count exceeds a threshold. This avoids requiring every application event to be emitted as a separate custom metric.

Exam-Relevant Takeaways

• Use the unified CloudWatch agent to collect log files and host-level metrics from EC2 or on-premises servers.
• CloudWatch Logs requires correct IAM permissions for publishing, reading, filtering, and exporting data.
• Configure log retention deliberately. No expiration policy can result in indefinite retention and unnecessary cost.
• Use a customer managed KMS key when explicit key ownership, policy control, or governance is required.
• Use Amazon S3 for durable, economical archival and later analysis.
• Use subscription filters for near-real-time delivery to Lambda, Kinesis Data Streams, Kinesis Data Firehose, or OpenSearch Service.
• Do not confuse a historical export with a streaming subscription.
• A Lambda function’s execution role commonly needs CloudWatch Logs permissions so the function can create log streams and publish log events.
• OpenSearch Service is appropriate for indexed, searchable log analytics; S3 is generally better for inexpensive long-term storage.

Architecture Decision Guide

Requirement	Recommended approach	Reason
Collect application and OS logs from EC2	Install and configure the unified CloudWatch agent	Supports file-based logs and host metrics
Collect logs from on-premises servers	Install the unified CloudWatch agent with appropriate AWS credentials and connectivity	Extends centralized logging to hybrid infrastructure
Store logs for a limited troubleshooting period	Configure a CloudWatch Logs retention policy	Controls cost and removes obsolete data
Retain logs for years at low cost	Export or stream logs to Amazon S3	S3 is suited to durable archival and data lake use cases
Process matching events with custom code	Use a CloudWatch Logs subscription filter to AWS Lambda	Enables event-driven processing
Fan out logs to custom stream consumers	Use Kinesis Data Streams	Provides a stream that consumers can read and process
Deliver logs through a managed pipeline	Use Kinesis Data Firehose	Reduces the need to build and operate delivery consumers
Search and visualize operational logs	Use a subscription filter to Amazon OpenSearch Service	Supports indexed search and dashboards
Trigger an alarm from a recurring log pattern	Use a CloudWatch Logs metric filter and CloudWatch alarm	Converts matching log events into measurable metrics
Control encryption keys independently	Associate the log group with a customer managed KMS key	Provides customer control over key policy and lifecycle

Common Exam Traps

• Assuming the CloudWatch agent is only for metrics: The unified agent can collect both metrics and log files.
• Forgetting IAM permissions: Installing an agent or deploying a Lambda function does not automatically solve authorization problems.
• Using S3 export for real-time processing: Export is intended for moving log data in a batch-oriented manner. Use subscription filters for ongoing low-latency delivery.
• Choosing OpenSearch for all retention needs: OpenSearch is useful for indexed search and analysis but can be more expensive and operationally involved than S3 for long-term archives.
• Confusing encryption with access control: KMS protects data encryption keys; IAM and resource policies control who can publish or read logs.
• Leaving retention indefinite unintentionally: A log group without an appropriate retention policy can accumulate large volumes of data.
• Ignoring the destination’s permissions: Streaming to Lambda, Kinesis, Firehose, or OpenSearch may require permissions and destination configuration beyond the source service’s setup.
• Treating logs and metrics as interchangeable: Logs provide detailed event context; metrics are aggregated numerical observations used efficiently for alarms and dashboards.

Real-World Engineer Notes

• Define log group naming, retention, tagging, and KMS standards through infrastructure as code so environments remain consistent.
• Separate log groups by application and environment when different teams, retention periods, or access policies apply.
• Use subscription filters selectively. Streaming every verbose debug event can increase downstream processing and storage costs.
• Design for delivery failures. Monitor agent health, subscription delivery errors, throttling, and destination availability.
• Keep sensitive data out of logs where possible. CloudWatch Logs is not a substitute for application-level data classification and redaction.
• Consider a two-tier design: short-term searchable logs in CloudWatch or OpenSearch, with longer-term records archived in S3.
• For hybrid collection, ensure on-premises hosts can reach the required AWS endpoints and that credentials are managed securely.
• Use metric filters for simple known patterns, but use Logs Insights or a dedicated stream-processing pipeline for more complex analysis.

Quick Reference Summary

• Collection: Unified CloudWatch agent for EC2 and on-premises logs and metrics; supported AWS services such as Lambda can publish directly.
• Organization: Log groups contain log streams, which contain log events.
• Retention: Configure expiration per log group to control cost and meet governance requirements.
• Encryption: CloudWatch Logs encrypts data at rest; customer managed KMS keys provide additional key control.
• Batch destination: Export logs to Amazon S3.
• Streaming destinations: Use subscription filters with Lambda, Kinesis Data Streams, Kinesis Data Firehose, or OpenSearch Service.
• Permissions: IAM roles and policies are required for publishing and accessing logs.
• Detection: Metric filters can convert log patterns into CloudWatch metrics and alarms.

Flashcards

1. Q: Which agent collects both logs and host metrics from EC2 and on-premises servers?

A: The unified CloudWatch agent.

1. Q: What is the difference between a log group and a log stream?

A: A log group is a container for related logs; a log stream is a sequence of log events from a specific source.

1. Q: Why configure a CloudWatch Logs retention policy?

A: To remove older events automatically, control cost, and meet data lifecycle requirements.

1. Q: Which destination is generally best for low-cost, long-term log archival?

A: Amazon S3.

1. Q: Which CloudWatch Logs feature delivers new matching events with low latency?

A: A subscription filter.

1. Q: Which destination is useful for custom real-time log processing?

A: AWS Lambda or Kinesis Data Streams, depending on the processing architecture.

1. Q: What service is commonly used for indexed log search and visualization?

A: Amazon OpenSearch Service.

1. Q: What type of permission does a Lambda function commonly need to write execution logs?

A: CloudWatch Logs permissions such as logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents.

1. Q: Does configuring a KMS key automatically grant users permission to read logs?

A: No. KMS controls encryption-key use; IAM and resource policies still control CloudWatch Logs access.

1. Q: How can a repeated error pattern become an alarmable CloudWatch metric?

A: Create a CloudWatch Logs metric filter and attach a CloudWatch alarm to the resulting metric.

Practice Questions

Question 1

A company runs applications on EC2 instances and on-premises servers. It needs centralized collection of application logs, operating system logs, memory metrics, and disk metrics in CloudWatch. Which solution best meets the requirement?

A. Install the unified CloudWatch agent on each server and assign the required IAM permissions
B. Configure VPC Flow Logs on each server and send them to CloudWatch Logs
C. Use CloudTrail to collect operating system and application logs
D. Export EC2 console logs to Amazon S3 and create CloudWatch metrics from the objects

Correct answer: A

Explanation: The unified CloudWatch agent supports both log-file collection and custom or host-level metrics on EC2 and on-premises servers. VPC Flow Logs and CloudTrail provide different types of data and do not collect general operating system or application logs.

Question 2

A security team requires continuous processing of selected application log events. Matching events must be sent to a Lambda function with minimal delay so the function can enrich them and route alerts. Which solution should an architect recommend?

A. Schedule a daily CloudWatch Logs export to Amazon S3
B. Configure a CloudWatch Logs subscription filter with Lambda as the destination
C. Run a nightly Lambda function that queries CloudWatch Logs Insights
D. Configure a CloudWatch Logs retention policy

Correct answer: B

Explanation: Subscription filters provide ongoing delivery of matching log events and can use Lambda for near-real-time processing. Export and scheduled queries are better suited to batch or investigative workflows.

Question 3

An organization must retain large volumes of application logs for seven years. The logs are rarely accessed, but they must remain available for later compliance analysis. Which design is generally the most cost-effective?

A. Keep all logs indefinitely in OpenSearch Service hot storage
B. Keep all logs indefinitely in CloudWatch Logs without a retention policy
C. Export or deliver the logs to Amazon S3 and apply an appropriate lifecycle policy
D. Send every log event to a CloudWatch custom metric

Correct answer: C

Explanation: Amazon S3 is well suited to durable, low-cost archival and can use lifecycle policies for further cost optimization. CloudWatch Logs and OpenSearch are generally more appropriate for active operational use than infrequently accessed, multi-year archives.

Question 4

A Lambda function successfully runs, but no execution logs appear in its CloudWatch log group. Which issue is the most likely cause?

A. The function’s execution role lacks CloudWatch Logs permissions
B. The function must use an EC2 instance profile
C. CloudWatch Logs only accepts logs from the unified CloudWatch agent
D. The function must export logs to Amazon S3 before CloudWatch can display them

Correct answer: A

Explanation: Lambda requires appropriate permissions in its execution role to create log groups or streams and publish log events. Lambda does not require an EC2 instance profile or the unified agent to write execution logs.

Question 5

An architect needs to encrypt CloudWatch Logs with a key controlled by the company’s security account and managed through explicit KMS key policies. What should the architect consider?

A. CloudWatch Logs cannot use KMS keys
B. Associate the log group with a customer managed KMS key and configure the necessary key and IAM permissions
C. Enable encryption only on the CloudWatch dashboard
D. Export logs to plain-text Amazon S3 because CloudWatch Logs cannot be encrypted

Correct answer: B

Explanation: CloudWatch Logs supports customer managed KMS keys for log-group encryption. The key policy, IAM permissions, Region placement, and key lifecycle must all be managed correctly. Encryption does not eliminate the need for CloudWatch Logs authorization.