Amazon CloudWatch Logs: Centralization, Retention, Analysis, and Delivery

Purpose of This Lesson

Amazon CloudWatch Logs centralizes application, system, AWS service, and on-premises log data. For the AWS Certified CloudOps Engineer Associate SOA-C03 context, the key skills are recognizing how logs enter CloudWatch, how they are organized and retained, how they are analyzed, and which downstream services or features should be selected for monitoring and remediation workflows.

Key Concepts

Log events

A log event is a record of activity produced by a monitored service, application, or system.

Log streams

A log stream is a sequence of log events that share the same source. Streams help separate event sequences from individual sources within a log group.

Log groups

A log group is a collection of log streams that share settings such as:

• Retention behavior
• Monitoring configuration
• Access control

Retention and access decisions are made at the log-group level, so selecting or configuring the correct group is important when organizing logs.

Retention settings

CloudWatch Logs retains log events indefinitely by default. A retention policy can be configured for each log group, with supported periods ranging from one day to ten years. Retention should be selected deliberately to balance operational investigation needs with storage-management requirements.

Metric filters

A metric filter extracts observations from ingested log events and converts them into data points for a CloudWatch metric. This makes it possible to monitor patterns in logs and create alarms based on matching log activity.

Subscription filters

Subscription filters support near-real-time processing of log data and can route log information to destinations such as Kinesis or Lambda. The lecture also identifies Amazon Elasticsearch Service as a destination for CloudWatch Logs data and real-time log processing workflows.

Log Collection and Delivery Architecture

CloudWatch Logs can receive data from several categories of sources:

• Applications
• Operating system and system logs
• AWS services
• On-premises systems
• EC2 instances
• AWS Lambda

For EC2 instances and on-premises servers, the unified CloudWatch agent can be installed to collect application and system information and send it to CloudWatch Logs.

AWS Lambda can also send information to CloudWatch Logs. Lambda must have the appropriate permissions to write to CloudWatch Logs; a missing or incorrect permission is a common cause of absent Lambda logs.

CloudWatch Logs data can be sent onward to:

• Amazon S3 through an export workflow
• Kinesis Data Streams
• Kinesis Data Firehose
• Amazon Elasticsearch Service

Use subscription filters when log data needs to be processed or delivered in near real time to supported downstream consumers such as Kinesis or Lambda.

Analysis, Monitoring, and AWS Service Integrations

CloudWatch Logs Insights

CloudWatch Logs Insights provides interactive searching and analysis of data stored in CloudWatch Logs. It is appropriate when an operator needs to investigate log data directly rather than only observe a preconfigured metric or alarm.

CloudTrail-driven alarms

CloudWatch can create alarms based on API activity captured by AWS CloudTrail. This connects API auditing with monitoring and alerting workflows.

Route 53 query logging

CloudWatch Logs can store information about DNS queries received by Amazon Route 53. This extends the logging workflow beyond hosts and applications to DNS activity.

Encryption with KMS

CloudWatch Logs data can be encrypted using an AWS Key Management Service (KMS) key. Encryption settings should be considered alongside access-control requirements for the relevant log group.

Exam- or Assessment-Relevant Takeaways

• Choose the unified CloudWatch agent when collecting application or system logs from EC2 instances or on-premises servers.
• Remember that Lambda logging depends on appropriate permissions to CloudWatch Logs.
• Distinguish the hierarchy: log events are individual records, log streams group events from the same source, and log groups organize streams under shared settings.
• CloudWatch Logs retention is indefinite by default, but each log group can be configured from one day through ten years.
• Use CloudWatch Logs Insights for interactive search and analysis of existing log data.
• Use metric filters when log-event patterns must become CloudWatch metric data points.
• Use subscription filters when logs need near-real-time delivery or processing by destinations such as Kinesis or Lambda.
• CloudWatch Logs supports application, system, AWS service, Lambda, on-premises, and Route 53 query logging use cases.
• CloudTrail API activity can provide the underlying information for CloudWatch alarms.

Tool / Feature Decision Guide

Requirement	Appropriate feature or destination	Decisive reason
Collect application and system logs from EC2 or on-premises servers	Unified CloudWatch agent	The agent sends host-based log information to CloudWatch Logs.
Search and investigate existing log data interactively	CloudWatch Logs Insights	It is designed for interactive log searching and analysis.
Turn matching log entries into measurable monitoring data	Metric filter	It extracts observations and produces CloudWatch metric data points.
Process or route logs in near real time	Subscription filter	It supports real-time log delivery to destinations such as Kinesis or Lambda.
Deliver logs for object-storage use	Amazon S3 export	The lecture identifies S3 as an export destination.
Stream logs through Kinesis services	Kinesis Data Streams or Kinesis Data Firehose	These are supported downstream delivery options for CloudWatch Logs data.
Protect log data with customer-controlled encryption configuration	KMS key	CloudWatch Logs encryption can be enabled using a KMS key.
Retain logs for a defined operational period	Log-group retention setting	Retention is configured per log group and can override indefinite default retention.

Common Traps / Misconceptions

• Assuming logs expire automatically: CloudWatch Logs retains data indefinitely by default. An explicit retention policy is needed when logs should expire.
• Confusing streams and groups: A stream represents a sequence from one shared source; a group contains streams and their shared retention, monitoring, and access-control settings.
• Using Logs Insights to create metrics: Logs Insights searches and analyzes log data. Metric filters convert matching ingested events into CloudWatch metric data points.
• Treating all delivery as real time: Subscription filters are the relevant mechanism for near-real-time processing and delivery; S3 is identified as an export destination.
• Forgetting Lambda permissions: Lambda must have the required permissions to send information to CloudWatch Logs.
• Assuming only AWS resources can be logged: On-premises systems can send logs when the unified CloudWatch agent is installed.
• Overlooking Route 53: CloudWatch Logs can also capture DNS query information received by Amazon Route 53.

Real-World Engineer / Analyst Notes

• Design log groups around shared retention, monitoring, and access-control requirements rather than treating them as arbitrary containers.
• Set retention intentionally. Indefinite retention may preserve useful investigative history, but it also leaves logs stored until the policy is changed.
• When troubleshooting missing logs, first identify the source type and confirm the collection path: agent installation for EC2 or on-premises systems, and permissions for Lambda.
• Use Logs Insights for exploratory investigation, then use metric filters when a recurring log pattern should become a monitored signal.
• Use subscription filters when another service must consume log data quickly, especially for processing or streaming workflows.
• Treat encryption and access control as part of the logging design, not as separate afterthoughts.

Quick Reference Summary

• CloudWatch Logs: Centralizes application, system, AWS service, Lambda, EC2, on-premises, and Route 53 query logs.
• Unified CloudWatch agent: Collects application and system logs from EC2 and on-premises systems.
• Log event: Individual activity record.
• Log stream: Sequence of events from a shared source.
• Log group: Collection of streams with shared retention, monitoring, and access-control settings.
• Default retention: Indefinite.
• Configurable retention: One day to ten years per log group.
• Logs Insights: Interactive search and analysis.
• Metric filter: Converts log observations into CloudWatch metric data points.
• Subscription filter: Supports near-real-time delivery and processing.
• Destinations: Amazon S3 export, Kinesis Data Streams, Kinesis Data Firehose, Amazon Elasticsearch Service, Lambda, and other supported workflows described in the lesson.
• Security: Encryption can be enabled with a KMS key; Lambda requires appropriate CloudWatch Logs permissions.

Flashcards

Q: A company needs application and operating-system logs from both EC2 instances and on-premises servers in CloudWatch Logs. Which collection approach should be considered?

A: Install the unified CloudWatch agent on the EC2 and on-premises systems. It can collect application and system logs from both environments.

Q: What is the relationship between a log event, log stream, and log group?

A: A log event is an individual activity record. A log stream is a sequence of events from the same source, and a log group contains streams that share retention, monitoring, and access-control settings.

Q: What happens to CloudWatch Logs when no retention policy is configured?

A: Events are retained indefinitely by default. A retention setting must be configured on the log group if events should expire.

Q: An operations team wants logs to expire after a defined period. Where is retention configured, and what range does the lesson identify?

A: Retention is configured for each log group, with periods ranging from one day to ten years.

Q: Which CloudWatch feature should an analyst use to interactively search and analyze existing log data?

A: CloudWatch Logs Insights, because it is designed for interactive searching and analysis of CloudWatch Logs data.

Q: When should a metric filter be used instead of Logs Insights?

A: Use a metric filter when matching log events must be transformed into CloudWatch metric data points for ongoing monitoring or alarms. Use Logs Insights for interactive investigation.

Q: A team needs to send log data for near-real-time processing by Kinesis. Which CloudWatch Logs feature is the key choice?

A: Use a subscription filter. Subscription filters support near-real-time log processing and delivery to destinations such as Kinesis.

Q: What is the operational trap when AWS Lambda is expected to write logs to CloudWatch Logs but no logs appear?

A: Lambda may lack the required permissions to send information to CloudWatch Logs. Verify the Lambda permissions before assuming the application failed to generate logs.

Q: Which log sources are supported beyond application and system logs?

A: CloudWatch Logs can receive logs from AWS services, Lambda, on-premises systems, and Route 53 DNS query logging, among the sources identified in the lesson.

Q: How can CloudWatch Logs data be protected using AWS KMS?

A: Encryption can be enabled using a KMS key. The choice should be considered together with the log group’s access-control requirements.

Q: A security team wants alarms based on API activity. Which AWS service supplies the captured API activity described in the lesson?

A: AWS CloudTrail captures the API activity, and CloudWatch can create alarms based on that information.

Q: When would an export to Amazon S3 be selected instead of a subscription filter?

A: Select an S3 export when the requirement is to export log data to Amazon S3. Select a subscription filter when near-real-time processing or delivery is required.

Practice Questions

Question 1

An operations team wants to investigate why an application produced errors during the last deployment. They need to search existing log data interactively across CloudWatch Logs. Which feature is the best fit?

A. CloudWatch metric filter
B. CloudWatch Logs Insights
C. Route 53 query logging
D. KMS encryption

Correct answer: B. CloudWatch Logs Insights

Explanation: Logs Insights is designed for interactive searching and analysis of data already stored in CloudWatch Logs. Metric filters are used to turn matching events into metric data points.

Question 2

A company wants every occurrence of a specific error pattern in application logs to contribute to a CloudWatch metric that can be monitored. Which feature should be configured?

A. Log stream
B. Subscription filter
C. Metric filter
D. Amazon S3 export

Correct answer: C. Metric filter

Explanation: Metric filters extract observations from ingested events and transform them into CloudWatch metric data points.

Question 3

A Lambda function runs successfully, but expected Lambda log information is missing from CloudWatch Logs. Which issue should be checked first based on the lesson?

A. Whether Route 53 query logging is enabled
B. Whether Lambda has permissions to CloudWatch Logs
C. Whether the log group retention is set to ten years
D. Whether an S3 export has completed

Correct answer: B. Whether Lambda has permissions to CloudWatch Logs

Explanation: Lambda requires appropriate permissions to send information to CloudWatch Logs. Retention and export settings do not solve a write-permission problem.

Question 4

An organization wants logs delivered for near-real-time processing by a downstream streaming service. Which option best matches this requirement?

A. Configure a subscription filter to Kinesis
B. Set log retention to one day
C. Use Logs Insights only
D. Configure a log stream without a log group

Correct answer: A. Configure a subscription filter to Kinesis

Explanation: Subscription filters support near-real-time log processing and can deliver data to Kinesis. Retention controls storage duration, not streaming behavior.

Question 5

An administrator wants logs to be removed after seven days rather than retained indefinitely. What is the correct configuration scope?

A. Configure retention on each individual log event
B. Configure retention on the relevant log group
C. Configure retention on the KMS key
D. Configure retention on the Lambda function only

Correct answer: B. Configure retention on the relevant log group

Explanation: Retention settings apply to log groups. The lesson identifies one day through ten years as the configurable range, so seven days is within that range.

WordPress Metadata

Suggested Slug:
amazon-cloudwatch-logs-soa-c03

Meta Description:
Study Amazon CloudWatch Logs for SOA-C03, including log sources, streams and groups, retention, metric filters, subscription filters, permissions, and analysis workflows.

Tags:
AWS CloudWatch, CloudWatch Logs, SOA-C03, AWS monitoring, AWS logging, CloudWatch Logs Insights, metric filters, subscription filters, AWS Lambda, Kinesis, AWS CloudTrail, Amazon Route 53