Study guide

Technical reference and lesson notes

Purpose of This Lesson

This hands-on lesson demonstrates an end-to-end monitoring workflow:

Launch an Amazon EC2 instance with the required IAM instance profile.
Install the CloudWatch agent, Apache HTTPD, and the required logging utility.
Configure the agent to stream operating-system, agent, and Apache access logs to Amazon CloudWatch Logs.
Create a metric filter that counts HTTP 404 responses.
Create a CloudWatch alarm that enters an alarm state when at least one 404 occurs during a five-minute period.

The workflow is especially useful for understanding how raw log events become metrics and then actionable alarms.

Key Concepts

CloudWatch agent

The CloudWatch agent runs on the EC2 instance and collects local log files. Its configuration determines which files are monitored and which CloudWatch Logs log groups receive the events.

In this exercise, the supplied configuration collects:

System messages from the system log.
The CloudWatch agent’s own log.
Apache HTTPD access-log entries.

The resulting log groups use names such as:

/apps/system/messages
/apps/cloudwatch-agent-log
/apps/web-service-httpd-access

The exact names depend on the supplied configuration file and should be verified in the CloudWatch console.

IAM instance profile

The EC2 instance needs an IAM role trusted by the EC2 service. The role used in the lab includes:

CloudWatchAgentServerPolicy, allowing the agent to send monitoring and log data.
AmazonSSMManagedInstanceCore, allowing Systems Manager-related management capabilities used by the instance.

If the instance does not have the correct role attached, the agent may run locally while log data fails to arrive in CloudWatch Logs.

CloudWatch Logs log groups and streams

A log group organizes related log data. Log streams within a group represent individual sources, commonly an instance or a log file. After the agent is configured and running, incoming Apache and agent events can be inspected by opening the relevant log stream.

Metric filters

A metric filter evaluates incoming log events against an event pattern. Matching events contribute a value to a CloudWatch metric.

The lab filters Apache access logs for status code 404. The filter uses:

Filter name: 404
Metric namespace: HTTPD status code
Metric name: 404 error
Metric value: 1
Unit: Count

Each matching 404 event contributes one count to the metric.

CloudWatch alarms

An alarm evaluates a metric against a threshold over a defined period. The lab configures an alarm for the 404 error metric with:

Period: 5 minutes
Condition: greater than or equal to 1
Notification: a new or existing notification topic

A newly created alarm may initially show INSUFFICIENT_DATA while CloudWatch gathers enough data. Generating additional 404 requests and waiting for metric processing allows the alarm to transition to IN_ALARM.

EC2 Log Collection and Alerting Workflow

1. Create the IAM role

Create an EC2-trusted role and attach the CloudWatch agent server policy and Systems Manager managed-instance policy. Attach the role as the instance profile when launching the instance.

2. Launch the EC2 instance

The demonstration uses an Amazon Linux 2023 AMI and a small instance type. The security group must allow:

TCP port 22 for SSH or instance-connect access.
TCP port 80 for access to the Apache web service.

The lesson uses EC2 user data for initial installation and setup. The downloaded course materials include the CloudWatch configuration file and the commands used in the lab.

3. Install and start the required software

The instance is prepared with:

The CloudWatch agent.
The syslog/system logging utility needed to provide the system messages file on the selected Amazon Linux 2023 AMI.
Apache HTTPD.

Apache is started and enabled so that it provides a web endpoint and generates access-log events.

4. Apply the CloudWatch agent configuration

Replace or recreate the agent configuration file at the path used by the CloudWatch agent. Copy the supplied CW-config.JSON contents into the configuration file, then stop and start the agent so it reads the updated configuration.

Restarting the agent after changing its configuration is important. Simply editing the file does not guarantee that a running agent will immediately use the new settings.

5. Verify log delivery

Open the EC2 instance’s public address in a browser. The default Apache page confirms that HTTPD is reachable. Repeatedly request nonexistent paths such as test.html to generate HTTP 404 entries.

In CloudWatch:

Open Log groups.
Locate the Apache access-log group.
Open its log stream.
Confirm that 404 events and request details are present.
Check the CloudWatch agent log group for agent activity.
Check the system messages group after confirming that the system log file exists and is receiving entries.

Log delivery can take a short time. A missing group or stream should first prompt verification of the instance role, agent configuration, agent restart, and source log-file path.

6. Create and test the metric filter

From the Apache access-log group, open Metric filters and create a filter using the event pattern supplied in the lab. The pattern identifies events whose HTTP status field is 404.

Use the pattern-testing function against an existing log stream before creating the filter. The test should identify matching events when the log contains generated 404 requests.

After creating the filter, locate the metric under the configured namespace, using the metrics-with-no-dimensions view. Metric publication is not necessarily instantaneous, so the graph may initially be empty.

7. Create and validate the alarm

Create an alarm on the HTTPD status code / 404 error metric:

Set the evaluation period to five minutes.
Set the threshold to greater than or equal to one.
Select an existing notification topic or create a new one.
Confirm a new notification subscription when required.

Generate another nonexistent-page request within the evaluation window. After CloudWatch processes the metric, the alarm should move from INSUFFICIENT_DATA to IN_ALARM, and the configured notification should be delivered.

Assessment-Relevant Takeaways

An EC2 instance must have an appropriate IAM instance profile before the CloudWatch agent can deliver logs to CloudWatch Logs.
The EC2 role needs an EC2 trust relationship; permissions alone are not sufficient.
Security groups affect both administration and validation: port 22 supports access to the instance, while port 80 allows HTTP traffic to generate Apache events.
The CloudWatch agent collects local files according to its configuration; it does not automatically collect every log file.
After changing the agent configuration, restart the agent to ensure the new configuration is loaded.
A metric filter converts matching log events into metric data; it does not itself create an alarm.
A CloudWatch alarm evaluates the resulting metric and can publish notifications through a topic.
INSUFFICIENT_DATA immediately after alarm creation is expected until metric data is available.
A count metric with a value of 1 per matching event means a threshold of >= 1 detects at least one matching event during the evaluation period.
CloudWatch Logs and CloudWatch Metrics have different processing paths, so a log event may appear before its metric and alarm state update.

Tool / Feature Decision Guide

Requirement	Use	Reason
Collect log files from an EC2 operating system or application	CloudWatch agent	The agent reads configured local files and sends their contents to CloudWatch Logs.
Organize and inspect incoming log events	CloudWatch Logs log groups and streams	Log groups organize sources, while streams contain the event data.
Detect a text or field pattern in logs	CloudWatch Logs metric filter	A filter turns matching events into a metric value.
Notify when the resulting count crosses a threshold	CloudWatch alarm	The alarm evaluates the metric over a period and can send a notification.
Manage or connect to the instance using AWS Systems Manager capabilities	IAM permission from the instance profile	The lab role includes `AmazonSSMManagedInstanceCore` for this purpose.
Test whether a filter pattern matches existing events	Metric filter test function	Testing helps validate the pattern before relying on the metric.

Common Traps / Misconceptions

The agent is installed, so logs must be arriving. Installation does not prove successful delivery. Verify the instance profile, configuration, agent state, and CloudWatch log groups.
Changing the configuration file automatically changes a running agent. Stop and start the agent after replacing the configuration so the new settings are loaded.
The system messages log is always present. The lesson needed to install the logging utility because the expected system log file was not installed by default on the selected Amazon Linux 2023 AMI.
A log group appearing proves every configured source works. Check each expected group separately; the Apache and agent groups may appear before the system messages group.
A metric filter immediately produces visible graph data. CloudWatch may need time to publish and display the metric.
An alarm in INSUFFICIENT_DATA means the configuration failed. It can simply mean that the alarm has not received enough metric data yet.
The metric filter itself sends notifications. The filter creates metric data; the alarm is the component that evaluates the metric and publishes notifications.
A 404 is necessarily an infrastructure failure. In this lab, 404s are intentionally generated as test events to validate the monitoring workflow.
Leaving the lab instance running is harmless. Terminate the EC2 instance and remove any temporary resources when testing is complete.

Real-World Engineer / Analyst Notes

Validate the entire pipeline in sequence: source file, agent configuration, agent process, IAM permissions, log group arrival, filter matches, metric publication, and alarm state.
Generate known test events instead of waiting for naturally occurring failures. Deliberate nonexistent URLs provide a repeatable way to validate 404 detection.
Use the metric-filter test tool before creating an alarm. A filter that matches no events will make downstream alarm troubleshooting confusing.
Keep the filter pattern aligned with the actual log format. The lab pattern depends on the Apache access-log fields, including the status-code field.
Expect delays between an HTTP request, log delivery, metric publication, and alarm notification. Do not assume the absence of an immediate alarm means the pipeline is broken.
Treat notification subscriptions as part of the validation process. A newly created topic may require subscription confirmation before messages are received.
Clean up temporary EC2 resources after the exercise to avoid unnecessary ongoing resource use.

Quick Reference Summary

Instance role: EC2 trust plus CloudWatchAgentServerPolicy and AmazonSSMManagedInstanceCore.
Required security-group access: TCP 22 for instance access and TCP 80 for Apache testing.
Agent inputs: System messages, CloudWatch agent log, and Apache HTTPD access log.
Agent action after configuration change: Stop and start the agent.
Test event: Request nonexistent Apache paths to generate HTTP 404 entries.
Metric filter: Match status code 404; publish value 1 to the HTTPD status code / 404 error metric.
Alarm: Five-minute period with a threshold of >= 1.
Expected progression: Log event → metric data → alarm evaluation → notification.
Cleanup: Terminate the EC2 instance and remove temporary lab resources.

Flashcards

Q: An EC2 instance is running the CloudWatch agent, but no log groups receive data. What should you verify first?

A: Verify that the instance profile is attached, has the EC2 trust relationship, and includes CloudWatchAgentServerPolicy. Then check the agent configuration, source file paths, and agent status.

Q: Why does the lab install a system logging utility before configuring the CloudWatch agent?

A: The selected Amazon Linux 2023 AMI did not have the expected system messages log available by default. The utility provides the log source that the agent is configured to collect.

Q: When should you restart the CloudWatch agent during this workflow?

A: Restart it after replacing or editing the agent configuration file so the running process loads the new settings.

Q: Which component should you choose when you need to turn Apache 404 log events into a numerical time-series value?

A: Use a CloudWatch Logs metric filter. It matches the 404 pattern and publishes a value, such as 1, to a CloudWatch metric.

Q: What is the difference between the metric filter and the CloudWatch alarm in this exercise?

A: The metric filter identifies matching log events and emits metric data. The alarm evaluates that metric against a threshold and can send a notification.

Q: An alarm is configured for 404 error >= 1 over five minutes. What event causes it to breach?

A: At least one matching 404 event must contribute to the metric during the five-minute evaluation period.

Q: Why might the Apache access-log group contain data while the system messages group is absent?

A: Apache may be generating logs correctly while the system-log source file is missing or the logging utility was not installed or started. Check the configured path and confirm that the file contains entries.

Q: What should you do if the metric-filter graph is empty immediately after the filter is created?

A: Generate additional matching 404 requests, wait for CloudWatch processing, and refresh the metric view. Log arrival and metric publication are not necessarily immediate.

Q: Which security-group ports are needed for this lab, and what does each support?

A: TCP 22 supports SSH or EC2 Instance Connect access, and TCP 80 permits browser requests to Apache so access-log events can be generated.

Q: What does a metric value of 1 represent in the 404 filter configuration?

A: Each log event matching the 404 pattern contributes one count to the metric.

Q: What is the expected initial state of a newly created alarm before sufficient metric data arrives?

A: It may show INSUFFICIENT_DATA. After matching events are processed, it can transition to IN_ALARM when the threshold is breached.

Q: A new notification topic was created for the alarm, but no email arrives. What lab-specific step may be missing?

A: The notification subscription may require confirmation. Confirm the subscription before expecting alarm messages.

Q: How can you validate that the metric-filter pattern is appropriate before creating an alarm?

A: Use the metric filter’s test function against an existing log stream and confirm that known 404 events are matched.

Practice Questions

Question 1

An operations team needs an alarm whenever Apache records at least one HTTP 404 within five minutes. Which implementation best matches the demonstrated workflow?

A. Create an alarm directly on the Apache log group with a >= 1 threshold.
B. Create a metric filter for status 404, publish value 1 to a count metric, and alarm on that metric.
C. Create a CloudWatch agent alarm that monitors port 80.
D. Create a Systems Manager association that restarts Apache after every 404.

Correct answer: B

The metric filter converts matching log events into a metric, and the CloudWatch alarm evaluates that metric.

Question 2

The CloudWatch agent log group and Apache access-log group appear, but the system messages group does not. Which investigation is most appropriate?

A. Close port 80 because system logs require HTTP access.
B. Delete the alarm and recreate the EC2 instance with a larger instance type.
C. Verify that the system logging utility created the expected log file and that the agent configuration points to the correct path.
D. Add an HTTPD metric filter to the system messages log group.

Correct answer: C

The system messages source may be unavailable or incorrectly configured even though the other configured files are being collected.

Question 3

A user replaces the CloudWatch agent configuration file while the agent is running. The new log group does not appear. What is the most likely corrective action from the lab procedure?

A. Restart the agent so it loads the new configuration.
B. Reboot the user’s browser.
C. Change the Apache status code from 404 to 200.
D. Remove TCP port 22 from the security group.

Correct answer: A

The lesson explicitly stops and starts the agent after replacing the configuration file.

Question 4

A 404 alarm remains in INSUFFICIENT_DATA after creation. The Apache log stream shows that requests are being recorded. What should the engineer do next?

A. Assume the IAM role is invalid because all alarms start in ALARM.
B. Generate additional 404 requests, wait for metric processing, and refresh the alarm.
C. Replace the EC2 instance with a T2 micro instance.
D. Remove the metric filter because alarms cannot use filtered metrics.