Querying Application Load Balancer Access Logs with Amazon Athena – SAP-C02 Study Guide

Purpose of This Lesson

Application Load Balancer (ALB) access logs provide a record of requests received by the load balancer. By storing these logs in Amazon S3 and querying them with Amazon Athena, architects can analyze traffic without provisioning database or analytics servers.

This pattern is useful for troubleshooting, security investigations, usage analysis, HTTP status reporting, and operational reporting.

Key Concepts

ALB access logging

An ALB can publish access log files to an Amazon S3 bucket. The records can include details such as:

• Request time and load balancer identity
• Client and target IP addresses and ports
• Request processing times
• HTTP methods, URLs, and protocol versions
• HTTP status codes returned by the load balancer and target
• User-agent information
• TLS-related details when HTTPS is used

Access logging is configured as an ALB attribute. It is not enabled automatically for every load balancer.

Log delivery is asynchronous. After traffic is generated, there can be a delay before log files appear in S3. An empty or recently configured log prefix does not necessarily indicate a configuration failure.

S3 destination requirements

The destination bucket must be configured to allow the Elastic Load Balancing log delivery service to write objects. The bucket policy should:

• Grant the required AWS log delivery principal permission to write
• Restrict writes to the intended bucket and log prefix
• Use the correct AWS account identifier and bucket ARN for the deployment
• Preserve the expected object path, including the AWSLogs/<account-id>/ structure when applicable

The bucket policy is a security boundary. Avoid granting broad permissions such as public write access or unrestricted access to all principals.

The S3 bucket and the ALB should normally be in the same AWS Region for ALB access log delivery. Confirm the current service-specific requirements when designing a cross-Region logging architecture.

Athena external tables

Amazon Athena queries data in place in Amazon S3. It does not import the ALB logs into a managed database. A table definition describes:

• The columns and their data types
• The delimiter or serialization format
• The location of the files in S3
• Any parsing rules required for the log format

The table is external, so deleting the Athena table removes metadata only; it does not delete the underlying S3 log files.

A typical workflow is:

1. Enable ALB access logging to an S3 bucket.
2. Generate or wait for application traffic.
3. Confirm that log objects are delivered to the expected S3 prefix.
4. Configure an Athena query result location in a separate S3 bucket or prefix.
5. Create an external table for the ALB log format.
6. Run SQL queries against the table.
7. Remove the table and infrastructure when the analysis is complete.

Athena query result location

Athena needs an S3 location where it can write query results and, depending on the query, metadata or temporary output. This is separate from the S3 location containing the source ALB logs.

If no valid query result location is configured, Athena queries can fail before the SQL itself is evaluated. The result bucket must also have appropriate permissions for the user, role, or workgroup running Athena queries.

Example analysis queries

Once the external table has been created, common analyses include:

• Viewing recent access log records ordered by request time
• Counting requests by client IP address
• Identifying the most frequently requested URL paths
• Filtering requests by browser or user-agent
• Grouping requests by HTTP status code
• Finding slow requests using request-processing duration fields
• Comparing load balancer responses with target responses

For large log collections, add time or partition filters wherever possible. Querying an entire historical S3 prefix can scan substantial amounts of data and increase Athena costs.

Exam-Relevant Takeaways

• ALB access logs are delivered to Amazon S3 and can be analyzed with Amazon Athena.
• Athena is serverless and queries data in place; it does not require an EC2 instance, RDS database, or EMR cluster for this use case.
• The S3 bucket policy must permit the Elastic Load Balancing log delivery service to write objects.
• Athena requires an S3 query result location before queries can run successfully.
• ALB log delivery is asynchronous, so logs may not appear immediately after requests are made.
• Creating an Athena external table does not copy or own the source data. The S3 objects remain the system of record.
• Keep source logs and Athena query results logically separate through different buckets or prefixes.
• Use IAM, S3 bucket policies, encryption, and potentially Lake Formation controls to restrict access to sensitive request data.
• CloudFormation is useful for repeatable creation of the ALB, target resources, and associated configuration, but the log destination and permissions still need to be modeled correctly.

Architecture Decision Guide

Requirement	Recommended design	Important considerations
Store ALB request history	Enable ALB access logs to Amazon S3	Delivery is asynchronous; allow time for objects to arrive
Query logs without managing servers	Amazon Athena over S3	Configure a query result location and control scanned data
Repeated reporting over large log volumes	S3, Glue Data Catalog, Athena, and partitioned data	Partition by date or another useful dimension where practical
Centralize logs from multiple accounts	Dedicated logging account with controlled S3 access	Use bucket policies, IAM, organization controls, and encryption
Protect log contents	S3 Block Public Access, least-privilege IAM, encryption, and restrictive bucket policies	URLs, IP addresses, and user agents may be sensitive data
Analyze near-real-time traffic	ALB metrics, access logs, or a streaming pipeline depending on latency needs	Athena over S3 is generally better for interactive or historical analysis than immediate alerting
Automate the environment	AWS CloudFormation or another infrastructure-as-code tool	Avoid hard-coded account IDs, Regions, bucket names, and AMI IDs

Common Exam Traps

• Confusing ALB access logs with CloudWatch metrics: Metrics provide aggregated operational measurements, while access logs provide request-level records in S3.
• Assuming Athena stores the data: Athena stores query results but reads source records from S3. The external table is metadata.
• Forgetting Athena’s output location: A valid source table is not enough; Athena also needs permission to write query results to S3.
• Using the ALB log bucket as an unrestricted result bucket: Separate source and output locations to simplify permissions, lifecycle policies, and governance.
• Expecting logs immediately: ALB log delivery is not synchronous with the request.
• Granting public S3 permissions: The load balancer’s ability to write logs should come from a narrowly scoped bucket policy, not public access.
• Ignoring query scan costs: A simple SQL query can scan a large number of S3 objects. Use prefixes, partitions, compressed formats, and time predicates where appropriate.
• Assuming a browser filter will always return data: The user-agent recorded in the log must match the filter value and the traffic must actually have been generated by that client.
• Deleting the Athena table to delete logs: Dropping an external table does not remove the S3 objects.

Real-World Engineer Notes

• Use an organization-wide logging bucket or logging account for production environments rather than creating ad hoc buckets per experiment.
• Apply S3 lifecycle policies to transition or expire logs according to compliance and operational requirements.
• Encrypt the S3 buckets with SSE-S3 or SSE-KMS. If using SSE-KMS, verify that the log delivery service and query principals have the required permissions.
• Enable S3 versioning and carefully evaluate whether Object Lock is required for regulatory or forensic retention.
• Use S3 Block Public Access and verify that bucket policies do not accidentally override the intended security posture.
• Consider AWS Glue Data Catalog for reusable table definitions and schema management across teams.
• Partition large datasets by date, account, Region, or load balancer where the access pattern supports it. Partitioning reduces the amount of data Athena must scan.
• For frequent analysis, transform raw text logs into columnar formats such as Parquet using an ETL process. This can reduce scan volume and improve query performance.
• ALB logs can contain client IP addresses, URLs, user agents, and other information that may be sensitive. Define retention, access, and privacy controls before enabling broad analyst access.
• Use CloudFormation parameters for the Region-specific AMI, VPC, subnets, bucket name, and account-specific values. Hard-coded lab values are fragile in production.

Quick Reference Summary

• Source: Application Load Balancer access logs
• Storage: Amazon S3
• Analysis engine: Amazon Athena
• Metadata: Athena external table, optionally managed through AWS Glue Data Catalog
• Required permissions: ALB log delivery can write to the destination prefix; Athena principals can read source data and write query results
• Operational delay: Log objects may take time to appear
• Cost control: Restrict scanned data with prefixes, partitions, filters, and columnar formats
• Cleanup: Delete the Athena table separately from the S3 objects; remove test ALBs, buckets, and CloudFormation stacks when no longer needed

Flashcards

1. Q: Where does an ALB deliver access logs?

A: To an Amazon S3 bucket configured through the ALB access logging attribute.

1. Q: What service can query ALB logs in S3 without managed servers?

A: Amazon Athena.

1. Q: Does Athena import ALB logs into a database when an external table is created?

A: No. The table stores schema and location metadata; Athena reads the files directly from S3.

1. Q: What must be configured before Athena queries can run?

A: An S3 query result location with appropriate permissions.

1. Q: Why might an ALB log prefix be empty immediately after enabling logging?

A: Log delivery is asynchronous and can take time.

1. Q: What does the S3 bucket policy allow the load balancer logging service to do?

A: Write log objects to the intended bucket and prefix.

1. Q: What happens to S3 log files when an Athena external table is dropped?

A: They remain in S3; only the table metadata is removed.

1. Q: Which provides request-level details: ALB metrics or ALB access logs?

A: ALB access logs.

1. Q: Why should Athena queries include date or partition filters?

A: To reduce the amount of S3 data scanned and therefore improve performance and control cost.

1. Q: What type of S3 access should be avoided for ALB log delivery?

A: Public or broadly unrestricted write access.

Practice Questions

Question 1

A company enables access logging on an Application Load Balancer. The ALB is healthy and receives requests, but no log files appear in the configured S3 prefix for several seconds. What is the most likely explanation?

A. Athena must create a table before ALB logs can be delivered
B. ALB access log delivery is asynchronous
C. ALB logs are delivered only when a target returns an HTTP 500 response
D. The S3 bucket must be configured as a static website

Correct answer: B

Explanation: ALB access logs are delivered asynchronously. A short delay between request processing and object creation in S3 is expected. Athena is not required for delivery.

Question 2

An architect creates an Athena external table over ALB log files in S3. Every query fails with an error indicating that Athena cannot write output. What should be checked first?

A. Whether the ALB has at least three targets
B. Whether the S3 bucket uses S3 Intelligent-Tiering
C. Whether an Athena query result location is configured and writable
D. Whether the ALB is internet-facing

Correct answer: C

Explanation: Athena needs an S3 output location and permissions to write query results. This is independent of the source table’s ability to read ALB log files.

Question 3

A security team requires ALB logs to be stored in a central S3 bucket. Which design provides the strongest baseline security?

A. Allow public write access to the bucket so every ALB can deliver logs
B. Grant the required log delivery principal permission only for the intended bucket prefix, while using Block Public Access and encryption
C. Give all IAM users s3:* on the bucket
D. Store the logs in an unencrypted public bucket and restrict access through the ALB security group

Correct answer: B

Explanation: ALB log delivery should use a narrowly scoped bucket policy. S3 Block Public Access, encryption, and least-privilege permissions protect the log data. Security groups do not control access to S3 objects.

Question 4

An analytics team frequently queries two years of ALB logs stored as many raw text files. Queries are slow and scan large amounts of data. Which improvement is most appropriate?

A. Add more ALB listener rules
B. Convert the data to a compressed columnar format and organize it with useful partitions
C. Replace Athena with an internet-facing ALB
D. Disable access logging during business hours

Correct answer: B

Explanation: Transforming raw logs into compressed columnar formats such as Parquet and partitioning by query-relevant attributes can significantly reduce scanned data and improve Athena performance and cost.

Question 5

An operations team drops an Athena table after completing an investigation and expects the underlying ALB logs to be removed automatically. What will happen?

A. The S3 objects remain because the table is external
B. The S3 objects are deleted after the next Athena query
C. The ALB deletes its historical logs automatically
D. The S3 bucket is deleted with the table

Correct answer: A

Explanation: Dropping an external Athena table removes metadata only. S3 lifecycle rules or explicit deletion are needed to remove the underlying log objects.