AWS SOA-C02 Drill: ELB Access Logs - The Troubleshooting SRE's Source-of-Truth

Jeff’s Note

#

Jeff’s Note

#

“Unlike generic exam dumps, ADH analyzes this scenario through the lens of a Real-World Site Reliability Engineer (SRE).”

“For SOA-C02 candidates, the confusion often lies in confusing CloudWatch metrics (aggregated data) with access logs (granular request data). In production, this is about knowing exactly which AWS service stores per-request forensic data when your Director of Engineering asks: ‘Who caused this traffic spike at 3 AM?’ Let’s drill down.”

The Certification Drill

#

Scenario

#

FinStreamX, a fintech startup processing cryptocurrency transactions, operates a three-tier web application behind an Application Load Balancer (ALB). During a recent product launch, the SRE team received PagerDuty alerts indicating an Auto Scaling group had triggered a scale-up event, spinning up 12 additional EC2 instances within 5 minutes.

The SRE lead reviewed Amazon CloudWatch dashboards and observed a sudden 600% spike in the RequestCount metric for the ALB between 02:47 UTC and 03:15 UTC. The security team suspects this might be a Layer 7 DDoS attempt or a misconfigured API client retry loop.

The Requirement

#

The SRE administrator needs to identify the source IP addresses of incoming requests during this time window to determine whether the traffic originated from legitimate users, a single rogue client, or a distributed attack pattern.

The Options

#

• A) Review Auto Scaling activity logs in the EC2 console
• B) Query AWS CloudTrail logs for API call origins
• C) SSH into EC2 instances and parse application logs in /var/log/app/
• D) Enable and analyze Elastic Load Balancer access logs stored in S3

Correct Answer

#

D) Enable and analyze Elastic Load Balancer access logs stored in S3.

Quick Insight: The SysOps Forensic Imperative

#

For SysOps/SRE: This is about understanding the data granularity hierarchy in AWS:

• CloudWatch Metrics = Aggregated time-series data (count, latency percentiles)
• CloudTrail Logs = AWS API control plane actions (who created/modified resources)
• ELB Access Logs = Per-request data plane logs (client IP, backend status, request path, user-agent)

In incident response, only ELB access logs contain the forensic-grade detail needed for source IP analysis.

Content Locked: The Expert Analysis

#

You’ve identified the answer. But do you know the implementation details that separate a Junior from a Senior SRE?

The Expert’s Analysis

#

Correct Answer

#

Option D: Elastic Load Balancer access logs

The Winning Logic

#

ELB access logs provide per-request forensic data including:

• Client IP address (client:port)
• Request timestamp (microsecond precision)
• Request line (HTTP method, URI, protocol)
• Backend status codes (target response and ELB response)
• User-Agent headers
• SSL cipher and protocol (for HTTPS)

For SysOps troubleshooting:

1. Access logs are disabled by default - you must explicitly enable them via the ELB console or CLI
2. Logs are stored in S3 with predictable prefix patterns:

   s3://bucket-name/prefix/AWSLogs/account-id/elasticloadbalancing/region/yyyy/mm/dd/
   

3. Athena integration allows SQL queries like:

   SELECT client_ip, COUNT(*) as request_count
   FROM alb_logs
   WHERE parse_datetime(time,'yyyy-MM-dd''T''HH:mm:ss.SSSSSS''Z')
         BETWEEN timestamp '2025-01-20 02:47:00' 
         AND timestamp '2025-01-20 03:15:00'
   GROUP BY client_ip
   ORDER BY request_count DESC
   LIMIT 10;
   

Key SysOps exam pattern: When you see “IP address investigation” + “Load Balancer” → Always think ELB access logs first.

The Trap (Distractor Analysis)

#

• Why not A (Auto Scaling logs)?
  Auto Scaling activity logs show when instances launched/terminated and why (scaling policies, health checks), but contain zero information about HTTP requests. You’d see “Launching instance i-0abc123 due to target tracking policy” but never a client IP.

• Why not B (CloudTrail logs)?
  CloudTrail records AWS API actions (control plane), like CreateLoadBalancer or ModifyRule. It shows who (IAM principal) made infrastructure changes, not end-user traffic. CloudTrail would never capture the IP of a mobile app user hitting your ALB.

• Why not C (EC2 instance logs)?
  Technically possible but operationally fragile:

  • Requires SSH access to every instance (violates immutable infrastructure patterns)
  • If instances were terminated post-scale-down, logs are lost (unless forwarded to CloudWatch Logs beforehand)
  • Application logs show backend IPs (the ALB’s private IP), not the original client IP (unless your app parses X-Forwarded-For headers)
  • High MTTR (Mean Time To Resolution) - not acceptable for production incidents

The Technical Blueprint

#

CLI Implementation for SysOps:

# Step 1: Enable access logs (idempotent operation)
aws elbv2 modify-load-balancer-attributes \
  --load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/finstream-alb/a1b2c3d4 \
  --attributes \
    Key=access_logs.s3.enabled,Value=true \
    Key=access_logs.s3.bucket,Value=finstream-elb-logs \
    Key=access_logs.s3.prefix,Value=prod-alb

# Step 2: Create S3 bucket policy (required for ELB to write)
# Reference: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-logging-bucket-permissions
aws s3api put-bucket-policy --bucket finstream-elb-logs --policy file://elb-bucket-policy.json

# Step 3: Query logs with Athena (after creating Athena table)
aws athena start-query-execution \
  --query-string "SELECT client_ip, COUNT(*) FROM alb_logs WHERE time BETWEEN '2025-01-20T02:47:00Z' AND '2025-01-20T03:15:00Z' GROUP BY client_ip" \
  --result-configuration OutputLocation=s3://finstream-athena-results/ \
  --query-execution-context Database=default

Critical SysOps gotcha:
The S3 bucket policy must allow the ELB service principal (specific to your region) to PutObject. Missing this causes silent log delivery failures.

The Comparative Analysis

#

Cost note for FinOps:

• ELB access logs: ~10 GB/day for 100M requests = $0.23/day storage + Athena query costs
• Alternative: Stream to CloudWatch Logs Insights (higher cost but real-time filtering)

Real-World Application (Practitioner Insight)

#

Exam Rule

#

“For the SOA-C02 exam, when you see ‘IP address investigation’ + ‘Load Balancer traffic analysis’ → Pick ELB access logs. Always. No exceptions.”

Real World

#

“In reality, we combine approaches:

1. ELB access logs for forensic post-mortem analysis
2. CloudWatch Logs Insights with subscription filters for real-time DDoS detection (trigger Lambda → update WAF IP sets)
3. AWS WAF logs (if WAF is enabled) for Layer 7 attack pattern analysis

We also configure Log Exporter Lambda to parse ELB logs into a DynamoDB table for sub-second dashboard queries (Athena has 1-2 second latency). But for the exam? Keep it simple: ELB access logs = source IP truth.”

Production tip: Use the X-Forwarded-For header value in access logs (field #3) to get the original client IP when behind CloudFront or another proxy.

Stop Guessing, Start Mastering

#

Disclaimer

This is a study note based on simulated scenarios for the AWS SOA-C02 exam. The company “FinStreamX” is fictional. All technical configurations reflect AWS best practices as of January 2025.