Chapter 10 — Observability

Day 1 — CloudWatch Logs and Metrics

1. Concept Primer

CloudWatch Logs organizes output into log groups (usually one per service/function) and log streams (usually one per instance/execution). CloudWatch Metrics stores time-series numeric data, either emitted automatically by AWS services or published manually as custom metrics, and Alarms watch a metric against a threshold.

2. Hands-on Exercise

Tail the log group from a Lambda invocation, then publish and read back a custom metric.

3. Exact Commands

floci start && eval $(floci env)

# Reuse hello-fn from Chapter 7, or create a quick log group directly
aws logs create-log-group --log-group-name /day1/manual-log
aws logs create-log-stream --log-group-name /day1/manual-log --log-stream-name run-1

TIMESTAMP=$(($(date +%s%N)/1000000))
aws logs put-log-events \
  --log-group-name /day1/manual-log \
  --log-stream-name run-1 \
  --log-events "[{\"timestamp\":$TIMESTAMP,\"message\":\"hello from day1\"}]"

aws logs tail /day1/manual-log --since 5m

# If Chapter 7's hello-fn still exists, tail its real invocation logs too
aws logs tail /aws/lambda/hello-fn --since 15m

# Custom metric
aws cloudwatch put-metric-data \
  --namespace "Day1/Custom" \
  --metric-name RequestCount \
  --value 1 \
  --unit Count

aws cloudwatch get-metric-statistics \
  --namespace "Day1/Custom" \
  --metric-name RequestCount \
  --start-time $(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 60 \
  --statistics Sum

4. Gotchas

• put-log-events requires a millisecond-epoch timestamp, not seconds — an off-by-1000x timestamp is the most common reason events silently don't show up in tail.
• get-metric-statistics needs a start/end time window that actually brackets when you published the data point, or it returns an empty datapoints list even though the metric exists.
• Lambda's own log group (/aws/lambda/<function-name>) is created automatically the first time the function runs — it won't exist yet if you haven't invoked it.

5. Self-Check

Why would a get-metric-statistics call return zero datapoints even though put-metric-data succeeded — what's the most likely culprit?

---

Day 2 — Alarms and Basic X-Ray Tracing

1. Concept Primer

A CloudWatch Alarm transitions between OK/ALARM/INSUFFICIENT_DATA states based on a metric crossing a threshold over a defined number of evaluation periods. X-Ray traces a request as it moves across services, letting you see latency contribution per hop.

2. Hands-on Exercise

Create an alarm on the custom metric from Day 1, then push a minimal manual X-Ray segment.

3. Exact Commands

eval $(floci env)

aws cloudwatch put-metric-alarm \
  --alarm-name day2-high-request-count \
  --namespace "Day1/Custom" \
  --metric-name RequestCount \
  --statistic Sum \
  --period 60 \
  --evaluation-periods 1 \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold

aws cloudwatch describe-alarms --alarm-names day2-high-request-count

# Push several datapoints to try to trip the alarm
for i in 1 2 3 4 5 6; do
  aws cloudwatch put-metric-data \
    --namespace "Day1/Custom" --metric-name RequestCount --value 1 --unit Count
done

aws cloudwatch describe-alarms --alarm-names day2-high-request-count \
  --query 'MetricAlarms[0].StateValue'

# Minimal manual X-Ray segment
SEGMENT=$(python3 -c "
import json, time, uuid
print(json.dumps({
  'trace_id': f'1-{int(time.time()):x}-{uuid.uuid4().hex[:24]}',
  'id': uuid.uuid4().hex[:16],
  'name': 'day2-manual-segment',
  'start_time': time.time() - 1,
  'end_time': time.time()
}))
")

aws xray put-trace-segments --trace-segment-documents "$SEGMENT"

4. Gotchas

• Alarm state transitions depend on the evaluation engine actually running on schedule — in a local emulator, state changes may lag or need a manual nudge (re-running describe-alarms) rather than updating instantly like a real CloudWatch alarm loop.
• X-Ray's real value comes from SDK auto-instrumentation across services; the manual segment above is just enough to prove the API accepts and stores trace data, not a realistic tracing setup.

5. Self-Check

What's the difference between evaluation-periods and period in a CloudWatch alarm — why would you need more than one evaluation period for a "flaky" metric?