Lab 04 — CloudTrail deep dive¶

✅ Done (Part C skipped) 🧭 Security Logging and Monitoring 💲 ≈ $0.004

Services: CloudTrail, Organizations, KMS, S3, Athena

In one sentence

Build the org-wide audit trail the way the exam expects — one trail, every account, every region, KMS-encrypted, tamper-evident — then scope data events tightly and answer "who did what" with Athena.

The idea¶

  member 222222222222 ──┐                          (sees the trail, can't stop or change it)
  mgmt   111111111111 ──┼──► sec4-org-trail  (home us-east-2, multi-region, org trail)
                        │        │ validation ON → hourly digest files (SHA-256 + RSA signature)
                        │        │ SSE-KMS with alias/sec4-trail
                        │        ▼
                        │   s3://sec4-trail-111111111111/AWSLogs/o-exampleorg1/<account>/CloudTrail/<region>/yyyy/MM/dd/
                        │                     └── …/CloudTrail-Digest/…
                        │        ▲
                        │        └── Athena table sec4_db.org_trail (partition projection)
                        │
                        └── data events: S3 objects in sec4-data-111111111111 ONLY

Key policy statement	Who	Why
`EnableIAMPolicies`	account root	IAM policies can grant use of the key — that's how an admin decrypts the logs
`CloudTrailEncryptLogs`	`cloudtrail.amazonaws.com`	`kms:GenerateDataKey`, only for this trail* (`aws:SourceArn`) and a CloudTrail encryption context
`CloudTrailDescribeKey`	`cloudtrail.amazonaws.com`	CloudTrail checks the key when the trail is created/updated

What happened in the lab run¶

Part A + B — key, bucket, org trail¶

Step	Result
Trusted access for `cloudtrail.amazonaws.com`, then `create-trail --is-organization-trail --is-multi-region-trail --enable-log-file-validation`	✅
Logging before `start-logging`	❌ a new trail doesn't log until started
First log files	✅ under `AWSLogs/o-exampleorg1/<account>/…` for both accounts
Member admin: `describe-trails`	✅ sees the org trail (read-only shadow)
Member admin: `stop-logging` / `delete-trail`	❌ denied
Member admin: read the trail bucket	❌ denied — the bucket lives in the management account

Part C — log file validation¶

Step	Result
`validate-logs`, tamper with a log, delete a log	⚠️ not run — the first digest file takes up to ~1 hour. Exam notes kept below.

Part D — advanced event selectors¶

Step	Result
Selector 1: management events, excluding `kms.amazonaws.com`	✅
Selector 2: S3 data events for `sec4-data-…/` only	✅
`PutObject` / `GetObject` sent ~5 s after `put-event-selectors`	❌ not logged — selector changes take a few minutes to take effect
Same calls re-run a minute later	✅ logged

Part E — querying¶

Tool	Result
Event history `lookup-events` (`StartLogging`)	✅ — this account, this region, management events only
Athena table with partition projection (no crawler, no `MSCK REPAIR`)	✅ who-did-what, AccessDenied list, S3 data events — KBs scanned thanks to `WHERE day = …`

Key lessons¶

An org trail is the exam's audit baseline: created in the management account (or a CloudTrail delegated admin), auto-applied to every current and future account, and members can't stop, change or read it.
SSE-KMS on the trail is a second, separate read control — S3 read access alone can't decrypt the logs; the reader also needs kms:Decrypt on the key.
Pin service principals with aws:SourceArn / aws:SourceAccount in bucket and key policies, or any trail anywhere could write to your bucket (confused deputy).
Data events are off by default, charged and huge — scope them with advanced selectors, and remember put-event-selectors replaces everything, and takes a few minutes to apply.
Pick the query tool by need: event history for a quick 90-day look, Athena for cheap SQL over years, CloudTrail Lake for a managed store, EventBridge for real-time alerts.

Exam notes¶

CloudTrail puts encryption context aws:cloudtrail:arn = <trail ARN> on each log's data key. The KMS key must be in the same region as the bucket. Default (no key) = SSE-S3; "a key we control and every decrypt audited" = SSE-KMS with a customer managed key.
The bucket policy needs a PutObject statement for AWSLogs/<mgmt-account>/* and for AWSLogs/<org-id>/* — the org trail writes every account's logs under the org ID.
Multi-region = every region, including ones enabled later. Global service events (IAM, STS, CloudFront) are logged in us-east-1.
"Stop anyone disabling CloudTrail": an SCP denying cloudtrail:StopLogging / DeleteTrail / UpdateTrail / PutEventSelectors for members plus an EventBridge rule on those calls → SNS. Remember the SCP doesn't cover the management account.
Log file validation (Part C): hourly digest files hold the SHA-256 of every log delivered that hour plus the previous digest's hash/signature (a chain), RSA-signed by AWS. It detects modification and deletion (and proves no log was delivered in an hour, or a digest is missing) — it does not prevent them. Prevent with S3 Object Lock, MFA Delete, a separate log-archive account, a tight bucket policy. Files can be moved if the folder structure is kept (validate-logs --s3-bucket/--s3-prefix).
Management events = control plane (CreateBucket, RunInstances); data events = data plane (GetObject, Lambda Invoke, DynamoDB items). "Who downloaded this object?" needs S3 data events (or best-effort S3 server access logs) — not in event history.
Advanced selectors filter on eventName, resources.ARN, readOnly, userIdentity.arn…; basic selectors only read/write/all per resource. Excluding KMS events cuts a big share of management-event volume.
Network activity events and Insights events (unusual API call / error rates) are extra, charged options.
Event history: free, always on, 90 days, management events, one account + one region per call, 2 req/s.
Athena + partition projection: partitions derived from the S3 path → no Glue crawler, scans only the filtered prefixes. CloudTrail Lake: managed, immutable, SQL, up to 10 years, charged per GB ingested.
First copy of management events per region is free; additional trails copying them are charged.
Delete the trail before scheduling its KMS key for deletion — a pending-deletion key can't encrypt, so delivery fails ("logs stopped arriving").

Review questions¶

Question 1

A member-account admin with AdministratorAccess runs cloudtrail stop-logging on the org trail. What happens, and what two controls stop or alert on it if someone gets into the management account?

Show answer

It's denied — an org trail is owned by the management account; members only see a read-only shadow copy and can't stop, update or delete it. For the management account itself: an SCP can't restrict it, so rely on (1) tight IAM in the management account (few principals, MFA, no workloads; optionally run the trail from a CloudTrail delegated admin security account) and (2) an EventBridge rule on StopLogging / DeleteTrail / UpdateTrail → SNS alert (optionally a Lambda that restarts logging, or the Config rule for CloudTrail enabled). The SCP denying those actions protects against member accounts.

Question 2

The security team must prove that last month's logs weren't changed. What feature, what command, and what does it not guarantee?

Show answer

Log file integrity validation (digest files) checked with aws cloudtrail validate-logs over the time range. It cryptographically proves each log file is unchanged and none is missing. It does not prevent modification or deletion — it only detects it after the fact. To prevent, use S3 Object Lock (compliance mode), MFA Delete and a separate log-archive account.

Question 3

Auditors ask who downloaded payroll.csv from S3 last week. Event history shows nothing. Why, and what should have been on?

Show answer

GetObject is a data event; event history only contains management events, and data events are off by default. The trail needed S3 data events (read) for that bucket/prefix via event selectors — or, at least, S3 server access logging as a cheaper best-effort record. Neither can be turned on retroactively.

Question 4

You change the trail's event selectors to log Lambda invokes. Next day, CreateUser events are missing from the trail. What went wrong?

Show answer

PutEventSelectors replaces all selectors. The new configuration only had the Lambda data-event selector and dropped the management-events selector, so the trail silently stopped logging management events like CreateUser. Always resend the management selector together with the new data selector.

Question 5

The log bucket policy lets cloudtrail.amazonaws.com write with no conditions. What's the risk, and what condition key fixes it?

Show answer

Confused deputy: any CloudTrail trail in any AWS account could be pointed at your bucket and write into it (polluting logs, costing you storage). Add aws:SourceArn (your trail's ARN) and/or aws:SourceAccount conditions so CloudTrail can only write on behalf of your trail/account.

Question 6

Logs stopped arriving after someone "cleaned up unused KMS keys". What happened, and how do you recover within the waiting period?

Show answer

The trail's KMS key was scheduled for deletion (or disabled). A key pending deletion can't be used, so CloudTrail can't generate data keys and log delivery fails (get-trail-status shows a delivery error). Within the 7–30 day window run CancelKeyDeletion — the key comes back disabled — then EnableKey. Delivery resumes; events from the gap may be lost from S3 but are still in event history (management events, 90 days).

Question 7

Pick a query tool: (a) org-wide SQL over 3 years of logs at the lowest cost, (b) alert within a minute when someone calls DeleteTrail, (c) quick check of who terminated an instance this morning.

Show answer

(a) Athena over the org trail's S3 bucket, with partition projection to keep scans small (CloudTrail Lake also works but is charged per GB ingested). (b) An EventBridge rule matching DeleteTrail → SNS (or a CloudWatch Logs metric filter + alarm if the trail sends to CloudWatch Logs). (c) Event history (lookup-events) in that account and region — free and immediate for management events.

Gotcha from the run

Advanced event selectors aren't instant: data events sent seconds after put-event-selectors were dropped. Wait a couple of minutes before testing — and don't mistake that gap for a broken selector.