Lab 11 — Incident response drill

✅ Done 🧭 Threat Detection and Incident Response 💲 ≈ $0.02

Services: IAM, STS, CloudTrail, EC2, SSM, EBS

In one sentence

Practise the two most-tested IR playbooks in the order AWS recommends — contain → preserve evidence → investigate → eradicate — for a leaked access key and for a compromised EC2 instance.

The idea

 Part A  leaked key                          Part B  compromised instance
 ─────────────────────────                   ─────────────────────────────────────────────
 attacker: AKIA… key                         victim ── 1 metadata + termination protection + tags
   ├─ list buckets, describe EC2                    ── 2 volatile data via SSM (ps, users, files, net)
   ├─ GetSessionToken → ASIA… creds  (!)            ── 3 EBS snapshot (evidence)
   ▼                                                ── 4 isolate: SG with no rules
 you: 1 deactivate key                                        │ snapshot
      2 session creds STILL work                              ▼
      3 deny-all policy → now dead          forensics box ◄── volume from snapshot, mounted read-only
      4 CloudTrail: what did it do?

What happened in the lab run

Part A — leaked key

Time	Who	Action	Result
08:16:17	attacker AKIA…	whoami, list buckets, describe EC2	✅ allowed by the user's policy
08:16:23	attacker	iam create-user backdoor	❌ AccessDenied — but logged
08:16:24	attacker	sts get-session-token → ASIA… credentials	✅
08:16:29	defender	deactivate the access key
08:16:43	attacker, old key	list buckets	❌ InvalidAccessKeyId
08:16:47	attacker, session creds	list buckets	✅ still works
08:16:49	defender	attach inline Deny * policy
08:17:05	attacker, session creds	list buckets	❌ explicit deny — attacker is out

CloudTrail event history showed all of it — except two surprises:

• CreateUser was not in us-east-2: IAM is global, its events are logged in us-east-1.
• The calls with the deactivated key weren't logged at all (failed authentication isn't attributed), and the session's ASIA… key ID differs from the leaked AKIA… one → search by user name, not key ID.

Part B — compromised instance

Step	Result
Termination protection	✅ terminate-instances → OperationNotPermitted
Volatile capture via SSM	✅ "miner" process running from /dev/shm, backdoor user support2, hidden /var/tmp/.x/miner.sh, attacker's bash history, open connections
Snapshot, then isolation SG	✅ snapshot in 53 s
SSM after isolation	⚠️ still answered — already-tracked connections survive an SG change
Mount the copy without nouuid	❌ "duplicate UUID" (copy and forensics box come from the same AMI)
Mount -o ro,nouuid	✅ hidden file, backdoor user, history found; write → Read-only file system
The running miner on the disk copy	❌ not there — it lived only in RAM (tmpfs); only the volatile capture saw it

Key lessons

1. Deactivating or deleting a key does not revoke temporary credentials minted from it. Only an explicit Deny (user policy or SCP) stops them immediately.
2. Volatile data first. Stopping, rebooting or terminating destroys RAM-only evidence.
3. An isolation SG doesn't cut existing connections (SSM agent, reverse shells). Stop the instance after memory capture, or use a deny-all NACL on a subnet holding only affected workloads.
4. Analyse a copy, read-only, ideally in a separate forensics account.

Exam notes

• Leaked key playbook: deactivate → deny-all / quarantine policy → investigate CloudTrail (incl. us-east-1) → remove what the attacker created (users, keys, roles, instances, Lambda) → rotate → delete the key.
• For roles: Revoke active sessions = AWSRevokeOlderSessions (Deny if aws:TokenIssueTime < now).
• AWS attaches AWSCompromisedKeyQuarantineV3 itself when it finds your key on public GitHub.
• EC2 order: termination protection → detach from ASG / deregister from ELB → capture memory → EBS snapshot → isolate → analyse a copy.
• Share evidence snapshots cross-account with a customer managed KMS key (AWS managed keys can't be shared).
• Detective = investigate root cause/scope over time (needs GuardDuty). GuardDuty = detect. Security Hub = aggregate. Inspector = CVEs. Config = configuration history.

Review questions

Question 1

A developer pushed an access key to a public repo 2 hours ago. You deactivate it, yet CloudTrail still shows new API calls from that user with ASIA… key IDs. What happened, and how do you stop it?

Show answer

The attacker used the key to call GetSessionToken and got temporary credentials (ASIA…). Deactivating or deleting the long-term key doesn't revoke them; they stay valid until they expire (up to 36 h). Attach an explicit deny-all policy to the user — it's evaluated on every call, so the session dies immediately — then investigate and clean up.

Question 2

Put these in order for a compromised EC2 instance: snapshot EBS, isolate with SG, capture memory, enable termination protection, detach from ASG.

Show answer

Termination protection → detach from ASG → capture memory → snapshot EBS → isolate. Protect it, stop AWS from replacing it, grab what disappears first (RAM), then what lasts (disk), then cut it off. (Isolating earlier is acceptable if memory is still captured before any stop/reboot.)

Question 3

Why should you not simply stop the instance as the first step?

Show answer

Stopping wipes memory: running processes, network connections, RAM-only files (/dev/shm, tmpfs /tmp) and fileless malware are lost. In the lab, the miner existed only in RAM — the disk snapshot didn't contain it.

Question 4

The forensics team in another account can't use the evidence snapshot you shared. It was encrypted with the default aws/ebs key. Why, and what do you do?

Show answer

AWS managed keys can't be shared across accounts (you can't edit their key policy). Copy the snapshot, re-encrypting with a customer managed key, allow the forensics account in that key's policy, share the new snapshot, and have them copy it into their account. Don't "decrypt it first" — that's not possible for a snapshot, and unencrypted evidence is bad practice.

Question 5

GuardDuty, Detective, Security Hub, Inspector: which one answers "what else did this compromised role do over the last 3 months?"

Show answer

Amazon Detective — its behaviour graph (CloudTrail, VPC Flow Logs, GuardDuty findings; up to a year) is built for "what else did this entity do". AWS Config is about resource configuration, not behaviour.

My first attempt: 1 / 5

Weak spots: temp credentials outlive the key · volatile evidence first · Detective (behaviour) vs Config (configuration) · sharing encrypted snapshots needs a customer managed key.