Lab 07 — VPC & network controls¶
In one sentence
Build a private subnet with no internet at all, reach AWS through VPC endpoints, lock an S3 bucket to the endpoint and the endpoint to the bucket, then break traffic with a stateless NACL and find the evidence in VPC Flow Logs.
The idea¶
sec7-vpc 10.77.0.0/24 (no IGW, no NAT) S3 (regional, public IPs)
┌──────────────────────────────────────────────────┐ ┌───────────────────────────────┐
│ subnet sec7-private 10.77.0.0/26 NACL sec7-nacl │ │ sec7-data-… bucket policy: │
│ │ gateway │ Get/Put only if │
│ sec7-box (EC2) ──443──► interface endpoints │ endpoint│ aws:SourceVpce = our vpce │
│ SG: no inbound ssm / ssmmessages / │◄───────►│ sec7-other-… (IAM allows it, │
│ ec2messages (ENIs here)│ (route │ endpoint policy doesn't) │
│ SG: 443 from VPC only │ table) └───────────────────────────────┘
└──────────────────────────────────────────────────┘
│ Flow Logs (all traffic, 1-min aggregation) → CloudWatch Logs /sec7/flowlogs
What happened in the lab run¶
Endpoints and the two-sided S3 perimeter¶
| # | From | Test | Result | Decided by |
|---|---|---|---|---|
| 1 | box | resolve ssm.us-east-2.amazonaws.com |
✅ 10.77.0.x (endpoint ENI) | private DNS |
| 2 | box | reach the internet | ❌ no internet | no IGW / NAT |
| 3 | box | resolve S3 | ⚠️ still a public S3 IP | gateway endpoint = a route, not DNS |
| 4 | box | put/get in the data bucket | ✅ | |
| 5 | box | put to the other bucket | ❌ AccessDenied — "no VPC endpoint policy allows" | endpoint policy (IAM allowed it) |
| 6 | laptop (admin) | GetObject from the data bucket | ❌ AccessDenied | bucket policy aws:SourceVpce |
| 7 | laptop | list the data bucket | ✅ | ListBucket isn't in the Deny |
SG vs NACL, seen in Flow Logs¶
| Step | Result |
|---|---|
| NACL: inbound TCP 443 only, all outbound | ❌ S3 copy → socket timeout |
| SSM command to the box, same NACL | ✅ still works — box and SSM endpoint ENIs are in the same subnet |
| Add inbound 1024–65535 | ✅ S3 works again |
| Flow logs for the failing connection | request 10.77.0.45:58248 → 198.51.100.10:443 ACCEPT egress (pkt-dst-aws-service=S3); reply 198.51.100.10:443 → 10.77.0.45:58248 REJECT ingress |
Logs Insights with my own parse … as src |
❌ "Ephemeral field is already defined" — Insights auto-discovers flow-log fields (srcAddr, dstPort, action, flowDirection…) even for a custom format |
Cost note: the 3 interface endpoints were most of the bill — each partial hour is billed as a full hour per endpoint per AZ.
Key lessons¶
- Effective access = IAM AND endpoint policy AND bucket policy. The endpoint policy stops data leaving to other buckets; the bucket policy stops data being read from outside the VPC.
- Gateway endpoints don't change DNS. S3 still resolves to public IPs; traffic is steered by the prefix-list route. Interface endpoints are the ones with private IPs and private DNS.
- NACLs are stateless and only filter at the subnet boundary. Forget the ephemeral inbound range and replies die; traffic inside the subnet is never filtered.
- ACCEPT out + REJECT on the reply = NACL. An SG can never produce that pattern.
Exam notes¶
- Interface endpoint (PrivateLink): ENIs in your subnets, has an SG, hourly per AZ + per GB, reachable
from on-prem/peered VPCs. Private DNS needs
enableDnsSupport+enableDnsHostnames. - Gateway endpoint: S3 and DynamoDB only, free, a route-table entry, no SG/ENI, not reachable from on-prem or peered VPCs (use an S3 interface endpoint for that).
- SSM in a private subnet: ssm + ssmmessages + ec2messages endpoints (+ S3 gateway for patching, + kms / logs if Session Manager encrypts or logs). No inbound SG rule needed — the agent connects outbound.
- Bucket policy:
aws:SourceVpce(one endpoint) oraws:SourceVpc(the VPC).aws:SourceIpdoesn't work through an endpoint — useaws:VpcSourceIpfor the private IP. - Scalable endpoint policy: allow only resources in your org (
aws:ResourceOrgID). - Lockout trap: bucket-wide
Deny s3:*unlessSourceVpcelocks out admins and the console too; only the root user can then delete the bucket policy. - SG: ENI level, stateful, allow-only, can reference SGs/prefix lists. NACL: subnet level, stateless, allow and deny, numbered, first match wins, CIDRs only. A new custom NACL denies all; the default NACL allows all.
- Block one IP → NACL deny with a lower rule number than the allows. Ephemeral ports: answer 1024–65535.
- Flow logs: VPC / subnet / ENI; → CloudWatch Logs (needs a role), S3 or Firehose. Metadata only — payload needs Traffic Mirroring. Not logged: Amazon DNS resolver queries, 169.254.169.254 metadata, Time Sync, DHCP, Windows licence activation. Format can't be changed after creation.
Review questions¶
Question 1
A Lambda function in a private subnet must write to DynamoDB without a NAT gateway, at no extra hourly cost. What do you add?
Show answer
A DynamoDB gateway endpoint, associated with the route table of the Lambda's subnets. Gateway endpoints (S3 and DynamoDB only) are free and work by adding a prefix-list route, so no NAT or internet path is needed. An interface endpoint would also work but costs per hour per AZ.
Question 2
A bucket policy allows aws:SourceIp = the company's public IP range. Instances in the VPC now get AccessDenied since someone added an S3 gateway endpoint. Why, and what's the fix?
Show answer
Through the endpoint, requests arrive with the instance's private source IP, not the NAT's public IP,
so aws:SourceIp no longer matches. Add a condition allowing aws:SourceVpce (or aws:SourceVpc)
for that endpoint/VPC alongside the IP condition (or aws:VpcSourceIp for private ranges).
Question 3
How do you stop an insider on an EC2 instance from copying data to their own S3 bucket in another account, even with valid credentials?
Show answer
Make the S3 endpoint the only way out (no IGW/NAT path to S3) and give it an endpoint policy that
allows only your buckets — scalably, aws:ResourceOrgID = your org (or aws:ResourceAccount). IAM can't
help here because the insider could use credentials from the other account; the endpoint policy
applies to every request through the endpoint regardless of whose credentials sign it.
Question 4
Flow logs show ACCEPT outbound to port 443 and REJECT inbound from the same IP on port 443 → ephemeral. What's misconfigured?
Show answer
The network ACL lacks an inbound rule for the return traffic on the ephemeral port range (1024–65535). NACLs are stateless, so replies must be allowed explicitly. A security group can't cause this — SGs are stateful and always allow replies to permitted outbound traffic.
Question 5
Security asks to block a single attacking IP for a web tier behind an ALB. SG or NACL? Why?
Show answer
NACL — security groups only have allow rules, they can't deny one IP. Add a deny entry with a rule number lower than the allows, on the ALB's subnets (behind the ALB, the web instances only see the ALB's IPs). For an HTTP app, a WAF IP-set rule on the ALB is often the better answer in practice.