Lab 09 — Secrets & certificates¶
In one sentence
Store config and credentials the right way — Parameter Store for cheap config and small secrets, Secrets Manager when you need automatic rotation — lock down who can read them, then see how ACM treats public certificates (auto-renew) versus imported ones (never renewed).
The idea¶
Parameter Store Secrets Manager ACM
───────────────────── ─────────────────────────────── ─────────────────────
/sec9/app/db-host String sec9/app/db {"username","password"} public cert request
/sec9/app/db-password SecureString versions: AWSCURRENT / AWSPENDING / → PENDING_VALIDATION
(KMS aws/ssm) AWSPREVIOUS (DNS CNAME to prove)
▲ ▲ rotation ◄── Lambda sec9-rotate imported self-signed cert
│ GetParameter (host only) │ GetSecretValue → no auto-renew
sec9-app-role ─────────────────────────────┘
What to expect when you run it¶
Part A — Parameter Store¶
| Test | Expected result |
|---|---|
GetParameter on the SecureString without WithDecryption |
⚠️ returns KMS ciphertext (AQICAH…), not the value |
| Same call with decryption | ✅ plaintext password |
describe-parameters key ID |
alias/aws/ssm — the AWS managed key, used when you don't pass one |
Read an old version with name:1 |
✅ the first password comes back from the parameter history |
App role reads /sec9/app/db-host |
✅ allowed by the path-scoped policy |
App role reads /sec9/app/db-password |
❌ AccessDenied — the policy only grants the host parameter |
The app role never needs kms:Decrypt for aws/ssm: that AWS managed key lets any principal in the account
use it through SSM. With a customer managed key, it would.
Part B — Secrets Manager rotation¶
| Test | Expected result |
|---|---|
App role GetSecretValue |
✅ |
Read the secret via /aws/reference/secretsmanager/sec9/app/db |
✅ Parameter Store can proxy Secrets Manager secrets |
RotateSecret with the Lambda, 30-day schedule |
✅ rotates immediately; 4 Lambda invocations: createSecret → setSecret → testSecret → finishSecret |
Read AWSCURRENT / AWSPREVIOUS |
✅ new password / the old one |
| Rotate a second time | ⚠️ the oldest version loses its label (becomes deprecated) |
Resource policy that grants public access, with BlockPublicPolicy |
❌ rejected |
| Resource policy that denies everyone except the app role | ❌ you (admin) can't read the secret · ✅ the app role can |
| Delete that resource policy | ✅ still allowed — the deny only covered GetSecretValue |
Part C — ACM¶
| Test | Expected result |
|---|---|
| Request a public cert for a domain you don't own (DNS validation) | ⚠️ stays PENDING_VALIDATION (or FAILED); shows the CNAME you'd have to create |
| Import a self-signed cert valid for 2 days | ✅ Type: IMPORTED, RenewalEligibility: INELIGIBLE |
Key lessons¶
- Rotation = Secrets Manager. Parameter Store has no built-in rotation. Rotation is a Lambda with four
steps, and apps keep reading
AWSCURRENT, so it's invisible to them. - Customer managed key = two permissions. Reading a SecureString or secret encrypted with a CMK needs the
service permission and
kms:Decrypton that key. - A resource-policy Deny beats your admin rights, and
BlockPublicPolicystops anyone from attaching a public policy to a secret. - ACM only renews what it issued. Imported certificates expire silently unless you monitor them.
Exam notes¶
- Parameter Store:
String,StringList,SecureString(KMS). Standard tier: free, 4 KB, 10,000 params. Advanced: 8 KB, parameter policies (expiration, notification), paid. Scope IAM by path (parameter/app/prod/*). Reference from CloudFormation with{{resolve:ssm-secure:…}}. - Secrets Manager: staging labels AWSCURRENT / AWSPENDING / AWSPREVIOUS. AWS-provided rotation functions for RDS/Aurora/Redshift/DocumentDB: single-user vs alternating-users. RDS can manage its master password in Secrets Manager itself.
- Rotation Lambda in a VPC times out → it needs a route to Secrets Manager: interface VPC endpoint or NAT (and network access to the DB).
- Cross-account secret: resource policy on the secret + a customer managed KMS key whose key policy
allows the other account. The
aws/secretsmanagerkey can't be shared. DeleteSecret→ recovery window 7–30 days (default 30);RestoreSecretundoes it.- Secrets Manager also does replica secrets (multi-region DR) and random password generation.
- ACM: prefer DNS validation (auto-renews while the CNAME stays). Certs are regional; CloudFront needs the cert in us-east-1, an ALB needs it in the ALB's region. ACM public certs can't be installed on EC2 directly (except Nitro Enclaves or the paid exportable certs).
- Imported cert monitoring: CloudWatch
DaysToExpirymetric, EventBridge "ACM Certificate Approaching Expiration" event, Config ruleacm-certificate-expiration-check. - ACM Private CA = internal TLS / mTLS, ~$400/month (short-lived certificate mode is cheaper).
- Which service? Rotation → Secrets Manager · free config/flags/AMI IDs → Parameter Store · encrypt your own data → KMS (not a secret store) · hard-coded secrets in code → CodeGuru Reviewer / Amazon Q, then move them to Secrets Manager.
Review questions¶
Question 1
A Lambda function in a private subnet rotates an RDS password but every rotation times out. The Lambda's role has all Secrets Manager permissions. What's missing?
Show answer
A network path to the Secrets Manager API. A Lambda in a private subnet has no internet access, so its
calls to Secrets Manager (GetSecretValue, PutSecretValue, UpdateSecretVersionStage) hang until the
function times out. Add a Secrets Manager interface VPC endpoint (or a NAT gateway), and make sure the
Lambda's security group can reach both the endpoint and the database. IAM is not the problem — a
permissions issue would return AccessDenied, not a timeout.
Question 2
A partner account must read one secret. You added a resource policy allowing their role, but they get AccessDenied. The secret uses the default key. Why?
Show answer
The secret is encrypted with the AWS managed key aws/secretsmanager, and AWS managed keys can't be used
by other accounts — you can't edit their key policy. Re-encrypt the secret with a customer managed KMS
key whose key policy allows the partner role kms:Decrypt. The partner also needs IAM permissions in
their own account for secretsmanager:GetSecretValue and kms:Decrypt. Cross-account = resource policy
+ key policy + caller's IAM policy.
Question 3
Your ECS task reads a SecureString encrypted with a customer managed key and gets AccessDenied, even though the task role allows ssm:GetParameter. Fix?
Show answer
Grant kms:Decrypt on that CMK to the task role — in the task role's IAM policy and allowed by the
key policy (directly, or because the key policy delegates to IAM). Decrypting a SecureString is a KMS
call made on the caller's behalf, so ssm:GetParameter alone isn't enough. With the AWS managed
aws/ssm key this extra permission isn't needed, which is why it often "works in dev".
Question 4
An imported certificate on an ALB expired over the weekend. What should have warned you, and how do you avoid it next time?
Show answer
ACM never renews imported certificates. Warnings were available from the CloudWatch metric
DaysToExpiry (alarm on it), the EventBridge "ACM Certificate Approaching Expiration" event
(sent daily from 45 days before expiry), and the Config rule acm-certificate-expiration-check. The
lasting fix: replace it with an ACM-issued public certificate with DNS validation, which renews
automatically. If you must import, alarm on those signals.
Question 5
You need a TLS certificate for a CloudFront distribution; you created it in eu-west-1 and can't select it. Why?
Show answer
CloudFront is a global service and only uses ACM certificates from us-east-1 (N. Virginia). ACM certs are regional and can't be moved or copied, so request (or import) a new certificate in us-east-1. Regional services like an ALB are the opposite: the cert must be in the ALB's own region.
Not run yet
Watch-outs when running it: the ACM request for example.com may go straight to FAILED (that's fine);
cleanup uses force-delete for the secret — in production use a 7–30 day recovery window.