AWS Secrets Manager and KMS: Protect Keys and Secrets Explained
Juniors still put database passwords in Lambda environment variables in plaintext. That is a gift to anyone who can read the config. AWS Secrets Manager stores the secret encrypted; AWS KMS holds the keys that encrypt. Together they are the lock and the key factory.
What you'll learn in this chapter
- Why plaintext secrets fail audits
- Secrets Manager vs Parameter Store
- KMS CMK concepts: key policy, grants
- Rotation and RDS integration basics
- Lab, errors, FAQ, interview
1. Everyday example first
A hospital app connects to RDS. Password rotates every 30 days. Secrets Manager can rotate with a Lambda; apps call GetSecretValue at runtime (or cache briefly). KMS CMK encrypts the secret string. If a laptop is stolen, the thief still needs IAM + KMS permission.
2. Why both services?
Why do we need this?
KMS is about keys and cryptographic operations. Secrets Manager is about storing rotating application secrets. Parameter Store (SSM) can store SecureStrings with KMS too — pick Secrets Manager when you need managed rotation and fine secret lifecycle.
| Service | Use for |
|---|---|
| KMS | Create/control CMKs, encrypt data keys |
| Secrets Manager | DB creds, API tokens, rotation |
| SSM Parameter Store | Config and simpler secrets |
3. How to wire it
How to do it (step by step)
- Create a CMK (or use aws/secretsmanager carefully). Prefer customer CMK for audit.
- Create a secret; choose RDS rotation if supported.
- Grant the app role secretsmanager:GetSecretValue on that ARN and kms:Decrypt.
- Never log the secret string.
- Alert on unusual GetSecretValue spikes.
- Document break-glass procedure.
{
"Effect": "Allow",
"Action": ["secretsmanager:GetSecretValue"],
"Resource": "arn:aws:secretsmanager:ap-south-1:111122223333:secret:prod/db-*"
}4. Industries
- BFSI: mandatory rotation + CMK where required
- SaaS multi-tenant: per-tenant API tokens
- Healthcare: audit who decrypted
- Startups: stop Slack-shared passwords
5. Lab: create a dummy secret and read it
Do this in ap-south-1 (Mumbai) unless your org forbids it. Set a budget alert first.
How to do it (step by step)
- KMS → create symmetric CMK alias alias/lab-secrets.
- Secrets Manager → store new secret lab/demo string not-a-real-password.
- Encrypt with your CMK.
- From CloudShell get-secret-value in ap-south-1 with a lab role.
- Note CloudTrail shows the API call.
Cleanup (stop billing)
- Delete the secret schedule and schedule CMK deletion (7–30 days).
- Remove lab IAM policies.
6. Common errors and fixes
Each row is a real mistake students and juniors hit. Read the fix before you panic.
| Error / symptom | Fix |
|---|---|
| Secret in Lambda env plaintext | Move to Secrets Manager; inject at runtime. |
| Resource star on GetSecretValue | Pin secret ARN. |
| Disabled CMK broke prod | Monitor CMK state; staged disable. |
| Rotation Lambda fails silently | Alarm on rotation errors. |
| Cached secret forever in memory | Short TTL cache; refresh on auth errors. |
| Shared secret across 40 apps | Split secrets; least blast radius. |
| Put secret in AMI | Bake no secrets; fetch at boot. |
| Forgot replica for multi-region | Replicate secret + key strategy. |
| Developers all Admin | Separate read roles; break-glass only. |
| Printed secret in CI logs | Mask; use OIDC roles. |
7. Troubleshooting checklist
- AccessDeniedException → IAM + key policy both.
- Secret pending deletion — restore if needed.
- Rotation: check Lambda VPC/SG to DB.
- Decrypt failures after key policy edit — rollback policy.
- Wrong region secret id.
- CloudTrail not showing? Include management events.
8. FAQ
Q. Secrets Manager vs SSM SecureString?
Secrets Manager shines at rotation and secret lifecycle.
Q. Who can decrypt?
Principals allowed by key policy AND IAM.
Q. Envelope encryption?
KMS encrypts data keys; data keys encrypt bulk data.
Q. Cost worry?
Secrets and API calls cost; delete unused.
Q. Can I use CMK with S3/EBS too?
Yes — same CMK discipline.
9. Interview Q&A
Q. Why not env vars for DB passwords?
Visible in console/config; hard to rotate; often logged.
Q. What does KMS do?
Manages keys and encrypt/decrypt/generate data keys.
Q. CMK vs AWS managed key?
CMK: you control policy and rotation options.
Q. How rotation works?
Lambda updates DB and secret versions in stages.
Q. Least privilege for secrets?
GetSecretValue on exact ARN + kms:Decrypt on that CMK.
Q. Detect misuse?
CloudTrail + metric alarms on GetSecretValue.
10. Tips in English, Marathi and Hindi
Ravindra Bagale's Tip (EN)
Treat every secret like a house key: know who has a copy and change the locks on a schedule.
Ravindra Bagale's Tip (MR)
Secret = घरची चावी: copy कोणाकडे, schedule वर locks बदला.
Tip (HI)
हर secret घर की चाबी है — किसके पास copy है जानो और समय पर ताले बदलो।
Try at home
- Scan one personal project for hardcoded passwords.
- Write IAM policy for one secret ARN only.
- Explain envelope encryption to a friend in 60 seconds.
In short
- Secrets Manager stores rotating secrets.
- KMS holds and controls encryption keys.
- IAM + key policy together gate decrypt.
- No plaintext secrets in env, AMI, or Git.
Revision, interview and comparison cards
Revise this lesson in a few minutes. Save the cards on your phone or download them as PNG files. The QR code on each card opens this lesson.
More cards: Ravi Sir's Cards.
Practice Tasks
Amravati Agro Market is a agriculture business in Amravati. Anjali Thakre looks after its IT and cloud work. They count money in INR and run their AWS resources in the ap-south-1 Region. In the tasks below you do real jobs for them using what this lesson teaches. All names, account IDs, IP addresses and numbers are made up for practice.
0 / 16
Easy List secrets
Why: Inventory secrets.
What to do: List secrets.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Easy Describe secret
Why: Metadata without value.
What to do: Describe secret amravatiagro/db.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Easy List keys
Why: See CMKs.
What to do: List KMS keys.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Easy List aliases
Why: Human names for keys.
What to do: List aliases.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Easy Min recovery window days
Why: Delete schedule.
What to do: Minimum recovery window days often used in labs (enter 7)?
- Write down the numbers the task gives you.
- Work it out step by step on paper.
- Type only the final answer in the box.
- Click Check answer.
Easy Plaintext env bad — enter 0 for allowed?
Why: Never plaintext.
What to do: Is plaintext DB password in Lambda env OK? Enter 0 for no.
- Write down the numbers the task gives you.
- Work it out step by step on paper.
- Type only the final answer in the box.
- Click Check answer.
Medium Get secret value
Why: Runtime fetch.
What to do: Get secret value amravatiagro/db.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Medium Describe key
Why: Check state/policy.
What to do: Describe key alias/lab-secrets.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Medium Decrypt on one key
Why: Apps need decrypt.
What to do: Allow kms:Decrypt on arn:aws:kms:ap-south-1:120348739971:key/abcd-efgh.
- Start with { "Version": "2012-10-17", "Statement": [ ... ] }.
- In each statement write Effect, Action and Resource.
- Give only what the task needs (least privilege).
- Use double quotes and commas correctly. Click Check answer.
Medium Secret string JSON
Why: Structured secrets.
What to do: JSON username app password changeme.
- Start from the top-level keys the task names.
- Indent child keys with 2 spaces (no tabs).
- Fill in each property with the value from the task.
- Click Check answer. Key order does not matter.
Medium IAM and key policy both needed — 1
Why: Two gates.
What to do: Enter 1 if decrypt needs IAM and key policy.
- Write down the numbers the task gives you.
- Work it out step by step on paper.
- Type only the final answer in the box.
- Click Check answer.
Medium Short TTL minutes classroom
Why: Do not cache forever.
What to do: A short classroom secret cache TTL in minutes (enter 5)?
- Write down the numbers the task gives you.
- Work it out step by step on paper.
- Type only the final answer in the box.
- Click Check answer.
Hard Rotate secret
Why: Trigger rotation.
What to do: Rotate secret amravatiagro/db.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Hard Delete secret schedule
Why: Cleanup labs safely.
What to do: Delete secret amravatiagro/lab with recovery window 7.
- Read the task and decide which AWS service it is about.
- Write aws, then the service name, then the operation (the action word).
- Add each needed flag (--name) followed by its value from the task.
- Flags can be in any order. Click Check answer.
Hard GetSecretValue pinned
Why: No star resources.
What to do: Allow secretsmanager:GetSecretValue on arn:aws:secretsmanager:ap-south-1:120348739971:secret:amravatiagro/db-*.
- Start with { "Version": "2012-10-17", "Statement": [ ... ] }.
- In each statement write Effect, Action and Resource.
- Give only what the task needs (least privilege).
- Use double quotes and commas correctly. Click Check answer.
Hard Rotation days JSON
Why: Schedule rotation.
What to do: JSON RotationDays 30.
- Start from the top-level keys the task names.
- Indent child keys with 2 spaces (no tabs).
- Fill in each property with the value from the task.
- Click Check answer. Key order does not matter.