IAM and KMS Across the Hybrid Lab–Cloud Boundary

Transfer logs said AccessDenied. The bucket policy “allowed S3.” Someone added s3:* on the role and it still failed—because the objects used SSE-KMS and the key policy never trusted the transfer identity.

This post is for teams moving lab data into S3 with encryption and least privilege—not an IAM tutorial for greenfield accounts. It assumes you already mapped the path and upload with verification. The goal is a boundary checklist you can hand to a teammate before the next security review or 2 a.m. deny.

Separate identities by job

Do not use one principal for everything.

Mixing these is how a Batch job deletes the only copy of a failed run, or a lab agent inherits production admin rights.

Prefer federation over long-lived keys

When the architecture allows it:

• On-premises or edge compute uses short-lived credentials (IAM Roles Anywhere, OIDC, or a vault that mints temporary keys)—not a static access key in a config file.
• If static keys are unavoidable for a legacy agent, they are scoped to one role, rotated on a calendar, and never shared across environments.
• Credentials are not baked into container images or checked into git.

Long-lived keys are still common on instrument networks. Treat them as tech debt with an owner and an expiry, not as “temporary.”

S3 policy checklist

• Bucket policy and IAM identity policy both allow the action—either side can deny.
• Prefix isolation: raw/, quarantine/, validated/ grant different principals.
• s3:ListBucket is constrained with s3:prefix conditions so one instrument identity cannot browse the whole bucket.
• Block Public Access remains on; no “quick test” public ACLs.
• Optional: deny s3:DeleteObject on raw/ for transfer roles—deletes belong to a controlled lifecycle or operator role.

KMS: the deny that looks like S3

SSE-KMS means S3 permission is not enough.

• Transfer role can call the KMS APIs required for encrypting new objects (often kms:GenerateDataKey / Encrypt depending on how you write).
• Reader roles can kms:Decrypt for the same key.
• Key policy trusts those roles (or the account with conditions)—identity policy alone does not grant use of a customer-managed key.
• Encryption context (if used) is documented and stable; changing context breaks decrypt for old objects.
• Key administrators and key users are different people/roles where your compliance model requires it.

Cross-account landing adds another layer: bucket policy, key policy, and identity in the caller account must align. Draw all three before you open a ticket titled “S3 is broken.”

Encryption and classification

• Default bucket encryption set to the intended CMK (or SSE-S3 if classification allows—and document why).
• Clients do not accidentally write with a different key than consumers expect.
• CloudTrail data events (where enabled) and access logs have an owner for investigations.
• Secrets for agents live in a secrets manager or OS-protected store—not next to the binary in a world-readable share.

Diagnosis quick reference

Use CloudTrail and the error message’s kms vs s3 fields before widening policies to *.

Least privilege that still ships

A practical sequence:

1. Start with prefix-scoped read/write on a non-prod bucket and a dedicated CMK.
2. Prove upload + validate + consumer read with the same roles you will use in prod.
3. Remove wildcards discovered during that test.
4. Promote via Terraform environments—do not recreate trust by hand in the console.

What’s next

Next: idempotent pollers, retries, and quarantine paths—so permission success does not turn into duplicate science or infinite retry storms.

Previous: Moving instrument data to S3 reliably.

If you only do one thing: prove Put + Get + Decrypt with the production transfer and consumer roles against SSE-KMS objects in a non-prod bucket before the first real instrument run—console admin access proves nothing about the agent.