Lab-to-AI: From Raw Zone to SageMaker and HealthOmics
The science team’s request arrived as four words: “AI on our data.” Behind it stood a platform team with a validated raw zone, a governance model built for audits, and no plan for what happens when a model wants to read everything.
Read MoreObservability as Audit Evidence
Mid-investigation, someone asked for the transfer logs from eleven weeks ago—the ones that would show exactly what happened to run R. The retention policy was thirty days. The logs that could have answered the question had been rotated into nothing, and the finding closed as “probable, not demonstrable.”
Read MoreValidated Delivery: CI/CD Under GxP Change Control
The validation auditor asked for the change record behind a production change made two weeks earlier. The honest answer was a Slack thread, a terraform apply from someone’s laptop, and a Jira ticket with no attachments. None of that is a change record.
Read MoreShould Your Raw Instrument Zone Become Iceberg?
The science team asked a reasonable question: “Can we query two years of run metadata without a week of Athena scans?” The raw prefix held forty million objects, LIST calls were timing out, and every dashboard paged through keys like it was 2013.
Read MoreInstrument Ingestion Patterns: Poller, DataSync, or S3 Events
Three instruments, three capabilities. One writes finished runs to a Windows share. One can call the S3 API directly. One ships files over SFTP to wherever you point it. The team forced all three through one custom poller—and spent a year paying for it in special cases.
Read MoreThe ALCOA+ Field Guide for Platform Engineers
The auditor asked a simple question: “How do you know this file was not modified after upload?” The team had versioning off, a shared transfer credential, and a fourteen-second silence that felt much longer.
Read MoreAmazon Linux 2 Is EOL: a Migration Checklist for Batch, ECS, and Lambda
The security scan flagged eleven CVEs on the ECS instances. There was nothing to patch with—Amazon Linux 2 went end-of-life on June 30, 2026, and the fixes simply stopped coming. The AMI had been “temporary” for two years.
Read MoreHybrid Lab-to-Cloud: a Go-Live and Recovery Checklist
The pilot worked for one instrument on a quiet Tuesday. Cutover week added three instruments, a VPN blip, and a KMS key rotation—and the only recovery plan was “call the person who built the agent.”
Read MoreObserving and Debugging Hybrid Data Transfers
Slack said “files aren’t in S3.” The agent host said “fine.” CloudWatch had three log groups and no shared id. Twenty minutes later someone found the run under a different instrument prefix—and a completion marker that never wrote.
Read MoreIdempotent Pollers, Retries, and Quarantine Paths
The agent crashed after uploading 40 of 42 files. On restart it uploaded everything again, overwrote nothing useful, emitted two completion markers, and the pipeline ran twice—once on a partial set that somehow got marked ready during the race.
Read MoreIAM and KMS Across the Hybrid Lab–Cloud Boundary
Transfer logs said AccessDenied. The bucket policy “allowed S3.” Someone added s3:* on the role and it still failed—because the objects used SSE-KMS and the key policy never trusted the transfer identity.
Read MoreMoving Instrument Data to S3 Reliably
The object existed in S3. Size looked right. Downstream parsing failed because the transfer cut off mid-write and a later retry never ran—or ran into a different key. Metrics said “uploaded.” Science said “garbage.”
Read MoreHybrid Lab-to-Cloud: Map the Data Path Before Building It
The transfer agent was “done.” S3 had objects. Downstream Batch jobs still saw empty prefixes for hours. Nobody could say whether the instrument was late, the poller was stuck, or a completion marker never arrived.
Read MoreAWS Batch on EC2: a Deployment Checklist
This post is for teams running compute-heavy workloads on AWS Batch with EC2—genomics pipelines, media processing, ETL, simulations—not for “hello world” Fargate tutorials. It assumes you already have a container image, a VPC, and a reason to use Batch instead of ECS or Lambda.
Read MoreCloudWatch Logs Insights: 8 Queries I Reuse in Production
This post is for teams already shipping workloads to AWS who reach for the CloudWatch console when something breaks—but re-type the same Logs Insights query from memory every time. It is not a replacement for APM or full distributed tracing, and it assumes your apps write structured or semi-structured logs to CloudWatch.
Read MoreTerraform Module Layout I Use for Multi-Environment AWS
This post is for teams already running Terraform in AWS who have felt the pain of three “similar but not quite the same” environment folders. It is not a Terraform 101 tutorial, and it assumes you know what a module, variable, and remote state backend are.
Read MoreGitHub Actions to ECR to ECS: a Deployment Checklist
This post is for teams already running workloads on Amazon ECS who want a reliable GitHub Actions pipeline without reinventing the wheel each release. It is not a greenfield Kubernetes guide, and it assumes you have a working cluster, service, and task definition—not a blank AWS account.
Read More