DevOps
- Home /
- Categories /
- DevOps
Hybrid Lab-to-Cloud: a Go-Live and Recovery Checklist
The pilot worked for one instrument on a quiet Tuesday. Cutover week added three instruments, a VPN blip, and a KMS key rotation—and the only recovery plan was “call the person who built the agent.”
Read MoreIdempotent Pollers, Retries, and Quarantine Paths
The agent crashed after uploading 40 of 42 files. On restart it uploaded everything again, overwrote nothing useful, emitted two completion markers, and the pipeline ran twice—once on a partial set that somehow got marked ready during the race.
Read MoreIAM and KMS Across the Hybrid Lab–Cloud Boundary
Transfer logs said AccessDenied. The bucket policy “allowed S3.” Someone added s3:* on the role and it still failed—because the objects used SSE-KMS and the key policy never trusted the transfer identity.
Read MoreMoving Instrument Data to S3 Reliably
The object existed in S3. Size looked right. Downstream parsing failed because the transfer cut off mid-write and a later retry never ran—or ran into a different key. Metrics said “uploaded.” Science said “garbage.”
Read MoreHybrid Lab-to-Cloud: Map the Data Path Before Building It
The transfer agent was “done.” S3 had objects. Downstream Batch jobs still saw empty prefixes for hours. Nobody could say whether the instrument was late, the poller was stuck, or a completion marker never arrived.
Read MoreAWS Batch on EC2: a Deployment Checklist
This post is for teams running compute-heavy workloads on AWS Batch with EC2—genomics pipelines, media processing, ETL, simulations—not for “hello world” Fargate tutorials. It assumes you already have a container image, a VPC, and a reason to use Batch instead of ECS or Lambda.
Read MoreFrom Varnish and Drupal Tuning to Platform Engineering
In 2013 I wrote about Varnish in front of Drupal and where Drupal sites actually lose performance. The stack was Apache, PHP, MySQL, SSH, and a lot of hand-tuning. Today I work on AWS Batch, multi-region Terraform, and hybrid lab-to-cloud integrations in life sciences. The logos changed; the job did not.
Read MoreGitHub Actions to ECR to ECS: a Deployment Checklist
This post is for teams already running workloads on Amazon ECS who want a reliable GitHub Actions pipeline without reinventing the wheel each release. It is not a greenfield Kubernetes guide, and it assumes you have a working cluster, service, and task definition—not a blank AWS account.
Read More