AWS Batch on EC2: a Deployment Checklist

This post is for teams running compute-heavy workloads on AWS Batch with EC2—genomics pipelines, media processing, ETL, simulations—not for “hello world” Fargate tutorials. It assumes you already have a container image, a VPC, and a reason to use Batch instead of ECS or Lambda.

If you have debugged a job that sat in RUNNABLE for twenty minutes or failed with AccessDenied on an S3 object encrypted with KMS, most of this will feel familiar. The goal is a checklist you can hand to a teammate before the next pipeline go-live.

Prerequisites

Before applying Terraform or submitting jobs, confirm these exist:

Checklist before terraform apply

Walk through these in order every time you stand up or change a Batch environment.

1. Network placement

• Subnets have outbound path to ECR, S3, and CloudWatch (NAT gateway, VPC endpoints, or hybrid routing—pick one deliberately).
• Application subnets vs database subnets: Batch compute does not belong in DB-only subnets unless you have verified routing and SG rules.
• Security group egress is explicit enough for your compliance model (open 0.0.0.0/0 egress is common but not always acceptable).

2. Compute environment

• Instance types are available in the target AZ/region (Batch will fail CE creation if types are invalid or capacity is restricted).
• min_vcpus / max_vcpus / desired_vcpus match workload burst patterns—desired_vcpus > 0 keeps warm capacity; 0 scales to zero but adds cold-start latency.
• Root EBS volume size fits container image layers plus scratch data; 100 GiB is a default, not a law.
• EBS encryption enabled; KMS key policy allows the instance role if using a customer-managed key.

3. Job definition

• Image URI pins a specific tag or digest—not only latest in production.
• CPU and memory match what the container actually needs (Batch schedules on vCPU/memory; OOM kills look like mysterious retries).
• Environment variables documented; secrets from SSM or Secrets Manager, not baked into the image.
• command override behavior understood—empty command uses the image CMD; pipeline wrappers should fail non-zero on application errors.

4. IAM: job role vs execution role

• Execution role — pull image, write logs, read secrets for container startup.
• Job role — runtime S3, KMS decrypt, RDS, or other AWS API calls inside the container.
• S3 GetObject on the bucket is not enough when objects use SSE-KMS—you need kms:Decrypt on the key and key policy trust for the job role.
• Optional additional_job_policy_arns for workload-specific least privilege instead of one giant policy.

5. Storage mounts

• EFS — file system ID, mount path, transit encryption if required; security groups allow NFS from Batch instances.
• FSx / Lustre — DNS name and mount path tested from a representative instance.
• Reference genomes or static datasets: confirm paths exist in the image or are mounted before the job starts—not “we will copy later.”

6. Observability and alerts

• Log group name matches job definition awslogs configuration.
• SNS (or equivalent) on Batch job state change for FAILED—not only on exit code 0 with errors buried in logs.
• Consider a CloudWatch Logs metric filter on "ERROR" if application code can succeed with a zero exit code while logging failures.

Checklist before submitting a test job

• Job name / queue name / job definition name triple-checked in CLI or console.
• Test job uses the same subnets, SGs, and roles as production—not a one-off admin role that hides IAM gaps.
• Input data path exists and is readable with the job role (test with aws s3 ls or a minimal container that only lists paths).
• Someone is watching CloudWatch Logs and Batch job status—not just waiting for an email.

Debugging quick reference

Terraform layout that scales

Patterns that survive more than one region:

1. Reusable module — compute environment, queue, job definition, IAM, SG, logging in one place.
2. deployment/<region>/<env>.tfvars — VPC ID, subnets, image tag, and sizing per location; keep the module generic.
3. Remote state — one backend bucket with per-workspace or per-key separation; document the naming convention.
4. Tags — cost center, owner, provisioning tool, and environment on every resource; future-you will need them for chargeback and audits.

Relationship to ECS

Use ECS when you have long-running services behind a load balancer. Use Batch when work is job-shaped: finite input, batch output, scale-to-zero between runs, and instance sizing driven by peak compute rather than steady HTTP traffic. Many platform teams run both.

What’s next

This series started with ECS deploy pipelines, moved through Terraform layout and Logs Insights for day-two operations. Batch is where those habits meet scientific and regulated workloads—the subject of a longer essay on how LAMP-era ops thinking still applies.

If you only do one thing: pin production job definitions to an immutable image tag, put the job role through an S3 + KMS read test before the first real run, and make CloudWatch Logs the default answer to “what happened?”—not the Batch console status alone.