Moving Instrument Data to S3 Reliably

The object existed in S3. Size looked right. Downstream parsing failed because the transfer cut off mid-write and a later retry never ran—or ran into a different key. Metrics said “uploaded.” Science said “garbage.”

This post assumes you already mapped the data path. It is for teams moving instrument or lab file output into Amazon S3 across a hybrid boundary—not an S3 101. The goal is a transfer checklist you can hand to a teammate before you trust Batch or analytics on that prefix.

Design for durable objects, not “copy finished”

S3 PutObject returning 200 is necessary and not sufficient for hybrid runs. You need:

1. Correct bytes — integrity check.
2. Stable identity — key that will not be overwritten ambiguously.
3. Set completeness — knowing when all files for a run are present.
4. Safe retry — re-running the agent must not corrupt consumers.

Object naming that survives retries

Prefer immutable keys:

raw/{instrument_id}/{run_id}/{relative_path}

Rules that prevent 2 a.m. debates:

• run_id comes from the instrument or LIMS, not from “upload timestamp.”
• Do not overwrite a completed object with a different checksum—fail and quarantine instead.
• Avoid mutable “latest” keys for scientific inputs; point consumers at run_id.
• Encode version or acquisition software build in metadata if reprocessing rules depend on it.

If you must support re-upload after a lab correction, use a new run_id or an explicit attempt segment—never silent overwrite.

Transfer mechanics

Multipart and large files

• Use multipart upload for large objects; tune part size to your WAN and memory constraints.
• Persist multipart upload IDs if the agent can restart mid-file; incomplete multipart uploads cost money until aborted.
• Cap concurrent uploads so you do not melt the lab uplink or hit S3 request rate surprises on tiny-file storms.

Checksums and integrity

• Compute a checksum at the source (or on the agent host) before or during upload.
• Store checksum in object metadata and/or a sidecar object (file.sha256) the consumer can verify.
• Prefer S3 checksum features your SDK supports so the service rejects mismatched parts.
• Reject zero-byte or truncated files against expected size when the source provides length.

Completion markers

Consumers should not guess. Pick one explicit pattern:

Rule: the marker is written only after every required object for that run is verified.

Agent / poller checklist

• Reads only files that meet the source done definition (from the mapping post).
• Tracks local state: pending, uploading, uploaded, failed, quarantined.
• Retries with backoff and a max attempt count; poison files go to quarantine, not infinite loops.
• Does not delete or archive the lab copy until cloud validation succeeds (or policy explicitly allows shorter retention).
• Emits structured logs with run_id, object_key, bytes, checksum, attempt.
• Uses its own cloud credentials—see the upcoming IAM/KMS post—not a shared console user.

Validation before consumers

A minimal gate before AWS Batch or other jobs:

1. List expected keys from the manifest (or instrument API).
2. Confirm each object exists, size matches, checksum matches.
3. Write the completion marker or emit the “ready” event.
4. Only then allow the work queue or job submission path.

If validation fails, leave objects in place, alert, and keep the lab copy. Do not start expensive compute on a partial set.

Common failure modes

Lifecycle and cost hygiene

• Lifecycle rule on incomplete multipart uploads (for example abort after 7 days).
• Clear retention for raw/, quarantine/, and validated/ prefixes.
• Inventory or metrics on object count and bytes per instrument—hybrid systems grow quietly.

What’s next

Next in this series: IAM and KMS across the hybrid boundary—the AccessDenied and key-policy failures that look like “transfer bugs” until you read the error once carefully.

Previous: Map the data path before building it.

If you only do one thing: treat a run as complete only when every object is verified and a completion marker exists—never when the last PutObject returned 200.