Hybrid Lab-to-Cloud: Map the Data Path Before Building It

The transfer agent was “done.” S3 had objects. Downstream Batch jobs still saw empty prefixes for hours. Nobody could say whether the instrument was late, the poller was stuck, or a completion marker never arrived.

That is the usual failure mode of hybrid lab-to-cloud work: teams start with upload code, not a data path. This post is for platform and infrastructure engineers connecting on-premises instruments, LIMS exports, or lab file shares to AWS—not a vendor product tour. The goal is a discovery checklist you can hand to a teammate before the first Terraform module or sync daemon.

Why mapping comes first

Hybrid pipelines fail at contracts, not at SDKs:

• Instruments write files that are still growing when you copy them.
• Two teams share a share; neither owns retention or naming.
• Cloud-side jobs assume “all files for run X are present” without a completion signal.
• Security reviews ask about encryption and identity after the path is already hardcoded.

If you cannot draw the path on a whiteboard with owners and failure modes, you are not ready to automate it.

Define the path end to end

Walk these stages in order. Fill gaps before you pick tools.

Draw arrows. Label each hop with protocol, identity, and who gets paged when it stalls.

Discovery checklist

Use this before writing code or buying a connector.

1. Source and file semantics

• File formats and typical sizes documented (small many files vs few huge files change everything).
• Done definition is explicit: closed file handle, checksum file present, instrument run status API, or time-based settle window.
• Partial writes and rename-on-complete patterns are known—copying *.tmp or open files is a classic silent corruption path.
• Clock skew between instrument hosts and cloud is understood if you rely on mtime.

2. Throughput and volume

• Peak bytes per hour and peak file count per hour estimated from real runs, not averages.
• Network path capacity measured (lab LAN → WAN → AWS region), including VPN or Direct Connect constraints.
• Growth plan: what happens when a new instrument doubles nightly volume.

3. Trust boundaries

• On-premises identity for the transfer process is named (service account, instance role, agent identity)—not a shared admin login.
• Cloud identity for write and for read are separate; least privilege starts at discovery, not after the first audit finding.
• Data classification known: which prefixes need SSE-KMS, which stay in a regulated account, which may leave the region.

4. Ownership and SLOs

• RACI for source share, transfer agent, cloud landing bucket, and consumer jobs.
• Stale-data SLO defined (for example: “run complete in lab → objects visible in S3 within N minutes”).
• Escalation path when the SLO breaks—who owns the instrument side vs the AWS side.

5. Failure domains

Map at least these cases:

Anti-patterns to reject early

1. “Just rsync to a bucket and figure it out later.” Later is when Batch burns money on empty inputs.
2. One shared IAM user for lab and cloud. Rotation and blast radius become political problems.
3. Using object LastModified as the only completeness signal. It does not mean the run is finished.
4. Deleting the lab copy before cloud validation succeeds. You have removed your only replay source.
5. Building the Batch pipeline before the landing contract exists. See the AWS Batch checklist—it assumes input paths already make sense.

Minimal architecture sketch

A path that usually survives contact with production:

1. Source — instrument writes to a controlled landing directory with rename-on-complete or a manifest.
2. Transfer — dedicated agent or poller with its own identity; retries with backoff; never assumes one-shot success.
3. Raw prefix — immutable object keys (instrument/run-id/filename); no overwrite of completed objects.
4. Validation — completeness check emits a marker or event.
5. Work prefix or queue — consumers only see validated sets.
6. Observability — correlation id from run id through logs (series post on observing transfers comes later).

You can implement this with many tools. The sketch matters more than the brand names.

What to write down before sprint planning

Hand this packet to the team:

• One-page path diagram with owners.
• File naming and “done” rules.
• Peak volume and latency SLO.
• Identity and encryption requirements (even if IAM details come in the next posts).
• Open risks: unknown settle time, shared shares, no lab contact for weekends.

What’s next

This is the first post in a hybrid lab-to-cloud operations series. Next: moving instrument data to S3 reliably—multipart uploads, checksums, immutable keys, and completion markers.

Related reading from the platform-notes series: AWS Batch deployment checklist and Terraform module layout for the cloud side once the path is clear.

If you only do one thing: write down the done definition at the source and the completeness signal in the cloud before you write a single line of upload code—everything else in hybrid transfer hangs off those two contracts.