Instrument Ingestion Patterns: Poller, DataSync, or S3 Events
- Shameem Abdul Salam
- Aws , Dev ops
- September 17, 2026
Table of Contents
Three instruments, three capabilities. One writes finished runs to a Windows share. One can call the S3 API directly. One ships files over SFTP to wherever you point it. The team forced all three through one custom poller—and spent a year paying for it in special cases.
This post extends the idempotent poller post into a decision framework: which ingestion pattern fits which instrument, and which parts you need regardless of pattern. The goal is a decision table you can hand to a teammate before the next instrument onboards.
The three patterns at a glance
| Custom poller | AWS DataSync | Direct PUT + S3 Events | |
|---|---|---|---|
| Instrument capability needed | Writes to a share/directory | Writes to NFS/SMB/HDFS/object store | Speaks S3 SDK (or gateway in front) |
| “File is done” definition | Yours, in code | File-stability heuristics only | Producer declares it (it’s writing) |
| State/quarantine control | Full (your state machine) | Limited—retries and filters, little domain logic | None at ingest; all downstream |
| Ops burden | You own the agent | Managed agent, task scheduling | Least—until validation fails |
| Bulk/backfill fit | Slow without work | Excellent (parallel, bandwidth-limited) | Depends on producer |
| Cost shape | Agent host + your time | Per-GB-moved + agent hours | S3 requests + event fan-out |
When the custom poller wins
- The source has a real done definition (LIMS flags, sidecar files, settle windows) that heuristics can’t see.
- You need per-file state, quarantine with reasons, and replay semantics.
- Instrument diversity is low and long-lived; the state machine is worth building once.
The poller is the most work and the most control. If your validation rules are domain-heavy, this is home.
When DataSync wins
- Bulk migration or periodic sync from file shares; terabytes, not tricky semantics.
- Bandwidth control matters (throttle against lab business hours).
- The producer can’t change but the share is stable.
What DataSync gives you: scheduled/incremental tasks, in-transfer verification, metadata preservation, a managed agent. What it does not give you: run-completeness. A DataSync task ending successfully says “bytes moved,” not “run R is verified.” You still need manifests, markers, and checksum gates on top—DataSync replaces transport, not the contract.
When direct PUT + S3 Events wins
- The instrument or its control software can call S3 with scoped credentials (IAM Roles Anywhere for on-prem identity—no long-lived keys in config files).
- You want event-driven validation: EventBridge/SQS on object creation kicks the validator immediately.
The trap: event-per-object tempts you into treating each object as complete. Don’t—per-object events still roll up into per-run manifests and markers, or you rebuild the partial-run bug the hard way.
Honorable mentions
- AWS Transfer Family (SFTP): vendor instruments that only ship SFTP get a managed endpoint landing into S3—then treat it as direct PUT (events + validation on top).
- Storage Gateway: when something genuinely needs POSIX-ish file semantics on premises; keep it out of the critical data path if you can.
The decision table
| Situation | Start with |
|---|---|
| Legacy instrument, shared folder, fuzzy “done” | Poller (or DataSync + poller for backfill) |
| Modern instrument, SDK-capable vendor | Direct PUT + events |
| SFTP-only vendor software | Transfer Family + events |
| One-time or periodic bulk moves | DataSync |
| High-stakes validation rules on every file | Poller regardless of transport |
Regardless of pattern, you still need
- Immutable keys and source checksums—no pattern exempts you from transfer integrity.
- Per-run manifests and completion markers—transport success ≠ run readiness.
- Quarantine and replay—a pattern that can’t quarantine can’t be trusted with poison files.
- One correlation id—
run_idfrom producer to consumer, whichever pipe it rode.
If you only do one thing: choose the pattern per instrument capability, not once for the lab—and keep the validation contract identical behind every pipe, so switching transports later is an implementation detail, not a re-architecture.
Next: whether your raw instrument zone should become Iceberg with S3 Tables.