Idempotent Pollers, Retries, and Quarantine Paths

The agent crashed after uploading 40 of 42 files. On restart it uploaded everything again, overwrote nothing useful, emitted two completion markers, and the pipeline ran twice—once on a partial set that somehow got marked ready during the race.

Hybrid transfer without idempotency turns every network blip into a data-quality incident. This post assumes you can land objects reliably and cross the IAM/KMS boundary. The goal is a retry and quarantine checklist you can hand to a teammate.

Model the file as a state machine

Give every source file (or every run_id) explicit states. Example:

discovered → eligible → uploading → uploaded → verified → (marker written)
                ↘ failed → quarantine
                ↘ skipped (not done yet)

Rules:

• State is stored durably (local DB, SQLite, DynamoDB, or a control prefix)—not only in memory.
• Transitions are logged with timestamps and error codes.
• uploaded without verified is not “done.”
• Re-discovery of an already verified object is a no-op, not a re-upload.

Idempotency keys

Pick a stable business key:

• Prefer instrument_id + run_id + relative_path.
• Persist the checksum of the last successful upload.
• If the source file changes checksum after verified, treat it as a new event (new attempt id or quarantine for human review)—do not silently overwrite.

Consumer-side idempotency matters too: Batch or workflow engines should key on run_id so a duplicate marker does not double-bill compute. See the Batch checklist for job identity habits.

Retry budgets

Unbounded retries create outages that look like “the lab is flooding us.”

• Non-retryable failures alert immediately; do not burn the budget.
• Retryable failures increment attempt and schedule next try.
• After max attempts, move to quarantine and page—do not block the whole instrument queue on one poison file unless policy requires stop-the-line.

Poison files and quarantine

Quarantine is a first-class prefix or path, not a forgotten folder:

quarantine/{instrument_id}/{run_id}/{filename}
quarantine/.../reason.json

Include in reason.json: error class, attempts, checksums, agent version, timestamps.

Workflow:

1. Isolate the bad object (and optionally leave a stub in raw).
2. Alert with enough context for lab + platform.
3. Allow explicit replay after fix—do not require editing production state by hand without an audit trail.

Safe replay

Operators will ask: “Can we re-send run X?”

• Replay is an intentional action (CLI flag, approved job, or ticket)—not automatic on every agent restart.
• Replay re-enters the state machine at eligible or uploading with a recorded reason.
• Completion markers for a run are rewritten only after full re-validation.
• Consumers that already processed run_id either no-op or require a new run_id / reprocess flag.

Duplicate and partial-run scenarios

Testing that catches races

Before production:

• Kill the agent mid-multipart and restart.
• Inject a non-retryable IAM deny and confirm quarantine + alert.
• Drop a truncated file that passes size heuristics and confirm checksum fail.
• Dual-submit the same run_id marker and confirm consumers run once.
• Fill the retry budget on one file and confirm other instruments still progress.

Automate what you can in CI against a disposable bucket. The tests that usually find production bugs are the ugly ones: process kill, clock skew around settle windows, and a second agent started “just to help catch up.”

What’s next

Next: observing and debugging hybrid data transfers—correlation ids, metrics, and the first fifteen minutes of a stalled pipeline.

Previous: IAM and KMS across the hybrid boundary.

If you only do one thing: persist a per-file state machine with a finite retry budget and a quarantine path—memory-only agents will eventually double-process or loop forever.