Observing and Debugging Hybrid Data Transfers

Slack said “files aren’t in S3.” The agent host said “fine.” CloudWatch had three log groups and no shared id. Twenty minutes later someone found the run under a different instrument prefix—and a completion marker that never wrote.

Hybrid systems fail in the gaps between lab and cloud. This post assumes you have idempotent transfer and verified landing. The goal is an observability checklist and first-fifteen-minutes playbook you can hand to on-call.

Correlation is the product feature

Pick a correlation id and carry it everywhere:

• Prefer run_id from the instrument or LIMS.
• Add instrument_id, agent_id, and attempt on every log line and metric dimension you can afford.
• Put the same ids in S3 object metadata and in Batch / workflow input.

If you cannot answer “show me everything for run X” in one query, you do not have hybrid observability yet—you have disconnected logs.

Structured logs (minimum fields)

Emit JSON or consistently keyed text from the agent and validators:

Ship agent logs to CloudWatch Logs (or equivalent) the same way you treat Batch container logs—not as optional SSH archaeology.

Metrics that catch stalls

Track at least:

• Files discovered vs uploaded vs verified per instrument (rates and lags).
• Bytes uploaded per minute vs capacity baseline.
• Retry count and quarantine count.
• Age of oldest pending file (stall signal).
• Time from source “done” to completion marker (SLO latency).
• Incomplete multipart upload count.

Alarms that matter more than CPU on the agent VM:

1. Stale pending — oldest eligible file older than SLO.
2. Quarantine spike — sudden poison rate.
3. Zero verified during expected production window.
4. Marker lag — uploads succeed but markers stop.

Dashboards for two audiences

One dashboard that only shows AWS service health will not end the “is it the instrument?” argument.

First fifteen minutes

When “data isn’t flowing” pages:

Minutes 0–5 — scope

• Which instrument_id / run_id? Single run or whole site?
• Is the source producing new files (lab), or is landing empty (transfer), or is marker missing (validation)?
• Any recent deploy of agent, IAM, bucket policy, or KMS key policy?

Minutes 5–10 — locate the hop

Use CloudWatch Logs Insights patterns: filter by run_id, count error_code, and chart event volume over the last hour.

Minutes 10–15 — stabilize

• Stop the stampede: reduce concurrency if the WAN is saturated.
• Quarantine confirmed poison; do not widen IAM to “fix” a deny without reading the error.
• Communicate: which hop is red, ETA for next check, whether lab should pause acquisition.
• Capture run_ids affected for replay after fix.

Sample Logs Insights starters

Adapt field names to your schema:

fields @timestamp, run_id, event, error_code, object_key
| filter instrument_id = "INSTRUMENT"
| sort @timestamp desc
| limit 100

fields error_code
| filter ispresent(error_code)
| stats count(*) by error_code

filter event = "verified" or event = "marker_written"
| stats count(*) by bin(5m)

Tie-in to consumers

When transfers look healthy but science is late, follow the path into Batch or workflows with the same run_id. Hybrid observability ends at “object exists”; platform observability ends at “consumer succeeded or failed with a clear log.”

What’s next

Series finale: go-live and recovery checklist—readiness, rollback, replay, and ownership before you cut traffic.

Previous: Idempotent pollers, retries, and quarantine paths.

If you only do one thing: require a run_id on every log line and metric from agent through validator to consumer—without it, hybrid debugging is archaeology.