Synchronous vs asynchronous document processing
Documents arrive through an API and a UI upload. A single document can require OCR, extraction, validation, and a human decision before its data is usable downstream.
A request-response call cannot hold open for the time this takes, and a failure mid-processing would lose work that has already been paid for in compute.
External OCR and model calls have variable latency and can fail. Callers need immediate acknowledgement. Nothing may be processed twice as a side effect of a retry.
Synchronous processing inside the request; asynchronous processing with an in-memory queue; asynchronous processing with persisted jobs and explicit state.
For this system I chose asynchronous processing with persisted jobs. Ingestion accepts the document, assigns an identity, and returns a job reference; every processing stage advances a stored state that survives a restart.
More moving parts and more state to reason about, and callers must handle completion through polling or webhooks rather than a single response.
Retries become safe because ingestion is idempotent on a client-supplied key. Failed stages can be resumed instead of restarted. Human review fits naturally as one more state rather than an exception path.
Making the processing state explicit turned out to matter more than the choice of queue. Once each document had a durable state, reliability work became a matter of moving between states correctly.
received → queued → extracting → validating
→ review → approved → delivered
└─ failed → retry