Python Infrastructure

Python Infrastructure

System-reliability concerns for Python services, grouped because real code uses them together: a task you queue (background-jobs) needs retries (resilience) and instrumentation (observability) on the same call path.

Scope routing

Decision tree

Operation can fail transiently (network/IO/3rd-party API)?
  -> resilience.md (retry policy)
Operation runs out-of-request (email, image processing, batch)?
  -> background-jobs.md (queue + worker)
Need to know what's happening in production?
  -> observability.md (logs/metrics/traces)
All three at once for one feature?
  -> all three references, in that order.

Cross-skill boundaries

• writing-python — how to write the function. This skill — how it survives in production.
• python-error-handling — what exception to raise. This skill — what to do when it's raised across a network boundary.
• python-resource-management — how to clean up resources (context managers). This skill — how to keep retrying when resources fail to acquire.

Gotchas

• Retry without backoff is a DoS amplifier. A failed downstream + immediate retry from N clients = traffic burst that keeps the downstream down. Default to exponential backoff + jitter from day one.
• Retrying non-idempotent operations duplicates side effects. A failed POST + retry can mean two charges. Always pair retry-on-failure with an idempotency key OR mark the operation non-retryable.
• Synchronous code inside an async worker blocks the event loop. A "fast" requests call in an asyncio worker kills throughput. Use the async client (httpx, aiohttp) or run sync code in an executor.
• Structured logs and metrics serve different audiences. Logs answer "what happened to this one request"; metrics answer "what's happening across all requests". Don't try to derive one from the other — instrument both.
• Trace context propagation needs explicit plumbing across the queue boundary. Pushing a task to Celery loses the current trace unless you serialize the trace context into the task headers and restore it in the worker. Read the OpenTelemetry-Celery propagator docs before assuming it "just works".