Async & data line
Celery + Redis
How a web app hands slow jobs to a background worker, so nobody sits watching a spinner.
- Request
- Queued message
- Reply or result
- Failure or retry
You ask for a report
You tap “Make my report”. Your request travels to the web app.
Next stop: The app writes a ticket
All stops
The everyday version
A busy restaurant runs the same way. Every station has a twin in the kitchen.
- The customeris the You
Orders food, then chats with friends instead of standing at the kitchen door.
- The waiteris the FastAPI app
Takes the order, hands you a token, and never cooks.
- The order railis the Redis queue
Tickets clip onto the rail and wait their turn.
- The cookis the Celery worker
Takes the next ticket and cooks. On a busy night you hire more cooks, not more waiters.
- The pickup counteris the Result store
Finished dishes wait here under your token number.
- A burnt dishis a retry
The cook clips the ticket back on the rail and makes it again.
Interview questions
Say the short answer first. Go deeper only if they ask.
Why not do the slow work inside the API request?
It holds the request open and ties up a server worker, so a few slow jobs can stall everyone. Celery moves the work out and the API replies in milliseconds.
Web servers have a limited number of workers. A 30-second PDF holds one for 30 seconds, and a proxy may time out before it finishes. A queue also gives you retries and lets you scale workers separately from web servers.
“Celery makes the code run faster.” It runs the same code, just somewhere else.
What is the difference between the broker and the result backend?
The broker carries task messages to workers. The result backend stores what happened afterwards. Redis can do both, but they are separate jobs.
They are configured separately (broker_url, result_backend). Many teams pick RabbitMQ as the broker for native acks and keep Redis for results. If Redis also serves as your cache, use a separate instance or DB so cache eviction can never delete queued tasks. If nothing reads results, set ignore_result=True.
“They are the same thing, both are just Redis.”
A worker crashes halfway through a task. What happens?
With default settings the message was already acknowledged, so the task is lost. Set acks_late=True and task_reject_on_worker_lost=True and it is delivered again.
acks_late alone is not enough: a worker that dies abruptly still acks on exit unless task_reject_on_worker_lost is on. Together they swap “maybe lost” for “maybe runs twice”, so tasks must be idempotent. On Redis, keep visibility_timeout longer than your slowest task or a second worker gets the same task.
“acks_late gives exactly-once delivery.” It gives at-least-once.
Why should tasks be idempotent?
A task can run more than once because of retries or redelivery, so running it twice must end the same way as running it once.
Charging a card twice or sending the same email twice is the classic bug. Guard it with a unique key: record “payment 42 done” and skip if it is already there, or pass the payment provider’s idempotency key.
“Each task runs exactly once.”
What should you pass to .delay()?
Small, JSON-friendly values such as IDs. Let the task load fresh data itself.
Arguments are serialized into the message (JSON by default since Celery 4). An ORM object will not serialize, bloats the queue, and is stale by the time the worker runs. Pass user_id=42 and query inside the task. Pickle is unsafe: anyone who can write to the broker can run code on your workers.
“Pass the user object and switch the serializer to pickle.”
Cheat-sheet
Commands you will actually type.
celery -A app worker --loglevel=INFOStart a worker that reads the default queue.
celery -A app worker -Q reports --concurrency=4Four processes that only take jobs from the “reports” queue.
celery -A app beatThe scheduler for periodic tasks. Run exactly one.
celery -A app inspect activeSee what every worker is running right now.
redis-cli LLEN celeryHow many tasks are waiting in the default queue.
@app.task(bind=True, autoretry_for=(TimeoutError,),
retry_kwargs={'max_retries': 5, 'countdown': 60},
acks_late=True)
def generate_report(self, user_id: int) -> str:
...
result = generate_report.delay(user_id=42)
AsyncResult(result.id).state # PENDING → SUCCESSDefine a retrying task, queue it, check its state.