Concurrency line
Multiprocessing pools
A head chef cannot cook every long dish alone, so it sets up side kitchens and passes each one a copied order.
Stops explain each step. The film plays the whole ride.
- Setting up a kitchen
- A slip passed along
- Finished dishes
- Something went wrong
The head chef asks for side kitchens
The head chef has eight heavy dishes, too slow to cook alone. It asks Kitchen setup to build two side kitchens.
Next stop: Both side kitchens open
All stops
The everyday version
The head chef cannot cook everything alone, so it sets up side kitchens. Kitchens share nothing: orders and dishes travel as copied slips.
- Head chefis the Parent process
Plans the dishes, hands out the work and collects the finished dishes.
- Kitchen setupis the Start method
Builds each side kitchen, empty and ready for orders.
- Slip windowis the Task queue
Order slips wait here until a side kitchen takes one.
- Side kitchen 1is the Worker 1
One cook working alone, with its own fridge that nobody else can open.
- Side kitchen 2is the Worker 2
Cooks at the same time in another kitchen. Whoever finishes first sends first.
- Pickup windowis the Result queue
Finished dishes arrive here, in the order they were done.
- A live cook through the windowis an order that cannot be copied
The head chef tries to pass an order that has no name, or a phone line that is still open. Only a written copy fits through.
Interview questions
Say the short answer first. Go deeper only if they ask.
Pool vs ProcessPoolExecutor?
Both are process pools. The executor has the Future API and fails loudly when a worker crashes.
multiprocessing.Pool has map, imap, apply_async and maxtasksperchild. ProcessPoolExecutor offers submit, map, futures, BrokenProcessPool and works with asyncio’s run_in_executor.
“They are unrelated; Pool uses threads.”
Why must arguments and results be picklable?
They cross a process boundary as bytes.
There is no shared memory. Each task is pickled into a pipe and unpickled in the worker. Lambdas, local functions, open files and locks cannot be pickled by default.
“Python just passes a reference.”
Why the if __name__ == "__main__": guard?
Spawn and forkserver children re-import the main module.
Without it, child start-up re-runs your pool-creation code and raises a bootstrapping RuntimeError. The main module must also be safely importable.
“It is just a style convention.”
What does chunksize do?
It groups items per task to cut communication overhead.
Bigger chunks mean fewer pickles and round trips but worse load balancing when item cost varies. Executor.map defaults to 1, which is slow for tiny items on a process pool.
“It is the number of workers.”
Does map preserve order?
Yes. Results come back in input order.
imap_unordered and as_completed return in completion order. That streams results and avoids head-of-line blocking behind one slow item.
“Results come back in whatever order they finish.”
What is the default start method on Linux in 3.14?
forkserver.
It was fork until 3.13. macOS and Windows use spawn. Forking a multithreaded parent can deadlock on inherited locks. Use get_context("fork") explicitly if you really need it.
“fork”, or “spawn on Linux”.
How does Celery’s prefork pool relate?
It is the same model: a parent plus child worker processes.
Prefork is Celery’s default pool. It runs tasks in child processes, built on billiard, a multiprocessing fork, and worker_max_tasks_per_child recycles them. For I/O-bound tasks there are gevent, eventlet and thread pools.
“Celery workers are threads.”
How many workers for CPU-bound work?
About the number of cores.
More processes than cores just context-switch, and memory per worker multiplies. In containers, os.process_cpu_count() can differ from os.cpu_count() because of affinity.
“As many as possible; 100 workers is faster.”
Cheat-sheet
Commands you will actually type.
with Pool(4) as p: print(p.map(f, range(100), chunksize=10))The canonical pool. Put it under `if __name__ == "__main__":`.
with ProcessPoolExecutor() as ex: futs = [ex.submit(f, i) for i in range(10)]The `Future` API. Read them with `as_completed(futs)`, in completion order.
p.imap_unordered(f, items, chunksize=20)Stream results as they finish. Avoids head-of-line blocking.
ctx = mp.get_context("spawn"); ctx.Pool(4)Choose a start method per pool without changing the global default.
mp.get_start_method() # forkserver on Linux 3.14What am I running? `spawn` on macOS and Windows.
Pool(4, maxtasksperchild=100)Recycle workers to cap memory growth. Celery calls it `worker_max_tasks_per_child`.
p.map(partial(f, k=3), items) # not a lambdaThe pickle-safe way to bind extra arguments.
celery -A app worker --pool=prefork --concurrency=4Celery’s prefork pool: 4 child processes.