Parallel Computing in Julia

Analogy categoryAnalogies

Julia's Parallelism Model: Threads, Tasks, and Processes

Julia offers three distinct, composable layers of parallelism: asynchronous tasks (@async, @sync, and coroutines via Channels) for overlapping I/O-bound work like network requests within a single thread; multithreading (Threads.@threads, Threads.@spawn) for CPU-bound work split across cores that share the same memory space within one process; and multiprocessing via the Distributed standard library (addprocs, @distributed, pmap) for spreading work across entirely separate Julia processes, which may run on different machines and communicate by explicit message-passing rather than shared memory. Choosing the right layer matters because they have very different overhead and failure characteristics — threads are cheap to spawn but share mutable state (and thus risk data races), while processes are heavier to start but fully isolated, making them safer for large, independent chunks of work or for scaling beyond a single machine.

🏏

Cricket analogy: A single team fielding with players covering different zones of the same ground simultaneously is like multithreading sharing one field, while fielding two entirely separate matches on two different grounds with no shared communication except scheduled updates is like multiprocessing across separate Julia processes.

Multithreading with Threads.@threads

Threads.@threads splits the iterations of a for loop across however many threads Julia was started with (set via the -t flag, e.g. julia -t 4, or the JULIA_NUM_THREADS environment variable), giving each thread a contiguous chunk of the iteration range to execute in shared memory with no communication overhead between threads for independent work. Because all threads share the same heap, writing to a shared mutable variable (like accumulating into a single sum without synchronization) from multiple threads simultaneously is a data race that produces silently wrong, non-deterministic results rather than an error — the loop body must either write to disjoint memory locations per iteration, or use synchronization primitives like Threads.Atomic or a ReentrantLock when a shared value truly must be updated.

🏏

Cricket analogy: Assigning four groundstaff each their own quarter of the outfield to mow independently and simultaneously, with no need to coordinate since their zones never overlap, mirrors Threads.@threads giving each thread a disjoint chunk of loop iterations to work on in shared memory.

julia

using Base.Threads

# Safe: each iteration writes to its own disjoint slot in the output array
result = Vector{Float64}(undef, 1_000_000)
@threads for i in 1:1_000_000
    result[i] = sqrt(i) + sin(i)
end

# UNSAFE: multiple threads racing on the same shared variable
total = 0.0
@threads for i in 1:1_000_000
    global total += sqrt(i)   # DATA RACE — result is wrong and non-deterministic
end

# Fixed with an atomic accumulator
atomic_total = Atomic{Float64}(0.0)
@threads for i in 1:1_000_000
    atomic_add!(atomic_total, sqrt(i))
end
println(atomic_total[])

A data race in Julia does NOT throw an error — it silently produces incorrect, non-reproducible results that can differ between runs. Never accumulate into a shared plain variable from inside Threads.@threads without an Atomic type or a ReentrantLock; when possible, prefer writing each iteration's result into a disjoint slot of a preallocated array instead of synchronizing access to a single shared value.

Distributed Computing with Distributed.jl

The Distributed standard library adds worker processes with addprocs(n) (or connects to remote machines via SSH with addprocs([("host1", n), ("host2", n)])), each an entirely separate Julia process with its own memory space and no shared state by default, so data must be explicitly sent across using @spawnat, remotecall, or the higher-level @distributed macro and pmap function. Because workers don't automatically have access to code or packages loaded only in the main process, @everywhere using PackageName and @everywhere function myfunc(...) ... end are required to make definitions available on every worker before using them in a distributed computation, and pmap in particular is well suited to embarrassingly parallel workloads where each unit of work is independent and roughly equal in cost.

🏏

Cricket analogy: Setting up entirely separate practice nets at different training grounds, each requiring its own equipment shipped out in advance since nothing is shared between grounds, mirrors addprocs() spinning up separate worker processes that need @everywhere to distribute code and packages before use.

julia

using Distributed
addprocs(4)   # spin up 4 worker processes

@everywhere using SharedArrays
@everywhere function slow_computation(x)
    return sum(sqrt(i) for i in 1:x)
end

# pmap distributes independent units of work across all workers
inputs = [10_000, 20_000, 30_000, 40_000]
results = pmap(slow_computation, inputs)

# @distributed with a reduction operator aggregates results across workers
total = @distributed (+) for i in 1:1_000_000
    sqrt(i)
end

Choosing Between Threads and Processes

Threads are the right choice when work is CPU-bound, fine-grained, and needs to share large in-memory data structures without the cost of copying or serializing them — spawning a thread is cheap (microseconds) and there's no message-passing overhead for read-only shared data. Processes are the right choice when you need true fault isolation (a crash in one worker process doesn't take down the whole computation), when scaling beyond a single machine's core count across a cluster, or when working with libraries that aren't thread-safe, since giving each unit of work its own process sidesteps shared-mutable-state hazards entirely at the cost of higher startup overhead and the need to explicitly serialize data across process boundaries.

🏏

Cricket analogy: Splitting fielding duties among players already on the ground costs nothing extra since they share the same pitch, mirroring threads' low overhead, while flying in a separate squad to play on another continent costs more setup but means a rain-out there doesn't cancel both matches — that isolation mirrors processes.

Quick decision guide: use @async/Channels for overlapping I/O-bound waits (network/disk); use Threads.@threads/Threads.@spawn for CPU-bound work over shared, large in-memory data on one machine; use Distributed.jl (addprocs, pmap, @distributed) when you need fault isolation, non-thread-safe libraries, or need to scale across multiple machines. Check Threads.nthreads() to confirm how many threads Julia was actually started with — a common gotcha is forgetting the -t auto (or -t N) flag and having Threads.@threads silently run on a single thread.

Julia offers three composable parallelism layers: async tasks for I/O overlap, multithreading for shared-memory CPU work, and multiprocessing (Distributed.jl) for isolated, multi-machine work.
Threads.@threads splits a for loop's iterations across threads started via the -t flag or JULIA_NUM_THREADS; Threads.nthreads() confirms how many are active.
Writing to a shared mutable variable from multiple threads without synchronization is a data race that silently produces wrong, non-deterministic results — it does not throw an error.
Fix shared-state races with Threads.Atomic, a ReentrantLock, or by restructuring the loop so each iteration writes to a disjoint slot of a preallocated array.
Distributed.jl's addprocs() creates separate worker processes with isolated memory; @everywhere is required to load code/packages on every worker before use.
pmap and @distributed distribute independent units of work across processes, well suited to embarrassingly parallel workloads.
Choose threads for cheap, fine-grained, shared-memory CPU work on one machine; choose processes for fault isolation, non-thread-safe libraries, or scaling across a cluster.