The Aggregation Pipeline

Analogy categoryAnalogies

What Is the Aggregation Pipeline

The aggregation pipeline is MongoDB's framework for processing documents through a sequence of stages, where each stage transforms the documents and passes the results to the next stage. Unlike a single find() query that filters and projects, a pipeline lets you chain operations such as filtering, grouping, reshaping, and computing derived fields into one server-side data processing workflow, conceptually similar to a Unix pipe where the output of one command becomes the input of the next.

🏏

Cricket analogy: Think of a T20 innings broken into overs: each over (stage) takes the match state left by the previous over and changes it further, so by the 20th over the scoreboard reflects every prior transformation, just as a pipeline's final stage reflects every earlier stage's work.

Stages and How Documents Flow Between Them

Each stage in the pipeline array is an operator such as $match, $group, $sort, $project, or $lookup, and stages execute strictly in the order they appear. A document entering $match is either kept or discarded; documents that survive flow to the next stage carrying whatever shape they had, until a reshaping stage like $project or $group changes their structure. Because stages run in sequence, placing a restrictive $match early reduces the number of documents every subsequent stage has to process, which has a direct and often dramatic effect on performance.

🏏

Cricket analogy: Placing your best death-overs bowler in the 19th over rather than the 5th over changes the whole match outcome, just as placing $match early versus late in a pipeline changes how many documents downstream stages must process.

Building and Running a Pipeline

You invoke a pipeline with db.collection.aggregate([ stage1, stage2, ... ]), passing an array of stage objects. The driver sends the whole pipeline to the server in one round trip, and MongoDB's query planner may reorder or combine certain stages internally for efficiency, though the logical order you specify is what determines correctness. The result of aggregate() is a cursor, just like find(), so you can iterate over it, and by default the pipeline runs on the primary unless a read preference routes it to a secondary.

🏏

Cricket analogy: Submitting a full batting order to the umpire before the toss commits to a sequence, but the actual outcome still depends on how each partnership plays out, similar to how you declare stage order but the engine executes it stage by stage.

javascript

db.orders.aggregate([
  { $match: { status: "shipped" } },
  { $group: { _id: "$customerId", totalSpent: { $sum: "$amount" } } },
  { $sort: { totalSpent: -1 } },
  { $limit: 10 }
]);

Aggregation pipelines run entirely on the MongoDB server, so heavy computation such as grouping and sorting large datasets avoids the network cost of pulling raw documents into application code first.

Pipeline Result Limits and allowDiskUse

By default, each pipeline stage is limited to 100 MB of RAM; if a stage like $sort or $group needs to process more data than that in memory, MongoDB throws an error unless you set allowDiskUse: true, which permits the server to write temporary files to disk for that operation. This matters for large collections where an in-memory $group over millions of documents would otherwise exceed the limit, though disk-based aggregation is measurably slower than in-memory execution, so it should be a deliberate choice rather than a default habit.

🏏

Cricket analogy: A stadium's fixed seating capacity forces overflow fans to stand outside unless organizers open an overflow section, similar to how a 100 MB memory cap forces MongoDB to spill to disk via allowDiskUse when data exceeds that limit.

Setting allowDiskUse: true on every pipeline as a habit can mask an underlying problem: an unindexed $match or an overly broad $group that should be narrowed. Investigate why a stage needs more than 100 MB before reaching for allowDiskUse as a permanent fix.

The aggregation pipeline processes documents through an ordered array of stages, each consuming the output of the previous stage.
Common stages include $match, $group, $sort, $project, $limit, and $lookup, executed via db.collection.aggregate([...]).
Placing restrictive stages like $match early in the pipeline reduces the document volume all later stages must handle.
aggregate() returns a cursor, similar to find(), and typically executes on the primary node.
Each stage defaults to a 100 MB memory limit; allowDiskUse: true permits spilling to disk for larger datasets at a performance cost.
MongoDB's query planner may internally reorder or optimize certain stages while preserving the pipeline's logical semantics.
A well-designed pipeline minimizes early document volume and avoids unnecessary reshaping stages before filtering.