Tokens in C++

Analogy categoryAnalogies

1. Intro

A token is the smallest individual unit in a C++ program that the compiler can recognize. Before a compiler can understand what a program does, it first breaks the raw source code into a stream of tokens — a process called lexical analysis (or "tokenizing"). Every keyword, variable name, number, symbol, and string literal you write is a token.

🏏

Cricket analogy: Just as a scorecard breaks a match down into individual deliveries before analysis, a C++ compiler breaks source code into individual tokens - keywords, names, numbers, symbols - before it can understand the program at all.

Think of tokens the way words are the smallest meaningful units of a sentence. A compiler cannot work directly with a line of raw text — it first has to split that line into recognizable pieces, just like a reader splits a sentence into words before understanding it.

2. Syntax

C++ defines six categories of tokens. Every valid line of C++ code is made up entirely of these:

🏏

Cricket analogy: Cricket officially recognizes distinct categories of dismissal - bowled, caught, LBW, run out, stumped, hit wicket - and C++ similarly recognizes exactly six categories of tokens that every valid program is built from.

Keywords — reserved words with special meaning (int, return, class, if)
Identifiers — names given to variables, functions, classes (age, calculateTotal)
Literals (Constants) — fixed values written directly in code (42, 3.14, 'A', "hello")
Punctuators (Separators) — symbols that structure code ( { } ( ) ; , )
Operators — symbols that perform operations (+, -, =, ==, &&)
Whitespace — spaces, tabs, and newlines that separate tokens (not a token itself, but a separator)

3. Explanation

When the compiler reads a statement like int total = 10 + count;, its lexer scans left to right and emits a stream of tokens: the keyword int, the identifier total, the operator =, the literal 10, the operator +, the identifier count, and the punctuator ;. Each token is classified independently, and this classified stream is what the next compiler stage (the parser) actually works with — not the raw characters.

🏏

Cricket analogy: Reading int total = 10 + count; token by token is like a scorer logging each ball of an over separately - 'keyword,' 'identifier,' 'operator,' 'literal' - building a classified sequence the analyst (parser) later interprets.

Whitespace and comments are stripped out during tokenizing — they exist for human readability, not for the compiler. That is why int total=10; and int total = 10; tokenize to the exact same sequence of tokens.

🏏

Cricket analogy: Whether a scorecard is written with extra spacing or tightly packed, the umpire's official record of runs and wickets is identical - just as int total=10; and int total = 10; tokenize to the exact same sequence.

4. Example

cpp

#include <iostream>
using namespace std;

int main() {
    int score = 90;          // keyword, identifier, operator, literal, punctuator
    const double PI = 3.14;  // keyword, keyword, identifier, operator, literal

    cout << "Score: " << score << endl;
    cout << "PI: " << PI << endl;

    return 0;
}

5. Output

text

Score: 90
PI: 3.14

6. Key Takeaways

A token is the smallest unit of meaning the compiler recognizes.
C++ has six token categories: keywords, identifiers, literals, punctuators, operators, and whitespace.
Tokenizing (lexical analysis) happens before parsing — the compiler never works with raw text directly.
Whitespace separates tokens but is not itself carried forward into the parsed program.