Why Serialization Matters
Every record that crosses a network boundary between operators, gets written into managed state, or is captured in a checkpoint must be serialized to bytes and later deserialized back into an object, and because this happens for every single record in a high-throughput job, the efficiency of the chosen serializer directly determines the job's achievable throughput. Flink analyzes the generic type of each DataStream at job-graph construction time through its TypeInformation system, choosing the most efficient available TypeSerializer for that type automatically wherever possible.
Cricket analogy: Serialization overhead is like the time a fielder takes to relay the ball from the boundary back to the wicketkeeper — a clean, direct throw (efficient serializer) saves crucial seconds compared to an awkward, fumbled relay through two extra fielders (a slow fallback serializer).
Flink's Built-in TypeSerializers
For a class to qualify as a Flink POJO and get the efficient PojoTypeSerializer, it must be public, have a public no-argument constructor, and have all non-static, non-transient fields either public or accessible via standard public getters and setters; if any of these conditions fail, Flink silently falls back to Kryo, a general-purpose reflection-based serializer that works for almost any class but is measurably slower and produces larger serialized payloads. Flink also integrates natively with Avro's SpecificRecord classes, which offer both efficient serialization and, unlike Kryo, robust support for schema evolution.
Cricket analogy: A well-formed POJO is like a player who fits neatly into the team's standard training regimen — the coaching staff (Flink) can apply their optimized program directly, whereas a player needing a bespoke program (Kryo fallback) takes more staff time and resources to manage.
// Qualifies for the efficient PojoTypeSerializer:
public class SensorReading {
public String sensorId; // public field
public double temperature;
public long timestamp;
public SensorReading() {} // public no-arg constructor required
public SensorReading(String sensorId, double temperature, long timestamp) {
this.sensorId = sensorId;
this.temperature = temperature;
this.timestamp = timestamp;
}
}
// Register a custom serializer explicitly if needed:
env.getConfig().registerTypeWithKryoSerializer(
LegacyPayload.class, LegacyPayloadKryoSerializer.class);State Serialization and Schema Evolution
Every keyed state value written into a checkpoint is persisted using the TypeSerializer active for that state at the time, along with a TypeSerializerSnapshot describing that serializer's configuration, and on job restart Flink uses this snapshot to check whether the current serializer is still compatible with what was checkpointed. POJO and Avro-based state serializers support well-defined schema evolution rules — such as adding a new field with a default value — allowing a job to be upgraded and restarted from an old checkpoint even after its state's class definition has changed, whereas Kryo's reflection-based serialization offers only weak, best-effort compatibility guarantees across such changes.
Cricket analogy: Schema evolution is like a scoring format adding a new statistic, say strike rate in the powerplay, to historical scorecards — a well-designed format (POJO/Avro) can backfill it with a sensible default, while an ad hoc handwritten scorebook (Kryo) may simply become unreadable against the new template.
For state that must survive schema evolution reliably in production, prefer well-formed POJOs or Avro's generated SpecificRecord classes over Kryo-backed state. Avro in particular is explicitly designed with forward and backward compatibility rules — such as required fields needing defaults when added — that make planned schema changes safe and predictable across job upgrades.
Custom TypeSerializer
For the hottest paths in a job where even the default PojoTypeSerializer's overhead is unacceptable, Flink allows implementing a fully custom TypeSerializer that hand-codes the exact byte layout for a type, paired with a TypeSerializerSnapshot that records the serializer's configuration and defines the compatibility check run on restore. This is a significant undertaking reserved for performance-critical hot paths — such as a compact fixed-width encoding for a high-volume tuple type — because a custom serializer must correctly implement copy, duplicate, and snapshot-compatibility logic to avoid subtle checkpoint corruption bugs.
Cricket analogy: Writing a custom TypeSerializer is like a team engineering a completely bespoke fielding drill instead of using the standard training manual — it can shave off crucial fractions of a second, but only a specialist coach can implement it correctly without introducing new errors.
Changing a POJO's field types, removing fields, or altering field order without ensuring the corresponding TypeSerializerSnapshot reports compatibility can cause a job restart from an old checkpoint to fail outright or, worse, silently deserialize corrupted state. Always test checkpoint/savepoint restore compatibility in a staging environment before deploying a schema change to a production job's state types.
- Every record crossing network, state, or checkpoint boundaries must be serialized, making serializer efficiency critical to throughput.
- Flink prefers the efficient PojoTypeSerializer for classes meeting strict POJO rules, falling back to slower Kryo otherwise.
- Avro's SpecificRecord classes offer efficient serialization plus robust, well-defined schema evolution support.
- Checkpoints store both the serialized state and a TypeSerializerSnapshot describing compatibility for restore.
- Kryo offers only weak, best-effort compatibility guarantees across schema changes compared to POJO or Avro.
- Custom TypeSerializers can maximize performance on hot paths but require careful, correct implementation.
- Schema changes to state types should always be validated against checkpoint/savepoint restore before production deployment.
Practice what you learned
1. What happens when a class does not meet Flink's strict POJO requirements?
2. What advantage does Avro's SpecificRecord offer over Kryo for state serialization?
3. What is the purpose of a TypeSerializerSnapshot stored in a checkpoint?
4. Which POJO requirement, if violated, causes Flink to fall back to Kryo?
5. Why should a custom TypeSerializer only be used for performance-critical hot paths?
Was this page helpful?
You May Also Like
DataStream Basics
An introduction to Flink's DataStream API, the core abstraction for processing unbounded and bounded streams of data in real time.
Keyed Streams and keyBy
Partitioning a DataStream by key using keyBy() to enable per-key state and correct windowed aggregations.
Connectors: Sources and Sinks
An overview of Flink's connector ecosystem for reading from and writing to external systems like Kafka, files, and databases.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingHadoop Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics