Data Engineering
Everything on SkillVeris tagged Data Engineering — collected across the glossary, study notes, blog, and cheat sheets.
20 resources across 1 library
Interview Questions(20)
What are Window Functions in SQL?
A window function computes a value across a set of related rows (its window) for each row individually, without collapsing those rows into a single output row…
What Does PARTITION BY Do in SQL?
PARTITION BY divides the rows visible to a window function into independent groups, so the function's calculation restarts from scratch for each group instead…
What are LEAD and LAG Functions in SQL?
LAG and LEAD are offset window functions that let a row directly access a column's value from a preceding row (LAG) or a following row (LEAD) within the same o…
How Do You Calculate a Running Total with Window Functions?
A running total is calculated with SUM(column) OVER (ORDER BY sequence_column), which sums the target column across all rows from the start of the window up to…
Data Lake vs Data Warehouse: What is the Difference?
A data warehouse stores structured, schema-validated data organized for fast, well-defined analytical queries, while a data lake stores raw data of any format…
ETL vs ELT Pipelines: What is the Difference?
ETL (Extract, Transform, Load) transforms data in a separate processing engine before loading it into the target system, while ELT (Extract, Load, Transform) l…
What is Change Data Capture (CDC)?
Change Data Capture (CDC) is a technique for identifying and streaming only the rows that changed — inserts, updates, and deletes — in a source database, typic…
What Are Database Compression Techniques and Why Do They Matter?
Database compression techniques shrink the physical size of stored data by removing redundancy — through methods like dictionary encoding, run-length encoding,…
How Does Run-Length Encoding Work in Columnar Compression?
Run-length encoding (RLE) compresses a column by replacing consecutive repeated values with a single stored value plus a count of how many times it repeats in…
What is Data Deduplication in Storage Systems?
Data deduplication is a storage technique that identifies duplicate chunks of data — whole files, fixed-size blocks, or variable-length content-defined chunks…
What Are Common Archival Strategies for Old Database Data?
Archival strategies move old, rarely accessed data out of the primary operational database into cheaper long-term storage — via time-based partition archiving,…
What is Data Lineage Tracking and Why Does It Matter?
Data lineage tracking is the practice of recording where each piece of data originated, every transformation it passed through, and every downstream table, rep…
How Do You Implement Data Quality Validation in Pipelines?
Data quality validation in pipelines means running automated checks — for null rates, uniqueness, referential integrity, value ranges, and row-count anomalies…
Batch vs Stream Processing: How Do You Choose?
Batch processing runs computations over a bounded, already-collected chunk of data on a schedule, trading latency for simplicity and easy correctness, while st…
What is Lambda Architecture in System Design?
Lambda architecture is a data-processing pattern that runs a slow, accurate batch layer over the complete historical dataset alongside a fast speed layer over…
What is Kappa Architecture and How Does It Differ from Lambda?
Kappa architecture simplifies Lambda architecture by processing all data — historical and real-time — through a single stream-processing pipeline, treating the…
Data Lake vs Data Warehouse: What Is the Difference?
A data lake stores raw data of any format — structured, semi-structured, or unstructured — cheaply at scale and applies schema only when the data is read, whil…
How Would You Design an ETL Pipeline?
An ETL pipeline extracts data from source systems, transforms it into a clean, validated, and correctly shaped form, and loads it into a target store like a wa…
What is Change Data Capture (CDC)?
Change data capture is a technique for detecting and streaming every row-level insert, update, and delete made to a database as an ordered sequence of events,…
What Is Apache Kafka and How Does It Work?
Apache Kafka is a distributed, durable event-streaming platform that stores records as an ordered, append-only log split into partitions, allowing many indepen…