Tokenization
Tokenization is the process of breaking text into smaller units, called tokens, that a language model can process — typically words, subwords, or characters.
7 resources across 3 libraries
Glossary Terms(4)
Attention Mechanism
The attention mechanism is a neural network technique that allows a model to dynamically weigh the importance of different parts of its input when producing ea…
Tokenization
Tokenization is the process of breaking text into smaller units, called tokens, that a language model can process — typically words, subwords, or characters.
Embedding
An embedding is a numeric vector representation of data — such as text, images, or audio — designed so that items with similar meaning or content are positione…
Natural Language Processing (NLP)
Natural Language Processing (NLP) is the field of artificial intelligence focused on enabling computers to understand, interpret, generate, and interact using…
Cheat Sheets(1)
Interview Questions(2)
How Does Full-Text Search Work in Relational Databases?
Full-text search in a relational database uses a dedicated inverted-index structure that maps normalized words to the rows containing them, letting the engine…
How Do You Design a Full-Text Search System?
A full-text search system is built around an inverted index — a mapping from each normalized term to the list of documents (and positions) containing it — popu…