Code Llama

Definition

Code Llama is a family of large language models from Meta, built by further training Llama 2 on code-heavy data, specialized for code generation, completion, and understanding, and released in multiple sizes and variants including Python-specific and instruction-tuned versions.

Overview

Code Llama takes Meta's general-purpose Llama 2 foundation model and continues training it on a large corpus dominated by publicly available source code, along with some natural-language data related to code such as discussions and documentation. This continued pretraining approach lets Code Llama inherit Llama 2's general language capability while substantially improving its accuracy and fluency on programming tasks compared to the base model. The family was released in three sizes — 7B, 13B, and 34B parameters (with a later 70B variant added) — and in several specialized flavors: the base Code Llama models for general code completion, Code Llama - Python, further specialized for Python specifically, and Code Llama - Instruct, fine-tuned to follow natural-language instructions for tasks like explaining code or generating code from a description. This structure lets developers choose a variant matched to their exact use case, whether that's raw code completion in an IDE or instruction-driven code generation in a chat interface. Code Llama supports long-context input via a technique called fill-in-the-middle training, which allows the model to complete code given both preceding and following context (not just what comes before), which is particularly useful for IDE-style autocomplete where the cursor sits in the middle of existing code. Released with open weights under Meta's community license, Code Llama has been widely adopted as a base for fine-tuned coding assistants and as a benchmark reference point for subsequent open code models such as StarCoder2 and WizardCoder.

Key Features

• Built by continued pretraining of Llama 2 on code-heavy data
• Released in 7B, 13B, 34B, and later 70B parameter sizes
• Specialized Python and Instruct variants alongside the base models
• Fill-in-the-middle training for cursor-aware code completion
• Long context window suited to multi-file code tasks
• Open-weight release under Meta's community license
• Widely used as a base for fine-tuned coding assistants
• Strong benchmark performance on code-generation tasks like HumanEval

Use Cases

Alternatives

Frequently Asked Questions

Related in AI Models