Why isn't offline evaluation enough to validate a new model?

Offline metrics measure performance against historical data and don't always predict real-world impact on business or user outcomes, which can be affected by factors like changed user behavior in response to the model that historical data can't capture.

How is A/B testing different from canary deployment?

A/B testing is designed to statistically compare outcomes between model versions to decide which is better, while canary deployment is primarily a risk-mitigation rollout strategy that gradually increases traffic to a new version to limit the impact of potential failures.

What is a guardrail metric?

A guardrail metric is a secondary metric tracked during an A/B test to catch unintended negative side effects, even if the primary metric being optimized shows improvement.

Why shouldn't you keep checking A/B test results continuously?

Repeatedly checking results before a test reaches its planned sample size inflates the risk of a false positive, a problem known as 'peeking'; proper practice is to determine sample size or duration in advance and evaluate significance once that threshold is reached.

What tools support ML A/B testing?

Platforms like Optimizely, Statsig, and GrowthBook provide experimentation infrastructure, alongside custom internal systems built by large technology companies for high-scale testing.

Can A/B testing be used for large language model changes?

Yes, teams increasingly A/B test different LLM versions, prompts, or fine-tuned models against metrics like user satisfaction, task completion rate, or engagement.

AI Fundamentals

A/B Testing (ML)

IntermediateTechnique11.1K learners

A/B testing in machine learning is a controlled experimentation method that compares two or more model versions by routing production traffic between them and measuring the impact on real business or user metrics.

#ABTesting #ML #AIFundamentals#Technique#Intermediate#CanaryModelDeployment #ShadowDeployment #ModelMonitoring #AUCScore #ArtificialIntelligence#Glossary#SkillVeris

Definition

Overview

While offline evaluation metrics like accuracy or AUC measure how well a model performs on historical data, they don't always predict how a model change will affect real business outcomes — user engagement, revenue, click-through rate, or task success. A/B testing (also called online experimentation, split testing, or controlled experimentation) addresses this by randomly splitting live production traffic between a control model (the current, incumbent version) and one or more treatment models (new candidates), then statistically comparing the resulting outcomes. A properly designed ML A/B test requires several elements: random and stable assignment of users or requests to variants (often via consistent hashing so a given user always sees the same variant), a pre-registered primary metric and a determined minimum sample size or test duration based on a power analysis, guardrail metrics to catch unintended harm even if the primary metric improves, and a statistical significance test (commonly a t-test, or a Bayesian approach) applied at the end of the experiment rather than continuously peeking at results, which inflates false-positive rates. A/B testing is the gold standard for validating that a model improvement measured offline actually translates into a real-world benefit, and it is standard practice at companies operating recommendation systems, search engines, ad-ranking systems, and any ML product with a large enough user base to reach statistical significance in a reasonable time. It is closely related to but distinct from canary deployment, which focuses on safely and gradually rolling out a new model version primarily to limit the blast radius of failures, and shadow deployment, where a new model's predictions are logged but never actually shown to users. Platforms like Optimizely, Statsig, GrowthBook, and internal experimentation frameworks at large tech companies provide the infrastructure for running and analyzing these tests at scale.

Key Concepts

Randomly splits production traffic between control and treatment model versions
Measures impact on real business or user metrics, not just offline accuracy
Requires stable, consistent user-to-variant assignment via techniques like hashing
Uses pre-registered primary metrics and guardrail metrics to catch unintended harm
Relies on statistical significance testing to validate observed differences
Considered the gold standard for validating real-world model improvements
Distinct from canary deployment, which focuses on safe rollout rather than comparison
Supported by platforms like Optimizely, Statsig, and GrowthBook

Use Cases

Comparing a new recommendation algorithm against the current production model

Validating that a search ranking model change improves click-through rate

Testing ad-ranking model changes against revenue and user engagement metrics

Evaluating whether a new fraud detection model reduces losses without harming legitimate users

Measuring the real-world impact of a fine-tuned LLM prompt or model version

Frequently Asked Questions

From the Blog

Programming

Testing Python Code with pytest: A Beginner's Guide

Untested code is legacy code from the moment it's written. This guide explains how to write effective Python tests with pytest — from your first test function through fixtures, parametrize, mocking, and measuring coverage.