# Overview

Experimentation lets you measure the impact of changes to your product or application, ensuring they improve key metrics, prevent regressions, and move you towards your business goals.

## Problem

Shipping features without experimentation creates several challenges:

* **Undetected regressions:** You don't know whether a feature rollout caused any regressions in key business, product, or system metrics, e.g. revenue, conversion rates, cost, latency, error rates, etc.
* **Uninformed decisions:** You don't know how a feature affects key metrics, so you can't make data-informed decisions about whether to ship, iterate, or pivot.
* **Slow decisions:** Without data, it's slower to get stakeholder alignment and the confidence needed to ship a feature, iterate, or pivot.
* **Limited learning:** You don't learn what kinds of changes result in the most impact, so you can't feed that into roadmap prioritization.
* **Recognition challenges:** You can't directly attribute impact to the work of specific teams and individuals.

## Solution

Before shipping a feature to everyone:

1. **Track events:** Ensure you're tracking the analytics events you care about, e.g. `SignUp`, `Purchase`, `AddToCart`, `PageLoadTime`.
2. **Create an experiment:** Create an experiment that randomly assigns users to a control group and a test group.
3. **Use the experiment:** Enable the feature for the test group, and keep it disabled for the control group.
4. **Analyze the results:** Compare metrics between the groups to see if the feature drove improvements or caused regressions.
5. **Make a decision:** Make a decision to ship, iterate, or pivot.

## Benefits

* **Regression detection:** Quickly see whether a feature rollout causes regressions in key business, product, or system metrics, e.g. revenue, conversion rates, cost, latency, error rates, etc.
* **Data-informed decisions:** Understand exactly how a feature affects key metrics, so you can decide whether to ship, iterate, or pivot.
* **Faster decisions:** Quickly get stakeholder alignment and the confidence needed to ship a feature, iterate, or pivot.
* **Accelerated learning:** Learn what kinds of changes result in the most impact and feed that into roadmap prioritization and product strategy.
* **Impact attribution:** Directly attribute impact to the work of specific teams and individuals to recognize contributions and reward high-impact work.

## ROI

These benefits help teams:

* **Improve key business metrics, e.g. revenue, conversion rates, cost, latency, error rates, etc.**
* **Protect key business metrics from regressions.**
* **Improve reliability for users.**
* **Make better and faster product decisions.**

## Beyond A/B testing

Experimentation isn’t limited to simple A/B tests. You can also:

* Run A/B/n tests to test multiple variants at the same time.
* Run multivariate tests to test all combinations of variants of different features, e.g. all 9 combinations of 3 button text variants and 3 button color variants.
* Run AI loops to automatically learn and shift traffic to the best performing variants, for each unique user.
* Add targeting rules to exclude specific users from an experiment, and override their variant.
* Add targeting rules to run experiments on specific user segments, e.g. new users, enterprise customers, etc.