Overview
Experimentation lets you measure the impact of changes to your product or application, ensuring they improve key metrics, prevent regressions, and move you towards your business goals.
Problem
Shipping features without experimentation creates several challenges:
Undetected regressions: You don't know whether a feature rollout caused any regressions in key business, product, or system metrics, e.g. revenue, conversion rates, cost, latency, error rates, etc.
Uninformed decisions: You don't know how a feature affects key metrics, so you can't make data-informed decisions about whether to ship, iterate, or pivot.
Slow decisions: Without data, it's slower to get stakeholder alignment and the confidence needed to ship a feature, iterate, or pivot.
Limited learning: You don't learn what kinds of changes result in the most impact, so you can't feed that into roadmap prioritization.
Recognition challenges: You can't directly attribute impact to the work of specific teams and individuals.
Solution
Before shipping a feature to everyone:
Track events: Ensure you're tracking the analytics events you care about, e.g.
SignUp,Purchase,AddToCart,PageLoadTime.Create an experiment: Create an experiment that randomly assigns users to a control group and a test group.
Use the experiment: Enable the feature for the test group, and keep it disabled for the control group.
Analyze the results: Compare metrics between the groups to see if the feature drove improvements or caused regressions.
Make a decision: Make a decision to ship, iterate, or pivot.
Benefits
Regression detection: Quickly see whether a feature rollout causes regressions in key business, product, or system metrics, e.g. revenue, conversion rates, cost, latency, error rates, etc.
Data-informed decisions: Understand exactly how a feature affects key metrics, so you can decide whether to ship, iterate, or pivot.
Faster decisions: Quickly get stakeholder alignment and the confidence needed to ship a feature, iterate, or pivot.
Accelerated learning: Learn what kinds of changes result in the most impact and feed that into roadmap prioritization and product strategy.
Impact attribution: Directly attribute impact to the work of specific teams and individuals to recognize contributions and reward high-impact work.
ROI
These outcomes help teams:
Improve key business metrics, e.g. revenue, conversion rates, cost, latency, error rates, etc.
Protect key business metrics from regressions.
Improve reliability for users.
Make better and faster product decisions.
Beyond A/B testing
Experimentation isn’t limited to simple A/B tests. You can also:
Run A/B/n tests to test multiple variants at the same time.
Run multivariate tests to test all combinations of variants of different features, e.g. all 9 combinations of 3 button text variants and 3 button color variants.
Run AI loops to automatically learn and shift traffic to the best performing variants, for each unique user.
Add targeting rules to exclude specific users from an experiment, and override their variant.
Add targeting rules to run experiments on specific user segments, e.g. new users, enterprise customers, etc.
Last updated