Experiments

The Experiments page shows every test running on your subscription program in one place: payment retry strategies, cancel flows and Workflows, whether you started the test or an Ordergroove agent did. Ordergroove does the statistics for you. It measures each test, tells you how likely it is that the new version is better, and tells you when a test has a result. This article explains what you can test, how to start a test, and how to read the page and your results.


What you can test

Every experiment is an A/B test. It splits subscribers between your current setup (the control) and a new version (the variant), then compares how the two groups perform.

What you test Where it's set up Who builds the variant What decides the winner When the variant wins
Order incentive milestones The A/B Test step in a Workflow You The Workflow's A/B test results You decide whether to apply it.
Cancel flows Retention Agent The agent recommends a test, and you start it Save rate You decide whether to apply it.
Payment retry strategy Payment Agent The agent Recovery rate The agent rolls it out automatically.

Each type is available only when its feature is turned on for your program. All three types run on your existing subscribers.

Note: If your program has run incentive tests, which compare two subscription discounts for new customers, those tests appear on the Experiments page too. Incentive testing isn't open to new programs right now.


Starting an experiment

  1. Go to Data > Experiments.
  2. Click Create Experiment.
  3. Choose the type of test. Each card opens the feature where you set the test up.
The Create Experiment page, showing four test types to choose from.

Here's what happens next for each type:

  • Payment Retry strategy (via Payment Agent): Turn on the Payment Agent in Settings > Payment Recovery. It starts its first test right away. See Payment Agent.
  • Cancel Flows (via Retention Agent): When the Retention Agent recommends a cancel flow test, review it and click Start test. While the test runs, your live cancel flow is locked. See Retention Agent.
  • Order Incentive Milestones: Add an A/B Test step to a Workflow and choose how to split subscribers. See A/B testing a Workflow.

Reading the Experiments page

The page has three parts: summary cards, a timeline and a table.

The top of the Experiments page, with the summary cards and the experiments timeline highlighted.

Summary cards

  • In Progress: how many tests are running, and how many drafts you have.
  • Finished: how many tests have ended.
  • Est. annualized incremental revenue: the extra revenue your winning variants are estimated to bring in over a year.

Timeline

The timeline shows your tests over time. Each bar is one test, running from its start date to its end date (or to today, if it's still running). Running tests are labeled In progress, and finished tests are labeled with their lift. Bars that overlap are tests that ran at the same time. The timeline shows in-progress and finished tests only. If your tests cover more time than fits on the screen, scroll the timeline sideways.

Hover over a bar to see the test's dates, the metric its lift is measured on and its estimated revenue. Click the test's name to open its details.

Table

The experiments table, with its filters, search and columns highlighted.

The table lists every experiment, with tests in progress at the top. For each test it shows the feature, status, lift, estimated annualized incremental revenue, probability of being better, estimated days to decision, and start and end dates. For finished tests, the lift is green when the variant did better than the control and red when it did worse.

To narrow the list, use the All features and All statuses filters, or search by name. Click a column header to sort by that column. To export the table as a CSV file, click the download icon above it.

Statuses

Status What it means
Draft The test is set up but hasn't started.
In progress The test is running and collecting data.
Stopped The test ended early without a result, for example because a safeguard stopped it to protect your results.
Finished The test reached a result: the variant won, the control won, or there was no clear winner.

Experiment details

Click a test's name on the timeline or in the table to open its details page. It shows:

  • The test's hypothesis, under its name, and a description of the control and the variant.
  • A summary: the status, lift, estimated annualized incremental revenue, probability of being better, feature, traffic split, start date, and estimated days to a decision (replaced by the end date once the test ends).
  • Charts comparing the control and the variant over time. The ship decision metric's chart is tagged Ship decision metric and labeled Leading metric. Metrics that take longer to mature, like order retention, are labeled Lagging metric and show Long-term metric. Check again soon. until they have data.
  • A metrics table comparing the control and the variant, with the difference between them in the Variant vs Control column. Hover over the info icon next to a metric to see what it measures.
The details page for a finished test, highlighting the hypothesis, control and variant, summary, ship decision metric, lagging metric, and Variant vs Control column.

When a test finishes, its status changes to Finished, and its lift and probability of being better show the result. For payment retry tests, a winning strategy is rolled out automatically. For other tests, you decide whether to apply it.


Understanding your results

  • Ship decision metric: the one metric that decides whether the variant wins. For payment retry tests it's recovery rate. For cancel flow tests it's save rate. The other metrics on the page give context. Some of them, like order retention, are lagging metrics that take longer to mature.
  • Save rate: the share of subscribers who reached your cancel flow and were still subscribed 24 hours later. It includes subscribers who left the flow without finishing it.
  • Lift: how much better or worse the variant does than the control on the ship decision metric. It's shown two ways: in percentage points and as a percentage. For example, if the control recovers 20% of failed payments and the variant recovers 22%, the lift is 2 percentage points, or 10%.
  • Probability of being better: how likely it is, based on the data so far, that the variant really beats the control. For payment retry tests, a variant needs at least 95% to be rolled out.
  • Estimated days to decision: about how long until the test has enough data to reach a result.
  • Estimated annualized incremental revenue: an estimate of the extra revenue the variant would bring in over a year if it replaced the control. It helps you size the impact of a test, but the winner is decided by the ship decision metric, not by revenue.
  • Traffic split: how subscribers are divided between the control and the variant. Payment retry tests split them evenly. In a Workflow, you choose the split.

How Ordergroove decides a winner

Ordergroove uses Bayesian statistics. Instead of a yes-or-no answer about statistical significance, you see the probability that the variant is better, and how that probability changes as data comes in.

Safeguards watch payment retry and cancel flow tests while they run. If a variant is hurting your results, the test is stopped and your current setup stays in place. Payment retry tests run for at least 30 days and at most 90 days.

Why early results swing

In the first days of a test, a handful of subscribers can move the numbers a lot. A variant that looks behind in its first week can still win, and one that looks ahead can still lose. Some metrics show no data at first, or show Long-term metric. Check again soon. Wait for the test to reach a decision rather than acting on early numbers.


Frequently asked questions

When will my test end?

Check Est. days to decision on the test's details page or in the table. Payment retry tests end within 90 days. When a test ends, its status changes to Finished and its details page shows the end date.

Should I stop a test that's losing?

Usually not, especially early on. Results swing a lot in a test's first days, and safeguards stop payment retry and cancel flow tests automatically if a variant is hurting your results. If you turn off the Payment Agent, a test that's already running continues until it's complete.

Can I test on my existing subscribers?

Yes. Payment retry, cancel flow and Workflow tests all run on your existing subscribers. Incentive tests are the exception: they include new customers only.

Why don't I see results yet?

A new test needs time to collect data, and some metrics, like order retention, take weeks to mature. A zero or a blank can mean the data isn't ready yet rather than that nothing happened.

How often do results update?

Results aren't live. New activity can take a day or more to appear on the Experiments page.

Why don't my cancel flow test numbers match Cancel Flow Analytics?

The two measure differently. A cancel flow test counts subscribers who left the flow without finishing it, and measures whether they're still subscribed 24 hours later. Cancel Flow Analytics doesn't, so its numbers can differ.

What's the difference between A/B testing in a Workflow and the Experiments page?

You run a Workflow A/B test by adding an A/B Test step to the Workflow. The Experiments page is where you see that test alongside every other test on your program, including the ones the Payment Agent and Retention Agent run.

Why can't I open some older tests?

Older payment retry tests, listed as Retry Flows, are kept in the table for your records but don't have a details page.

The Experiments page won't load. What should I do?

Your browser may be using an old copy of the Ordergroove admin from before an update. Try these steps in order:

  1. Do a hard refresh on the page: Cmd+Shift+R on a Mac, or Ctrl+Shift+R on Windows.
  2. Open the page in a private or incognito window. If it loads there, clear your browser's cached files for the Ordergroove admin.
  3. Turn off browser extensions, such as ad blockers or script blockers, for the Ordergroove admin.
I can't start or open a test. What should I do?

You may not have permission for the feature that runs the test. Contact Ordergroove Support to check your user permissions. Permission changes take effect the next time you sign in.


Related articles