A/B tests

Open Online Store › A/B tests. A test compares your live theme with another theme version, such as an earlier one or a redesign, on real visitors.

You need a published theme first: the live theme is what every test compares against.

Plan a test

New A/B test opens a form. Nothing changes for visitors until you start it.

• Name, and What you expect (optional), for example “a bigger buy button sells more”.
• Theme versions to try: up to three, each against the live theme. Preview them first under Themes.
• Traffic for the live theme: Evenly, 80% live theme or 90% live theme.
• How it decides, fixed before the test starts so a lucky streak is not taken for a winner:
  • Confidence for a winner: 90%, 95% (usual) or 99%. Higher needs more visitors.
  • Smallest improvement worth finding (%): 20 means a 2% conversion becoming 2.4%. Smaller needs more visitors.
• When there is a winner: tick Publish the winner on its own, or the test tells you and waits for you to publish it.

Visitors needed estimates how many visitors the live theme and each version need, and about how many days that takes at your recent number of visitors a day. With too few visits for that to be within a year, it says there are not enough visits to tell. A test runs for six weeks at most. With fewer than 20 orders in four weeks, a 2% conversion is assumed.

Run a test

• Start test starts splitting visitors. Each visitor keeps the version they first saw, on that browser. Search engines and other bots always see the live theme.
• Pause shows everyone the live theme; Resume carries on.
• Stop test ends it without a winner, after you confirm. The live theme stays.
• Delete draft removes a test that has not started.
• While a test runs or is paused, the theme or page design it shows cannot be deleted or renamed: the message names the test. End the test first. A draft whose design was deleted meanwhile does not start until you choose the designs again.

How a test decides

• No decision in the first 7 days, so every day of the week is seen.
• Before the planned number of visitors, only an overwhelming difference ends the test.
• At the planned number, a version wins if it is better at the confidence you chose. Otherwise the live theme stays.
• If visitors are not split in the planned shares, the test holds and decides nothing until the split recovers. Check that nothing else changes which theme visitors see.
• Publishing another theme during the test stops it.
• After six weeks without enough visitors, the live theme stays.

The test’s page shows, for the live theme and each version, the Visitors, how many Converted, the Revenue per visitor and the Chance it is better. A visitor counts once they are shown a version while the test runs, even if their visit began a moment before it started. Their orders count while the test runs and not while it is paused, so Converted is never more than Visitors.

Test one page’s designs

What to test can also be one page instead of a whole theme version, while the rest of your store stays as it is:

• Homepage designs: your homepage against other homepage designs.
• Product page designs: product pages against other product page designs (products given a template of their own are left out of the test).
• Landing page: …: one of your landing pages against other page designs.

Make the other designs in the theme editor first: open the page there, then choose Create a template… in the page list and change the copy. Under Designs to try, tick up to three; Open the theme editor takes you there. On the test’s page, Look at it opens each design in the editor.

When a design wins, Use … (or the test on its own, when it may) makes it that page’s design, as a new theme version; Roll back puts the page back as it was. Keep the current design ends the test and changes nothing.

After a winner

When a winner is published, Roll back publishes the earlier live theme again, after you confirm.

Who can use A/B tests

Owners and admins can start A/B tests, because they change what visitors see. Others whose role can see themes see the tests and a note saying who can start them; a role that cannot see themes is told so.

A/B tests come with the Scale and Enterprise plans. On another plan, New A/B test is greyed out and says so, with a link to Plan.