Subscription Experiences allow you to simply and intuitively build comprehensive customer journeys. A/B testing is a core functionality of all experiences, including pre-built templates. Run A/B tests on different incentives and offers to determine which marketing efforts increase retention and AOV throughout the customer journey.

This guide covers the benefits of A/B testing and how to run an A/B test when building Subscription Experiences. 

Before you start

• Review the guide How to use the A/B test new reward template to learn more about Recharge's prebuilt template that streamlines the A/B test creation process.

A/B testing overview 

A/B testing, also known as split testing, is a marketing research methodology that compares two versions of a marketing campaign to determine the optimal experience. This is done by randomly dividing the target audience into two groups, and showing each group a different version of the campaign. Test results are then analyzed to determine which campaign performs better.

A/B testing makes it easy and accessible for anyone to build a hypothesis for a new customer experience and to test different iterations with your customers to optimize your engagement & revenue metrics.

Each customer’s journey in the A/B split is unique. For example, setting a 50/50 split does not mean customers are evenly distributed among the two groups. Instead, each customer has a 50% chance of falling into either group. As a result, your overall customer split may lean towards ratios like 43/57 or 52/48.

If you set your A/B testing with a 75/25 distribution, for example, it is likely that the first few customers will follow the 75% path. Keep in mind that each instance operates independently, and as the volume of customers increases, the distribution will not follow a strict 75/25 split.

Benefits of A/B testing

The following lists some benefits of using A/B testing:

• Increased engagement: A/B testing can help you identify the elements of your store that are most engaging to customers, so you can focus your efforts on those areas.
• Improved conversion rates: By testing different campaign versions, you can identify the changes that lead to more next-order conversions.
• Increased revenue: A/B testing helps you increase revenue by optimizing your growth and retention strategies specific to next-order conversions and average order value (AOV).

A/B testing

Step 1 - Determine the goals of your test

1. What: hypothesis would you like to test?
   • Determine if offering a free gift on a specific order sequence improves retention. Set up a Surprise and Delight Subscription Experience or Rewards strategies to test this hypothesis. 
   •  Identify which cross-sell offers increase subscriber AOV and LTV. Set up an Upcoming Order Cross-sell Subscription Experience to test this hypothesis. 
2. When: What part of the customer journey would you like to test? When do you want the experience to trigger? Common questions to ask when determining trigger order number:
   • Is there a specific charge where most of your customers cancel? I.e. after 2, 3, 4 orders? Would incentivizing customers with a free gift after this order help improve retention? 
   • Do you want to expose all loyal customers to a new product and allow them to add to their next subscription at a discounted price?
3. Who: Which customers do you want to target? Gain deeper insights by targeting users based on their characteristics, such as their recurring order number, AOV, LTV, location, or product usage.
   1. Order filters: Target customers who ordered or didn’t order a specific product(s).
      • Do certain products complement each other? Add a free gift or cross-sell offer to customers who already purchased one of those products.
      • Avoid sending customers duplicate products by excluding customers who subscribe to the incentive product.
   2. Customer filters
      • Target your most loyal customers and limit the incentives to customers with a higher-than-average AOV or LTV.
      • Target customers based on location.
4. Time: How long do I want to run the experience and how many customers do I want to participate?
   • Do you only want the test to run for a limited time? Set a start and end date.
   • Do you want to limit the number of customers that can enter the experience? Do you have a limited number of free gifts to send out? Set a customer limit.

Once you have answers to these questions, you can configure the experience in Recharge.

Step 2 - Create an A/B test

To create an A/B test:

1. Create a Subscription Experience in Recharge using the flows canvas.
2. Drag the A/B Test Node onto the canvas to create a second path. 
3. Set the incentive or offer for each path.
4. Modify the notifications using Recharge notifications or the Klaviyo notification node.
5. Update the A/B test percentages as needed. A 50/50 split will reach statistical significance faster than other splits, but the percentages can be updated based on your business needs.

Step 3 - Test and activate

1. Open the experience in your merchant portal, and click the pencil icon for the Paths to configure the percentages for each path. 
2. Set the percentage for Path A to 100%.
   
   
3. Save your changes.
4. Perform a test order to trigger the experience to enter Path A using a test customer account configured to accept marketing. Review the Klaviyo email sent to confirm all variables and links work correctly. Ensure Path A works as expected.
5. Access the experience again and set the percentage for Path B to 100%.
   
6. Save your changes.
7. Perform a test order to trigger the experience to enter Path A using a test customer account configured to accept marketing. Review the Klaviyo email sent to confirm all variables and links work correctly. Ensure Path A works as expected.
8. After testing both paths, update your A/B Test flow back to the 50/50 split. Save your changes and activate your Subscription Experience.
   
9. Save and click Activate flow. 

Step 4 - Review results

To determine the success of the incentive or offer, review your branch results within the experience. Easily see how many customers entered the experience and how many accepted the offer and processed their next charge.

There will naturally be a delay in when the next charge branch results will be complete. You must wait until the customer reaches their next charge, so if the charge frequency is every 2 months, there will be at least a 2-month delay between when a customer enters the experience and completes their next charge. This timeframe is further delayed if a customer delays their next charge date.

Once you determine if the incentive or offer was successful, consider creating an ongoing “Basic experience” to apply the incentive or offer to 100% of the customer segment.