Sell on Amazon Sign in
This article applies to selling in: United States

Experiment Results

We’ll update your experiment results every few weeks until it ends. You can access your results by clicking on the experiment name in the experiments dashboard. Don’t end your experiment based on early results, as those can be misleading. Instead, wait for your experiment to end before deciding what content to publish.

Tip: When your experiment ends, make sure to publish the winning content by re-submitting it in the A+ content manager.

Interpreting Results

Based on the data collected during the experiment, we calculate a range of possible impacts of publishing each piece of content. Results are aggregated for all enrolled ASINs in an experiment. We provide a few kinds of results:

  1. Probability that one version of content is better. For example, if we say there is a 75% probability that Version A is better, that means that 75% of the possible impacts that we calculated show a likely positive units/sales lift from publishing Version A.
  2. For each version of content, we show: Units, sales, conversion rate, units sold per unique visitor, and sample size assigned to that version. The conversion rate is the percentage of unique visitors in the experiment who saw the A+ content and made a purchase. Sample size is the count of unique visitors who saw each version of content. Units per unique visitor is units divided by sample size.
  3. Projected one-year impact. This section is only populated for completed experiments. It shows an estimate of the possible incremental units and sales over the next year from publishing the winning version of content. For high-confidence winners, you will notice that most of the projected impacts are positive. For low-confidence winners, you will notice that the ‘worse case’ impact may be negative. This is because there is still a chance that the content that performed worse during the experiment may actually perform better over time.

Projected One-Year Impact

To project one year impact, we calculate the average daily sales increase of the winning content and multiply by 365. This is an estimate which doesn’t take into account seasonality, price changes, or other factors that would affect your business in the real world; it is provided for informational purposes only and we cannot guarantee any incremental benefits.

The Likely column shows the median (the 50th percentile) of the range of possible outcomes we calculated. The Best Case and Worse Case columns show the 95% confidence interval of those outcomes.

Inconclusive Results

An experiment can end with results that are inconclusive, or results that show a low confidence that one version of content is better than another. However, these results can still be valuable.

Here are some reasons why an experiment may have inconclusive results:

  • The change you made to your content was too small to significantly change customer behavior
  • There wasn’t enough traffic to determine the winning content with high confidence
  • The two versions of content you tested were similarly effective in driving sales
  • The change you made to your content isn’t something that most customers care about when making a purchase decision

Refer to your experiment hypothesis when trying to make sense of inconclusive results. For example, depending on what you changed, an inconclusive result can tell you that a certain type of content isn’t worth investing in because it doesn’t affect customer behavior. Or, it can tell you that two ways of merchandising your product are similarly effective. You can run additional experiments to confirm what you’ve learned from your earlier tests.

Experiment Methodology

These notes on experiment methodology may help you understand how we choose an experiment winner and project impact; however, this is not required to run an experiment.

Experiments are based on individual customer accounts. During an experiment, each customer account that sees your content is considered part of the experiment. Customers are randomly assigned to view one version of content will see that content persistently for the duration of the experiment regardless of device type or other factors, as long as the customer can be identified. Visits to your page where a customer cannot be identified are not included in the sample size. We may automatically remove certain types of data from the sample to improve the accuracy of results, such as statistical outliers.

We use a Bayesian approach to analyze experiment results. This means we construct a probability distribution based on a model as well as the actual results of the experiment. We report the mean effect size (in terms of change in units) as well as the 95% confidence interval (also known as a credible interval) of the posterior probability distribution, which is updated weekly during the experiment based on all experiment data collected since the start. The confidence of a winning treatment is the percentage of outcomes in the probability distribution that show a positive unit sales impact.

To project one year impact, we compute the average difference between the winning and losing treatment sales per day for the duration of the experiment so far and multiply by 365. We provide a 95% confidence interval for the impact which is based on the posterior probability distribution.

Sign in to use the tool and get personalized help (desktop browser required). Sign in


Was this article helpful?
Thank you for your feedback
How can we improve this help page? Note: This form doesn’t create a support case. Please specify what you think is wrong with the content itself.
1000
What was helpful?
1000
Thanks again for your help!
Thank you for your feedback. If you have more questions, please Contact Us.

Reach Hundreds of Millions of Customers

Start Selling On Amazon


© 1999-2020, Amazon.com, Inc. or its affiliates