Creating experiments
This article applies to selling in: United States

Creating experiments

Overview

Manage Your Experiments lets you run A/B tests (also known as split tests) on your brand’s listing content. Experiments help you compare two versions of content against each other so you can see which performs better. At the end of an experiment, you can review which version of content performed the best and then publish the winning content. By running experiments, you can learn how to build better content that appeals to your customers and helps to drive more sales.

What are good experiments?

The best experiments share these characteristics:

  1. Version A and Version B are very different from each other. For example, with A+ Content, use different modules or different orders of modules. For product titles, try significantly shortening the title length to reduce noise and encourage more customers to visit your detail page. For image experiments, try alternatives that make your product easier to understand and more information-richFor Brand Story, use different background images and different modules to tell the story behind the brand.
    Important: The more that your content and treatments are different, the more likely it will be that any performance differences detected are meaningful and not caused by random chance.
  2. The duration of the experiment is typically 8-10 weeks. This allows us time to collect as much data as possible. If you do not want to select a fixed duration, you can choose to significance. Amazon will conclude the experiment when we have statistically relevant results. This feature is available for title, image, and bullet point experiments. Combine Experiment to Significance with Auto-Publish to automatically publish your winning content as soon as your experiment reaches statistical-relevance in as little as 4 weeks.

Set up an experiment

Step 1: Start an Experiment: From the Manage Your Experiments main screen, click the Create a New Experiment drop-down. Then select your desired experiment type. You can select among A+ Content, Bullet Points, Product Images, Product Description, Product Title, A+ Brand Story and Multi-Attribute.

Step 2a: Select an eligible ASIN to experiment on: You will be prompted to select a Reference ASIN. A “Reference ASIN” is the primary product that will be included in the experiment.

Step 2b: For A+ Content and Brand Story experiments, you can either select from ASIN-matched existing contents or create new content in A+ Content manager. You can navigate to A+ Content manager by clicking “Start by duplicating Version A” on the same page.

Step 2c: For Multi-Attribute experiments, you can A/B test various experiment types (title, images, bullet points) under one experiment.

Step 2d: Recommendation. Manage Your Experiments also offers machine learning-based suggestions for your Title and A+ Content experiments on eligible ASINs. These suggestions are based on the content level insights generated from thousands of experiments that have already been conducted in the past testing similar changes.

Step 2e: Auto-Publish: : You may opt into using this feature while creating an experiment. You may also turn this feature on or off for any experiment in progress. If you opt-in to this feature, Amazon will publish the winning content version on behalf of the Brand Representative only if the winning version is at least 66% better.

You may choose a variational ASIN for experimentation. If you choose a variational ASIN, the system will detail which child ASINs can be included in the experiment. The system will display ASIN eligibility status, along with details about why some ASINs may be ineligible.

Note: ASINs with low traffic may not show up in the available ASINs list and are not eligible for experimentation.

Step 3:Add Experiment Details: To create your experiment, enter the following details:

  • Experiment name: This name will only be visible to you. This name is important as you will use it to identify your experiment when it is running to review the results.
  • Hypothesis: A hypothesis is one of the most important parts of your experiment. Your hypothesis asks a question that you expect to evaluate with your experimental content. An example hypothesis could be “Changing my product title from being vague to being more descriptive will drive more sales.” By stating and validating the hypothesis, you can work to create learnings that you can apply to products beyond those under experimentation.
  • Experiment duration and start dates: The recommended experiment duration is 8-10 weeks. You can always change your duration or end your experiment early. However, the longer you run your experiment, the more confident you’ll be in the results. If you don't want to select a fixed duration, you can choose to significance. This feature is in beta and available for title and image experiments. Amazon concludes the experiment when we have statistically relevant results.

Note: Depending on the validation time of the experimental content type, the earliest start day may be several days into the future. This gives Amazon time to validate that all submitted content meets our guidelines.

Step 4: Select experimental content: Based on the experimental content type, select content this way:

  • Product Titles: Enter your proposed titles for experimental content into each associated box.
  • A+ Content: Use the selection drop-downs for Version A, to select content that has been previously approved, already has ASINs applied, and is not part of a current experiment. We will only show you content that is associated with the Reference ASIN. For Version B content, you can create new content from scratch, or duplicate and modify the existing content, or select from existing content variations. In all cases, ASINs for Version B are automatically inherited from Version A to prevent a mismatch between the two versions of content.
  • A+ Brand Story: Use the selection drop-downs for Version A, to select brand story that has been previously approved, already has ASINs applied, and is not part of a current experiment. For Version B brand story, it is possible to create new brand story, duplicate the existing Version A brand story to then modify, or select from existing brand story variations that are different from Version A.
  • Product Images: Click Upload image and then use the file picker to select a compliant product image.
  • Product Bullet Points: Enter your proposed bullet points for experimental content into each associated box.
  • Product Description: Enter your proposed description for experimental content into each associated box.
  • Multi-Attributes: Select attributes to test and create version A and B for each attribute, including title, image and bullet points.

For variational ASINS, you can submit product title or product image content for some or more of the child ASINs, but you always have to submit content for the parent. This is because the parent ASIN’s content is used more broadly across the customer experience.

Typically, you’ll have to create new content to use as Version B. The easiest way to do this is to click the link on our page that says "Start by duplicating Version A". That will create a copy of your Version A content with the same set of ASINs (both versions of content must have the same ASINs applied to submit a valid experiment).

Do not worry if you cannot complete setting up experiment in one go. You can save it as a draft, and return to it to add missing details or final touches at a later date. Your drafts are available for up to 2 weeks, offering you greater convenience without the rush.

Step 5: Submit your experiment: At this point your experiment will be scheduled pending content validation.

Important: Make sure to return to Manage Your Experiments in the days after you’ve submitted your experiment to validate that content validation passed. If it failed (for example, submitting an image with a non-white background in a category where this is required), modify the content to be compliant with the failed validation and submit your experiment again.

Simplified experiment setup

Setting up title experiments as simple as updating your product title directly on the catalog page for eligible ASINs. This makes the new title your Version B and the winning title is automatically published. You can find these experiments in your experiment dashboard with experiment source that says “Listings Page”.

You can easily opt out - just uncheck the box under the input section. During peak seasons, if you would like to add specific key words (e.g., Mother's Day, Valentine's Day) for immediate title updates, you are encouraged to opt-out of title experiments. For other scenarios, we recommend opting in for title experiments. This ensures that decisions regarding listing optimization are well-informed and data-driven.

Content validation

Any content submitted during an experiment must meet the same guidelines as any regular content not part of an experiment.

Specific guidelines for experimental content types is as follows:

  • Product Titles: Product titles must not have more than 200 characters, including spaces. This upper limit applies to all categories. Some categories might have a limit of even fewer characters.
  • Product Images: Images are very important to customers, so quality matters. Choose images that are clear, easy to understand, information-rich, and attractively presented.
  • A+ Content: Amazon has specific terms and policies regarding types of A+ Content that may not be allowed. Version A must already be approved. Version B can be submitted at the time the experiment is set-up, but both versions of content need to be approved before the experiment can begin. Product title and image experiments will have their content validated as part of the experiment submission.
  • Brand Story: Same as A+ Content, Amazon has specific terms and policies regarding types of brand story that may not be allowed. Version A must already be approved. Version B can be submitted at the time the experiment is set-up, but both versions of content need to be approved before the experiment can begin.
  • Product Bullet Points: Bullet points help you to sell features and benefits of your product. Bullet Points are descriptive text about specific aspects of a product, which appear on the detail page. Up to five bullet points can be included for each product. It is better to keep bullet points clear and concise. A general piece of advice is to keep your bullet points under 1,000 characters in total such as, for all five bullets, not per bullet. Being less than 1,000 characters improves their readability.
  • Product Description: The Product Description is a text-only field with a limited number of characters that varies by product category. Give concise, honest, and friendly overview of its uses and where it fits in its category. Discuss the features and benefits of the product and focus on your product's unique properties. Don't mention competitors. Highlight the best applications for the product. If the product has limitations, you can say so and upsell.

Edit an experiment

To edit an experiment, start by viewing the experiment details. There are multiple states that your experiment can potentially be in, and these states will affect what detail can be edited. Your experiment can be edited in these states:

  • Scheduled, waiting content validation: In this state, you can edit any of your experiment details including its content.
  • Failed content validation: In this state, content that you submitted failed validation and you need to revise your experimental content for your experiment to run.
  • Scheduled, successful content validation: In this state, your content has passed validation and your experiment schedule is locked. If you choose to change the experiment contents, you must select a new experiment start date to allow time to repeat the content validation process.
  • Experiment in-progress (partial editing available): While an experiment is in-progress, the content can no longer be revised. The hypothesis and duration of the experiment can still however be revised as desired. If you want to change the experimental content once the experiment is in-progress, cancel the existing experiment and create a new one.

Experiments in these states can’t be edited:

  • Canceled: Your experiment was terminated before the end date.
  • Completed: You experiment reached the end date. At this point, all customers will see your original (control) content again until you make a decision about what content to publish.

Important: Regardless of the experiment results, when your experiment ends, your experimental content won’t publish automatically. Make sure to publish your experimental content if you want customers to see it post-experiment.

Cancelling an experiment

To cancel an experiment, start by viewing the experiment details. Then select Cancel experiment and provide a reason for cancellation. For example, you might indicate cancellation was due to an experimental content error or that the experiment realized its desired result before the end date. After you have canceled the experiment, the test will be over and results will no longer be collected. Customers viewing your product page will only view the original content. You’ll still be able to see all canceled experiments in your experiment dashboard, along with any results that were collected until the time of cancellation.

Top