Manage Your Experiments lets you run A/B tests (also known as split tests) on your brand’s listing content. Experiments help you compare two versions of content against each other so you can see which performs better. At the end of an experiment, you can review which version of content performed the best and then publish the winning content. By running experiments, you can learn how to build better content that appeals to your customers and helps to drive more sales.
What are good experiments?
The best
experiments share these characteristics:
Version A and Version B are very different from each other. For
example, with A+ Content, use different modules or different orders of
modules. For product titles, try significantly shortening the title length
to reduce noise and encourage more customers to visit your detail page. For
image experiments, try alternatives that make your product easier to
understand and more information-richFor Brand Story, use different
background images and different modules to tell the story behind the
brand.
Important: The more that your content and treatments
are different, the more likely it will be that any performance
differences detected are meaningful and not caused by random
chance.
The duration of the experiment is typically 8-10 weeks. This allows us time
to collect as much data as possible. If you do not want to select a fixed
duration, you can choose to significance. Amazon will conclude the
experiment when we have statistically relevant results. This feature is
available for title, image, and bullet point experiments. Combine Experiment
to Significance with Auto-Publish to automatically publish your winning
content as soon as your experiment reaches statistical-relevance in as
little as 4 weeks.
Set up an experiment
Step 1: Start an Experiment:
From the Manage Your Experiments main screen, click the
Create a New Experiment drop-down. Then select your
desired experiment type. You can select among A+ Content, Bullet Points, Product
Images, Product Description, Product Title, A+ Brand Story and
Multi-Attribute.
Step 2a:Select an eligible ASIN to experiment on: You will be
prompted to select a Reference ASIN. A “Reference ASIN” is
the primary product that will be included in the experiment.
Step
2b: For A+ Content and Brand Story experiments, you can either
select from ASIN-matched existing contents or create new content in A+ Content
manager. You can navigate to A+ Content manager by clicking “Start by duplicating
Version A” on the same page.
Step 2c: For
Multi-Attribute experiments, you can A/B test various experiment types (title,
images, bullet points) under one experiment.
Step 2d:
Recommendation. Manage Your Experiments also offers machine learning-based
suggestions for your Title and A+ Content experiments on eligible ASINs. These
suggestions are based on the content level insights generated from thousands of
experiments that have already been conducted in the past testing similar
changes.
Step 2e:Auto-Publish: : You may opt into using this feature while
creating an experiment. You may also turn this feature on or off for any experiment
in progress. If you opt-in to this feature, Amazon will publish the winning content
version on behalf of the Brand Representative only if the winning version is at least 66%
better.
You may choose a variational ASIN for experimentation. If you choose a
variational ASIN, the system will detail which child ASINs can be included in the
experiment. The system will display ASIN eligibility status, along with details
about why some ASINs may be ineligible.
Note: ASINs with low traffic may not show up in the available
ASINs list and are not eligible for experimentation.
Step 3:Add Experiment Details: To
create your experiment, enter the following details:
Experiment name: This name will only be visible to
you. This name is important as you will use it to identify your experiment
when it is running to review the results.
Hypothesis: A hypothesis is one of the most important
parts of your experiment. Your hypothesis asks a question that you expect to
evaluate with your experimental content. An example hypothesis could be
“Changing my product title from being vague to being more descriptive will
drive more sales.” By stating and validating the hypothesis, you can work to
create learnings that you can apply to products beyond those under
experimentation.
Experiment duration and start dates: The recommended
experiment duration is 8-10 weeks. You can always change your duration or
end your experiment early. However, the longer you run your experiment, the
more confident you’ll be in the results. If you don't want to select a fixed duration, you can choose
to significance. This feature is in beta and
available for title and image experiments. Amazon concludes the
experiment when we have statistically relevant results.
Note: Depending on the validation time of the experimental
content type, the earliest start day may be several days into the future. This
gives Amazon time to validate that all submitted content meets our
guidelines.
Step 4: Select experimental content: Based on the
experimental content type, select content this way:
Product Titles: Enter your proposed titles for experimental content into
each associated box.
A+ Content: Use the selection drop-downs for Version A, to select content
that has been previously approved, already has ASINs applied, and is not
part of a current experiment. We will only show you content that is
associated with the Reference ASIN. For Version B content, you can create
new content from scratch, or duplicate and modify the existing content, or
select from existing content variations. In all cases, ASINs for Version B
are automatically inherited from Version A to prevent a mismatch between the
two versions of content.
A+ Brand Story: Use the selection drop-downs for Version A, to select brand
story that has been previously approved, already has ASINs applied, and is
not part of a current experiment. For Version B brand story, it is possible
to create new brand story, duplicate the existing Version A brand story to
then modify, or select from existing brand story variations that are
different from Version A.
Product Images: Click Upload image and then use the
file picker to select a compliant product image.
Product Bullet Points: Enter your proposed
bullet points for experimental content into each associated box.
Product Description: Enter your proposed
description for experimental content into each associated box.
Multi-Attributes: Select attributes to test and create version A and B for
each attribute, including title, image and bullet points.
For variational ASINS, you can submit product title or product image
content for some or more of the child ASINs, but you always have to submit content
for the parent. This is because the parent ASIN’s content is used more broadly
across the customer experience.
Typically, you’ll have to create new content
to use as Version B. The easiest way to do this is to click the link on our page
that says "Start by duplicating Version A". That will create a copy of your Version
A content with the same set of ASINs (both versions of content must have the same
ASINs applied to submit a valid experiment).
Do not worry if you cannot
complete setting up experiment in one go. You can save it as a draft, and return to
it to add missing details or final touches at a later date. Your drafts are
available for up to 2 weeks, offering you greater convenience without the
rush.
Step 5: Submit your experiment: At this point
your experiment will be scheduled pending content validation.
Important: Make sure to return to Manage Your
Experiments in the days after you’ve submitted your experiment to validate that
content validation passed. If it failed (for example, submitting an image with a
non-white background in a category where this is required), modify the content
to be compliant with the failed validation and submit your experiment
again.
Simplified experiment setup
Setting up title experiments as simple as updating your product title directly on the catalog page for eligible ASINs. This makes the new title your Version B and the winning title is automatically published. You can find these experiments in your experiment dashboard with experiment source that says “Listings Page”.
You can easily opt out - just uncheck the box under the input section. During peak seasons, if you would like to add specific key words (e.g., Mother's Day, Valentine's Day) for immediate title updates, you are encouraged to opt-out of title experiments. For other scenarios, we recommend opting in for title experiments. This ensures that decisions regarding listing optimization are well-informed and data-driven.
Content validation
Any content submitted during an experiment must meet the same guidelines as any regular content not part of an experiment.
Specific guidelines for experimental content types is as follows:
Product Titles: Product titles must not have more than 200 characters, including spaces. This upper limit applies to all categories. Some categories might have a limit of even fewer characters.
Product Images: Images are very important to customers, so quality matters. Choose images that are clear, easy to understand, information-rich, and attractively presented.
A+ Content: Amazon has specific terms and policies regarding types of A+ Content that may not be allowed. Version A must already be approved. Version B can be submitted at the time the experiment is set-up, but both versions of content need to be approved before the experiment can begin. Product title and image experiments will have their content validated as part of the experiment submission.
Brand Story: Same as A+ Content, Amazon has specific terms and policies regarding types of brand story that may not be allowed. Version A must already be approved. Version B can be submitted at the time the experiment is set-up, but both versions of content need to be approved before the experiment can begin.
Product Bullet Points: Bullet points help you to sell features and benefits of your product. Bullet Points are descriptive text about specific aspects of a product, which appear on the detail page. Up to five bullet points can be included for each product. It is better to keep bullet points clear and concise. A general piece of advice is to keep your bullet points under 1,000 characters in total such as, for all five bullets, not per bullet. Being less than 1,000 characters improves their readability.
Product Description: The Product Description is a text-only field with a limited number of characters that varies by product category. Give concise, honest, and friendly overview of its uses and where it fits in its category. Discuss the features and benefits of the product and focus on your product's unique properties. Don't mention competitors. Highlight the best applications for the product. If the product has limitations, you can say so and upsell.
Edit an experiment
To edit an experiment, start by viewing the experiment details. There are multiple states that your experiment can potentially be in, and these states will affect what detail can be edited. Your experiment can be edited in these states:
Scheduled, waiting content validation: In this state, you can edit any of your experiment details including its content.
Failed content validation: In this state, content that you submitted failed validation and you need to revise your experimental content for your experiment to run.
Scheduled, successful content validation: In this state, your content has passed validation and your experiment schedule is locked. If you choose to change the experiment contents, you must select a new experiment start date to allow time to repeat the content validation process.
Experiment in-progress (partial editing available): While an experiment is in-progress, the content can no longer be revised. The hypothesis and duration of the experiment can still however be revised as desired. If you want to change the experimental content once the experiment is in-progress, cancel the existing experiment and create a new one.
Experiments in these states can’t be edited:
Canceled: Your experiment was terminated before the end date.
Completed: You experiment reached the end date. At this point, all customers will see your original (control) content again until you make a decision about what content to publish.
Important: Regardless of the experiment results, when your experiment ends, your experimental content won’t publish automatically. Make sure to publish your experimental content if you want customers to see it post-experiment.
Cancelling an experiment
To cancel an experiment, start by viewing the experiment details. Then select Cancel experiment and provide a reason for cancellation. For example, you might indicate cancellation was due to an experimental content error or that the experiment realized its desired result before the end date. After you have canceled the experiment, the test will be over and results will no longer be collected. Customers viewing your product page will only view the original content. You’ll still be able to see all canceled experiments in your experiment dashboard, along with any results that were collected until the time of cancellation.