A/B Test Walkthrough Guide

A step-by-step, overview guide for creating your very first A/B test on Hackle.


Before reading our on getting started with A/B testing, we recommend that you read our glossary page first.


The following guide is based on the production environment.

Prepare for A/B testing

1. Choosing Areas to A/B Test

If there are any changes that have the potential to affect existing users, it's a good idea to A/B test them.
Representative examples include:

  • Logic, Algorithm change
  • UI/UX change
  • Changed text and banners for marketing

2. Create Metric Goals and Hypotheses

There should be a criterion of measurement in order to conclude which version is better, the original or the newly change changed version.
This could be in the form of more users, more button clicks, less loading times, etc.
These criteria are called metrics.

If you find it difficult to set metrics, it is helpful to generate the hypothesis first.
Establish a hypothesis by recalling why the proposed change is better than the existing one and why the change was made.

An example hypothesis to have in an A/B test can be:
"Changing the purchase button color from red to blue will increase the number of clicks."

As you can see from the above hypothesis, we included that the new change will lead to a "better" outcome. Whether this hypothesis is true or false will be determined by the results collected from our metrics data in the A/B test.

How to Carry Out an A/B Test


1. Prerequisites & Set Up

Step 1: SDK Integration β†’
Step 2: Manage Workspace β†’
Step 3: Events β†’
Step 4: User Identifier β†’

2. Create A/B Test

When creating an A/B test, you need to create & set the version (variant) you want to compare with the current version.
Please refer to the document below for more details.

Step 1: Create an A/B Test

3. Set Metrics

Metrics are a means for measuring the performance of different test groups within the A/B test. You can register multiple metrics you want to measure, and you can easily re-register previously registered metrics from other A/B tests.
Please refer to the document below for more details.

Step 2: Set Metrics

4. Set Test Groups

Step 4: Set Test Groups

5. Allocate Traffic

There is only one step left until you press start and begin your A/B test; setting your desired traffic amount.
In Hackle, you can use the traffic allocation function to expose your A/B test or experiment to only a desired percentage of users.

Step 5: Allocate Traffic

You can now start A/B testing.
Check the document below on starting, pausing, and terminating your A/B test.

Status of an Ongoing A/B Test


Run the test for at least a week

It is recommended to run the test for at least one week to avoid distortion of test results due to the influence of the days of the week.

6. Interpret Results


The results of the A/B test can be viewed on the Metrics tab of the A/B test detail page.
You can find out what data you can get from your results and how to interpret them in the article below.

Step 7: Interpret Results β†’


Raw data extraction

With a paid plan (Pro plan or higher), you can extract data regarding user distribution to test groups and events.
For more information about data extraction, please refer to **Frequently Asked Questions (SDK Integration).

7. Remove Code (important)

After deriving results and selecting the winning test group at the end of the A/B test, all users from that point onwards will be shown the version of the page of the winning test group.

Therefore, make sure to remove the code related to the A/B test after the experiment is terminated.