Hackle vs. Google Optimize

These are some of the additional features provided by Hackle A/B Testing that are not available with Google Optimize’s A/B Testing.

Brief Overview Description

Hackle's A/B test does not affect the loading speed of your service.
Hackle's A/B test reflects changes made to an A/B test setting near real-time.
Hackle's A/B test frequently updates your A/B test metric results data (at least once per hour).
Hackle supports SPA or Single Page Applications.
Hackle allows you to create metrics with a high level of customization.
Hackle allows the segmentation analysis of metric results in an A/B test.
Hackle provides more advanced targeting capabilities.
Hackle's A/B test is possible in any mobile app, web browser, or server-based platform through various SDKs.
Hackle's A/B test takes into account unique visitor counts that you can easily manipulate.
The data for Hackle's A/B Testing is not subjected to sampling.
Hackle provides technical support to customers through our 24/7 hotline.
Hackle's A/B test does not have any limitations on the number of metrics that can be set up.
Hackle's A/B test allows you to run an unlimited number of experiments at the same time.
Hackle provides both development and production environments for each single A/B test experiment.

Detailed Description

Hackle's A/B test does not affect the loading speed of your service.

The implementation of Hackle’s A/B testing tools into your website or app will never slow down the original speed of your service on your respective channel. On the other hand, when A/B tests created with a visual editor are accumulated in the system, the loading speed of your pages will significantly slow down, affecting the quality of the overall service to your customers.

Hackle's A/B test reflects changes made to an A/B test setting near real-time.

The SDK's role is to periodically receive the set-up information or changes made from the dashboard to reflect the changes to your code.
Compared to Google Optimize, the Hackle has a significantly shorter time cycle to update configuration changes made to the A/B test settings, allowing you to control all changes and updates in real-time.

Hackle's A/B test frequently updates your A/B test metric results data (at least once per hour).

Google Optimize calculates A/B test metric results from data collected from the previous day with Google Analytics. In this case, it is difficult for the user to immediately judge and respond to the current situation.
Hackle updates the experimental results at least once every hour.

SPA is not supported in Google Optimize

SPA (Single Page Application) is not supported by the automated framework of Google Optimize. This is because Google Optimize usually focuses on static objects and creates new variants or versions with its Visual Editor. However, SPA is a website that is not static and interacts with users by dynamically recreating the current web page with new data rather than traditionally having a web browser load up a new web page. Although there is a way of working around this on Google Optimize, it takes up a lot of resources and the appearance of objects in SPA will be unpredictable while using Google Optimize. Meanwhile, on Hackle, we support smooth A/B testing with SPA pages.

Hackle's A/B test allows you to create metrics with a high level of customization.

With Hackle, you can select desired events as the denominator/numerator to set as your A/B metrics. Moreover, through the various calculation methods provided by Hackle, you can also customize how you calculate the metric values. For example, calculations for the conversion rate, average order amount (AOV), the average purchase price per user (ARPU), and the average purchase price per purchaser (ARPPU), can all be set up.

A filtering feature is also provided so that you can measure certain metrics for specific user segments.

Hackle allows the segmentation analysis of metric results in an A/B test.

A segmentation analysis for the results of each metric set in the A/B test can be conducted on the basis of platform (iOS, Android, Web, etc.), browser, app version, or property information managed internally by your company (ex. membership member status, first purchase status, gender, age group, etc.)

This allows you to determine whether the results for certain metrics in your A/B testing are only affecting specific user segments.

Hackle provides more advanced targeting capabilities.

Hackle supports customization of specific targeting conditions than the user targeting settings provided by Google Optimize. You can set up several conditions for targeting in order to conduct A/B tests on desired users.

Hackle's A/B test is possible in any mobile app, web browser, or server-based platform through various SDKs.

Google Optimize only supports A/B testing based on web browsers. Google Optimize does not support A/B testing on mobile app environments. On the other hand, Hackle provides various SDKs to support the implementation of A/B testing on any mobile app, web, or server-based platform through SDKs provided in various languages.

Unlike Google Optimize, Hackle’s SDKs that are available for each programming language allow developers to directly code the different variations for A/B testing. Although directly coding the different variants can be a nuisance, this means that users can have more flexibility with the changes made on the website, and an accurate rendition of their desired version of the page is possible.

Hackle's A/B test takes into account "unique user counts" that you can easily manipulate.

Google Optimize is based on Google Analytics where unique user/visitor IDs are not taken into account and only sessions are considered. This is one of the biggest limitations of Google Optimize as the same visitor who visits a website twice with a span of 4 hours in between each visit will be logged as 2 different visitors on Google Optimize rather than 1 unique user. This has many implications as listed on the importance of unique user IDs in Hackle’s Managing User Identifiers. Google Optimize’s sessions make it hard to obtain meaningful data when calculating metric results.

The data for Hackle's A/B test is not subjected to sampling.

Google Optimize A/B Testing samples data, which in some cases may provide statistically insignificant data.
Hackle A/B testing provides accurate calculations because it uses full data to calculate results.
Google Optimize results are dependent on Google Analytics. Unless you pay for Analytics 360, there is a high chance that your data is being sampled. Sampling is the practice of using a subset of the full traffic and reporting results derived from the sample set. Although sampling may lead to similar results as when the whole data set is being used, when you need precise data that you can trust such as the conversion rate of your website or the total revenue spent in a page, data sampling can be problematic.

Hackle provides technical support to customers through our 24/7 hotline.

With the free Google Optimize tool, users will only be able to use the Google Optimize online community to troubleshoot their issues.

In Hackle, we care about every single one of our customers. We provide private, expert-level support channels for each customer in Slack Messenger so that you can conveniently ask any questions regarding the SDK integration process, A/B test code implementation, and experimenting.

Hackle's A/B test does not have any limitations to the number of metrics that can be set up.

Google Optimize allows for up to 3 pre-configured metrics per experiment. On the other hand, Hackle A/B Testing can allow users to set up an unlimited number of metrics/objectives per experiment

Hackle's A/B test allows you to run an unlimited number of experiments at the same time.

Google Optimize only allows a total of 5 different experiments to run at the same time, while Hackle allows users to set up an unlimited number of experiments to run simultaneously.

Hackle's A/B test provides both development and production environments for each A/B test experiment.

Google Optimize does not separate the production environment and the development environment, and hence there is a possibility that developers will make mistakes due to confusion where they expose the experiment to real users prior to conducting the A/B test in the development environment in advance.

Hackle provides both the production environment and the development environment when conducting a single experiment, and hence this problem will not occur.