Hackle vs. Firebase

These are some of the additional features provided by Hackle A/B Testing that are not available with Firebase’s A/B Testing.

Brief Overview Description

Hackle's A/B test allows you to customize any type of metric you want.
Hackle's A/B test allows you to analyze your metrics by segment.
Hackle's A/B test provides more advanced user targeting features.
Hackle's A/B test allows you to run a single experiment in both the Android and iOS platforms simultaneously.
Hackle's A/B test reflects changes made to an A/B test setting near real-time.
Hackle's A/B test frequently updates the results data for each A/B test (at least once per hour).
Hackle provides both client-side and server-side SDK (Firebase only provides client-side SDK).
Hackle's A/B test provides results based on full data without sampling.
Hackle's A/B test is not prone to errors when calculating data for conversion rates.
Hackle provides technical support to customers through the 24/7 Hotline.
With Hackle's A/B test, there is no limit to the number of metrics per A/B test.
Hackle allows you to easily set up A/B tests and feature flags to run simultaneously in both the production and development environment.

Detailed Description

Hackle's A/B test allows you to customize any type of metric you want.

With Hackle, you can select desired events as the denominator/numerator to set as your A/B metrics. Moreover, through the various calculation methods provided by Hackle, you can also customize how you calculate the metric values. For example, calculations for the conversion rate, average order amount (AOV), the average purchase price per user (ARPU), and the average purchase price per purchaser (ARPPU), can all be set up.

Additionally, there is also a "filter" setting that allows you to only calculate the metrics for specific user segments.

Hackle's A/B test allows you to analyze your metrics by segment.

A segmentation analysis for the results of each metric set in the A/B test can be conducted on the basis of platform (iOS, Android, Web, etc.), browser, app version, or property information managed internally by your company (ex. membership member status, first purchase status, gender, age group, etc.)

This allows you to determine whether the results for certain metrics in your A/B testing are only affecting specific user segments.

Hackle's A/B test provides more advanced user targeting features.

Hackle provides a far more advanced targeting feature than that provided by Firebase. Hackle supports a highly customized A/B testing to a very specific desired pool of users by allowing multiple conditions and rules to be set during targeting.

You can experiment in both the Android and iOS platforms simultaneously in one A/B test.

With Firebase, it is cumbersome to run the same experiment separately on two different platforms, Android and iOS.
Hackle supports cross-platform experiments, so you can conduct A/B testing on Android and iOS simultaneously with one experiment without having to create two separate experiments.

Hackle A/B test reflects changes in an experimental setup near real-time.

The SDK's role is to periodically receive the set-up information or changes made from the dashboard and reflect the changes to your code.

Compared to the Firebase SDK, the Hackle SDK has a significantly shorter cycle to update configuration changes, allowing you to control the changes and updates in near real-time.

Hackle A/B test frequently updates experimental results data (at least once per hour).

Firebase provides experimental results based on data collected from the previous day on Google Analytics. In some cases, the data is collected almost 24 hours ago. In this case, it is difficult for the user to immediately judge and respond to the current situation.
Hackle updates the experimental results at least once an hour, hence any time differences between data results provided and the current situation are limited to a maximum of one hour.

Hackle provides both client SDK and server SDK (Firebase only provides client SDK).

It is much more efficient to utilize the server SDK when performing A/B testing on the logic that is performed on the server, such as search logic and recommended store listing logic. This is because the code work only needs to be done on the server.
On the other hand, if you use the client SDK when performing A/B testing of server logic, code work must be done on both the server and the client (Android, iOS). This means working with the code on the client (Android, iOS) requires a significant amount of development resources. This is because you need to consider the compatibility of the lower version of the app and the timing of distribution, and when you end the A/B test, you need to make sure the code is deleted.

Hackle A/B test provides results based on full data without sampling.

Firebase A/B Testing samples and uses data, which in some cases may provide statistically insignificant data.
Hackle A/B testing provides accurate calculations because it uses full data to calculate results.

Hackle A/B test is not prone to errors that occur due to differences in the time measured when calculating conversion rates

In order to calculate a specific conversion rate (numerator/denominator | eg button click rate, purchase conversion rate, etc.), the denominator and numerator values must coincide.
However, in Firebase A/B Testing, there is a possibility that the time when users are distributed to a specific test group (A or B) of the A/B test (Denominator 1) and the time when the user is exposed to the screen where the experiment is in progress (Denominator 2) may not match. In this case, the denominator and numerator values are measured at different points, which may contaminate the results.
In the Hackle A/B test, the point at which users are distributed to a specific test group (A or B) of the A/B test (Denominator 1) and the point at which the user is exposed to the screen where the experiment is in progress (Denominator 2) always match. Since this time coincides with the time the numerator is measured, Hackle's data is usually always accurate.

Hackle provides technical support to customers through the 24/7 Hotline.

In Hackle, we provide private channels for each customer via Slack Messenger so that anyone can conveniently ask questions during the SDK interworking process, A/B test code implementation process, or during their A/B test.

In Hackle A/B test, there is no limit to the number of results that can be checked per experiment.

Firebase is limited to a total of 6 metrics (1 primary metric and 5 addition metrics) that can be tracked and reported in a single A/B test.
With Hackle's A/B testing, there is no limit to the number of metrics created and hence, various metrics can be tracked and monitored simultaneously.

Hackle's A/B test allows you to easily set up A/B tests and feature flags to run simultaneously in both the production and development environment.

Firebase does not separate the production environment from the development environment, developers can easily be confused between different environments. Making a clear distinction between different environments is especially important when implementing A/B testing. Usually, developers like to test out the A/B tests in the development environment for QA purposes. However, if the distinction between the two environments is unclear, mistakes can be made where the A/B test is released to real users in the production environment.

Hackle provides both a production environment and a development environment for each and every A/B test, eliminating the possibility of such mistakes.