Frequently Asked Questions
Hackle A/B Test Questions
1) What kind of A/B tests can we do with Hackle?
Hackle supports many different types of A/B testing conducted on various platforms and channels. With Hackle, you can apply A/B tests with different versions of a page within a website or mobile app and experiment with changes in APIs from the server-side.
Server/client-based A/B testing with screen/page changes, new feature launches, and algorithm changes are also possible.
2) What type of companies use Hackle’s A/B test platform?
Regardless of the size of the company’s respective in-house development team, various companies that provide many different types of services and products currently use Hackle.
Our current clients include Naver Z (Zepeto: virtual reality platform), Kakao Style(Fashion), LG Uplus(Telecom), Yogiyo (Food Delivery), Olive Young(Beauty), Spartan coding club (EduTech for coding) and GameDuo(Game).
3) What are the specific types of functions Hackle provides?
Once you install our SDK, your software will be linked to the Hackle server. After the SDK integration, A/B tests can be easily turned on and off, and you can also control the traffic without a separate code operation.
When the A/B test begins, each unique user is randomly distributed into each test group, and the user's data is collected and statistically analyzed together with the target metric values that are set at the beginning of the experiment. You can receive the calculated results of the indicators you set as the metrics for your A/B test in an hour without the help of a separate data analyst.
4) What are the advantages of Hackle compared to other A/B test platforms?
First, Hackle supports various types of SDKs, so carrying out your A/B test is not only limited to certain websites that are created with a certain programming language. Our service also encompasses both mobile app and server-based A/B tests. For the full list of SDKs provided by Hackle, please refer to What is a SDK?.
Second, Hackle updates configuration changes made on an A/B test setting from the dashboard to your service in real-time. In comparison, a similar A/B service provided by Google Optimize takes more than several hours for the actual experimentation to be applied to the respective channels used for experimentation.
Third, Hackle updates the data collected from each A/B test every hour. The hourly updated result can be accessed from the Hackle dashboard. Google Optimize calculates and presents A/B test results based on data collected a day ago from Google Analytics, making it difficult to stay updated with recent results.
Fourth, Hackle allows you to freely create any type of A/B test metric to be measured for free, regardless of the plan you are subscribed to.
Finally, Hackle is the most affordable A/B testing platform available. Other A/B testing platforms like Optimizely and Google's Optimize 360 are known to cost at least tens of thousands of dollars per year.
For more information, please refer to the document that compares Google Optimize, Firebase, and Optimizely with Hackle.
General A/B Test Questions
1) What is the minimum number of users needed to conduct an A/B test?
There is no predetermined, minimum number of users required for an A/B test, but a daily traffic of 100+ into your channel should be more than enough. If there are enough users, you are more likely to get faster test results in A/B testing, however, even with fewer users, you should always use A/B testing when implementing new features or elements that have large “ripple effects”.
2) Can I do A/B testing without prior statistical knowledge?
You can conduct A/B testing without any statistical knowledge. The Hackle dashboard will provide you with insight on whether the results of an A/B test for indicators such as click-through rate or purchase conversion rate of a specific button/banner are statistically significant.
However, in cases where you would like to extract the raw data related to A/B testing and perform additional analysis, it may be helpful to have some statistical background.
3) What should I keep in mind when analyzing A/B test results?
A test period of at least one to two weeks is required to avoid distortion of test results due to the impact of the days-of-the-week factor. For A/B tests with a short test period, the results may be temporary, and not reflective nor representative of the real situation.
Terminating the test prematurely creates the risk that the results of the test may be misleading. Therefore, it is recommended to observe the test for a minimum period of time (of at least 1 week).
4) I am curious about the difference between a marketing-based A/B test and a product-based A/B test.
From a marketing point of view, A/B testing is an experiment conducted to maximize the performance of landing pages and pop-up messages in apps that are used as user inflow channels. Techniques such as marketing and branding are also useful for marketing A/B testing, as they have one clear success metric: click-through rate. In conclusion, marketing A/B tests are experiments that center around the information that is provided on your website or page.
Product A/B testing means testing whether or not screens, algorithms, and functions work effectively for users in order to maximize various different business metrics on your website or mobile app. Here you can assume the product as your online platform, whether that be your website or app.
5) Where can I inquire about development-related issues such as SDK integration?
First, check the Frequently Asked Questions (SDK Integration) and Troubleshooting documents in the SDK section of this User Guide and check to see if there is already an answer to your problem.
For any unresolved questions, please contact us via [email protected].
In-App Messages
- I have set up the in-app message, but I see "[WARN] Hackle: SDK not ready." in the console and the message is not being displayed.
After initializing the SDK, it fetches information about in-app messages from the Hackle server. However, if there are event triggers before the SDK initialization, this warning message may appear. To resolve this issue, you can make the following code changes.
//as-is
hackleClient.track({ key: "event key" });
//to-be
hackleClient.onReady(() => {
hackleClient.track({ key: "event key" });
});
-
In the app environment, if you specify a web link URL in the click action or button, it will typically open in the default browser of the mobile device (such as Safari, Chrome, etc.). Usually, to display a web view within the app, you need to implement a scheme internally, and by entering the URL according to that scheme, you can navigate to the in-app web view (for example, myApp://webview?url=abcd).Can I link the external web page (e.g. google form survey) as outlink in the app environment?
Updated over 1 year ago