📘

About this document

This document can be used as a reference when trying to understand supported functions, implementation know-hows or advice during the SDK integration process.
Please refer to the Troubleshooting document if you suspect abnormal behavior or bugs.

1. SDK integration

1.1. Is it possible to link the SDK to an outsourced website?

As long as it is possible to modify the code for the website, integration is possible. Since code work is required for SDK integration, the integration depends on whether the code can be modified or not.
An example where SDK linkage is possible is a website based on Cafe24, and an example where it is not possible is Naver Smart Store.

1.2. When should a Hackle Instance be created? Can it be used for other files in the project after creation?

It is recommended to create an instance at the time of initial startup. Since it is implemented as a Singleton, there is no need to re-create it in another file.
When using the SDK that requires import, import it within the file you want to use.

1.3. I'm not sure which SDK key to use.

First, select the right environment (production or development), and select Server, App, or Browser according to the development platform used. The documentation Keys Used by SDK Functions tells you which SDK key to use.
If the app is built by a WebView retrieving the web service information, the Browser key should be selected.

1.4. Can you verify that the integration is successful before starting the test?

Both development/production environments can undergo quality assurance (QA) checks by leveraging the manual assignment function. Manual assignment is a method for forcibly assigning a user identifier specified in the dashboard to a specific test group. For more information, please refer to the Step 6: Register Test Devices (optional) document.

First of all, in the case of the development environment, the user identifiers of test devices of your team member who will participate in the verification are entered and stored for each test group. Then, after starting the experiment, your team can verify that the intended results for each test group are being correctly shown.

For production environments, you should start your experiment after setting the traffic allocation to 0%. This is to ensure that the experiment is not being conducted on actual users but only to the test devices that were manually assigned and registered. The subsequent steps to follow are the same for the steps to take in the development environment.

If you want to know if the event for a particular action is being fired and sent to the server successfully, you can check the real-time delivery status through accessing the detailed page of the corresponding event key in the event menu of the Hackle dashboard. By having the manually assigned test device perform a particular action corresponding to the event, you can check whether this is being sent properly from this menu.

2. User identifier

2.1. What should I set as the user identifier for the mobile app?

One way is to use the user identifier provided by Hackle. However, a new user identifier is given to users who delete and reinstall the app. In other words, in such cases, the user will not be viewed as the same user which may skew results.
If it is difficult to use the user identifier provided by Hackle for cases like above, it is recommended to create a unique value that can differentiate each user and set that value as the user identifier.

2.2. For non-login users, how should I determine the user identifier?

There is no correct answer, but to give one example, in the case of a web browser environment, a unique cookie is baked for each browser and the corresponding value is used, and in the case of a mobile app environment, you can use the unique value is generated when the user installs the app.
There may be many other methods, so please apply the method that suits your service the best.

2.3. Any tips on creating user identifiers based on cookies?

The digits in a user identifier should not be too short or long. Generally it consists of either 16 or 32 digits.
Also, it is recommended that the cookie domain be in the form of .{company name}.kr. This is because the value can be easily shared even if there is a subdomain, and the same value is used regardless of the event being sent out from whichever environment, client or server.
For reference, an example code for generating a user identifier based on a cookie is provided in the Setting up User Identifiers document.

2.4. How should I set the user identifiers in order to measure by sessions?

In general, it is recommended to define user identifiers on a per-user basis. However, if the user identifier is set as the Session ID, it is also possible to analyze the results for each session. In this case, you need to define the criteria for the session or define a session logic, etc. in order to directly set the criteria for calculating the metric.

3. Design and Implement A/B Test

3.1. I am using the server-side SDK. If the distribution result affects not only the server but also the client, what is the best way to implement it?

An easy way to think about is to send the test group value from the server to the client and handle separate logic for each group, but this is not a good way. This is because, if it is confirmed with a specific logic later, changes are required not only in the server but also in the client.

Therefore, it is important to design the code to be modified only on the server. For example, suppose you want to apply red for group A and blue for group B.

# Good case
# After the server assigns a value for the group, the server sends the value to the client side.
# In case of test group A
HTTP/1.1 200 OK
{
  "color" : "red",
  ...
}
# In case of test group B
HTTP/1.1 200 OK
{
  "color" : "blue",
  ...
}

# Client-side processing
fill_color(response.color)

# Bad case
# The server immediately assigns the assigned group value from the server to the client.
HTTP/1.1 200 OK
{
  "variation" : "B"
  ...
}

# Client-side processing
if response.variation == 'A'
  fill_color(red)
elif response.variation == 'B'
  fill_color(blue)

In the bad case of the example, the server sends the distributed test group value to the client and the client branches according to the value, but in the good case, the server internally finishes branching according to the distributed test group , you can see that the client is sending a value that needs to be processed.

3.2. I am using the server-side SDK. I want to test a new function on the client-side, is it better to create an API for each new function and use it?

It can be divided into two cases. If an API to manage the value to be tested already exists, it is easier to manage by adding a response for each test group in the API. However, if such an API does not exist, it is recommended to create a new API, distribute a test group within it, and then send a response for each group.

For example, suppose you created an A/B test where you want to see the reaction of changing the number of characters you can enter. If there is an API that returns the number of inputable characters among existing APIs, the number of characters in test group A and the number of characters in test group B are returned within the API.
On the other hand, if there is no API that returns the number of inputable characters because there is no need to change the number of inputable characters, create an API and distribute test groups within it. In this case, you can also control that only users going through the new API enter the population.

3.3. A/B testing is in progress in the production environment, and I want to check whether the traffic distribution is working well.

After entering the details of the A/B test in the A/B test list on the dashboard, select the Real-time exposure status tab to see the distribution status by test group in the graph that appears. Please refer to the Realtime Exposure Status document.

4. Data

4.1. I want to extract the raw data from my A/B tests, where can I get them?

With a paid plan (Pro plan or higher), you can extract data regarding user distribution to test groups and events.