Step 8-1 Interpret Retention Chart

🚧

Set Metric is only available in Production Environment

A/B Test Retention

You can check the retention chart for each group of exposed users to the test. In addition, you can compare the improvement rate and p-value of the treatment group (group B) compared to the control group (group A), which gives you insight for test result analysis.

Interpreting Retention Chart

Once you save your metric, you can view the retention data on a daily/weekly basis.
The result is calculated one hour after saving and the data is updated on a daily basis. Result data will be collected up to 30 days after the start of the A/B test and discontinued at the close of the test.

• Day0 : The number/percentage of users who occurred the return event on the same day(Day0) when they occurred start event
• Week0 : The number/percentage of users who occurred in the return event on the same week(Week0) when they occurred start event

1. Users : The number of unique users that caused the set start event.
2. Retention : You can check the retention rate and the number of users by the group for each set period unit. Retention rate by group refers to the proportion of users who have generated a start event and who have generated a return event
3. Improvement : The relative difference between the retention rate of the control group (group A) versus the treatment group (group B). The absolute values are displayed in parentheses. For Day1, the retention rate of the treatment group (group B) increased by 98.4% (17 people) than the control group (group A). Depending on whether the retention rate increases or decreases, the number is preceded by a + or -, and if the increase or decrease is statistically significant, it is displayed in green and red respectively
4. p-value : A value that expresses the reliability of the A/B test results. The lower the p-value, the higher the reliability. If the p-value is less than 0.05, the improvement rate is interpreted as statistically significant and if the p-value is more than 0.05, the improvement rate is interpreted as statistically not significant

You can check the retention rate trend at a glance. You can hover over the graph to see the retention rate, the number of each group in that period.