Feature Flag Data Analysis

The 'Data Analysis' tab on the feature flag detailed page is where you can compare and verify the differences between user groups that have exposed to the existing feature and user groups that have exposed to the new feature.

The main purpose of this functionality is to determine whether users exposed to the new feature through feature flags are experiencing any negative impact compared to users exposed to the existing feature. If any issues are identified, the feature flag can be turned off for mitigation. Conversely, if there are no issues, it serves as a guide to progressively allocate more traffic to the new feature.

The 'Data Analysis' tab screen is structured as shown in the image below.

Set Events

When you click the 'Set Events' button, you will see a modal similar to the image below. In this modal, select the events that require data analysis and click 'Save'.

For the selected events, conversion rates and average count are calculated. The analysis results for metrics are provided for user groups exposed to the new feature (True, On) and user groups exposed to the existing feature (False, Off).

Data Analysis Tab - Detailed

Version

By default, the analysis results in the 'Data Analysis' tab are provided on a per-feature flag's version basis. The version of the feature flag changes when there are modifications to the targeting conditions. (e.g. Version 1 → Version 2)

Changes to the targeting conditions of a feature flag include:

• Changing the feature flag state (On, Off)
• Modifying rules in the feature flag's user targeting, including changes to traffic distribution ratios

Improvement Rate

On the graph, you can assess how much improvement has been achieved for the selected metrics between user groups exposed to the new feature (True, On) and those exposed to the existing feature (False, Off).

You can view data from Version 1 to the latest version on the graph. (Image below)

• Select Box: You can choose either the conversion rate metric or the average count metric.
• Cumulative Picker: You can see the improvement rate calculated based on the data from the beginning to the calculation point for each feature flag version.
• Daily Picker: You can see the improvement rate calculated based on daily data from the beginning to the calculation point for each feature flag version.

Metric Analysis Results

You can view the calculated results for all metrics in a table.

While the 'Improvement Rate' graph area at the top only allows you to observe the improvement trend, the metric analysis results table provides additional graphs showing p-values and trend changes for each version's metrics.

You can review the metric analysis results for each feature flag version, and by clicking the select box located in the upper right corner of the table, you can choose the version.

By clicking the 'View Graph' button, you can observe the trend of metrics values, improvement rate, and p-values for the True (On) and False (Off) groups for each metric.

📘

Feature Flag data analysis functionality supported SDK versions