Data Import

The Data Import feature allows you to send event data stored in your AWS S3, GCP, or GCS to Hackle. The Data Import feature imports data on a daily basis.

Supported Cloud Storage

Preparation

The following tasks are required before data extraction

• Create a storage to store the event data (AWS S3, GCP GCS, etc.).
• Create and authorize a key to access the storage to store event data.
• Process the event data into a standardized format and store it by day (e.g. 2023-01-01, 2023-01-02, etc.)

Generating keys and authorizing : GCP GCS

For GCP GCS, you can generate a key by referring to GCP IAM > Generating and Managing Service Account Keys.

The following authorizations are required when creating a key to access GCS.

storage.buckets.get
storage.objects.get
storage.objects.create
storage.objects.delete
storage.objects.list

Generating keys and authorizing : AWS S3

For AWS S3, you can refer to the following documents to create a key and grant the necessary permissions.

1. Create an AWS IAM User by following the documentation in AWS Docs: Create an IAM User.
2. follow AWS Docs: Creating an IAM Policy to create a policy and include the IAM Policy policy attached as code below. Then add the IAM Policy policy to the IAM Role created in the previous step.
3. Follow AWS Docs: Creating an IAM Key to create a key

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
              "s3:GetObject",
              "s3:GetObjectVersion",
              "s3:DeleteObject",
              "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::<bucket>/<prefix>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::<bucket>",
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "<prefix>/*",
                        "<prefix>/",
                        "<prefix>"                      
                    ]
                }
            }
        }
    ]
}

Data Import Format

Data import currently supports the Apache Parquet format. Below is a schema of the Parquet format data that is passed. It is processed and stored in the format described in the table below

Below is a summary of the data formats described in the table above.

root
 |-- ts: timestamp (nullable = false)
 |-- event_key: string (nullable = false)
 |-- identifiers: string (nullable = false)
 |-- insert_id: string (nullable = false)
 |-- metric_value: decimal(24,6) (nullable = false)
 |-- user_properties: map<string, string> (nullable = false)
 |-- event_properties: map<string, string> (nullable = false)
 |-- platform_properties: map<string, string> (nullable = false)

Processing for Data Import

Process data according to the aforementioned Parquet Format and stored daily in the Bucket.

• When the data processing is complete, create a '_SUCCESS' (Signal) file of 0 Byte.
• Data processing includes D-1 data. For example, if you run the data import on January 2nd, you can process the data on January 1st.

The following is an example of a saved partition and file.

# 2023-01-01 data
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01/_SUCCESS
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01/000000000000.parquet
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01/000000000001.parquet

# 2023-01-02 data
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-02/_SUCCESS
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-02/000000000000.parquet
gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-02/000000000001.parquet

How to Request Data Import

Please contact Hackle team for data import requests. Below information is required to import the data

• Key authorized for access
• AWS S3, GCS Bucket name and partition path where data in the Bucket will be loaded (e.g. 'gcs://customer-data-hackle/test/prefix-custom/dt=2023-01-01`)
• Data loading time (e.g. loading completed before 13:00 KST)