The SDK supports a variety of different languages, although these SDKS differ in the interface provided externally, but the internal implementation uses a unified data format, and all support the direct import of data stored in the form of files that meet the required format, here, we give a more detailed description of the data format.

• Note: The definition of the underlying data transfer format described here is independent of the call interface of the specific SDK

1. Data global format

A log file is a line of JSON that physically corresponds to a piece of data, logically corresponds to an event that describes user behavior, or a Profile operation that describes one or more user attributes.

1.1. track：

Records an Event and its associated Properties.

Sample data:

{ 	"distinct_id": "123456", 	"login_id": "123456", 	"anonymous_id": "0f485d4daaadedae5f", 	"time": 1434556935000, 	"type": "track", 	"event": "ViewProduct", 	"project": "ebiz_test", 	"time_free": true, //建议在导入历史数据时使用，SDK 采集的实时数据不建议使用 "identities":{ 		"$identity_android_id":"0f485d4daaadedae5f", 		"$identity_login_id":"123456" }, 	"properties": { 		"$is_login_id":true, 		"$app_version":"1.3", 		"$wifi":true, 		"$ip":"180.79.35.65", 		"$province":"湖南", 		"$city":"长沙", 		"$user_agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_2 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/58.0.3029.113 Mobile/14F89 Safari/602.1", 		"$screen_width":320, 		"$screen_height":568, 		"product_id":12345, 		"product_name":"苹果", 		"product_classify":"水果", 		"product_price":14.0 	} }

The above fields are described as follows:

• distinct_id：The type is string , for the user's identity, for the user who is not logged in, you can fill the device ID, CookieID, etc., for the login user, you should fill the registered account; In this example, we assume a logon user , so we fill in a logon ID;
• login_id：Type is string, the login ID of the user should be filled with the login ID of the corresponding user;
• anonymous_id：Type is string, the anonymous identity of the user should be filled with the device ID, CookieID, etc.; In this example, we assume that it is an Android device, so it is filled with an Android ID;
• time：The type is the value , the actual timestamp of the event, accurate to milliseconds;
• type：track means to record an Event. Here we assume that it is a commodity browsing behavior.
• event：The event name must be a valid variable name, that is, it cannot start with a digit, and contains only uppercase and lowercase letters, digits, underscores (_), and $. The value starting with $indicates the preset event of the system. The value of the event field cannot start with $.
• project：Name of the project to which the data belongs. If this parameter is not specified, the value is required when this field is used default, this is the default item. The specified item must be an existing item in the system; otherwise, the data will be invalid. For more information about the item, seeMultiple Projects；
• time_free：This field is optional. It indicates that the event is not filtered based on the occurrence time.If the time_free key is displayed and the value is not null, the time is no longer checked whether the time range allowed for import. This field may be used when importing historical data;
• identities：The user ID field in the global user association service can contain multiple user ids. This structure is used only by customers who use the global user association scheme. This structure is ignored by customers who use the simple user association scheme. For more information, seeGlobal user association；

• properties：The specific properties of this Event are in the form of dict. The value starting with $indicates a preset attribute of the system, whose type and Chinese name have been pre-defined. The user-defined attribute name must be a valid variable name, cannot start with a digit, and contains only uppercase and lowercase letters, digits, and underscores. The user-defined attribute name cannot start with $. property with the same name must have the same definition and type in different events. The case of the property of the same name cannot be the same. If the lower-case property already exists, the corresponding upper-case property cannot be imported (for example, if the metadata contains the Abc property name, the ABC property name cannot be passed). Otherwise, the data will fail to be verified and will not be stored.

  • $is_login_id：Indicates whether the distinct_id corresponds to a login ID.
  • $app_version：The version of the App used by the user;
  • $wifi：Whether the user was using wifi at the time of this event;
  • $ip：The IP address of the device used by the user. If $ip appears in the data and there is no $province or $city field in the data, the IP address will be used to parse out the provincial and city information and fill in the missing field.
  • $province、$city：Province and city, if the two fields are not filled, will be resolved according to the IP address;
  • $user_agent：This parameter is optional. If this parameter is passed, the User-Agent is resolved, and the parsing results include: device manufacturer, device model, operating system, operating system version, browser, browser version, and crawler name (if it is a crawler); At present, the ghost is determined by UA and has a default attribute of $bot_name (crawler name), but there are two cases that can not be determined, the first is if the UA is not marked, and will trigger the JS script illegal crawler. Second: If the crawler does not trigger the JS script, then it will not trigger our SDK, so it will not be counted. For reptile species, it is not possible to add all the species in advance, the mainstream divine policies are added, and the others are less common and less abundant.
  • $screen_width、$screen_height：Width and height of the screen;
  • product_id、product_name、product_classify、product_price：Specific properties associated with the product.

1.2. track_signup：

This interface is a more complex feature, if you are using simple user association, please refer toSimple user association, if you are using global user association, seeGlobal user association, contact our technical support staff if necessary.

Sample data:

{ 	"distinct_id":"12345", 	"original_id":"0f485d4d12345e5f", 	"login_id": "12345", 	"anonymous_id": "0f485d4d12345e5f", 	"time": 1434557935000, 	"type": "track_signup", 	"event": "$SignUp", 	"project": "ebiz_test", 	"identities":{ 		"$identity_android_id":"0f485d4d12345e5f", 		"$identity_login_id":"12345" 	}, 	"properties": { 		"$manufacturer":"Apple", 		"$model": "iPhone5,2", 		"$os":"iOS", 		"$os_version":"7.0", 		"$app_version":"1.3", 		"$wifi":true, 		"$ip":"180.79.35.65", 		"$province":"湖南", 		"$city":"长沙", 		"$screen_width":320, 		"$screen_height":568 	} }

This data indicates that a user whose device ID (anonymous ID) is 0f485d4d12345e5f has successfully completed registration, and the login ID after registration is 12345. After the device ID and login ID are successfully associated, the system background will take the anonymous_id of 0f485d4d12345e5f and the login_id of 12345 as a user.

Note that the distinct_id and original_id fields of this interface are mandatory. distinct_id=login_id, original_id=anonymous_id.

1.3. Profile-related operations

Profile-related operations are mainly used to set a user's Profile, and provide the following series of interfaces:

1.3.1. profile_set：

Configure a user Profile. If the user or Profile field already exists, the profile is overwritten. If the profile field does not exist, the profile is automatically created.

Sample data:

{ 	"distinct_id": "12345", 	"login_id": "12345", 	"anonymous_id": "0f485d4da1111fe5f", 	"type": "profile_set", 	"time": 1435290195610, 	"project": "ebiz_test", 	"identities":{ 		"$identity_android_id":"0f485d4da1111fe5f", 		"$identity_login_id":"12345" 	}, "properties": { 		"$province":"湖南", 		"FavoriteFruits": ["苹果","香蕉","芒果"], 		"Age":33, 		"$city":"长沙", 		"IncomeLevel": "3000~5000", 		"$name": "小明", 		"Gender":"男", 		"$signup_time": "2015-06-26 11:43:15.610" 	} }

1.3.2. profile_set_once

Directly set a user's Profile. Unlike the profile_set interface, this record is ignored and does not overwrite existing data if the user or Profile field already exists, or is created automatically if the Profile does not exist. Therefore, the profile_set_once comparison is suitable for setting the first activation time, first registration time, and other attributes that are only valid when set for the first time.

Sample data:

{ 	"distinct_id": "12345", 	"login_id": "12345", 	"anonymous_id": "0f485d4da1111fe5f", 	"type": "profile_set_once", 	"time": 1435290195610, 	"project": "ebiz_test", 	"identities":{ 		"$identity_android_id":"0f485d4da1111fe5f", 		"$identity_login_id":"12345" 	}, "properties": { 		"register_time": "2015-06-26 11:43:15.610", 		"activation_time": "2015-06-26 11:43:15.610" 	} }

1.3.3. profile_increment：

Increase or decrease a Profile value of type NUMBER for a user, such as adding or subtracting 1 from the value of the user attribute Age. If the user does not exist in the users table, the system automatically creates an id for the user in the user table and sets a Profile attribute value for the user. The default value 0 is added to the uploaded Profile value.

Sample data:

{ 	"distinct_id": "12345", 	"login_id": "12345", 	"anonymous_id": "0f485d4da1111fe5f", 	"type": "profile_increment", 	"time": 1435290200354, 	"project": "ebiz_test", 	"identities":{ 		"$identity_android_id":"0f485d4da1111fe5f", 		"$identity_login_id":"12345" 	}, "properties": { 		"age": 1 	} }

1.3.4. profile_delete：

Delete the entire Profile of a user.

Sample data:

{ 	"distinct_id": "12345", 	"login_id": "12345", 	"anonymous_id": "0f485d4da1111fe5f", 	"type": "profile_delete", 	"time": 1437290200354, 	"project": "ebiz_test", 	"identities": { 		"$identity_android_id": "0f485d4da1111fe5f", 		"$identity_login_id": "12345" 	}, 	"properties": {} }

1.3.5. profile_append：

Adds one or more values to a Profile of an array type for a user. If the value uploaded this time is the same as an existing value in the system, it will not be reduplicated by default. If the value uploaded this time, there are duplicate items, it will not be reweighted.

Sample data:

{ 	"distinct_id": "12345", 	"login_id": "12345", 	"anonymous_id": "0f485d4da1111fe5f", 	"type": "profile_append", 	"time": 1437280200354, 	"project": "ebiz_test", 	"identities":{ 		"$identity_android_id":"0f485d4da1111fe5f", 		"$identity_login_id":"12345" 	}, "properties": { 		"FavoriteFruits": ["橘子","西瓜"] 	} }

1.3.6.  profile_unset：

Set some property values for a user to null. Also, to be consistent with other interfaces, set the value of the property to any value that is not null on the submitted data, such as true.

Sample data:

{ 	"distinct_id":"12345", 	"login_id": "12345", 	"anonymous_id": "0f485d4da1111fe5f", 	"type":"profile_unset", 	"time":1437280200354, 	"project": "ebiz_test", 	"identities":{ 		"$identity_android_id":"0f485d4da1111fe5f", 		"$identity_login_id":"12345" 	}, "properties":{ 		"age":true, 		"favorite_fruits":true 	} }

1.4. Item-related operations

Item-related operations, mainly used to set the specific content of the Item, provide the following series of interfaces:

1.4.1. item_set:

Directly set an Item. If the fields of the Item already exist, they will be overwritten. If the fields do not exist, they will be automatically created.

Data sample:

{ 	"type":"item_set", 	"item_id":"12", 	"item_type":"dub", 	"project": "ebiz_test", 	"properties":{ 		"title":"because of u", 		"sub_title":"st", 		"xxx":"xxx" 	} }

Explanation of the fields above:

• type: item_set indicates that it is setting an item;
• item_id: represents the id of the item;
• item_type: the type of the item table, distinguishing different item tables. It needs to be a valid variable name, which cannot start with a number and can only contain uppercase and lowercase letters, numbers, underscores, and $. The length of the item_type field is a maximum of 100 characters;
• project: the project name to which this data belongs. If this parameter is not specified, the value will be "default" when the field is used. The specified project must be an existing project in the system, otherwise this data will be invalid. For more information about projects, please refer to .multi-projects v1.13';
• properties: the specific attributes of this item, presented as a dictionary. The attribute names need to be valid variable names and cannot start with a number. They can only contain uppercase and lowercase letters, numbers, and underscores;

1.4.2. item_delete:

Delete the entire content of the Item.

Data sample:

{ "type":"item_delete", "item_id":"16", "item_type":"dub", "project": "ebiz_test" }

1.5. Property Data Types

1.5.1. Notes

The sender uses JSON as the data transmission format. This system is based on the JSON data type with additional restrictions. It defines several data types, which are not completely equivalent to JSON types. Please refer to the following attribute type description. The (table_name, property_name) tuple uniquely identifies an attribute. table_name is the table name of the data table, such as users, events, which can be seen in custom SQL queries. This means that attributes with the same name must have the same type in the same table, but attributes with the same name can have different types in different tables. The relationship between message type and the target data table is shown in the following table:

The type of an attribute is determined by the type at the first import. The attribute type displayed on the metadata-event attribute/user attribute page is the type at the first import. When importing data later, if the type does not match the type displayed in the metadata, an attempt will be made to convert the data type. If the conversion fails, the input data will be rejected as a whole. The attempted type conversions are as follows (spaces are not converted):

• Target type corresponds to the data type in the metadata, and the original type corresponds to the attribute value type during data uploading.
• When to use numeric properties:
  • Values that need to be aggregated (such as sum, average) or grouped by intervals, typical examples include price, duration, age, etc.
  • Unless there are special requirements, it is not recommended to store various IDs (such as order ID) as numeric types.

1.5.2. Definition of data types in SensData

1.6. Predefined Properties

In order to help users make better use of our product, we currently provide some predefined fields for both Events and Profiles.

Note: There are many predefined properties for the JavaScript SDK that are not listed here. For detailed information, please refer to the related documentation .Web Predefined Properties v1.13

Predefined fields for Events are:

Profile's pre-set fields include:

Item's pre-defined fields:

The meanings of the above Y superscript are as follows:

① The relevant attributes starting with $utm_ are automatically collected by the App Channel Tracking function, please refer to the relevant document .Channel Tracking v1.13.

② When a new user visits for the first time, autoTrack needs to be enabled and the URL needs to contain UTM parameters in order to collect, please refer to the relevant document Web JS SDK.

③ When a new user visits for the first time, autoTrack needs to be enabled in order to collect, please refer to the relevant document Web JS SDK.

④ When a new user starts the App for the first time, if the trackInstallation interface is called, it will automatically assign a value to this attribute.

⑤ The Mini Program SDK version 1.3 or later is supported.

⑥ If the SDK is initialized with is_track_device_id:true, it will be collected.

1.6.1. Collection method of built-in properties

In iOS, the collection method of built-in properties is roughly as follows:

NSMutableDictionary *p = [NSMutableDictionary dictionary]; UIDevice *device = [UIDevice currentDevice]; NSString *deviceModel = [self deviceModel]; struct CGSize size = [UIScreen mainScreen].bounds.size; CTCarrier *carrier = [[[CTTelephonyNetworkInfo alloc] init] subscriberCellularProvider]; // Use setValue semantics to avoid adding keys where value can be nil. [p setValue:[[NSBundle mainBundle] infoDictionary][@"CFBundleShortVersionString"] forKey:@"$app_version"]; [p setValue:carrier.carrierName forKey:@"$carrier"]; [p addEntriesFromDictionary:@{ @"$lib": @"iOS", @"$lib_version": [self libVersion], @"$manufacturer": @"Apple", @"$os": [device systemName], @"$os_version": [device systemVersion], @"$model": deviceModel, @"$screen_height": @((NSInteger)size.height), @"$screen_width": @((NSInteger)size.width), }];

In Android, the collection method of built-in properties is roughly as follows:

{ deviceInfo.put("$lib", "Android"); deviceInfo.put("$lib_version", VERSION); deviceInfo.put("$os", "Android"); deviceInfo.put("$os_version", Build.VERSION.RELEASE == null ? "UNKNOWN" : Build.VERSION.RELEASE); deviceInfo .put("$manufacturer", Build.MANUFACTURER == null ? "UNKNOWN" : Build.MANUFACTURER); deviceInfo.put("$model", Build.MODEL == null ? "UNKNOWN" : Build.MODEL); try { final PackageManager manager = mContext.getPackageManager(); final PackageInfo info = manager.getPackageInfo(mContext.getPackageName(), 0); deviceInfo.put("$app_version", info.versionName); } catch (final PackageManager.NameNotFoundException e) { Log.e(LOGTAG, "Exception getting app version name", e); } final DisplayMetrics displayMetrics = context.getResources().getDisplayMetrics(); deviceInfo.put("$screen_height", displayMetrics.heightPixels); deviceInfo.put("$screen_width", displayMetrics.widthPixels); TelephonyManager telephonyManager = (TelephonyManager) mContext.getSystemService(Context .TELEPHONY_SERVICE); String operatorString = telephonyManager.getSimOperator(); if (operatorString == null) { // DO NOTHING } else if (operatorString.equals("46000") || operatorString.equals("46002")) { deviceInfo.put("$carrier", "中国移动"); } else if (operatorString.equals("46001")) { deviceInfo.put("$carrier", "中国联通"); } else if (operatorString.equals("46003")) { deviceInfo.put("$carrier", "中国电信"); } else { deviceInfo.put("$carrier", "其他"); } }

1.7. Import data restrictions

1.7.1. General restrictions

1. The event variable name (the value of the event) and the attribute variable name (the value of the key in properties) must be valid variable names, meaning they cannot start with a number and can only contain: uppercase and lowercase letters, numbers, underscores, and $ signs. The maximum length of event variable names and attribute variable names is 100, and custom event names or attribute names cannot start with $;
2. The variable name cannot be the same as the variable name of an existing virtual event or virtual attribute;
3. The system has special requirements for the case handling of variable names, where the lowercase and uppercase letters must be identical, but the case difference in variable names will be intercepted;
4. The value of the type field can only be one of the several listed above (track, profile_set, etc.), and it is case-sensitive;
5. The properties field must exist and can be empty ( {} );
6. The allowed range of the event time field is 1000000000000(2001-09-09 09:46:40) to 10000000000000(2286-11-21 01:46:40).
7. The variable name of a user-defined event or attribute cannot be the same as that of a system reserved field. This section 1.7.7 lists reserved attribute names.

1.7.2. Event Time Restriction

The accuracy of data will be affected if user events with unreasonable time are imported (such as importing future data due to client time error). Therefore, by default, there are restrictions on the event time when importing:

1. For data imported using client SDKs (iOS, Android), the server will only receive data with event time within 10 days before the receiving time and 1 hour in the future;
2. For data imported using backend language SDKs (e.g. Java, Python) or import tools (e.g. LogAgent, BatchImporter, HdfsImporter), the default is to import data with event time within 2 years before the current time and 1 hour in the future;

Note:

• If you want to import data outside the default time window described above, you can contact the on-duty team to modify the window restriction, or add the `time_free` field to the data (see section 2. track example in this document). Make sure that the format of `time` meets the requirements stated in section 8.1.
• Because on the App side, only the client's time can be used as the event time, if the client's time is inaccurate, it will cause anomalies in the collected data. Therefore, SensData enables time correction mechanism: When an event occurs on the App side, the value of time (t1), the time when the data is sent (_flush_time) (client time, and the _flush_time is not stored in the database), and the time when the server receives the data time ($receive_time) (server time) are recorded. If |t3-t2|>60s, it is considered that the client's time is inaccurate, and the event trigger time will be corrected. The corrected event time t1' = t1 + (t3-t2).
  The following two scenarios will not correct the event time:
  • If the client time is inaccurate when collecting the event (i.e., t1 is inaccurate, such as when the phone has just been turned on and the time is inaccurate), but the client time is accurate when sending data (_flush_time is accurate, such as when the network time correction function is enabled), as long as |t3-t2|<60s, t1 will not be corrected.
  • If the data is delayed reporting (for example, if the data is forced to be killed by the user before sending, causing the data to not be sent in time, it will be cached locally, and the cached data will be re-attempted to be sent when the app is opened next time and the network is normal), the _flush_time when sending the data is accurate, and the event trigger time will not be corrected.

1.7.3. Limitations on Display Names

1. In addition to variable names, events and properties also have display names to facilitate customers in managing events and properties.
2. The display names must correspond one-to-one with the variable names and do not allow one-to-many or many-to-one relationships. Display names support Chinese, spaces, English, various symbols, and other characters.

1.7.4. Same Name Property, Same Type

For event properties, a property name can only have one type (for different specific events, properties with the same name must also have the same type);
For profile properties, a property name can only have one type;
For the same property name, it can have different types in events and profiles;

1.7.5. Property Length Limit

The data types of properties, and the length restrictions for special fields are as follows:

1.7.6. Maximum Number of Properties

It is recommended to set a reasonable number of properties for each Event / Profile / Item. Too many properties will affect import and query performance, and reaching the limit will cause import exceptions.

1.7.7. Reserved Fields

In order to ensure that property names do not conflict with system variable names during queries, the following reserved fields are set. Please avoid using them as event names or property names (keys in properties):

date datetime distinct_id event events event_id first_id id original_id device_id properties second_id time user_id users user_group 开头 user_tag 开头