Data validation logic
|
Collect
1. Data strong validation mode specific errors and handling methods
Data reporting scenarios | Reporting process | Is metadata unable to be created | Does the code return an error | Does the buried point data query report an error | Is the data discarded | Can discarded data be recovered |
---|---|---|---|---|---|---|
Corresponding event name or attribute name in the original event, and the reported data conforms to the SenseData format |
| No | No | No | / | / |
Event not created, data reported for non-existent event in original event |
| / | Yes | Yes | Whole data is discarded | Yes |
Created a custom event with $ character in metadata and reported data | If the event name with $ character is not set before enabling strong verification, an error will occur when creating the event | Yes | / | / | / | / |
If the event name with $ character has been set before enabling strong verification
| No | No | No | / | / | |
The attribute name created in the metadata contains reserved fields for Sensors Analytics and reports data |
| Yes | Yes | No | / | / |
The event name created in the metadata is illegal and illegal event name data is reported |
| Yes | Yes | No | / | / |
Create an event with the same name as the virtual event and report data |
| Yes | Yes | Yes | The entire data is discarded | Yes |
The case case of the created attribute name is inconsistent with that of the existing attribute name in the meta event, and the data is reported |
| Yes | Yes | Yes | The error attribute is not stored. Other data is stored normally | No |
The reported data contains an attribute name that does not exist in the meta event |
| / | Yes | Yes | Error attributes are not stored, other data is stored normally | No |
The attribute in the reported data is inconsistent with the existing attribute type in the meta-event | The type conversion is successful and stored normally (e.g., number converted to string type) | / | No | No | / | / |
The type conversion fails (e.g., string converted to list), and the previous property has data:
PROPERTY_WITH_WRONG_TYPE, data type error, it was previously defined as xxx, but this time the value passed is...
| / | Yes | Yes | Default intercepts the entire data. It can be modified to only intercept attributes | No | |
Failed type conversion (e.g. string to list), and the property has not reported data before:
Data type not match with pre defined properties
| / | Yes | Yes | Error attribute is not stored, other data is normally stored | No | |
The length of reported data attribute exceeds the limit | The length of string type exceeds the limit
| / | Yes | No | / | / |
String type exceeds the length limit
| / | No | No | Truncated the bytes exceeding the length limit (truncated exceeding 1024 bytes) and normally stored in database | / | |
Number type exceeds the limit (number attribute exceeds the length limit or has more than 3 decimal places)
| / | Yes | Yes | Error attributes are not stored in the database, other data can be stored normally | No | |
Custom properties created in the metadata with the character $ are reported |
| Yes | Yes | Yes | Error attributes not stored, other data stored as normal | No |
Create attribute name in metadata is invalid and report invalid attribute name data |
| Yes | / | / | / | / |
Newly added preset attributes by the SDK, but the version of Sensdata analysis is relatively old | Same as "Create custom attributes in metadata with $ character and report data" | / | yes | yes | Error attributes do not enter the database, other data are normally entered | no |
Create an attribute name with the same name as the virtual attribute and report data |
| yes | yes | yes | Error attributes do not enter the database, other data are normally entered | no |
Create an attribute name with the prefix user_tag or user_group and report data | no | no | no | Attributes with the user_tag or user_group prefix are discarded, and other data is stored normally | no | |
Data verification for normal_token or super_token is configured. | Metadata creation is not affected Reference document:https://manual.sensorsdata.cn/sa/latest/数据导入常见问题-47677442.html#id-.数据导入常见问题v2.3-判断数据Token类型 Error scenario:
{"type":"INVALID_IMPORT_TOKEN","reason":"Invalid import token "}
2. If the super_token is set, but the imported data token is empty or inconsistent with the set super_token, and the meta event that does not exist in the metadata is reported
| No | Yes | Yes | The entire data is discarded | Yes |
If the super_token is set, but the imported data token is empty or inconsistent with the set super_token, and an event/user attribute that does not exist in the reported metadata is reported
| No | Yes | Yes | Incorrect attribute is not stored, other data is stored normally | No |
2. Detailed error and handling method in free reporting mode
Data reporting scenario | Screenshot | Does the code return an error | Is there an error in the buried data query | Is the data discarded | Can the discarded data be retrieved |
---|---|---|---|---|---|
In the reporting data, the data that meets the Sensors Data format | Normal reporting data, the buried data query shows no errors, and the no error display is as follows: | No | No | / | / |
In the reporting data, custom events in raw data added the $ character | Normal reporting data | No | No | / | / |
In the reporting data, the property name in the raw data contains the reserved fields of Sensors | Data cannot be reported: The property 'xx' is invalid | Yes | No | / | / |
The event name in the original data reported is invalid. | The data cannot be reported: The Event Name 'xx' is invalid | Yes | No | / | / |
The original data reported contains sub-events that do not exist in the parent event. | Normal Data Reporting | No | No | / | / |
The original data in the reporting data includes event names that are the same as virtual events |
| Yes | Yes | The entire data is discarded | Yes |
The property name in the original data in the reporting data is inconsistent with the case of the existing property name in the original event |
| Yes | Yes | The entire data is discarded | Yes |
In the reported data, the original data contains attribute names that do not exist in the meta event | Normal reported data | No | No | / | / |
In the reported data, the data types of attributes in the original data are inconsistent with those in the meta event | The type conversion is successful, and the data is reported normally (such as converting number to string type) | No | No | / | / |
The type conversion fails (such as converting string to list)
| Yes | Yes | The whole data is discarded | Yes | |
In the reported data, the length of the attribute in the original data exceeds the limit | The length of the string type exceeds the limit
| Yes | no | / | / |
The length of the string type exceeds the limit
| no | no | Truncate the overflow byte length and other data is normally stored in the database | / | |
The number type exceeds the limit: number property 'xx' value 'xxx' overflow
| Yes | Yes | Error attributes not stored, other data can be stored normally | No | |
In the reported data, the custom attributes in the raw data added the $ character |
| Yes | Yes | Incorrect attributes will not be stored in the database, other data can be stored normally | No |
Report data with illegal attribute names | Data cannot be reported
| Yes | No | ||
SDK newly added preset attributes, but the version of Sensors Analytics is relatively old | Same as "Creating custom attributes in the metadata added $ characters and reported data" | Yes | Yes | New preset attributes are not stored, other data is normal | No |
The reported data contains event attributes with the same name as the virtual attributes | No | No | Clear the data of the virtual attribute | / | |
In the reported data, the attribute name uses the prefix user_tag or user_group | No | No | Attributes with the prefix user_tag or user_group are discarded, other data is stored normally | No | |
Data validation with normal_token or super_token | Reference document:https://manual.sensorsdata.cn/sa/latest/数据导入常见问题-47677442.html#id-.数据导入常见问题v2.3-判断数据Token类型 Error scenario:
{"type":"INVALID_IMPORT_TOKEN","reason":"Invalid import token "}
2. If super_token is set, but the imported data token is empty or inconsistent with the set super_token, and a meta event that does not exist in the metadata is reported
| Yes | Yes | The whole data is abandoned | Yes |
If super_token is set, but the imported data token is empty or inconsistent with the super_token, and events/user properties that do not exist in the reported metadata are reported
| Yes | Yes | Error attribute is not stored, other data can be stored normally | No |
Note: The content of this document is a technical document that provides details on how to use the Sensors product and does not include sales terms; the specific content of enterprise procurement products and technical services shall be subject to the commercial procurement contract.