1. Overview

Data Management > Data Import > Syncs module primarily offers configuration capabilities for data synchronization in general data access scenarios, supporting the synchronization of external data sources to the Sensor system via a visual interface.

1.1. Syncs Type

1.2. Syncs Usage Scenarios

Task Management supports synchronizing External Data Sources and Sensor Data Tables to Sensor Data Tables or Event Tables (events) or User Tables (users), meeting various scenarios:

2. Create Sync

2.1. Create Sync

1. Select Data Fusion > General Data Ingestion > Task Management.
2. Click the Create Task button at the top right, and select Create Sync Task.
3. Fill in Basic Task Information.
   1. Fill in Task Name.
   2. Fill in Description. Optional.
4. Set Data Source and Target.
   1. Set Data Sync Source
      1. Select Source Connection. You can select External Data Source or Sensor Data Table. External data sources support Hive, MySQL, ClickHouse, and FTP.
      2. Select Source Data.
   2. Set Data Sync Target
      1. Select Target Type.
         1. Entity
         2. Existing tables: support primary key tables andnon primary key tables.
         3. Create New Table
            
            1. Enter Table Name.
            2. Select Table Type: Supports creating primary key or non-primary key tables.
            3. Select Primary Key Field: When selecting primary key table type, choose a field from the source table as the primary key field.
      2. Select Target Data. When selecting new table as the target type, table name must be entered.
5. Configure Field Mapping. When selecting new table as the target type, the system will automatically add all fields from the source table. You can manually modify the field names in the target table or delete unnecessary fields.
6. Configure Task Execution Rules.
   1. Select Execution Frequency.
      • Manual Execution: Tasks can be manually executed multiple times within the effective period in the Execution Records module.
        1. Execute Once Upon Effective Time: Sets whether the task will automatically execute once at the start of the effective period.
      • Repeat Execution: Supports configuring task repeat execution frequency using Basic Configuration and Expression Configuration.
   2. Select Effective Cycle.
   3. Configure Advanced Settings.
      1. Set Number of Retry Attempts. Optional.
      2. Set Retry Interval. Optional.
      3. Set Concurrency. Optional.
   4. Select Read Logic.
   5. Select Write Logic. For different types of synchronization targets, the optional write logic varies.
      • When writing data tables with a primary key and a target entity user table, onlyupdates or inserts are supported.
      • When writing to an entity event table, only support Incremental Write.
      • When writing to a data table without a primary key, support for selection Full coverage or Incremental Write. When writing to a partitioned table, you need to first select the partition field and then select the write logic.
   6. Set Alarm monitoring.When the data import task encounters operational problems, it supports the function of alerting maintenance personnel through email, enterprise WeChat, DingTalk, and Feishu to facilitate timely problem investigation and handling.
      1. Turn on/off alarms: Turn on alarms, need in Platform Management > Inbox Management/Touch Channel Management complete the configuration of the corresponding notification channel, which can be referenced 发件箱管理 and Touch Channel Management, need in Platform Management > Alarm management complete notification method configuration, please refer to Alarm Management.
      2. Select Alarm conditions. Support Task execution failed Trigger alarms and Task timeout Trigger alarm.
      3. Notification method
7. Click Data preview button.
8. Click Submit button.

• When you use the Divine Strategy data table as a synchronization source or target, you can only view and choose to haveTable Data Permissions data table.
• Alarm records can be accessed inPlatform Management > Alarm management > Alarm History to view.

2.2. Create SQL Sync

1. Select Data Fusion > Universal Data Access > Task Management.
2. Click on the top right corner Create Task button, and selectCreate SQL task.
3. Fill in Basic task information.
   1. Fill in Task Name.
   2. Fill in Description. optional.
4. Set Read Logic. Support simultaneous reading of multiple databases from one data source. External data source supportHive, ClickHouse, MySQL, FTP.
   1. Select the data source and click on the right side of the table add button.
   2. Click Analysis SQL. After analysingData synchronization source will automatically fill in.
   3. Click Run SQLto view the running results.
5. Set Data synchronization target
   1. Select target type.
      1. Entity
      2. Existing table: Support primary key table and non-primary key table.
      3. Create new table
         
         1. Enter table name.
         2. Select table type: Support creating primary key table or non-primary key table.
         3. Select primary key field: When the table type is selected as primary key table, you need to select a field from the source table as the primary key field.
   2. Select target data. When the target type is selected as a new table, the table name needs to be filled.
6. Configure task execution rules.
   1. Select execution frequency.
      • Manual execution: Within the effective period, the task can be executed manually multiple times in the run record module.
        1. Immediately execute once after reaching the effective time: Set whether to automatically execute the task once when the effective period starts.
      • Repeat execution: Support configuring the task repeat frequency through basic configuration and expression configuration methods.
   2. Select effective period.
   3. Configure advanced settings.
      1. Set retry times when failure. Optional.
      2. Set Retry interval . Optional.
      3. Set Execution concurrency. Optional.
   4. Select Read Logic.
   5. Select Write Logic. For different types of synchronization targets, the optional write logic varies.
      • When writing data tables with primary keys targeting entity user tables, onlyupdates or insertsare supported.
      • When writing to an entity event table, only incremental writing is supported with.
      • When writing to a data table without a primary key, support for selection Full coverage or Incremental write. When writing to a partitioned table, you need to first select the partition field and then select the write logic.
7. Set Alarm monitoring.When the data import task encounters operational problems, it supports the function of alerting maintenance personnel through email, enterprise WeChat, DingTalk, and Feishu to facilitate timely problem investigation and handling.
   1. Turn on/off alarms: Turn on alarms, need in Platform Management > Inbox Management/Touch Channel Management Complete the configuration of the corresponding notification channel, which can be referenced 发件箱管理 and Touch Channel Management; need in Platform Management > Alarm management complete notification method configuration, please refer to Alarm Management.
   2. Select Alarm conditions. Support Task execution failed Trigger alarms and Task timeout Trigger alarm.
   3. Notification method
8. Click Submit button.

2.3. cron expression

When create task configuration Task execution rules, if Execution frequency select Repeated execution, support cron expression collocation method.

The cron expression is a standard expression widely used to describe time and date information. The cron expression is a string consisting of 6 necessary settings and supports optional year settings. Usually, cron expressions are separated by spaces, and the main structure is: seconds, minutes, hours, dates, months, weeks (years, optional).

Currently, only hourly level execution tasks are supported at most.

Each field uses numbers, and special characters such as:

• *: matches any value. If used in sub fields*, this means that it will be executed every minute.
• ?: indicates no specified field and can only be used in the date and week fields.
• -: represents the matching range. For example, using the minute field5-20, indicating execution every minute between 5 minutes and 20 minutes.
• /: Indicates execution starting from the start time and every fixed time. For example, using the minute field5/20Starting from 5 minutes, execute every 20 minutes, that is, every 25 minutes and every 45 minutes.
• ,: Represents listing enumeration values. For example, using the minute field5,20, indicating execution once at 5 and 20 respectively.

For example:

3. Sync Management

3.1. Sync List

Select Data Integration > General Data Access > Task Management to view the task list.

In the task management list, you can perform the following operations:

• Run/Pause Task: Tasks can be paused and restarted multiple times.
• View Run Records: View run records and logs, and supplement data.
• View Tasks: Navigate to the task detail page.
• Edit Tasks: After pausing tasks, all tasks can be modified including task names, descriptions, field mapping, task execution rules, and alerts. SQL tasks can also modify SQL statements.
• Delete Tasks: After deleting tasks, the related sync data tables will not be deleted together.

3.2. Run Records

3.2.1. Run Record Details

In the task list, click View Run Records on the right side of the task to view the task's basic information and run records.

Also supports the following features:

• Execute immediately: Click to execute immediately once.
• Supplementary data: ForTasks with time macro parameters can be clickedSupplementary data Click the button to select the corresponding business date for data supplementation operation.

3.2.2. Running logs

Click on the operation column Running logs button, you can view the specific execution status of each task instance.

4. Permissions

• If you have the permission to view syncs, you can enter data fusion> Universal Data Access> View existing tasks and running records on the task management page.
• If you have the authority to manage syncs, you can enter data fusion> Universal Data Access> The task management page performs a series of management operations such as creation, editing, and deletion.