Sensors Analytics provides several different tools to facilitate the import of various forms of data into the system. You can determine which import tool to use based on the [*]environment in which it runs, the format of the source data, and the size of the imported data.

Import Tool

Runtime Environment Requirements

Applicable Scenarios and Data Types

Advantages

Disadvantages

FormatImporterPython 3 runtime environment is required.For import CSV、MySQL 、Oracle、Nginx Logs、Log files in json format.Flexible. It can be run as a format converter without using it as an import tool, saving the work of data cleansing.Slow data import. Actually, it uses the BatchConsumer of the Python SDK to import data.
BatchImporterIt runs on the server where Sensors Analytics is deployed and can be used in both standalone and cluster editions, but not in the cloud edition.

Generally, it is used to import historical data in batches. Log files that conform to the JSON format are one JSON line。

Faster import speed compared to FormatImporter and LogAgent.Can only run on standalone and cluster editions, not on the cloud edition.
HdfsImporterIt runs on the server where Sensors Analytics is deployed and can only be used in the cluster edition.It is suitable for mass data import in the cluster version, and conforms to the JSON format of the log file, one line of JSON.The fastest import speed and data directly enters HDFS.Can only run on Sensing cluster environments, cannot be used in cloud and standalone environments.
LogAgentGenerally used on the client's business server or production log server.Used to import real-time data, log files in the format of Sensing JSON, one JSON per line, generally used in conjunction with Sensing backend SDK's fileConsumer.Real-time, no duplication or loss, with progress management and breakpoint resume, can read newly appended content in the same file.Java program, consumes more system resources; not suitable for container scenarios.
Fliebeat+LogstashGenerally used on the client's business server or production log server, supports containerized deployment.Log files in the format of Sensing JSON, one JSON per line, generally used for reporting real-time data from containerized deployment of server-side.Fliebeat+Logstash is a mature open-source tool combination with rich configuration, it can receive various inputs, and for file-based inputs, one component can collect logs from multiple directories. Compared to LogAgent, filebeat is lighter and decoupled from the server-side.Requires a certain understanding of the tool, with various usage methods, and troubleshooting problems may not be easy.