例如,查询按小时的例行导入任务:
hdfs_importer list \
--project_name test_project \
--job_name hourly_import \
--status SUCCESS \
--start_time '2018-08-30 00:00:00' \
--max_num 2 \
--dump /data/work/list_output_file1
输出到文件中的结果为:
[
{
"id": 12,
"session_id": 320,
"job_name": "hourly_import",
"scheduler_job_record_id": null,
"import_path": "/sa/tmp/hdfsImport",
"parameters": "{\"path\":\"/sa/tmp/hdfsImport\",\"project\":\"test_project\",\"user_define_job_name\":\"hourly_import\"}",
"start_time": "2018-09-11 18:46:50",
"end_time": "2018-09-11 18:49:46",
"counter_info": "{\"event_read_count\":252600,\"event_import_count\":252600,\"event_skipped_count\":0,\"profile_read_count\":10104,\"profile_import_count\":10104,\"profile_skipped_count\":0}",
"log_path": "/data/sa_cluster/logs/tools/hdfsimporter/hdfsimporter-43c7d4ea-0b14-48f6-8b03-764178e927ae.log",
"event_job_id": "job_1536029076029_3074",
"profile_job_id": "job_1536029076029_3077",
"event_job_status": "SUCCESS",
"profile_job_status": "SUCCESS",
"event_data_load_status": "SUCCESS",
"project_name": "test_project"
},
{
"id": 10,
"session_id": 317,
"job_name": "hourly_import",
"scheduler_job_record_id": null,
"import_path": "/sa/tmp/hdfsImport",
"parameters": "{\"path\":\"/sa/tmp/hdfsImport\",\"project\":\"test_project\",\"user_define_job_name\":\"hourly_import\"}",
"start_time": "2018-09-11 10:23:20",
"end_time": "2018-09-11 10:26:21",
"counter_info": "{\\"event_read_count\\":252600,\\"event_import_count\\":252600,\\"event_skipped_count\\":0,\\"profile_read_count\\":10104,\\"profile_import_count\\":10104,\\"profile_skipped_count\\":0}",
"log_path": "/data/sa_cluster/logs/tools/hdfsimporter/hdfsimporter-67a00f94-67d8-415e-a004-c9ca82a17a2a.log",
"event_job_id": "job_1536029076029_3044",
"profile_job_id": null,
"event_job_status": "SUCCESS",
"profile_job_status": null,
"event_data_load_status": "SUCCESS",
"project_name": "test_project"
}
]
参数 |
说明 |
样例 |
id |
HDFS Importer 表主键 |
8 |
session_id |
会话 id |
306 |
job_name |
导入任务名 |
hourly_import |
scheduler_job_record_id |
定时任务记录 id |
null |
import_path |
导入的数据所在的 HDFS 路径 |
/sa/tmp/hdfsImport |
parameters |
导入任务配置参数 |
{\"path\":\"/sa/tmp/hdfsImport\",\"project\":\"test_project\",\"user_define_job_name\":\"hourly_import\"} |
start_time |
导入任务开始时间 |
2018-09-10 19:02:08 |
end_time |
导入任务完成时间 |
2018-09-10 19:02:46 |
counter_info |
导入数据条数 |
{\"profile_read_count\":10104,\"profile_import_count\":10104,\"profile_skipped_count\":0} |
log_path |
导入任务运行日志 |
/data/sa_cluster/logs/tools/hdfsimporter/hdfsimporter-0634112a-6b90-40db-a26d-5492dbc7b995.log |
event_job_id |
导入事件数据任务 id |
job_1536029076029_3082 或 null(没有导入事件数据) |
profile_job_id |
导入用户属性任务 id |
job_1536029076029_3083 或 null(没有导入用户属性数据) |
item_job_id |
导入物品属性任务 id |
job_1536029076029_3085 或 null(没有导入物品属性数据) |
event_job_status |
导入事件数据任务状态 |
SUCCESS、FAILED 或 null(没有导入事件数据) |
profile_job_status |
导入用户属性任务状态 |
SUCCESS、FAILED 或 null(没有导入用户属性数据) |
item_job_status |
导入物品属性任务状态 |
SUCCESS、FAILED 或 null(没有导入物品属性数据) |
event_data_load_status |
事件数据入库状态 |
SUCCESS、FAILED 或 null(没有导入事件数据) |
project_name |
项目名 |
test_project |
再如,查询指定 session_id 的所有导入任务状态:
hdfs_importer list \
--session_id 306 \
--full
输出到控制台的结果为
[
{
"id": 8,
"session_id": 306,
"job_name": "hourly_import",
"scheduler_job_record_id": null,
"import_path": "/sa/tmp/hdfsImport",
"parameters": "{\"session\":\"HdfsImport-306\"}",
"start_time": "2018-09-10 19:02:08",
"end_time": "2018-09-10 19:02:46",
"counter_info": "{\"profile_read_count\":10104,\"profile_import_count\":10104,\"profile_skipped_count\":0}",
"log_path": "/data/sa_cluster/logs/tools/hdfsimporter/hdfsimporter-0634112a-6b90-40db-a26d-5492dbc7b995.log",
"event_job_id": null,
"profile_job_id": "job_1536029076029_3084",
"event_job_status": null,
"profile_job_status": "SUCCESS",
"event_data_load_status": null,
"project_name": "test_project"
},
{
"id": 7,
"session_id": 306,
"job_name": "hourly_import",
"scheduler_job_record_id": null,
"import_path": "/sa/tmp/hdfsImport",
"parameters": "{\"path\":\"/sa/tmp/hdfsImport\",\"project\":\"test_project\",\"user_define_job_name\":\"hourly_import\"}",
"start_time": "2018-09-10 18:58:45",
"end_time": "2018-09-10 19:01:10",
"counter_info": "{\"event_read_count\":252600,\"event_import_count\":252600,\"event_skipped_count\":0}",
"log_path": "/data/sa_cluster/logs/tools/hdfsimporter/hdfsimporter-c7514335-5a55-42b8-bfd3-0ad7a27ec1a3.log",
"event_job_id": "job_1536029076029_3082",
"profile_job_id": "job_1536029076029_3083",
"event_job_status": "SUCCESS",
"profile_job_status": "FAILED",
"event_data_load_status": "SUCCESS",
"project_name": "test_project"
}
]