data_stream/picture_download.md.md

@@ -55,35 +55,50 @@ http://tech.pingansec.com/granite/project-gravel/-/tree/develop_dyf/app_picture_
 
 ## 任务来源
 <!--说明爬虫任务的输入。如：来自某个数据库表等。如果来自某个数据库表则应该简要说明该表内的数据是如何维护的。-->
-由使用方掉taskhub 接口输入
+
+* 由使用方掉taskhub 接口输入
 
 
 ## 任务输入参数（样例）
+```json
 {
-        "spider_name":"picture_download",
-        "key": "",
-        "url": "",
-        "bucket": "patent",
-        "proxy":1,   
-    }
+    "spider_name":"picture_download",
+    "key": "",
+    "url": "",
+    "bucket": "patent",
+    "proxy":1,   
+}
+```
+
+
 
 
 ### 任务样例
 <!--注意是爬虫拿到的完整任务，而不仅仅是task_params-->
+
+```json
 {       
-        "key": "",
-        "url": "",
-        "bucket": "patent",
-        "proxy":1, 
- }
+    "key": "",
+    "url": "",
+    "bucket": "patent",
+    "proxy":1, 
+}
+```
+
+
 
 
 ### 任务参数说明
 <!--特有参数说明，通用参数比如spider_name,task_params,task_src,task_result等不需说明-->
+
+```
 "key": 图片唯一键
 "url": 图片url
 "bucket": 图片来源
 "proxy": 是否使用代理，1为使用，0为不使用，默认为1
+```
+
+
 
 
 ## data_type说明
@@ -94,20 +109,86 @@ http://tech.pingansec.com/granite/project-gravel/-/tree/develop_dyf/app_picture_
 <!--包含所有字段的json数据，每个value都要有样例值-->
 
 ```json
-
+{
+	"spider_name": "picture_download",
+	"platform_name": "picture",
+	"http_code": 200,
+	"error_msg": "successful",
+	"task_result": 1000,
+	"data_type": "",
+	"bucket": "patent",
+	"spider_start_time": "2021-10-13 15:48:56",
+	"spider_end_time": "2021-10-13 15:49:08",
+	"spider_used_time_ms": 12,
+	"spider_ip": "10.8.6.30",
+	"task_params": {
+		"key": "CN204671179U",
+		"url": "http://qxb-img.oss-cn-hangzhou.aliyuncs.com/dlpatents/009c730db9230747893f2356324376ba.jpg",
+		"bucket": "patent"
+	},
+	"metadata": {},
+	"data": {
+		"key": "CN204671179U",
+		"bucket": "patent",
+		"store_path": "patent/ff/b1/d0/ffb1d035d18b1d8a37ad2ac54218adb9.jpg",
+		"content": "",
+		"basket_host": "10.8.8.59:31010"
+	}
+}
 ```
 
 ## 实际爬虫结果的数据结构
 <!--可能与超级数据一致，可能不同的data_type的爬虫结果结构不同，超级数据是把所有data_type的结果组合在一起-->
 
+```
+{
+	"spider_name": "picture_download",
+	"platform_name": "picture",
+	"http_code": 200,
+	"error_msg": "successful",
+	"task_result": 1000,
+	"data_type": "",
+	"bucket": "patent",
+	"spider_start_time": "2021-10-13 15:48:56",
+	"spider_end_time": "2021-10-13 15:49:08",
+	"spider_used_time_ms": 12,
+	"spider_ip": "10.8.6.30",
+	"task_params": {
+		"key": "CN204671179U",
+		"url": "http://qxb-img.oss-cn-hangzhou.aliyuncs.com/dlpatents/009c730db9230747893f2356324376ba.jpg",
+		"bucket": "patent"
+	},
+	"metadata": {},
+	"data": {
+		"key": "CN204671179U",
+		"bucket": "patent",
+		"store_path": "patent/ff/b1/d0/ffb1d035d18b1d8a37ad2ac54218adb9.jpg",
+		"content": "",
+		"basket_host": "10.8.8.59:31010"
+	}
+}
+```
+
+
+
 
 ## 爬虫运行环境
 <!--udm模块？scrapy？或其他-->
 
+udm
+
 
 ## 爬虫部署信息
 <!--部署在哪些机器？每个机器多少进程？项目名称是什么？-->
 
+```
+部署机器：10.8.6.30
+进程数：1
+项目名称：app_picture_download
+```
+
+
+
 
 
 ## Taskhub地址
@@ -129,22 +210,24 @@ http://tech.pingansec.com/granite/project-gravel/-/tree/develop_dyf/app_picture_
 
 ## 数据归集方式
 
-- [ ] 爬虫直接写kafka
-
-- [x] 爬虫写文件logstash采集
+- [x] 爬虫直接写kafka
 
+- [ ] 爬虫写文件logstash采集
 
 ## 归集后存放目录
 
+/data2_227/grvael_spider_result/picture_download
+
 
 ## logstash配置文件名称
 
 
 ## logstash文件采集type
 
-
 ## 数据归集的topic
 
+collie_picture_download
+
 
 ## ES日志索引及筛选条件
 
@@ -172,7 +255,7 @@ http://tech.pingansec.com/granite/project-gravel/-/tree/develop_dyf/app_picture_
 ## 部署方法及说明
 <!--运行方法及运行命令、supervisor配置、supervisor的program等-->
 - [ ] crontab + data_pump
-- [ ] supervisor + data_pump
+- [x] supervisor + data_pump
 - [ ] supervisor + consumer
 
 ## 数据接收来源