|
|
|
# **基本信息**
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
总局pc公告爬虫
|
|
|
|
zongju_notice,通过scrapy部署
|
|
|
|
项目名称：project-ic-spiders
|
|
|
|
分支：dev_ic_pc_py3
|
|
|
|
```
|
|
|
|
|
|
|
|
## 数据名称（中文）
|
|
|
|
|
|
|
|
<!-- 该项数据的中文标准名称，后续沟通交流使用的规范名称，如：工商公示股东信息、失信被执行人、一般纳税人等-->
|
|
|
|
|
|
|
|
###
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
总局pc公告信息
|
|
|
|
```
|
|
|
|
|
|
|
|
## 数据英文名称
|
|
|
|
|
|
|
|
<!--英文名称，后续流程中所有涉及到英文名称均以此为准，如：partner、shixin、general_taxpayer等-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
zongju_notice
|
|
|
|
```
|
|
|
|
|
|
|
|
## 采集网站（采集入口）
|
|
|
|
|
|
|
|
<!--采集的入口地址，不能只是一个网站域名，具体到该网站的数据入口-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
首页 http://www.gsxt.gov.cn/corp-query-entprise-info-xxgg-100000.html
|
|
|
|
各省的地址不一样
|
|
|
|
```
|
|
|
|
|
|
|
|
## 采集频率及采集策略
|
|
|
|
|
|
|
|
<!--说明该项数据更新频率，存量数据的更新频率及策略、新增数据的采集频率及策略-->
|
|
|
|
|
|
|
|
### 存量更新策略
|
|
|
|
|
|
|
|
<!--无需更新？每天全量更新？逐条轮询更新？多久更新完一轮？或其他-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
无存量更新
|
|
|
|
```
|
|
|
|
|
|
|
|
### 增量采集策略
|
|
|
|
|
|
|
|
<!--新增数据怎么来？无需单独采集新增数据？-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
每日20：00提交一批任务
|
|
|
|
```
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
# **爬虫**
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
工商信息 zongju_notice
|
|
|
|
```
|
|
|
|
|
|
|
|
## 责任人
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
杨龙斌
|
|
|
|
```
|
|
|
|
|
|
|
|
## 爬虫名称
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
zongju_notice
|
|
|
|
```
|
|
|
|
|
|
|
|
<!--spider_name-->
|
|
|
|
|
|
|
|
## 代码地址
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
项目地址:http://tech.pingansec.com/granite/project-ic-spiders/-/blob/dev_ic_pc_py3/scrapy_spiders/gravel_spiders/spiders/ic_notice.py
|
|
|
|
```
|
|
|
|
|
|
|
|
## 队列名称及队列地址
|
|
|
|
|
|
|
|
<!--redis host port db key 优先级说明-->
|
|
|
|
-
|
|
|
|
|
|
|
|
* redis host: redis://:utn@0818@bdp-mq-001.redis.rds.aliyuncs.com:6379/0
|
|
|
|
* redis port: 6379
|
|
|
|
* redis db: 0
|
|
|
|
* redis key:
|
|
|
|
* zongju_notice_spider
|
|
|
|
|
|
|
|
### 优先级队列说明
|
|
|
|
|
|
|
|
* 支持队列优先级
|
|
|
|
|
|
|
|
## 任务来源
|
|
|
|
|
|
|
|
<!--说明爬虫任务的输入。如：来自某个数据库表等。如果来自某个数据库表则应该简要说明该表内的数据是如何维护的。-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
例行任务
|
|
|
|
```
|
|
|
|
|
|
|
|
## 任务输入参数（样例）
|
|
|
|
|
|
|
|
```json
|
|
|
|
{
|
|
|
|
"task_params": {
|
|
|
|
"sub_city_code": "110101",
|
|
|
|
"notice_type": "abnormal_include"
|
|
|
|
},
|
|
|
|
"task_src": 22
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
### 任务样例
|
|
|
|
|
|
|
|
<!--注意是爬虫拿到的完整任务，而不仅仅是task_params-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
```
|
|
|
|
|
|
|
|
### 任务参数说明
|
|
|
|
|
|
|
|
<!--特有参数说明，通用参数比如spider_name,task_params,task_src,task_result等不需说明-->
|
|
|
|
|
|
|
|
```json
|
|
|
|
{
|
|
|
|
"sub_city_code": "省份代码",
|
|
|
|
"notice_type": "公告类型"
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
## data_type说明
|
|
|
|
|
|
|
|
<!--可能产生的data_type说明-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
detail: 详情信息
|
|
|
|
```
|
|
|
|
|
|
|
|
## 爬虫结果的超级数据
|
|
|
|
|
|
|
|
<!--包含所有字段的json数据，每个value都要有样例值-->
|
|
|
|
|
|
|
|
```json
|
|
|
|
{
|
|
|
|
"spider_start_time": "2022-01-19 11:27:36.051569",
|
|
|
|
"task_result": 1000,
|
|
|
|
"task_src": 22,
|
|
|
|
"spider_name": "zongju_notice",
|
|
|
|
"task_params": {
|
|
|
|
"sub_city_code": "150423",
|
|
|
|
"notice_type": "abnormal_remove"
|
|
|
|
},
|
|
|
|
"task_type": "经营异常",
|
|
|
|
"spider_end_time": "2022-01-19 11:27:57.254657",
|
|
|
|
"data": {
|
|
|
|
"search_info": {
|
|
|
|
"datafrom": "",
|
|
|
|
"entName": "",
|
|
|
|
"judAuth": "150430002",
|
|
|
|
"judAuth_CN": "赤峰敖汉旗四道湾子市场监督管理所",
|
|
|
|
"judDate": null,
|
|
|
|
"lastModifiedTime": null,
|
|
|
|
"nodeNum": "150000",
|
|
|
|
"noticeContent": "敖汉旗长久育林绿化有限公司\r\n你单位因未依照《企业信息公示暂行条例》第八条规定的期限公示年度报告，违反了《企业经营异常名录管理暂行办法》 的相关规定，被列入经营异常名录。现企业已履行相关义务，申请移出。\r\n本局依据《企业经营异常名录管理暂行办法》规定，经审查作出决定：将该企业移出经营异常名录。",
|
|
|
|
"noticeDate": 1615132800000,
|
|
|
|
"noticeId": "e5332bb2-017a-1000-e000-0bf17f000001-ycyc",
|
|
|
|
"noticeNO": "",
|
|
|
|
"noticeTitle": "关于敖汉旗长久育林绿化有限公司移出经营异常名录的公告",
|
|
|
|
"noticeType": "12",
|
|
|
|
"simpleCancelUrl": ""
|
|
|
|
},
|
|
|
|
"detail_info": {
|
|
|
|
"datafrom": "",
|
|
|
|
"entName": "",
|
|
|
|
"judAuth": "150430002",
|
|
|
|
"notice_authority": "赤峰敖汉旗四道湾子市场监督管理所",
|
|
|
|
"judDate": 1615132800000,
|
|
|
|
"lastModifiedTime": null,
|
|
|
|
"nodeNum": "",
|
|
|
|
"notice_content": "敖汉旗长久育林绿化有限公司\n你单位因未依照《企业信息公示暂行条例》第八条规定的期限公示年度报告，违反了《企业经营异常名录管理暂行办法》 的相关规定，被列入经营异常名录。现企业已履行相关义务，申请移出。\n本局依据《企业经营异常名录管理暂行办法》规定，经审查作出决定：将该企业移出经营异常名录。",
|
|
|
|
"notice_time": 1615132800000,
|
|
|
|
"notice_id": "e5332bb2-017a-1000-e000-0bf17f000001-ycyc",
|
|
|
|
"noticeNO": "敖四道湾子移（2021）第20000261520001号",
|
|
|
|
"notice_title": "关于敖汉旗长久育林绿化有限公司移出经营异常名录的公告",
|
|
|
|
"noticeType": "12",
|
|
|
|
"simpleCancelUrl": "",
|
|
|
|
"company_name": "敖汉旗长久育林绿化有限公司",
|
|
|
|
"notice_type": 12
|
|
|
|
}
|
|
|
|
},
|
|
|
|
"spider_ip": "192.168.56.1"
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
## 实际爬虫结果的数据结构
|
|
|
|
|
|
|
|
<!--可能与超级数据一致，可能不同的data_type的爬虫结果结构不同，超级数据是把所有data_type的结果组合在一起-->
|
|
|
|
|
|
|
|
```json
|
|
|
|
{
|
|
|
|
"spider_start_time": "2022-01-19 11:27:36.051569",
|
|
|
|
"task_result": 1000,
|
|
|
|
"task_src": 22,
|
|
|
|
"spider_name": "zongju_notice",
|
|
|
|
"task_params": {
|
|
|
|
"sub_city_code": "150423",
|
|
|
|
"notice_type": "abnormal_remove"
|
|
|
|
},
|
|
|
|
"task_type": "经营异常",
|
|
|
|
"spider_end_time": "2022-01-19 11:27:57.254657",
|
|
|
|
"data": {
|
|
|
|
"search_info": {
|
|
|
|
"datafrom": "",
|
|
|
|
"entName": "",
|
|
|
|
"judAuth": "150430002",
|
|
|
|
"judAuth_CN": "赤峰敖汉旗四道湾子市场监督管理所",
|
|
|
|
"judDate": null,
|
|
|
|
"lastModifiedTime": null,
|
|
|
|
"nodeNum": "150000",
|
|
|
|
"noticeContent": "敖汉旗长久育林绿化有限公司\r\n你单位因未依照《企业信息公示暂行条例》第八条规定的期限公示年度报告，违反了《企业经营异常名录管理暂行办法》 的相关规定，被列入经营异常名录。现企业已履行相关义务，申请移出。\r\n本局依据《企业经营异常名录管理暂行办法》规定，经审查作出决定：将该企业移出经营异常名录。",
|
|
|
|
"noticeDate": 1615132800000,
|
|
|
|
"noticeId": "e5332bb2-017a-1000-e000-0bf17f000001-ycyc",
|
|
|
|
"noticeNO": "",
|
|
|
|
"noticeTitle": "关于敖汉旗长久育林绿化有限公司移出经营异常名录的公告",
|
|
|
|
"noticeType": "12",
|
|
|
|
"simpleCancelUrl": ""
|
|
|
|
},
|
|
|
|
"detail_info": {
|
|
|
|
"datafrom": "",
|
|
|
|
"entName": "",
|
|
|
|
"judAuth": "150430002",
|
|
|
|
"notice_authority": "赤峰敖汉旗四道湾子市场监督管理所",
|
|
|
|
"judDate": 1615132800000,
|
|
|
|
"lastModifiedTime": null,
|
|
|
|
"nodeNum": "",
|
|
|
|
"notice_content": "敖汉旗长久育林绿化有限公司\n你单位因未依照《企业信息公示暂行条例》第八条规定的期限公示年度报告，违反了《企业经营异常名录管理暂行办法》 的相关规定，被列入经营异常名录。现企业已履行相关义务，申请移出。\n本局依据《企业经营异常名录管理暂行办法》规定，经审查作出决定：将该企业移出经营异常名录。",
|
|
|
|
"notice_time": 1615132800000,
|
|
|
|
"notice_id": "e5332bb2-017a-1000-e000-0bf17f000001-ycyc",
|
|
|
|
"noticeNO": "敖四道湾子移（2021）第20000261520001号",
|
|
|
|
"notice_title": "关于敖汉旗长久育林绿化有限公司移出经营异常名录的公告",
|
|
|
|
"noticeType": "12",
|
|
|
|
"simpleCancelUrl": "",
|
|
|
|
"company_name": "敖汉旗长久育林绿化有限公司",
|
|
|
|
"notice_type": 12
|
|
|
|
}
|
|
|
|
},
|
|
|
|
"spider_ip": "192.168.56.1"
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
## 爬虫运行环境
|
|
|
|
|
|
|
|
<!--udm模块？scrapy？或其他-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
scrapy
|
|
|
|
```
|
|
|
|
|
|
|
|
## 爬虫部署信息
|
|
|
|
|
|
|
|
<!--部署在哪些机器？每个机器多少进程？项目名称是什么？-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
10.8.6.19 50个进程 zongju_notice
|
|
|
|
```
|
|
|
|
|
|
|
|
## Taskhub地址
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
```
|
|
|
|
|
|
|
|
## Taskhub调度规则说明
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
task_result=1000 # 正常获取到详情任务
|
|
|
|
task_result=1101 # 无结果信息
|
|
|
|
task_result=9101 # 超时错误，需要进行重试，目前重试5次
|
|
|
|
task_result=8000 # 参数错误
|
|
|
|
task_result=8004 # 请求到空白链接
|
|
|
|
```
|
|
|
|
|
|
|
|
## 爬虫监控指标设计
|
|
|
|
|
|
|
|
<!--监控爬虫正常运行的指标是什么？报警规则是什么？-->
|
|
|
|
|
|
|
|
```buildoutcfg
|
|
|
|
(先观察，待补充)
|
|
|
|
索引:
|
|
|
|
监控频率:
|
|
|
|
监控起止时间:
|
|
|
|
报警条件:
|
|
|
|
报警群:
|
|
|
|
报警内容:
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
# **数据归集**
|
|
|
|
|
|
|
|
## 责任人
|
|
|
|
|
|
|
|
```html
|
|
|
|
范召贤
|
|
|
|
```
|
|
|
|
|
|
|
|
## 数据归集方式
|
|
|
|
|
|
|
|
- [] 爬虫直接写kafka
|
|
|
|
|
|
|
|
- [] 爬虫写文件logstash采集
|
|
|
|
|
|
|
|
## 爬虫结果目录
|
|
|
|
|
|
|
|
```html
|
|
|
|
/data/ic_spider_data/zongju_notice
|
|
|
|
```
|
|
|
|
|
|
|
|
## 归集后存放目录
|
|
|
|
|
|
|
|
```html
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
## logstash配置文件名称
|
|
|
|
|
|
|
|
```html
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
## logstash文件采集type
|
|
|
|
|
|
|
|
```html
|
|
|
|
zongju_pc_spider
|
|
|
|
```
|
|
|
|
|
|
|
|
## 数据归集的topic
|
|
|
|
|
|
|
|
```
|
|
|
|
collie-ic-data-to-mongo
|
|
|
|
```
|
|
|
|
|
|
|
|
## ES日志索引及筛选条件
|
|
|
|
|
|
|
|
```
|
|
|
|
collie-ic-spider-business-log-*
|
|
|
|
```
|
|
|
|
|
|
|
|
## 监控指标看板
|
|
|
|
|
|
|
|
## 数据保留策略
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
# **数据清洗**
|
|
|
|
|
|
|
|
## 责任人
|
|
|
|
|
|
|
|
## 代码地址
|
|
|
|
|
|
|
|
## 部署地址
|
|
|
|
|
|
|
|
<!--机器及线上代码地址-->
|
|
|
|
|
|
|
|
## 部署方法及说明
|
|
|
|
|
|
|
|
<!--运行方法及运行命令、supervisor配置、supervisor的program等-->
|
|
|
|
|
|
|
|
- [ ] crontab + data_pump
|
|
|
|
- [ ] supervisor + data_pump
|
|
|
|
- [ ] supervisor + consumer
|
|
|
|
|
|
|
|
## 数据接收来源
|
|
|
|
|
|
|
|
<!--来源于kafka还是归集的文件、topic的group？-->
|
|
|
|
|
|
|
|
## 数据存储表地址
|
|
|
|
|
|
|
|
* 数据库地址：
|
|
|
|
* 表名： |
|
|
\ No newline at end of file |
