Skip to content

GitLab

hub · Changes

Page history
Update hub authored by 郭本江's avatar 郭本江
Hide whitespace changes
Inline Side-by-side
Showing with 131 additions and 104 deletions
data_stream/ic_icpsp/hub.md
View page @ 23a25f5e
# **基本信息**
```buildoutcfg
一般纳税人--上海
general_etax_reqs/sh 通过详情url的id遍历搜索
sh_verify_code/sh 通过破解验证码根据三要素搜索
工商.亮照服务--湖北
ic_icpsp_req/hub 通过提交任务参数中的credit_no进行搜索，如果credit_no参数值为空,则通过company_code或者company_name通过亮照服务查询获取credit_no再进行查询。
```
......@@ -10,20 +9,20 @@ sh_verify_code/sh 通过破解验证码根据三要素搜索
<!-- 该项数据的中文标准名称，后续沟通交流使用的规范名称，如：工商公示股东信息、失信被执行人、一般纳税人等-->
###
```buildoutcfg
上海--一般纳税人
湖北--工商.亮照服务
```
## 数据英文名称
<!--英文名称，后续流程中所有涉及到英文名称均以此为准，如：partner、shixin、general_taxpayer等-->
```buildoutcfg
general_taxpayer
ic_icpsp
```
## 采集网站（采集入口）
<!--采集的入口地址，不能只是一个网站域名，具体到该网站的数据入口-->
```buildoutcfg
官网PC端入口:
http://shanghai.chinatax.gov.cn/newxbwz/wzfw/YhscxCtrl-yhsCx.pfv
http://wsdj.egs.gov.cn/ICPSP/queryEnt.action
```
......@@ -35,13 +34,14 @@ http://shanghai.chinatax.gov.cn/newxbwz/wzfw/YhscxCtrl-yhsCx.pfv
### 存量更新策略
<!--无需更新？每天全量更新？逐条轮询更新？多久更新完一轮？或其他-->
```buildoutcfg
db_host: bdp-rds-003.mysql.rds.aliyuncs.com
db_name: utn_ng_biz_info
db_host: bdp-rds-007.mysql.rds.aliyuncs.com
db_name: utn_ic
db_user: shuidi
db_password:
数据库表名:tb_search_company_icpsp
全省的主体信息作为搜索条件
逐条更新
目前每年更新一轮即可。可以根据全量数据的爬取情况，再通过补充详情id顺序号的方式，补充其它主体信息
目前全量更新一轮即可.
```
......@@ -51,14 +51,13 @@ db_password:
```buildoutcfg
1.新成立的主体
2.补充的主体
3.通过已有爬取数据的详情url，分析详情url中的id顺序号，再补充
```
---
# **爬虫**
```buildoutcfg
上海--一般纳税人 gravel_spiders
湖北--工商.亮照服务 ic_icpsp_spiders
```
......@@ -69,18 +68,14 @@ db_password:
## 爬虫名称
```buildoutcfg
gravel_spiders
ic_icpsp_spider
```
<!--spider_name-->
## 代码地址
```buildoutcfg
项目地址:http://192.168.109.199/granite/project-gravel/tree/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/general_etax_reqs/sh
百度移动端:
破码版本: http://192.168.109.199/granite/project-gravel/blob/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/general_etax_reqs/sh_verify_code
遍历详情url顺序号版本: http://192.168.109.199/granite/project-gravel/blob/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/general_etax_reqs/sh/sh_etax_detail.py
项目地址:http://192.168.109.110/granite/project-gravel/-/tree/develop_ic_icpsp/scrapy_spiders/gravel_spiders/spiders/ic_icpsp_reqs/hub
```
## 队列名称及队列地址
......@@ -90,51 +85,64 @@ gravel_spiders
* redis port: 6379
* redis db: 7
* redis key:
* general_taxpayer
* ic_icpsp_keys
### 优先级队列说明
* general_taxpayer 支持队列优先级
* ic_icpsp_keys 支持队列优先级
## 任务来源
<!--说明爬虫任务的输入。如：来自某个数据库表等。如果来自某个数据库表则应该简要说明该表内的数据是如何维护的。-->
```buildoutcfg
taskhub 上海的全量主体信息
taskhub 湖北的全量主体信息
```
## 任务输入参数（样例）
## 任务输入参数（样例） 其中credit_no、province、company_name、company_code为必需
```json
破码版本
{
"company_name": "上海翔星建设工程有限公司",
"company_code": "",
"company_name": "武汉鸿雁房地产代理有限公司",
"company_code": "4201002111621",
"credit_no": "",
"province": "SH",
"company_name_digest": "c139fb8844006aa87dbe5f4d90473c57",
"province": "HUB",
"data_detail_url": "http://xyjg.egs.gov.cn/ECPS_HB/company/detail.jspx?id=E8D47B3680F92E8CE588BF508DCB88BE",
"company_major_type": "3",
"company_status": "已吊销",
"legal_person": "刘桂强",
"company_address": "武昌区紫阳路１４７号",
"create_time": "2016-11-09 07:00:45.194000+00:00",
"lastupdatetime": "2021-03-28 14:57:26+00:00",
"submit_time": "",
"authority": "武汉市工商行政管理局",
"n_company_status": "吊销",
"company_name_digest": "3c7e408bcaedd3a960f0948ad81cbb74",
"district_code": ""
}
顺序号版本
{"province":"SH", "credit_no":"", "company_code":"", "company_name":"", "djxh":"10013101001000048329"}
```
### 任务样例
<!--注意是爬虫拿到的完整任务，而不仅仅是task_params-->
```buildoutcfg
{
"data": {
"taxinfo_items": [
"icpsp_items": [
{
"tax_code": "",
"company_name": "李关金",
"company_type": "内资个体",
"legal_person": "李关金",
"register_address": "",
"bussiness_address": "",
"tax_authority": "国家税务总局上海市黄浦区税务局第四税务所",
"tax_status": "注销",
"tax_type": "按征收率征收增值税小规模纳税人",
"register_date": "",
"detail_url": "http://shanghai.chinatax.gov.cn/newxbwz/wzfw/YhscxCtrl-yhscxXx.pfv?djxh=10013101001010040630"
"search_keyword": "91420115MA49RG6T9L",
"province": "HUB",
"company_name": "武汉中肯商贸有限公司",
"credit_no": "91420115MA49RG6T9L",
"legal_person": "张耘",
"establish_date": "2021-05-17",
"detail_url": "/ICPSP/pdf.action?id=160000020925690093&lzcx=lzcx",
"data_id": "160000020925690093",
"company_address": "湖北省武汉市江夏区郑店街黄金南路8号汽车制造专用模具、夹具、检具生产线4号厂房",
"company_type_code": "1152",
"company_type": "有限责任公司(自然人投资或控股的法人独资)",
"capital": 50,
"company_code": "",
"company_status": "",
"legal_person_certno": "F3A10E8FD2D7882064EC67721F9DEBE9"
}
]
},
......@@ -142,40 +150,43 @@ taskhub 上海的全量主体信息
"error_msg": "",
"task_result": 1000,
"data_type": "detail",
"spider_start_time": "2021-03-20 04:04:44.464",
"spider_end_time": "2021-03-20 04:04:44",
"spider_start_time": "2021-05-24 18:04:27.775",
"spider_end_time": "2021-05-24 18:04:31",
"task_params": {
"province": "SH",
"credit_no": "",
"province": "HUB",
"credit_no": "91420115MA49RG6T9L",
"company_code": "",
"company_name": "",
"djxh": "10013101001010040630"
"company_name": "武汉中肯商贸有限公司"
},
"metadata": {},
"spider_name": "general_taxpayer",
"spider_ip": "10.8.6.56"
}```
"spider_name": "ic_icpsp_spider",
"spider_ip": "10.8.0.26"
}
```
### 任务参数说明
<!--特有参数说明，通用参数比如spider_name,task_params,task_src,task_result等不需说明-->
```json
破码版本
{
"company_name": "上海翔星建设工程有限公司",
"company_code": "",
"company_name": "武汉鸿雁房地产代理有限公司",
"company_code": "4201002111621",
"credit_no": "",
"province": "SH",
"company_name_digest": "c139fb8844006aa87dbe5f4d90473c57",
"province": "HUB",
"data_detail_url": "http://xyjg.egs.gov.cn/ECPS_HB/company/detail.jspx?id=E8D47B3680F92E8CE588BF508DCB88BE",
"company_major_type": "3",
"company_status": "已吊销",
"legal_person": "刘桂强",
"company_address": "武昌区紫阳路１４７号",
"create_time": "2016-11-09 07:00:45.194000+00:00",
"lastupdatetime": "2021-03-28 14:57:26+00:00",
"submit_time": "",
"authority": "武汉市工商行政管理局",
"n_company_status": "吊销",
"company_name_digest": "3c7e408bcaedd3a960f0948ad81cbb74",
"district_code": ""
}
顺序号版本
{
"province": "SH",
"credit_no": "",
"company_code": "",
"company_name": "",
"djxh": "10013101001000048329" --详情url的顺序号
}
```
## data_type说明
......@@ -189,38 +200,41 @@ detail: 详情信息
```buildoutcfg
{
"data": {
"taxinfo_items": [
"icpsp_items": [
{
"tax_code": "91310101703102855L", -- 纳税人识别号（统一社会信用代码）
"company_name": "上海东安国际旅行社有限公司", -- 纳税人名称
"company_type": "私营有限责任公司", -- 登记注册类型
"legal_person": "张锡华", -- 法定代表人
"register_address": "上海市黄浦区淮海东路99号714-715室", -- 注册地址
"bussiness_address": "上海市黄浦区淮海东路99号714-715室", -- 生产经营地址
"tax_authority": "国家税务总局上海市黄浦区税务局第十三税务所", -- 主管税务机关
"tax_status": "正常", -- 纳税人税务登记状态
"tax_type": "增值税一般纳税人", -- 一般纳税人资格状态
"register_date": "2016-10-26", -- 一般纳税人资格登记时间
"detail_url": "http://shanghai.chinatax.gov.cn/newxbwz/wzfw/YhscxCtrl-yhscxXx.pfv?djxh=10013101001010040675" -- 详情url链接
"search_keyword": "91420115MA49RG6T9L", --搜索关键词
"province": "HUB", --省份 此处固定为HUB
"company_name": "武汉中肯商贸有限公司", -- 公司名称
"credit_no": "91420115MA49RG6T9L", -- 统一社会信用代码
"legal_person": "张耘", -- 法人名称
"establish_date": "2021-05-17", --成立日期
"detail_url": "/ICPSP/pdf.action?id=160000020925690093&lzcx=lzcx", -- 详情url
"data_id": "160000020925690093", -- 详情url中的id参数
"company_address": "湖北省武汉市江夏区郑店街黄金南路8号汽车制造专用模具、夹具、检具生产线4号厂房", -- 登记地址
"company_type_code": "1152", -- 企业类型编码
"company_type": "有限责任公司(自然人投资或控股的法人独资)", --企业类型
"capital": 50, -- 注册资本
"company_code": "", -- 注册号
"company_status": "", -- 登记状态
"legal_person_certno": "F3A10E8FD2D7882064EC67721F9DEBE9" --法人ID加密后的内容
}
]
},
"http_code": 200,
"error_msg": "",
"task_result": 1000,
"task_result": 1000, -- 任务结果状态 1000成功 1101查询无结果 其它：任务爬取失败
"data_type": "detail",
"spider_start_time": "2021-03-20 04:04:40.008",
"spider_end_time": "2021-03-20 04:04:40",
"spider_start_time": "2021-05-24 18:04:27.775",
"spider_end_time": "2021-05-24 18:04:31",
"task_params": {
"province": "SH",
"credit_no": "",
"province": "HUB",
"credit_no": "91420115MA49RG6T9L",
"company_code": "",
"company_name": "",
"djxh": "10013101001010040675"
"company_name": "武汉中肯商贸有限公司"
},
"metadata": {},
"spider_name": "general_taxpayer",
"spider_ip": "10.8.6.56"
"spider_name": "ic_icpsp_spider",
"spider_ip": "10.8.0.26"
}
```
......@@ -230,19 +244,21 @@ detail: 详情信息
```buildoutcfg
{
"data": {
"taxinfo_items": [
"icpsp_items": [
{
"tax_code": "91310101703102855L", -- 纳税人识别号（统一社会信用代码）
"company_name": "上海东安国际旅行社有限公司", -- 纳税人名称
"company_type": "私营有限责任公司", -- 登记注册类型
"legal_person": "张锡华", -- 法定代表人
"register_address": "上海市黄浦区淮海东路99号714-715室", -- 注册地址
"bussiness_address": "上海市黄浦区淮海东路99号714-715室", -- 生产经营地址
"tax_authority": "国家税务总局上海市黄浦区税务局第十三税务所", -- 主管税务机关
"tax_status": "正常", -- 纳税人税务登记状态
"tax_type": "增值税一般纳税人", -- 一般纳税人资格状态
"register_date": "2016-10-26", -- 一般纳税人资格登记时间
"detail_url": "http://shanghai.chinatax.gov.cn/newxbwz/wzfw/YhscxCtrl-yhscxXx.pfv?djxh=10013101001010040675" -- 详情url链接
"search_keyword": "91422822MA490LYE01",
"province": "HUB",
"company_address": "建始县业州镇团结路新马路",
"company_name": "湖北文图档案服务有限公司",
"company_type_code": "1151",
"company_type": "有限责任公司（自然人独资）",
"establish_date": "2017-07-17",
"capital": 80,
"company_code": "",
"credit_no": "91422822MA490LYE01",
"legal_person": "侯贤才",
"company_status": "",
"legal_person_certno": "5B031B5292CCAF1EE26CAD8B09820982"
}
]
},
......@@ -250,18 +266,29 @@ detail: 详情信息
"error_msg": "",
"task_result": 1000,
"data_type": "detail",
"spider_start_time": "2021-03-20 04:04:40.008",
"spider_end_time": "2021-03-20 04:04:40",
"spider_start_time": "2021-05-25 09:24:57.669",
"spider_end_time": "2021-05-25 09:25:00",
"task_params": {
"province": "SH",
"credit_no": "",
"company_code": "",
"company_name": "",
"djxh": "10013101001010040675"
"company_name": "湖北文图档案服务有限公司",
"company_code": "422822000035850",
"credit_no": "91422822MA490LYE01",
"province": "HUB",
"data_detail_url": "http://xyjg.egs.gov.cn/ECPS_HB/company/detail.jspx?id=4F4A1397BB7B4B77A08DD1F6F5A1CE94",
"company_major_type": "3",
"company_status": "吊销，未注销",
"legal_person": "侯贤才",
"company_address": "建始县业州镇团结路新马路",
"create_time": "2017-11-10 01:18:18+00:00",
"lastupdatetime": "2021-03-23 00:44:22+00:00",
"submit_time": "",
"authority": "建始县市场监督管理局",
"n_company_status": "吊销",
"company_name_digest": "c1cda85ab5600d17b4b9f28784f41406",
"district_code": "422822"
},
"metadata": {},
"spider_name": "general_taxpayer",
"spider_ip": "10.8.6.56"
"spider_name": "ic_icpsp_spider",
"spider_ip": "10.8.6.51"
}
```
......@@ -276,14 +303,14 @@ scrapy
## 爬虫部署信息
<!--部署在哪些机器？每个机器多少进程？项目名称是什么？-->
```buildoutcfg
gravel_spiders: 10.8.6.56 5个进程
ic_icpsp_spiders: 10.8.6.51 35个进程
```
## Taskhub地址
```buildoutcfg
提交任务地址: http://10.8.6.222:18518/task/
代码编写地址: http://192.168.109.199/granite/project-gravel/blob/develop_app_10jqka_20210121/app_general_taxpayer/data_pump/general_taxpayer.yml
代码编写地址: http://192.168.109.110/granite/project-gravel/blob/develop_app_10jqka_20210121/app_general_taxpayer/data_pump/general_taxpayer.yml
```
## Taskhub调度规则说明
......