Skip to content

GitLab

A_taxpayer · Changes

Page history
add A_taxpayer authored by 杨龙斌's avatar 杨龙斌
Hide whitespace changes
Inline Side-by-side
Showing with 681 additions and 0 deletions
data_stream/general_taxpayer/A_taxpayer.md 0 → 100644
View page @ 003c5bb9
# **基本信息**
```buildoutcfg
A级纳税人
A_taxpayer,通过scrapy部署
项目名称：project-gravel
分支：develop_general_taxpayer
```
## 数据名称（中文）
<!-- 该项数据的中文标准名称，后续沟通交流使用的规范名称，如：工商公示股东信息、失信被执行人、一般纳税人等-->
###
```buildoutcfg
A级纳税人
```
## 数据英文名称
<!--英文名称，后续流程中所有涉及到英文名称均以此为准，如：partner、shixin、general_taxpayer等-->
```buildoutcfg
A_taxpayer
```
## 采集网站（采集入口）
<!--采集的入口地址，不能只是一个网站域名，具体到该网站的数据入口-->
```buildoutcfg
post http://hd.chinatax.gov.cn/service/findCredit.do
```
## 采集频率及采集策略
<!--说明该项数据更新频率，存量数据的更新频率及策略、新增数据的采集频率及策略-->
### 存量更新策略
<!--无需更新？每天全量更新？逐条轮询更新？多久更新完一轮？或其他-->
```buildoutcfg
每个月跑一轮，已经包括找新和全量更新
```
### 增量采集策略
<!--新增数据怎么来？无需单独采集新增数据？-->
```buildoutcfg
遍历各省code和page_num,page_num来源于响应
```
---
# **爬虫**
```buildoutcfg
A级纳税人 A_taxpayer
```
## 责任人
```buildoutcfg
杨龙斌
```
## 爬虫名称
```buildoutcfg
A_taxpayer
```
<!--spider_name-->
## 代码地址
```buildoutcfg
项目地址:http://tech.pingansec.com/granite/project-gravel/-/blob/develop_general_taxpayer/scrapy_spiders/gravel_spiders/spiders/A_taxpayer_spider.py
```
## 队列名称及队列地址
<!--redis host port db key 优先级说明-->
-
* redis host: redis://:utn@0818@bdp-mq-001.redis.rds.aliyuncs.com:6379/7
* redis port: 6379
* redis db: 7
* redis key:
* A_taxpayer
### 优先级队列说明
* A_taxpayer 不支持队列优先级
## 任务来源
<!--说明爬虫任务的输入。如：来自某个数据库表等。如果来自某个数据库表则应该简要说明该表内的数据是如何维护的。-->
```buildoutcfg
province_page_num = [
['320000', 21116], ['330000', 17073], ['440000', 16448], ['310000', 15221], ['370000', 12314],
['440300', 9878], ['110000', 9279], ['130000', 8472], ['510000', 7033], ['410000', 6574],
['420000', 5455], ['350000', 4723], ['330200', 4464], ['430000', 4147], ['610000', 4006],
['120000', 3852], ['340000', 3628], ['210000', 3209], ['370200', 3174], ['450000', 2675],
['140000', 2319], ['360000', 2205], ['530000', 2184], ['650000', 2154], ['350200', 1615],
['210200', 1605], ['620000', 1563], ['150000', 1518], ['520000', 1371], ['230000', 1088],
['220000', 993], ['640000', 860], ['500000', 763], ['460000', 643], ['630000', 379],
['540000', 94]
]
```
## 任务输入参数（样例）
```json
{
"province_code": "540000",
"page_num": 1
}
```
### 任务样例
<!--注意是爬虫拿到的完整任务，而不仅仅是task_params-->
```buildoutcfg
{"province_code":"540000", "page_num":1}
```
### 任务参数说明
<!--特有参数说明，通用参数比如spider_name,task_params,task_src,task_result等不需说明-->
```json
{
"province_code": "省份代码",
"page_num": "页数"
}
```
## data_type说明
<!--可能产生的data_type说明-->
```buildoutcfg
detail: 详情信息
```
## 爬虫结果的超级数据
<!--包含所有字段的json数据，每个value都要有样例值-->
```json
{
"data": {
"content": [
{
"code": "913306027044810927",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15985975,
"level": "A类",
"location": "330000",
"name": "绍兴市万通纺织有限公司"
},
{
"code": "91330602704481543H",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16010777,
"level": "A类",
"location": "330000",
"name": "绍兴市红日装饰广告有限公司"
},
{
"code": "91330602704481703Q",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16010779,
"level": "A类",
"location": "330000",
"name": "中国人民财产保险股份有限公司绍兴市分公司"
},
{
"code": "91330602704481738B",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018053,
"level": "A类",
"location": "330000",
"name": "绍兴正泰电器销售有限公司"
},
{
"code": "913306027044824231",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15985977,
"level": "A类",
"location": "330000",
"name": "绍兴宝兴化工有限公司"
},
{
"code": "913306027044829681",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15959911,
"level": "A类",
"location": "330000",
"name": "绍兴市华众空压机有限公司"
},
{
"code": "91330602704482976U",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15985979,
"level": "A类",
"location": "330000",
"name": "绍兴市繁特利轴承有限公司"
},
{
"code": "91330602704482992H",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16010781,
"level": "A类",
"location": "330000",
"name": "浙江诚锦信息技术有限公司"
},
{
"code": "91330602704483012D",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018055,
"level": "A类",
"location": "330000",
"name": "绍兴市越城雄丰物资销售部"
},
{
"code": "913306027044830988",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018315,
"level": "A类",
"location": "330000",
"name": "绍兴市越城海荣机电物资供应站"
},
{
"code": "9133060270448332X3",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018915,
"level": "A类",
"location": "330000",
"name": "绍兴菲尼克进出口有限公司"
},
{
"code": "91330602704483426H",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16010783,
"level": "A类",
"location": "330000",
"name": "绍兴中国轻纺城贸易发展有限公司"
},
{
"code": "91330602704483581X",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018917,
"level": "A类",
"location": "330000",
"name": "绍兴市越城铸造有限公司"
},
{
"code": "9133060270448375XW",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018321,
"level": "A类",
"location": "330000",
"name": "绍兴市城区三星汽车配件有限公司"
},
{
"code": "913306027044838646",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15959913,
"level": "A类",
"location": "330000",
"name": "绍兴市上工机电设备有限公司"
}
],
"empty": false,
"facets": [],
"first": false,
"last": false,
"maxScore": null,
"number": 10993,
"numberOfElements": 15,
"pageable": {
"offset": 164895,
"pageNumber": 10993,
"pageSize": 15,
"paged": true,
"sort": {
"empty": true,
"sorted": false,
"unsorted": true
},
"unpaged": false
},
"size": 15,
"sort": {
"$ref": "$.pageable.sort"
},
"totalElements": 256082,
"totalPages": 17073
},
"http_code": 200,
"error_msg": "",
"task_result": 1000,
"data_type": "detail",
"spider_start_time": "2021-10-20 11:53:09.266",
"spider_end_time": "2021-10-20 11:53:10",
"task_params": {
"province_code": "330000",
"page_num": 10993,
"warmhole_sessionid": "507327c8-a2e0-4976-a481-13bfc1f3165b"
},
"metadata": {},
"spider_name": "A_taxpayer",
"spider_ip": "10.8.6.51"
}
```
## 实际爬虫结果的数据结构
<!--可能与超级数据一致，可能不同的data_type的爬虫结果结构不同，超级数据是把所有data_type的结果组合在一起-->
```json
{
"data": {
"content": [
{
"code": "913306027044810927",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15985975,
"level": "A类",
"location": "330000",
"name": "绍兴市万通纺织有限公司"
},
{
"code": "91330602704481543H",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16010777,
"level": "A类",
"location": "330000",
"name": "绍兴市红日装饰广告有限公司"
},
{
"code": "91330602704481703Q",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16010779,
"level": "A类",
"location": "330000",
"name": "中国人民财产保险股份有限公司绍兴市分公司"
},
{
"code": "91330602704481738B",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018053,
"level": "A类",
"location": "330000",
"name": "绍兴正泰电器销售有限公司"
},
{
"code": "913306027044824231",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15985977,
"level": "A类",
"location": "330000",
"name": "绍兴宝兴化工有限公司"
},
{
"code": "913306027044829681",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15959911,
"level": "A类",
"location": "330000",
"name": "绍兴市华众空压机有限公司"
},
{
"code": "91330602704482976U",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15985979,
"level": "A类",
"location": "330000",
"name": "绍兴市繁特利轴承有限公司"
},
{
"code": "91330602704482992H",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16010781,
"level": "A类",
"location": "330000",
"name": "浙江诚锦信息技术有限公司"
},
{
"code": "91330602704483012D",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018055,
"level": "A类",
"location": "330000",
"name": "绍兴市越城雄丰物资销售部"
},
{
"code": "913306027044830988",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018315,
"level": "A类",
"location": "330000",
"name": "绍兴市越城海荣机电物资供应站"
},
{
"code": "9133060270448332X3",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018915,
"level": "A类",
"location": "330000",
"name": "绍兴菲尼克进出口有限公司"
},
{
"code": "91330602704483426H",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16010783,
"level": "A类",
"location": "330000",
"name": "绍兴中国轻纺城贸易发展有限公司"
},
{
"code": "91330602704483581X",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018917,
"level": "A类",
"location": "330000",
"name": "绍兴市越城铸造有限公司"
},
{
"code": "9133060270448375XW",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 16018321,
"level": "A类",
"location": "330000",
"name": "绍兴市城区三星汽车配件有限公司"
},
{
"code": "913306027044838646",
"createTime": 1619758800000,
"delFlag": "0",
"evalyear": "2020",
"id": 15959913,
"level": "A类",
"location": "330000",
"name": "绍兴市上工机电设备有限公司"
}
],
"empty": false,
"facets": [],
"first": false,
"last": false,
"maxScore": null,
"number": 10993,
"numberOfElements": 15,
"pageable": {
"offset": 164895,
"pageNumber": 10993,
"pageSize": 15,
"paged": true,
"sort": {
"empty": true,
"sorted": false,
"unsorted": true
},
"unpaged": false
},
"size": 15,
"sort": {
"$ref": "$.pageable.sort"
},
"totalElements": 256082,
"totalPages": 17073
},
"http_code": 200,
"error_msg": "",
"task_result": 1000,
"data_type": "detail",
"spider_start_time": "2021-10-20 11:53:09.266",
"spider_end_time": "2021-10-20 11:53:10",
"task_params": {
"province_code": "330000",
"page_num": 10993,
"warmhole_sessionid": "507327c8-a2e0-4976-a481-13bfc1f3165b"
},
"metadata": {},
"spider_name": "A_taxpayer",
"spider_ip": "10.8.6.51"
}
```
## 爬虫运行环境
<!--udm模块？scrapy？或其他-->
```buildoutcfg
scrapy
```
## 爬虫部署信息
<!--部署在哪些机器？每个机器多少进程？项目名称是什么？-->
```buildoutcfg
general_taxpayer: 10.8.6.51 1个进程
```
## Taskhub地址
```buildoutcfg
```
## Taskhub调度规则说明
```buildoutcfg
task_result=1000 # 正常获取到详情任务
task_result=1101 # 无结果信息
task_result=9101 # 超时错误，需要进行重试，目前重试5次
task_result=8000 # 参数错误
```
## 爬虫监控指标设计
<!--监控爬虫正常运行的指标是什么？报警规则是什么？-->
```buildoutcfg
(先观察，待补充)
索引:
监控频率:
监控起止时间:
报警条件:
报警群:
报警内容:
```
---
# **数据归集**
## 责任人
```html
范召贤
```
## 数据归集方式
- [ ] 爬虫直接写kafka
- [ ] 爬虫写文件logstash采集
## 爬虫结果目录
```html
/data/gravel_spiders/A_taxpayer
```
## 归集后存放目录
```html
```
## logstash配置文件名称
```html
```
## logstash文件采集type
```html
type=>"general-taxpayer"
```
## 数据归集的topic
```
topic_id => ""
```
## ES日志索引及筛选条件
```html
```
## 监控指标看板
## 数据保留策略
---
# **数据清洗**
## 责任人
## 代码地址
## 部署地址
<!--机器及线上代码地址-->
## 部署方法及说明
<!--运行方法及运行命令、supervisor配置、supervisor的program等-->
- [ ] crontab + data_pump
- [ ] supervisor + data_pump
- [ ] supervisor + consumer
## 数据接收来源
<!--来源于kafka还是归集的文件、topic的group？-->
## 数据存储表地址
* 数据库地址：
* 表名：
\ No newline at end of file