This is an old version of this page. You can view the most recent version or browse the history.

risk_zhixing

基本信息

执行公开爬虫(包含：'zhixing','shixin','xianxiao','zhongben')
zhixing: 通过增加的id和数据库已有id进行找新和更新,如果没有id，通过pname查找
shixin: 通过增加的id进行找新，更新
zhongben: 通过增加的id进行找新，更新

数据名称（中文）

执行公开信息

数据英文名称

risk_zxgk

采集网站（采集入口）

网站数据接口:
http://zxgk.court.gov.cn

采集频率及采集策略

存量更新策略

* 被执行人
  * 根据表里面状态仍是被执行状态的数据，逐条循环滚动更新

* 终本案件
  * 根据表里面状态，逐条循环滚动更新

* 限制高消费
  * 根据表里面状态仍是被限制消费的数据，逐条循环滚动更新

* 失信被执行人
  * 根据表里面状态仍是被执行状态的数据，逐条循环滚动更新


存量连续更新，预计两周更新一轮，逐条更新目前仍是失信状态的。
db_host: bdp-rds-001.mysql.rds.aliyuncs.com
db_name: utn_ng_risk
db_user: shuidi
db_password:

增量采集策略

* 被执行人
  * 每天发布四次任务，每次发布的zhixing_id为距今四天内且爬取失败次数小于30次且爬取结果为1101或null的zhixing_id
  * 每次发布任务前，判断爬取结果为1000的最大zhixing_id与表内最大zhixing_id,若两个id只差小于七万，就补充差额的zhixing_id到loss表内

* 终本案件
  * 根据表中zhongben_id的最大值每天生成十万任务

* 限制高消费
  * 循环失信被执行人表，按名字查询

* 失信被执行人
  * 根据表中最大ep_id进行生成10万任务进行更新

爬虫

责任人

杨龙斌

爬虫名称

risk_zxgk

代码地址

项目地址:http://tech.pingansec.com/granite/project-gravel/-/tree/develop_general_taxpayer/scrapy_spiders/gravel_spiders/spiders

队列名称及队列地址

redis host: redis://:utn@0818@bdp-mq-001.redis.rds.aliyuncs.com:6379/7
redis port: 6379
redis db: 7
redis key:
- risk_zxgk

优先级队列说明

risk_zxgk 支持队列优先级
- 找新任务
  - zhixing：risk_zxgk:101
  - shixin：risk_zxgk:102
  - zhongben：risk_zxgk:103
  - xianxiao：risk_zxgk:104
- 存量循环更新任务
  - zhixing：risk_zxgk:51
  - shixin：risk_zxgk:52
  - zhongben：risk_zxgk:53
  - xianxiao：risk_zxgk:54
- 补漏及其他任务
  - risk_zxgk:60

任务来源

任务输入参数（样例）

通用任务
zhixing
{
  "zhixing_id": "1371041114",
  "task_type": "spider_zhixing_task"
}

shixin
{
  "shixin_id": "1371041114",
  "task_type": "spider_shixin_task"
}

xianxiao
{
    "pname": "刘春志",
    "task_type": "spider_xianxiao_task"
}

zhongben
{
  "zhongben_id": "1371041114",
  "task_type": "spider_zhongben_task"
}


zhixing 通过pname查询
{
  "zhixing_id": "",
  "task_type": "spider_zhixing_task",
  "pname": "阮芙",
  "pname_id_task": true
}

任务样例

任务参数说明

{
  "zhixing_id": "",
  "task_type": "spider_zhixing_task",
  "pname": "阮芙",
  "pname_id_task": true
}
pname_id_task: 这个任务的类型，是否是通过pname查询

data_type说明

detail: 详情信息 列表,其中每个字典为一条数据,只有这一种类型

爬虫结果的超级数据

{
  "data": [
    {
      "zhixing_id": 1370997737,
      "case_code": "(2021)苏0402执1978号",
      "case_state": "0",
      "pname": "徐琰炜",
      "court_name": "常州市天宁区人民法院",
      "exec_money": 100000,
      "gist_id": "(2021)苏0402刑初97号",
      "cardnum_view": "32048219981****2812",
      "sex_name": "男性",
      "spider_time": "2021-07-06 19:40:09",
      "deleted": 0,
      "id": 713347040,
      "iname": "翟振海",
      "caseCode": "（2021）桂0981执400号",
      "age": 57,
      "sexy": "男性",
      "cardNum": "3421291962****761X",
      "courtName": "北流市人民法院",
      "areaName": "广西",
      "partyTypeName": "0",
      "gistId": "（2020）桂0981刑初147号",
      "regDate": "2021年01月14日",
      "gistUnit": "北流市人民法院",
      "duty": "（2020）桂0981刑初147号",
      "performance": "全部未履行",
      "performedPart": "暂无",
      "unperformPart": "暂无",
      "disruptTypeName": "违反财产报告制度",
      "publishDate": "2021年07月06日",
      "case_code": "2006年高执字第00497号",
      "cysah": "（2005）民三终字第5号",
      "zhongben_id": 2,
      "name": "北京嘉裕东方葡萄酒有限公司",
      "zhongben_date": "2007年12月20日",
      "lian_date": "2006年10月16日",
      "cardnum_view": "72261909-1",
      "exec_money": 7474752,
      "court_name": "北京市高级人民法院",
      "case_code": "(2021)鲁0202执633号",
      "lian_date": "2021-03-02 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 923679130,
      "file_url": "/1863/2021-03-18/3f42e43cebe44e8693c0fe3585dd1101.pdf",
      "sex": "男性",
      "company_info": "青岛志宇投资管理有限公司",
      "cardnum": "91370202MA3M2LWR50"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-06 19:40:08.217",
  "spider_end_time": "2021-07-06 19:40:09",
  "task_params": {
    "zhixing_id": "1370997737",
    "task_type": "spider_zhixing_task"
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "10.8.6.51"
}

实际爬虫结果的数据结构

zhixing
- 通过zhingxing_id爬取

{
  "data": [
    {
      "zhixing_id": 1370997737,
      "case_code": "(2021)苏0402执1978号",
      "case_state": "0",
      "pname": "徐琰炜",
      "court_name": "常州市天宁区人民法院",
      "exec_money": 100000,
      "gist_id": "(2021)苏0402刑初97号",
      "cardnum_view": "32048219981****2812",
      "sex_name": "男性",
      "spider_time": "2021-07-06 19:40:09",
      "deleted": 0
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-06 19:40:08.217",
  "spider_end_time": "2021-07-06 19:40:09",
  "task_params": {
    "zhixing_id": "1370997737",
    "task_type": "spider_zhixing_task",
    "pname": "徐琰炜",
    "pname_id_task": false
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "10.8.6.51"
}

通过pname爬取

{
  "data": [
    {
      "zhixing_id": 1369630353,
      "case_code": "(2021)黔0329执1098号",
      "pname": "丁一三",
      "spider_time": "2021-07-09 15:05:06",
      "deleted": 0
    },
    {
      "zhixing_id": 1368029763,
      "case_code": "(2021)黔0329执840号",
      "pname": "丁一三",
      "spider_time": "2021-07-09 15:05:06",
      "deleted": 0
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-09 15:05:05.833",
  "spider_end_time": "2021-07-09 15:05:06",
  "task_params": {
    "zhixing_id": "",
    "task_type": "spider_zhixing_task",
    "pname": "丁一三",
    "pname_id_task": true
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "192.168.56.1"
}

shixin

{
  "data": [
    {
      "id": 713347040,
      "iname": "翟振海",
      "caseCode": "（2021）桂0981执400号",
      "age": 57,
      "sexy": "男性",
      "cardNum": "3421291962****761X",
      "courtName": "北流市人民法院",
      "areaName": "广西",
      "partyTypeName": "0",
      "gistId": "（2020）桂0981刑初147号",
      "regDate": "2021年01月14日",
      "gistUnit": "北流市人民法院",
      "duty": "（2020）桂0981刑初147号",
      "performance": "全部未履行",
      "performedPart": "暂无",
      "unperformPart": "暂无",
      "disruptTypeName": "违反财产报告制度",
      "publishDate": "2021年07月06日"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-12 14:05:50.961",
  "spider_end_time": "2021-07-12 14:05:53",
  "task_params": {
    "shixin_id": 713347040,
    "task_type": "spider_shixin_task"
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "192.168.56.1"
}

xianxiao

{
  "data": [
    {
      "case_code": "（2020）冀0828执1600号",
      "lian_date": "2020-07-07 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 919132679,
      "file_url": "/249/2020-07-10/4685a18c32f14326b822852f4340f29e.pdf",
      "sex": "男性"
    },
    {
      "case_code": "（2020）冀1122执312号",
      "lian_date": "2020-07-07 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 920726038,
      "file_url": "/284/2020-09-29/9742a8505c9a405ebba077b54f352158.pdf",
      "sex": "男性"
    },
    {
      "case_code": "(2019)内0525执4154号",
      "lian_date": "2019-11-05 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 915306323,
      "file_url": "/515/2019-11-08/3c941051b8644d57bcaee9ff5ffab76c.pdf",
      "sex": "男性"
    },
    {
      "case_code": "（2019）桂0502执恢322号",
      "lian_date": "2019-06-25 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 914688361,
      "file_url": "/2795/2019-10-10/7ee4fdacea30435c84fa78d1ca33ef83.pdf",
      "sex": "男性"
    },
    {
      "case_code": "（2019）吉0104执997号",
      "lian_date": "2019-04-18 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 912231532,
      "file_url": "/754/2019-06-06/cf31bec774bf4b70808176314c2f9cd4.pdf",
      "sex": "男性"
    },
    {
      "case_code": "（2019）黑0822执592号",
      "lian_date": "2019-04-26 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 912333674,
      "file_url": "/958/2019-06-13/cdd4a5301a7943bba4f82ebed9b6a8ac.pdf",
      "sex": "男性"
    },
    {
      "case_code": "(2019)苏0803执2718号",
      "lian_date": "2019-09-12 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 915741535,
      "file_url": "/1230/2019-11-28/ae23f8178c6b49bf9c0d3ea04d7aa948.pdf",
      "sex": "男性"
    },
    {
      "case_code": "(2019)鲁0112执3546号",
      "lian_date": "2019-08-07 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 913501558,
      "file_url": "/1856/2019-08-09/ceb6572bdfd24bffa5517219f838792f.pdf",
      "sex": "男性",
      "company_info": "济南凯智不动产经纪有限公司",
      "cardnum": "91370112MA3MUQEJ1M"
    },
    {
      "case_code": "(2021)鲁0202执633号",
      "lian_date": "2021-03-02 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 923679130,
      "file_url": "/1863/2021-03-18/3f42e43cebe44e8693c0fe3585dd1101.pdf",
      "sex": "男性",
      "company_info": "青岛志宇投资管理有限公司",
      "cardnum": "91370202MA3M2LWR50"
    },
    {
      "case_code": "(2021)鲁0202执632号",
      "lian_date": "2021-03-02 00:00:00",
      "name": "刘春志",
      "xianxiao_id": 923679133,
      "file_url": "/1863/2021-03-18/a85f93e089ac46e3a7a04e4ab785db3c.pdf",
      "sex": "男性",
      "company_info": "青岛志宇投资管理有限公司",
      "cardnum": "91370202MA3M2LWR50"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-14 14:38:08.851",
  "spider_end_time": "2021-07-14 14:38:09",
  "task_params": {
    "pname": "刘春志",
    "task_type": "spider_xianxiao_task"
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "192.168.56.1"
}

zhongben

{
  "data": [
    {
      "case_code": "2006年高执字第00497号",
      "cysah": "（2005）民三终字第5号",
      "zhongben_id": 2,
      "name": "北京嘉裕东方葡萄酒有限公司",
      "zhongben_date": "2007年12月20日",
      "lian_date": "2006年10月16日",
      "cardnum_view": "72261909-1",
      "exec_money": 7474752,
      "court_name": "北京市高级人民法院"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-12 17:08:04.833",
  "spider_end_time": "2021-07-12 17:08:06",
  "task_params": {
    "zhongben_id": "2",
    "task_type": "spider_zhongben_task"
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "192.168.56.1"
}

爬虫运行环境

scrapy

爬虫部署信息

机器：node_51
项目名称：gravel_spider

Taskhub地址

Taskhub调度规则说明

task_result=1000    # 正常获取到详情任务
task_result=1101    # 无结果信息,表示官网查不到这个数据,同样需要解析入库


task_result=9101    # 超时错误，需要进行重试，目前重试5次
task_result=8000    # 参数错误

爬虫监控指标设计

(先观察，待补充)
索引: 
监控频率: 
监控起止时间: 
报警条件: 
报警群:  
报警内容:

数据归集

责任人

范召贤

数据归集方式

爬虫直接写kafka
爬虫写文件logstash采集

爬虫结果目录

/data/gravel_spiders/risk_zxgk

归集后存放目录

/data2_227/grvael_spider_result/risk_zxgk

logstash配置文件名称

project-deploy/logstash/10.8.6.246/conf.d/collie_spider_data_to_kfk.conf(入topic)
project-deploy/logstash/10.8.6.229/conf.d/grvael_spider_to_es.conf(入es)

logstash文件采集type

type=>"risk_zxgk"

数据归集的topic

topic_id => "general-taxpayer"

ES日志索引及筛选条件

index => "public-company-spider-data-%{log_date}"
{
"query": {
"match": {
"spider_name.keyword": {
"query": "risk_zxgk",
"type": "phrase"
}
}
}
}

监控指标看板

数据保留策略

数据清洗

责任人

刘治强

代码地址

http://192.168.109.110/granite/project-collie-app/-/tree/master/app_risk/udms/risk_zhixing_to_redis
http://192.168.109.110/granite/project-collie-app/-/tree/master/app_risk/udms/risk_zxgk_loss

部署地址

部署方法及说明

crontab + data_pump
supervisor + data_pump
supervisor + consumer

数据接收来源

1.爬虫数据进行数据更新
2.代码生成新的执行id进行爬虫任务发布

数据存储表地址

数据库地址：bdp-rds-001.mysql.rds.aliyuncs.com
表名：risk_zhixing_loss