Skip to content

GitLab

Last edited by liuzhiqiang
Page history
This is an old version of this page. You can view the most recent version or browse the history.

risk_zhixing

基本信息 

执行公开爬虫(包含：'zhixing','shixin','xianxiao','zhongben')
zhixing: 通过增加的id和数据库已有id进行找新和更新,如果没有id，通过pname查找
shixin: 通过增加的id进行找新，更新
zhongben: 通过增加的id进行找新，更新

数据名称（中文）

执行公开信息

数据英文名称

risk_zxgk

采集网站（采集入口）

网站数据接口:
http://zxgk.court.gov.cn

采集频率及采集策略

存量更新策略

* 被执行人
  * 根据表里面状态仍是被执行状态的数据，逐条循环滚动更新

* 终本案件
  * 根据表里面状态，逐条循环滚动更新

* 限制高消费
  * 根据表里面状态仍是被限制消费的数据，逐条循环滚动更新

* 失信被执行人
  * 根据表里面状态仍是被执行状态的数据，逐条循环滚动更新


存量连续更新，预计两周更新一轮，逐条更新目前仍是失信状态的。
db_host: bdp-rds-001.mysql.rds.aliyuncs.com
db_name: utn_ng_risk
db_user: shuidi
db_password:

增量采集策略

* 被执行人
  * 根据表里面最大zhixing_id每天生成十万任务进行更新

* 终本案件
  * 根据表中zhongben_id的最大值每天生成十万任务

* 限制高消费
  * 待补充

* 失信被执行人
  * 根据表中最大ep_id进行生成10万任务进行更新

爬虫

责任人

杨龙斌

爬虫名称

risk_zxgk

代码地址

项目地址:http://tech.pingansec.com/granite/project-gravel/-/tree/develop_general_taxpayer/scrapy_spiders/gravel_spiders/spiders

队列名称及队列地址

优先级队列说明

  • risk_zxgk 支持队列优先级

    • 找新任务

      • zhixing：risk_zxgk:101
      • shixin：risk_zxgk:102
      • xianxiao：risk_zxgk:103
      • zhongben：risk_zxgk:104

    • 存量循环更新任务

      • zhixing：risk_zxgk:51
      • shixin：risk_zxgk:52
      • xianxiao：risk_zxgk:53
      • zhongben：risk_zxgk:54

    • 补漏及其他任务

      • risk_zxgk:60

任务来源

任务输入参数（样例）

通用任务
zhixing
{
  "zhixing_id": "1371041114",
  "task_type": "spider_zhixing_task"
}

shixin
{
  "shixin_id": "1371041114",
  "task_type": "spider_shixin_task"
}

xianxiao
{
  "xianxiao_id": "1371041114",
  "task_type": "spider_xianxiao_task"
}

zhongben
{
  "zhongben_id": "1371041114",
  "task_type": "spider_zhongben_task"
}


zhixing 通过pname查询
{
  "zhixing_id": "",
  "task_type": "spider_zhixing_task",
  "pname": "阮芙",
  "pname_id_task": true
}

任务样例

任务参数说明

{
  "zhixing_id": "",
  "task_type": "spider_zhixing_task",
  "pname": "阮芙",
  "pname_id_task": true
}
pname_id_task: 这个任务的类型，是否是通过pname查询

data_type说明

detail: 详情信息 列表,其中每个字典为一条数据,只有这一种类型

爬虫结果的超级数据

{
  "data": [
    {
      "zhixing_id": 1370997737,
      "case_code": "(2021)苏0402执1978号",
      "case_state": "0",
      "pname": "徐琰炜",
      "court_name": "常州市天宁区人民法院",
      "exec_money": 100000,
      "gist_id": "(2021)苏0402刑初97号",
      "cardnum_view": "32048219981****2812",
      "sex_name": "男性",
      "spider_time": "2021-07-06 19:40:09",
      "deleted": 0,
      "id": 713347040,
      "iname": "翟振海",
      "caseCode": "（2021）桂0981执400号",
      "age": 57,
      "sexy": "男性",
      "cardNum": "3421291962****761X",
      "courtName": "北流市人民法院",
      "areaName": "广西",
      "partyTypeName": "0",
      "gistId": "（2020）桂0981刑初147号",
      "regDate": "2021年01月14日",
      "gistUnit": "北流市人民法院",
      "duty": "（2020）桂0981刑初147号",
      "performance": "全部未履行",
      "performedPart": "暂无",
      "unperformPart": "暂无",
      "disruptTypeName": "违反财产报告制度",
      "publishDate": "2021年07月06日",
      "case_code": "2006年高执字第00497号",
      "cysah": "（2005）民三终字第5号",
      "zhongben_id": 2,
      "name": "北京嘉裕东方葡萄酒有限公司",
      "zhongben_date": "2007年12月20日",
      "lian_date": "2006年10月16日",
      "cardnum_view": "72261909-1",
      "exec_money": 7474752,
      "court_name": "北京市高级人民法院"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-06 19:40:08.217",
  "spider_end_time": "2021-07-06 19:40:09",
  "task_params": {
    "zhixing_id": "1370997737",
    "task_type": "spider_zhixing_task"
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "10.8.6.51"
}

实际爬虫结果的数据结构

  • zhixing
    • 通过zhingxing_id爬取
{
  "data": [
    {
      "zhixing_id": 1370997737,
      "case_code": "(2021)苏0402执1978号",
      "case_state": "0",
      "pname": "徐琰炜",
      "court_name": "常州市天宁区人民法院",
      "exec_money": 100000,
      "gist_id": "(2021)苏0402刑初97号",
      "cardnum_view": "32048219981****2812",
      "sex_name": "男性",
      "spider_time": "2021-07-06 19:40:09",
      "deleted": 0
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-06 19:40:08.217",
  "spider_end_time": "2021-07-06 19:40:09",
  "task_params": {
    "zhixing_id": "1370997737",
    "task_type": "spider_zhixing_task",
    "pname": "徐琰炜",
    "pname_id_task": false
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "10.8.6.51"
}
  • 通过pname爬取
{
  "data": [
    {
      "zhixing_id": 1369630353,
      "case_code": "(2021)黔0329执1098号",
      "pname": "丁一三",
      "spider_time": "2021-07-09 15:05:06",
      "deleted": 0
    },
    {
      "zhixing_id": 1368029763,
      "case_code": "(2021)黔0329执840号",
      "pname": "丁一三",
      "spider_time": "2021-07-09 15:05:06",
      "deleted": 0
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-09 15:05:05.833",
  "spider_end_time": "2021-07-09 15:05:06",
  "task_params": {
    "zhixing_id": "",
    "task_type": "spider_zhixing_task",
    "pname": "丁一三",
    "pname_id_task": true
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "192.168.56.1"
}
  • shixin
{
  "data": [
    {
      "id": 713347040,
      "iname": "翟振海",
      "caseCode": "（2021）桂0981执400号",
      "age": 57,
      "sexy": "男性",
      "cardNum": "3421291962****761X",
      "courtName": "北流市人民法院",
      "areaName": "广西",
      "partyTypeName": "0",
      "gistId": "（2020）桂0981刑初147号",
      "regDate": "2021年01月14日",
      "gistUnit": "北流市人民法院",
      "duty": "（2020）桂0981刑初147号",
      "performance": "全部未履行",
      "performedPart": "暂无",
      "unperformPart": "暂无",
      "disruptTypeName": "违反财产报告制度",
      "publishDate": "2021年07月06日"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-12 14:05:50.961",
  "spider_end_time": "2021-07-12 14:05:53",
  "task_params": {
    "shixin_id": 713347040,
    "task_type": "spider_shixin_task"
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "192.168.56.1"
}
  • xianxiao
待补充
  • zhongben
{
  "data": [
    {
      "case_code": "2006年高执字第00497号",
      "cysah": "（2005）民三终字第5号",
      "zhongben_id": 2,
      "name": "北京嘉裕东方葡萄酒有限公司",
      "zhongben_date": "2007年12月20日",
      "lian_date": "2006年10月16日",
      "cardnum_view": "72261909-1",
      "exec_money": 7474752,
      "court_name": "北京市高级人民法院"
    }
  ],
  "http_code": 200,
  "error_msg": "",
  "task_result": 1000,
  "data_type": "detail",
  "spider_start_time": "2021-07-12 17:08:04.833",
  "spider_end_time": "2021-07-12 17:08:06",
  "task_params": {
    "zhongben_id": "2",
    "task_type": "spider_zhongben_task"
  },
  "metadata": {},
  "spider_name": "risk_zxgk",
  "spider_ip": "192.168.56.1"
}

爬虫运行环境

scrapy

爬虫部署信息

机器：node_51
项目名称：gravel_spider

Taskhub地址

Taskhub调度规则说明

task_result=1000    # 正常获取到详情任务
task_result=1101    # 无结果信息,表示官网查不到这个数据,同样需要解析入库


task_result=9101    # 超时错误，需要进行重试，目前重试5次
task_result=8000    # 参数错误

爬虫监控指标设计

(先观察，待补充)
索引: 
监控频率: 
监控起止时间: 
报警条件: 
报警群:  
报警内容:

数据归集

责任人

范召贤

数据归集方式

  • 爬虫直接写kafka

  • 爬虫写文件logstash采集

爬虫结果目录

/data/gravel_spiders/risk_zxgk

归集后存放目录

/data2_227/grvael_spider_result/risk_zxgk

logstash配置文件名称

project-deploy/logstash/10.8.6.246/conf.d/collie_spider_data_to_kfk.conf(入topic)
project-deploy/logstash/10.8.6.229/conf.d/grvael_spider_to_es.conf(入es)

logstash文件采集type

type=>"risk_zxgk"

数据归集的topic

topic_id => "general-taxpayer"

ES日志索引及筛选条件

index => "public-company-spider-data-%{log_date}"
{
"query": {
"match": {
"spider_name.keyword": {
"query": "risk_zxgk",
"type": "phrase"
}
}
}
}

监控指标看板

数据保留策略

数据清洗

责任人

代码地址

部署地址

部署方法及说明

  • crontab + data_pump
  • supervisor + data_pump
  • supervisor + consumer

数据接收来源

数据存储表地址

  • 数据库地址：
  • 表名：