hearing_announcement

基本信息

开庭公告
每天轮询所有已知的网站的某些页面，爬到的数据爬虫部分将能解析的字段均已解析。如果存在剩下的爬虫环节难以提取的字段，需要后续清洗环节处理的字段，
会将原始数据存放在source_data字段中，具体见爬虫结果数据。

基本字段说明

class CourtNoticeItem(PinganBaseSpiderItem):
    court_name = Field()        # 法院
    court_room = Field()        # 法庭
    contractors = Field()       # 承办部门
    start_date = Field()        # 开庭日期
    plan_date = Field()         # 排期日期
    case_code = Field()         # 案号
    action_cause = Field()      # 案由
    case_type = Field()         # 案件类型
    case_area = Field()         # 案件管辖区域
    plaintiff = Field()         # 原告/上诉人
    defendant = Field()         # 被告/上诉人
    litigant = Field()          # 当事人
    judge = Field()             # 审判长/主审人
    title = Field()             # 标题
    article = Field()           # 公告内容
    province = Field()          # 省份
    url = Field()               # 数据链接
    domain = Field()               # 域名
    source_data = Field()               # 如果存在需要清洗环节需要解析的字段， 存放在此字段。

归集到227目录的数据直接入mysql表

CREATE TABLE `hearing_announcement` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `unique_id` varchar(50) NOT NULL COMMENT '唯一索引，根据规则生成的MD5',
  `court_name` varchar(255) DEFAULT NULL COMMENT '法院',
  `court_room` varchar(50) DEFAULT NULL COMMENT '法庭',
  `contractors` varchar(50) DEFAULT NULL COMMENT '承办部门',
  `start_date` datetime DEFAULT NULL COMMENT '开庭日期',
  `plan_date` date DEFAULT NULL COMMENT '排期日期',
  `case_code` varchar(255) DEFAULT NULL COMMENT '案号',
  `action_cause` varchar(50) DEFAULT NULL COMMENT '案由',
  `case_type` varchar(30) DEFAULT NULL COMMENT '案件类型',
  `case_area` varchar(100) DEFAULT NULL COMMENT '案件管辖区域',
  `plaintiff` varchar(1023) DEFAULT NULL COMMENT '原告/上诉人',
  `defendant` varchar(1023) DEFAULT NULL COMMENT '被告/被上诉人',
  `litigant` varchar(1023) DEFAULT NULL COMMENT '当事人',
  `judge` varchar(50) DEFAULT NULL COMMENT '审判长/主审人',
  `title` varchar(2000) DEFAULT NULL COMMENT '标题',
  `article` longtext COMMENT '公告内容',
  `province` varchar(20) DEFAULT NULL COMMENT '省份',
  `source_data` longtext COMMENT '原始数据',
  `domain` varchar(50) DEFAULT NULL COMMENT '域名',
  `use_flag` int(11) DEFAULT NULL COMMENT '使用标志0正常，1人工已修正，2调试中，3未解析,9删除废弃',
  `create_time` datetime DEFAULT NULL COMMENT '入库时间',
  `update_time` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间',
  `is_hide` tinyint(4) DEFAULT '0' COMMENT '是否不展示',
  `desc02` varchar(255) DEFAULT NULL COMMENT '预留字段',
  `desc03` varchar(255) DEFAULT NULL COMMENT '预留字段',
  PRIMARY KEY (`id`),
  UNIQUE KEY `idx_unique_id` (`unique_id`) USING BTREE,
  KEY `idx_use_flag` (`use_flag`) USING BTREE,
  KEY `idx_start_date` (`start_date`) USING BTREE,
  KEY `idx_update_time` (`update_time`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;

数据名称（中文）

开庭公告

数据英文名称

hearing_announcement

采集网站（采集入口）

参考: "开庭公告采集"群  的 开庭公告采集网站-初步分工.xlsx 文件

采集频率及采集策略

对于不同的网站，针对其更新速度不同，指定1，3，6，12，24小时跑一遍根据数据量指定翻页数，依据内容MD5去重

存量更新策略

存量无需更新
增量按照网站的翻页数进行遍历更新 
 根据具体的网站情况, 会在crontab中确定网站更新频率。

增量采集策略

增量按照网站的翻页取最新的几页

爬虫

责任人

袁波

爬虫名称

hearing_announcement

代码地址

http://office.pingansec.com:30080/granite/project-judicature/-/tree/dev_court_notice_v3

队列名称及队列地址

redis host: redis://:utn@0818@bdp-mq-001.redis.rds.aliyuncs.com:6379/7
redis port: 6379
redis db: 7
redis key:
- hearing_announcement

优先级队列说明

任务来源

任务样例

{
	"domain": "xyfy.hicourt.gov.cn",
	"limit": 3
}

任务结果样例

{
	"task_result": 1000,
	"data_type": "detail",
	"data": [{
		"court_name": "海口市秀英区人民法院",
		"court_room": "少年法庭",
		"contractors": null,
		"start_date": "2020-07-01 14:30",
		"plan_date": null,
		"case_code": "(2020)琼0105民初328号",
		"action_cause": "离婚纠纷",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "章君华;胡天兵",
		"judge": "陈国琼",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}, {
		"court_name": "海口市秀英区人民法院",
		"court_room": "少年法庭",
		"contractors": null,
		"start_date": "2020-06-23 14:30",
		"plan_date": null,
		"case_code": "(2020)琼0105民初500号",
		"action_cause": "离婚纠纷",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "王惠宜;叶朝辉",
		"judge": "陈国琼",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}, {
		"court_name": "海口市秀英区人民法院",
		"court_room": "第二法庭",
		"contractors": null,
		"start_date": "2020-06-09 08:30",
		"plan_date": null,
		"case_code": "(2020)琼0105民初295号",
		"action_cause": "买卖合同纠纷",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "崔削山;海口盛源丰农业生产资料有限公司",
		"judge": "黎雪",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}, {
		"court_name": "海口市秀英区人民法院",
		"court_room": "第三法庭",
		"contractors": null,
		"start_date": "2020-06-03 09:00",
		"plan_date": null,
		"case_code": "(2019)琼0105民初6745号",
		"action_cause": "建设工程合同纠纷",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "王刚;朱善略;王康孙;海南南北通投资有限公司",
		"judge": "韩诚",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}, {
		"court_name": "海口市秀英区人民法院",
		"court_room": "第三法庭",
		"contractors": null,
		"start_date": "2020-05-29 09:00",
		"plan_date": null,
		"case_code": "(2020)琼0105民初486号",
		"action_cause": "买卖合同纠纷",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "海口托肯建材有限公司;广东亿润网络技术有限公司;佛山市顺德区高亮装饰材料有限公司",
		"judge": "吴松林",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}, {
		"court_name": "海口市秀英区人民法院",
		"court_room": "第四法庭",
		"contractors": null,
		"start_date": "2020-05-28 15:00",
		"plan_date": null,
		"case_code": "(2020)琼0105民初1507号",
		"action_cause": "共有人优先购买权纠纷",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "海口光通光电子研究所;国网四川省电力公司达州供电公司;赵相红;胡林涛",
		"judge": "王则朝",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}, {
		"court_name": "海口市秀英区人民法院",
		"court_room": "第三法庭",
		"contractors": null,
		"start_date": "2020-05-28 15:00",
		"plan_date": null,
		"case_code": "(2020)琼0105民初1203号",
		"action_cause": "案外人执行异议之诉",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "辛玉琢;曹津诚;海南银泉进出口贸易有限公司",
		"judge": "吴松林",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}, {
		"court_name": "海口市秀英区人民法院",
		"court_room": "第五法庭",
		"contractors": null,
		"start_date": "2020-05-28 09:00",
		"plan_date": null,
		"case_code": "(2020)琼0105民初1376号",
		"action_cause": "房屋买卖合同纠纷",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "蔡巧英;海南国神华实业有限公司;海南天白翼未来科技有限公司",
		"judge": "吴松林",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}, {
		"court_name": "海口市秀英区人民法院",
		"court_room": "第三法庭",
		"contractors": null,
		"start_date": "2020-05-27 15:00",
		"plan_date": null,
		"case_code": "(2020)琼0105民初1380号",
		"action_cause": "买卖合同纠纷",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "陈创甘;黄四川",
		"judge": "吴松林",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}, {
		"court_name": "海口市秀英区人民法院",
		"court_room": "第三法庭",
		"contractors": null,
		"start_date": "2020-05-27 15:00",
		"plan_date": null,
		"case_code": "(2020)琼0105民初1383号",
		"action_cause": "买卖合同纠纷",
		"case_type": null,
		"case_area": "海口市秀英区",
		"plaintiff": null,
		"defendant": null,
		"litigant": "陈创甘;杜爱民",
		"judge": "吴松林",
		"title": null,
		"article": null,
		"province": "海南省",
		"url": "http://xyfy.hicourt.gov.cn/preview/channel?columnId=5a0b2af8-fb49-4ecf-90fe-6694cb3d1cf8&&parentId=dd7fd507-d464-48a4-becc-cb01aff0c901&&siteId=a82def60-9a0b-48e8-aba2-909978e3e199",
		"source_data": null,
		"domain": "xyfy.hicourt.gov.cn"
	}],
	"http_code": 200,
	"error_msg": "",
	"metadata": {},
	"spider_start_time": "2022-04-01 15:07:08",
	"spider_end_time": "2022-04-01 15:07:08",
	"spider_used_time_ms": 570,
	"task_params": {
		"domain": "xyfy.hicourt.gov.cn",
		"limit": 3
	},
	"spider_name": "hearing_announcement",
	"spider_ip": "192.168.0.196"
}

任务参数说明

domain代表网站域名, limit表示翻页数。

data_type说明

data_type只有detail

爬虫结果的超级数据

数据来源很多，首次统一解析后结果如下

爬虫运行环境

scrapy

爬虫部署信息

10.8.6.19

Taskhub地址

Taskhub调度规则说明

爬虫监控指标设计

(先观察，待补充)
索引: 
监控频率: 
监控起止时间: 
报警条件: 
报警群:  
报警内容:

数据归集

责任人

数据归集方式

爬虫直接写kafka
爬虫写文件logstash采集

爬虫结果目录

/data/judicature_spiders/hearing_announcement

归集后存放目录

/data/227/data2/grvael_spider_result/hearing_announcement

logstash配置文件名称

logstash文件采集type

数据归集的topic

topic_id => "general-taxpayer"

ES日志索引及筛选条件

judicature-spider-data-*

监控指标看板

数据保留策略

数据清洗

责任人

代码地址

部署地址

部署方法及说明

crontab + data_pump
supervisor + data_pump
supervisor + consumer

supervisor路径

http://office.pingansec.com:30080/granite/project-judicature/-/blob/dev_court_notice_v3/app_hearing_announcement/supervisor/app_hearing_announcement.conf

数据接收来源

数据存储表地址

数据库地址：
表名：