Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
K
kb
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 2
    • Issues 2
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • Operations
    • Operations
    • Incidents
  • Analytics
    • Analytics
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • granite
  • kb
  • Wiki
    • Data_stream
    • Risk
  • risk_court_notice

risk_court_notice · Changes

Page history
update risk court notice authored Aug 18, 2021 by 杨龙斌's avatar 杨龙斌
Hide whitespace changes
Inline Side-by-side
Showing with 15 additions and 526 deletions
+15 -526
  • data_stream/risk/risk_court_notice.md data_stream/risk/risk_court_notice.md +15 -526
  • No files found.
data_stream/risk/risk_court_notice.md
View page @ 60b5ea84
......@@ -9,7 +9,7 @@
## 数据名称(中文)
<!-- 该项数据的中文标准名称,后续沟通交流使用的规范名称,如:工商公示股东信息、失信被执行人、一般纳税人等-->
法院公告
###
```buildoutcfg
......@@ -19,7 +19,7 @@
## 数据英文名称
<!--英文名称,后续流程中所有涉及到英文名称均以此为准,如:partner、shixin、general_taxpayer等-->
risk court notice
```buildoutcfg
risk_court_notice
```
......@@ -37,11 +37,12 @@ http://www.guizhoucourt.gov.cn/
## 采集频率及采集策略
<!--说明该项数据更新频率,存量数据的更新频率及策略、新增数据的采集频率及策略-->
对于不同的网站,针对其更新速度不同,指定1,3,6,12,24小时跑一遍
根据数据量指定翻页数,依据内容MD5去重
### 存量更新策略
<!--无需更新?每天全量更新?逐条轮询更新?多久更新完一轮?或其他-->
```buildoutcfg
存量无需更新
增量按照网站的翻页数进行遍历更新
......@@ -97,7 +98,9 @@ risk_court_notice
<!--说明爬虫任务的输入。如:来自某个数据库表等。如果来自某个数据库表则应该简要说明该表内的数据是如何维护的。-->
```buildoutcfg
待完善
对于已经写的爬虫,会在对应频率的文件下添加如下一行
涉及url,是否翻页
{"url_split": ["http://hnzzy.chinacourt.gov.cn/article/index/id/MzQ1NTBINiAOAAA/page/", "####", ".shtml"], "start_index": 1, "pages": 5, "task_type": "开庭公告", "method": "GET", "this_page": 3}
```
### 任务样例
......@@ -105,6 +108,7 @@ risk_court_notice
<!--注意是爬虫拿到的完整任务,而不仅仅是task_params-->
```buildoutcfg
{"url_split": ["http://hnzzy.chinacourt.gov.cn/article/index/id/MzQ1NTBINiAOAAA/page/", "####", ".shtml"], "start_index": 1, "pages": 5, "task_type": "开庭公告", "method": "GET", "this_page": 3}
```
### 任务参数说明
......@@ -119,7 +123,7 @@ risk_court_notice
<!--可能产生的data_type说明-->
```buildoutcfg
data_type只有detail
```
## 爬虫结果的超级数据
......@@ -127,6 +131,8 @@ risk_court_notice
<!--包含所有字段的json数据,每个value都要有样例值-->
```json
数据来源很多,首次统一解析后结果如下
{"type": "court_notices", "court": "连城县人民法院", "courtroom": "第五法庭", "startDate": "2019-10-11 09:00:00", "caseNo": "(2019)闽0825民初2319号", "caseReason": "机动车交通事故责任纠纷", "litigant": [{"name": "邹雄根", "role": ["原告"]}, {"name": "江锦生", "role": ["被告"]}], "content": "原告邹雄根与被告江锦生机动车交通事故责任纠纷", "publishdate": "2019-10-09 09:47:10", "source": "http://fjlcfy.chinacourt.gov.cn", "data_url": "http://fjlcfy.chinacourt.gov.cn/article/detail/2019/10/id/4500822.shtml", "judge": "", "uid": "", "data_md5": "2dd3d7348f35109bf600265dbfb682e1", "is_ok": true, "error_info": ""}
```
......@@ -135,525 +141,8 @@ risk_court_notice
<!--可能与超级数据一致,可能不同的data_type的爬虫结果结构不同,超级数据是把所有data_type的结果组合在一起-->
```json
{
"data": {
"court_items": [
{
"source": "http://www.cqfygzfw.com",
"data": {
"count": 2266,
"prePage": 189,
"nextPage": 191,
"currentPage": 190,
"pageSize": 10,
"pageCount": 227,
"data": [
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M57a8543afa-58b6-424b-b031-b150762a11f7",
"ajbs": "M5727b30594-e934-4439-8c7d-5216cf4f3c92",
"ahqc": "(2021)渝0116民初8949号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日09:30到10:00在第四人民法庭二审判法庭开庭审理(2021)渝0116民初8949号原告罗远林诉被告王开碧,杨军房屋买卖合同纠纷一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
},
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M5748e4ad11-49ae-4086-98ad-7ee6701e9da6",
"ajbs": "M5725653880-7d72-4636-87b6-06bb9067aaad",
"ahqc": "(2021)渝0116民初11913号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日14:30到15:30在第三人民法庭一审判法庭开庭审理(2021)渝0116民初11913号原告黄来毅诉被告周胜辉,高进民间借贷纠纷一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
},
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M57dfd4ebbd-88e8-40a6-9c89-fc732aa92eac",
"ajbs": "M57283867b8-a85b-41ba-80f4-14d6b0324173",
"ahqc": "(2021)渝0116民初9532号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日16:00到17:30在第九审判法庭开庭审理(2021)渝0116民初9532号原告蒲德琼诉被告重庆市江津区东风装饰工程有限公司,翁昌纯民间借贷纠纷一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
},
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M571533c068-5b80-4e98-832d-0dba1e2d9714",
"ajbs": "0c2a2c55-c6a1-4c72-916f-747011d9af25",
"ahqc": "(2021)渝0116民初9784号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日09:30到12:00在第二十二审判法庭开庭审理(2021)渝0116民初9784号原告孙兴福诉被告深圳市宝安区杨静便利店产品责任纠纷一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
},
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M577b4ea5d4-dc70-4bf5-8d8e-4f5a5e41a16c",
"ajbs": "fba8d204-1b20-4d15-88a0-765c339a4d99",
"ahqc": "(2021)渝0116民初10444号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日14:30到15:30在第十审判法庭开庭审理(2021)渝0116民初10444号原告中国农业银行股份有限公司重庆江津支行诉被告代龑信用卡纠纷一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
},
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M57bbd9cf46-6f17-43c3-b634-e4605cb8cad0",
"ajbs": "79559639-d046-4f56-91d6-67ec7c9142ed",
"ahqc": "(2021)渝0116民初10431号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日10:00到11:00在第十四审判法庭开庭审理(2021)渝0116民初10431号原告中国农业银行股份有限公司重庆江津支行诉被告樊涛信用卡纠纷一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
},
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M576399da41-7dc5-483f-b646-bf9d923bece2",
"ajbs": "M57f0d527a0-8bde-4d47-8c68-39abe3c20746",
"ahqc": "(2021)渝0116民初10243号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日09:30到12:00在第二十六审判法庭开庭审理(2021)渝0116民初10243号原告李兴诉被告中建致弘重庆建设开发有限公司,德尔未来科技控股集团股份有限公司,重庆合亨装饰工程有限公司,重庆鼎用装饰工程有限公司劳动争议一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
},
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M57f930f5d2-b593-491c-b7dd-e0acf4e5f49e",
"ajbs": "8d4c3560-dc86-4c7b-9293-b92100806fcb",
"ahqc": "(2021)渝0116民初10256号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日14:30到15:00在第九审判法庭开庭审理(2021)渝0116民初10256号原告重庆天旗实业有限公司诉被告重庆正商实业有限公司,李秋忱买卖合同纠纷一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
},
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M572696f301-3530-47c1-b128-bead5db7fa81",
"ajbs": "e07c3d44-a0b0-4bf7-9c45-5decb09d658d",
"ahqc": "(2021)渝0116民初10962号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日09:30到12:00在第六审判法庭开庭审理(2021)渝0116民初10962号原告重庆市江津区鹏程物业管理有限公司诉被告曾维财物业服务合同纠纷一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
},
{
"count": 0,
"pageSize": 10,
"pageCount": 0,
"page": 1,
"activePage": false,
"start": 0,
"end": 10,
"startDate": null,
"endDate": null,
"dateColName": null,
"retCode": 0,
"retMsg": null,
"bsdf": null,
"cbbm": null,
"cbrmc": null,
"pflag": false,
"sqlServec": 0,
"id": "M5799819ada-d1ea-4d80-909f-6fb9d66d4d19",
"ajbs": "f4b3a0e1-8aaf-4ab5-a920-6d6163d2b60d",
"ahqc": "(2021)渝0116民初10578号",
"ndh": null,
"bzzh": null,
"ajbh": null,
"ktxh": null,
"ggdw": "重庆市江津区人民法院",
"gglx": null,
"gglxmc": null,
"ggnr": "本院定于二〇二一年七月二十七日15:00到17:30在第十一审判法庭开庭审理(2021)渝0116民初10578号原告万世友诉被告敬敏抚养费纠纷一案。",
"fbrxm": null,
"ggrq": 1627315200000,
"xsksrq": null,
"xsjsrq": null,
"xsmtkssj": "08:30",
"xsmtjssj": null,
"xsms": null,
"ggdpbz": null,
"zzrbm": null,
"zzrq": null,
"qybz": null,
"xssx": null,
"sm": null,
"sysTime": null,
"fydm": null,
"fymc": "重庆市江津区人民法院",
"fydms": null,
"startdate": null,
"enddate": null
}
]
}
}
]
},
"http_code": 200,
"error_msg": "",
"task_result": 1000,
"data_type": "detail",
"spider_start_time": "2021-07-27 13:43:15.442",
"spider_end_time": "2021-07-27 13:43:16",
"task_params": {},
"metadata": {},
"spider_name": "court_notice",
"spider_ip": "192.168.56.1"
}
{"type": "court_notices", "court": "连城县人民法院", "courtroom": "第五法庭", "startDate": "2019-10-11 09:00:00", "caseNo": "(2019)闽0825民初2319号", "caseReason": "机动车交通事故责任纠纷", "litigant": [{"name": "邹雄根", "role": ["原告"]}, {"name": "江锦生", "role": ["被告"]}], "content": "原告邹雄根与被告江锦生机动车交通事故责任纠纷", "publishdate": "2019-10-09 09:47:10", "source": "http://fjlcfy.chinacourt.gov.cn", "data_url": "http://fjlcfy.chinacourt.gov.cn/article/detail/2019/10/id/4500822.shtml", "judge": "", "uid": "", "data_md5": "2dd3d7348f35109bf600265dbfb682e1", "is_ok": true, "error_info": ""}
```
## 爬虫运行环境
......@@ -661,7 +150,7 @@ risk_court_notice
<!--udm模块?scrapy?或其他-->
```buildoutcfg
scrapy
scrapy,udm初步解析
```
## 爬虫部署信息
......@@ -723,7 +212,7 @@ task_result=8000 # 参数错误
## 爬虫结果目录
```html
/data/judicature_spiders/court_notice
/data/judicature_spiders/risk_court_notice/BDP-C-1001_10.8.6.51
```
## 归集后存放目录
......
Clone repository
  • README
  • basic_guidelines
  • basic_guidelines
    • basic_guidelines
    • dev_guide
    • project_build
    • 开发流程
  • best_practice
  • best_practice
    • AlterTable
    • RDS
    • azkaban
    • create_table
    • design
    • elasticsearch
    • elasticsearch
      • ES运维
    • logstash
View All Pages