Skip to content

GitLab

Last edited by 王鹏举
Page history

law_office

基本信息

数据名称（中文）

律师事务所

数据英文名称

law_office           
bj_law_office   
sh_law_office

采集网站（采集入口）

http://12348.moj.gov.cn/#/publicies/lawdept/lawdept
http://xkyw.bjsf.gov.cn/lawofficeaction.do?method=queryService&chaxun=lvsuocx&ifHq=1&page.currentPage=1&page.totalPerPage=19
http://sh.12348.gov.cn/sites/12348/team.jsp?typeId=cf2ec647762641e897c0d1fd7f474141&zoneId=&sort=All&servant=All&normal=true&award=&member=All&businessArea=All&jbusinessArea=All&partWord=#

采集频率及采集策略

基本是一次性工作，暂不需要周期调度

存量更新策略

增量采集策略

爬虫名称以及平台 

爬虫名称: law_office  
平台: 中国法律服务网-律师事务所

爬虫名称: bj_law_office
平台: 北京市司法局律师管理系统律所查询

爬虫名称: sh_law_office
平台: 上海法网-律师事务所

责任人

袁波

代码地址

项目地址
http://192.168.109.110/granite/project-gravel/-/tree/develop_adsearch_20210517/scrapy_spiders/gravel_spiders/spiders

项目入口脚本
http://192.168.109.110/granite/project-gravel/-/blob/develop_adsearch_20210517/scrapy_spiders/gravel_spiders/spiders/law_office.py
http://192.168.109.110/granite/project-gravel/-/blob/develop_adsearch_20210517/scrapy_spiders/gravel_spiders/spiders/bj_law_office.py
http://192.168.109.110/granite/project-gravel/-/blob/develop_adsearch_20210517/scrapy_spiders/gravel_spiders/spiders/sh_law_office.py

代码具体实现板块
http://192.168.109.110/granite/project-gravel/-/tree/develop_adsearch_20210517/scrapy_spiders/gravel_spiders/spiders/law_office_reqs

队列名称及队列地址

优先级队列说明

law_office
bj_law_office
sh_law_office
说明:无特别处理，都是默认优先级10

任务来源

中国法网
ajax请求，json数据格式返回
http://12348.moj.gov.cn/lawerdeptlist/getlawerdeptlist

北京律所与上海律所是从html页面中提取

任务输入参数（样例）

{
    "platform": "law_office",
    "data_type": "list"
}
{
    "platform": "bj_law_office",
    "data_type": "list"
}
{
    "platform": "sh_law_office",
    "data_type": "list"
}

任务样例

{
    "platform": "law_office",
    "data_type: "list",
}
{
    "platform": "bj_law_office",
    "data_type: "list",
}
{
    "platform": "sh_law_office",
    "data_type: "list",
}

任务参数说明

{
    "platform": "law_office",
    "data_type": "list"
}
说明: 任务直接到redis,不经过taskhub

data_type说明

list: 入口url
detail: 某个详情页的数据
log: 日志记录

爬虫结果的超级数据

中国法网
{
	"data": [{
		"jj": "",
		"pzslsj": "2010-05-13",
		"zzxs": "02",
		"lsswsmc": "重庆道润律师事务所",
		"zsdh": "023-67686602",
		"zyzh": "25001201010281592",
		"cz": "023-67686602",
		"zsd": "重庆市北部新区高新园区宝华大道10号1幢1区17-6、17-7、17-8、17-9、17-10",
		"jd": "106.55640000004154",
		"tyshxydm": "31500000552049937N",
		"wd": "29.610119006729243",
		"ywzclist": [],
		"nums": 5
	}],
	"http_code": 200,
	"error_msg": "",
	"task_result": 1000,
	"data_type": "detail",
	"spider_start_time": "2021-08-20 13:14:27.338",
	"spider_end_time": "2021-08-20 13:16:30.341",
	"task_params": {
		"platform": "law_office",
		"data_type": "list"
	},
	"metadata": {
		"current_page": 2515
	},
	"spider_name": "law_office",
	"spider_ip": "10.8.1.10",
	"proxy_ip": "http://127.0.0.1:8888"
}

北京律所
{
	"data": {
		"bj_law_office": {
			"company_name": "北京腾骏律师事务所",
			"credit_no": "31110000306433298G",
			"company_status": "正常",
			"en_name": " ",
			"company_address": "北京市朝阳区霄云里8号1单元12层1501室216",
			"organizer": "朝阳区司法局",
			"issue_date": " 2014-06-30 00:00:00.0 ",
			"capital": "50.0",
			"legal_person": "李雯",
			"composion_form": "个人",
			"e_mail": "liwen20@hotmail.com",
			"web_url": null,
			"phone": "85188812",
			"fax": " 85188812",
			"brief_introduction": ""
		}
	},
	"http_code": 200,
	"error_msg": "",
	"task_result": 1000,
	"data_type": "detail",
	"spider_start_time": "2021-08-26 11:44:50.409",
	"spider_end_time": "2021-08-26 11:44:51.591",
	"task_params": {
		"platform": "bj_law_office",
		"data_type": "list"
	},
	"metadata": {
		"current_page": 15
	},
	"spider_name": "bj_law_office",
	"spider_ip": "10.8.1.42",
	"proxy_ip": "http://10.8.6.219:38080"
}

上海律所
{
	"data": {
		"sh_law_office": {
			"company_name": "上海维盈律师事务所",
			"lawyer_num": "27",
			"license_no": "23101201010322666",
			"company_address": "东方路971号钱江大厦24D",
			"phone": "50812019",
			"legal_person": "赵德强",
			"composion_form": "普通合伙",
			"company_status": "正常",
			"org_code": "55000011-X",
			"issue_date": "2010-01-11",
			"organizer": "浦东新区司法局"
		}
	},
	"http_code": 200,
	"error_msg": "",
	"task_result": 1000,
	"data_type": "detail",
	"spider_start_time": "2021-08-27 14:18:19.156",
	"spider_end_time": "2021-08-27 14:18:19.801",
	"task_params": {
		"platform": "sh_law_office",
		"data_type": "list"
	},
	"metadata": {
		"current_page": 9
	},
	"spider_name": "sh_law_office",
	"spider_ip": "10.8.1.42",
	"proxy_ip": "http://10.8.6.219:38080"
}

实际爬虫结果的数据结构

爬虫运行环境

scrapy

爬虫部署信息

10.8.6.76   3个进程

Taskhub地址

Taskhub调度规则说明

爬虫监控指标设计

爬虫待采集结果目录

/data/gravel_spiders/law_office
/data/gravel_spiders/bj_law_office
/data/gravel_spiders/sh_law_office

数据归集

责任人

范召贤

数据归集方式

  • 爬虫直接写kafka

  • 爬虫写文件logstash采集

归集后存放目录

/data2_227/grvael_spider_result/law_office
/data2_227/grvael_spider_result/bj_law_office
/data2_227/grvael_spider_result/sh_law_office

logstash配置文件名称

project-deploy/logstash/10.8.6.246/conf.d/collie_spider_data_to_kfk.conf(入topic)

logstash文件采集type

type=>"law_office"
type=>"bj_law_office"
type=>"sh_law_office"

数据归集的topic

topic_id => "general-taxpayer"
分区数: 12
数据清洗group => "law_office_etl"

ES日志索引及筛选条件

监控指标看板

数据保留策略

数据清洗

责任人

王鹏举

代码地址

http://tech.pingansec.com/granite/project-collie-app/-/tree/master/app_law_office

http://tech.pingansec.com/granite/project-collie-app/-/tree/master/app_online_lake

部署地址

10.8.6.228

部署方法及说明

run.sh

数据接收来源

文件

数据存储表地址

  • 数据库地址：10.8.6.87
  • 数据库名：utn_special
  • 表名：special_law_office