|
|
|
# **基本信息**
|
|
|
|
```buildoutcfg
|
|
|
|
se_platform spider_name 平台名称
|
|
|
|
baidu_pc ad_baidu_pc 百度 PC端
|
|
|
|
baidu_h5 ad_baidu_h5 百度 移动端
|
|
|
|
360_pc ad_360_pc 360 PC端
|
|
|
|
360_h5 ad_360_h5 360 移动端
|
|
|
|
sougou_pc ad_sougou_pc 搜狗 PC端
|
|
|
|
sougou_h5 ad_sougou_h5 搜狗 移动端
|
|
|
|
shenma ad_shenma_h5 神马 移动端
|
|
|
|
toutiao ad_toutiao_h5 头条 移动端
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 数据名称（中文）
|
|
|
|
<!-- 该项数据的中文标准名称，后续沟通交流使用的规范名称，如：工商公示股东信息、失信被执行人、一般纳税人等-->
|
|
|
|
###
|
|
|
|
```buildoutcfg
|
|
|
|
搜索广告
|
|
|
|
```
|
|
|
|
|
|
|
|
## 数据英文名称
|
|
|
|
<!--英文名称，后续流程中所有涉及到英文名称均以此为准，如：partner、shixin、general_taxpayer等-->
|
|
|
|
```buildoutcfg
|
|
|
|
search_ads
|
|
|
|
```
|
|
|
|
|
|
|
|
## 采集网站（采集入口）
|
|
|
|
<!--采集的入口地址，不能只是一个网站域名，具体到该网站的数据入口-->
|
|
|
|
```buildoutcfg
|
|
|
|
百度h5入口:
|
|
|
|
https://www.baidu.com/from=844b/s?word=python&ts=8478142&t_kt=0&ie=utf-8&fm_kl=021394be2f&rsv_iqid=2183853714&rsv_t=376a8VEY8%252FDOKsDMkKlCpa2B%252FASIiibUR4bHEz4HRUJlSfO8rYgnUcCdQw&sa=ih_2&ms=1&rsv_pq=2183853714&rsv_sug4=2159&ss=011&tj=1
|
|
|
|
360 PC入口:
|
|
|
|
https://www.so.com/s?q=java&src=srp&fr=so.com&psid=69bd05cd22757efaf20f73ad38ab739d
|
|
|
|
360 H5入口:
|
|
|
|
https://m.so.com/s?q=python&src=suggest_msearch&sug_pos=0&sug=&nlpv=&ssid=&srcg=home_next
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 采集频率及采集策略
|
|
|
|
<!--说明该项数据更新频率，存量数据的更新频率及策略、新增数据的采集频率及策略-->
|
|
|
|
|
|
|
|
### 存量更新策略
|
|
|
|
<!--无需更新？每天全量更新？逐条轮询更新？多久更新完一轮？或其他-->
|
|
|
|
```buildoutcfg
|
|
|
|
db_host: bdp-rds-103.mysql.rds.aliyuncs.com
|
|
|
|
db_name: aicha_ad
|
|
|
|
db_user: aicha
|
|
|
|
db_password: aicha@0611
|
|
|
|
se_word数据表中任务作为初始任务
|
|
|
|
逐条更新
|
|
|
|
目前2周更新一轮
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### 增量采集策略
|
|
|
|
<!--新增数据怎么来？无需单独采集新增数据？-->
|
|
|
|
```buildoutcfg
|
|
|
|
暂无
|
|
|
|
```
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
# **爬虫名称以及平台**
|
|
|
|
```buildoutcfg
|
|
|
|
spider_name 平台名称
|
|
|
|
ad_baidu_pc 百度 PC端
|
|
|
|
ad_baidu_h5 百度 移动端
|
|
|
|
ad_360_pc 360 PC端
|
|
|
|
ad_360_h5 360 移动端
|
|
|
|
ad_sougou_pc 搜狗 PC端
|
|
|
|
ad_sougou_h5 搜狗 移动端
|
|
|
|
ad_shenma_h5 神马 移动端
|
|
|
|
ad_toutiao_h5 头条 移动端
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 责任人
|
|
|
|
```buildoutcfg
|
|
|
|
袁波
|
|
|
|
```
|
|
|
|
|
|
|
|
<!--spider_name-->
|
|
|
|
|
|
|
|
## 代码地址
|
|
|
|
```buildoutcfg
|
|
|
|
项目地址:http://192.168.109.199/granite/project-gravel/tree/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders
|
|
|
|
|
|
|
|
百度移动端:
|
|
|
|
项目入口脚本: http://192.168.109.199/granite/project-gravel/blob/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/ad_baidu_h5.py
|
|
|
|
代码具体实现板块: http://192.168.109.199/granite/project-gravel/tree/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/search_ads_reqs/baidu
|
|
|
|
(说明: 同理，其它爬虫都根据爬虫名称在同级目录或板块下)
|
|
|
|
```
|
|
|
|
|
|
|
|
## 队列名称及队列地址
|
|
|
|
<!--redis host port db key 优先级说明-->
|
|
|
|
-
|
|
|
|
* redis host: redis://:utn@0818@bdp-mq-001.redis.rds.aliyuncs.com:6379/7
|
|
|
|
* redis port: 6379
|
|
|
|
* redis db: 7
|
|
|
|
* redis key:
|
|
|
|
* ad_baidu_pc
|
|
|
|
* ad_baidu_h5
|
|
|
|
* ad_360_pc
|
|
|
|
* ad_360_h5
|
|
|
|
* ad_sougou_pc
|
|
|
|
* ad_sougou_h5
|
|
|
|
* ad_shenma_h5
|
|
|
|
* ad_toutiao_h5
|
|
|
|
|
|
|
|
### 优先级队列说明
|
|
|
|
```buildoutcfg
|
|
|
|
ad_baidu_pc
|
|
|
|
ad_baidu_h5
|
|
|
|
ad_360_pc
|
|
|
|
ad_360_h5
|
|
|
|
ad_sougou_pc
|
|
|
|
ad_sougou_h5
|
|
|
|
ad_shenma_h5
|
|
|
|
ad_toutiao_h5
|
|
|
|
说明:无特别处理，都是默认优先级10
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## 任务来源
|
|
|
|
<!--说明爬虫任务的输入。如：来自某个数据库表等。如果来自某个数据库表则应该简要说明该表内的数据是如何维护的。-->
|
|
|
|
```buildoutcfg
|
|
|
|
db_host: bdp-rds-103.mysql.rds.aliyuncs.com
|
|
|
|
db_name: aicha_ad
|
|
|
|
db_user: aicha
|
|
|
|
db_password: aicha@0611
|
|
|
|
se_word数据表中任务作为初始任务
|
|
|
|
```
|
|
|
|
|
|
|
|
## 任务输入参数（样例）
|
|
|
|
```json
|
|
|
|
{
|
|
|
|
"se_platform": "baidu_h5",
|
|
|
|
"word": "1200目普通硅微粉",
|
|
|
|
"area": "全国"
|
|
|
|
}
|
|
|
|
说明: 其它任务同理，填不同的se_platform和word即可
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
### 任务样例
|
|
|
|
<!--注意是爬虫拿到的完整任务，而不仅仅是task_params-->
|
|
|
|
```buildoutcfg
|
|
|
|
{
|
|
|
|
"se_platform": "baidu_h5",
|
|
|
|
"word": "1200目普通硅微粉",
|
|
|
|
"area": "全国",
|
|
|
|
"outbound": "ad_all_platform",
|
|
|
|
"routed_count": 1,
|
|
|
|
"submitter": "taskhub",
|
|
|
|
"group_retry_times": 0,
|
|
|
|
"submit_time": "2021-04-15 15:11:02",
|
|
|
|
"token_scope": "ad",
|
|
|
|
"retry_limits": 2,
|
|
|
|
"rt": false,
|
|
|
|
"priority": null,
|
|
|
|
"task_uuid": "c1577311-f58b-4235-8778-ce08d54df889",
|
|
|
|
"retry_times": 0
|
|
|
|
}
|
|
|
|
说明: 其它爬虫同理，se_platform和word不同即可
|
|
|
|
```
|
|
|
|
|
|
|
|
### 任务参数说明
|
|
|
|
<!--特有参数说明，通用参数比如spider_name,task_params,task_src,task_result等不需说明-->
|
|
|
|
```json
|
|
|
|
{
|
|
|
|
"se_platform": "baidu_h5", # 平台名称
|
|
|
|
"word": "1200目普通硅微粉", # 搜索关键词
|
|
|
|
"area": "全国" # 广告范围
|
|
|
|
}
|
|
|
|
说明: 其它爬虫同理，se_platform和word不同即可
|
|
|
|
```
|
|
|
|
|
|
|
|
## data_type说明
|
|
|
|
<!--可能产生的data_type说明-->
|
|
|
|
```buildoutcfg
|
|
|
|
list: 暂未使用
|
|
|
|
detail: 默认正常出来的广告数据
|
|
|
|
recommend: 搜索推荐的关键词
|
|
|
|
```
|
|
|
|
|
|
|
|
## 爬虫结果的超级数据
|
|
|
|
<!--包含所有字段的json数据，每个value都要有样例值-->
|
|
|
|
```buildoutcfg
|
|
|
|
{
|
|
|
|
"data": [
|
|
|
|
{
|
|
|
|
"find_time": "2021-04-14 21:40:46.358",
|
|
|
|
"match_word": "牡丹江,招聘信息,招聘,招聘,招聘信息,人才招聘,职位信息",
|
|
|
|
"land_url": "https://mkt.51job.com/tg/sem/lp2020_H5_jc/index.php?partner=sem_baidu20_126156",
|
|
|
|
"ad_content": "【前程无忧51job】高薪招聘信息,专业人才招聘网站,海量职位信息,快速投递简历.手机轻松找工作!",
|
|
|
|
"title": "牡丹江_招聘信息_热门招聘_高薪招聘【51Job】招聘网"
|
|
|
|
}
|
|
|
|
],
|
|
|
|
"http_code": 200,
|
|
|
|
"error_msg": "",
|
|
|
|
"task_result": 1000,
|
|
|
|
"data_type": "detail",
|
|
|
|
"spider_start_time": "2021-04-14 21:42:39.901",
|
|
|
|
"spider_end_time": "2021-04-14 21:42:41.688",
|
|
|
|
"task_params": {
|
|
|
|
"se_platform": "baidu_h5",
|
|
|
|
"word": "0453牡丹江最新招聘信息",
|
|
|
|
"area": "全国",
|
|
|
|
"outbound": "ad_all_platform",
|
|
|
|
"routed_count": 1,
|
|
|
|
"submitter": "taskhub",
|
|
|
|
"group_retry_times": 0,
|
|
|
|
"submit_time": "2021-04-14 21:19:45",
|
|
|
|
"token_scope": "ad",
|
|
|
|
"retry_limits": 2,
|
|
|
|
"rt": false,
|
|
|
|
"priority": null,
|
|
|
|
"task_uuid": "a2bd1b18-e671-4b77-81ee-3e636372f385",
|
|
|
|
"retry_times": 0
|
|
|
|
},
|
|
|
|
"metadata": { },
|
|
|
|
"spider_name": "ad_baidu_h5",
|
|
|
|
"spider_ip": "192.168.108.155"
|
|
|
|
}
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
## 实际爬虫结果的数据结构
|
|
|
|
<!--可能与超级数据一致，可能不同的data_type的爬虫结果结构不同，超级数据是把所有data_type的结果组合在一起-->
|
|
|
|
```buildoutcfg
|
|
|
|
baidu_h5 样例链接: http://192.168.109.199/granite/project-gravel/wikis/ad-baidu-h5-data-example
|
|
|
|
360_pc 样例链接: http://192.168.109.199/granite/project-gravel/wikis/ad-360-pc-data-example
|
|
|
|
360_h5 样例链接: http://192.168.109.199/granite/project-gravel/wikis/ad-360-h5-data-example
|
|
|
|
```
|
|
|
|
|
|
|
|
## 爬虫运行环境
|
|
|
|
<!--udm模块？scrapy？或其他-->
|
|
|
|
```buildoutcfg
|
|
|
|
scrapy
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 爬虫部署信息
|
|
|
|
<!--部署在哪些机器？每个机器多少进程？项目名称是什么？-->
|
|
|
|
```buildoutcfg
|
|
|
|
baidu_h5: 10.8.6.57 10个进程
|
|
|
|
360_pc: 10.8.6.23 10个进程
|
|
|
|
360_h5: 10.8.6.23 10个进程
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## Taskhub地址
|
|
|
|
```buildoutcfg
|
|
|
|
提交任务地址: http://10.8.6.222:18518/task/
|
|
|
|
代码编写地址: http://192.168.109.199/granite/project-taskhub/blob/master/taskhub/config/ad/config_ad.d/01_task.yaml
|
|
|
|
```
|
|
|
|
|
|
|
|
## Taskhub调度规则说明
|
|
|
|
```buildoutcfg
|
|
|
|
task_result=1000 # 正常获取到详情任务
|
|
|
|
task_result=1101 # 本条关键词没有搜索到广告数据
|
|
|
|
task_result=9101 # 超时错误，需要进行重试，目前重试5次
|
|
|
|
task_result=8000 # 参数错误
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## 爬虫监控指标设计
|
|
|
|
<!--监控爬虫正常运行的指标是什么？报警规则是什么？-->
|
|
|
|
```buildoutcfg
|
|
|
|
(先观察，待补充)
|
|
|
|
索引:
|
|
|
|
监控频率:
|
|
|
|
监控起止时间:
|
|
|
|
报警条件:
|
|
|
|
报警群:
|
|
|
|
报警内容:
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
# **数据归集**
|
|
|
|
|
|
|
|
## 责任人
|
|
|
|
```
|
|
|
|
范召贤
|
|
|
|
```
|
|
|
|
|
|
|
|
## 数据归集方式
|
|
|
|
|
|
|
|
- [ ] 爬虫直接写kafka
|
|
|
|
|
|
|
|
- [x] 爬虫写文件logstash采集
|
|
|
|
|
|
|
|
## 爬虫结果目录
|
|
|
|
/data/gravel_spiders/ad_baidu_h5
|
|
|
|
|
|
|
|
## 归集后存放目录
|
|
|
|
/data2_227/ad_spider_result/ad_360_h5
|
|
|
|
|
|
|
|
## logstash配置文件名称
|
|
|
|
project-deploy/logstash/10.8.6.229/conf.d/collie_spider_data_to_kfk.conf
|
|
|
|
|
|
|
|
## logstash文件采集type
|
|
|
|
collie-spider-aicha-ad
|
|
|
|
|
|
|
|
## 数据归集的topic
|
|
|
|
collie-ad-result
|
|
|
|
|
|
|
|
## ES日志索引及筛选条件
|
|
|
|
ad_spider_log-*
|
|
|
|
|
|
|
|
## 监控指标看板
|
|
|
|
|
|
|
|
|
|
|
|
## 数据保留策略
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
# **数据清洗**
|
|
|
|
|
|
|
|
## 责任人
|
|
|
|
|
|
|
|
|
|
|
|
## 代码地址
|
|
|
|
|
|
|
|
|
|
|
|
## 部署地址
|
|
|
|
<!--机器及线上代码地址-->
|
|
|
|
|
|
|
|
|
|
|
|
## 部署方法及说明
|
|
|
|
<!--运行方法及运行命令、supervisor配置、supervisor的program等-->
|
|
|
|
- [ ] crontab + data_pump
|
|
|
|
- [ ] supervisor + data_pump
|
|
|
|
- [ ] supervisor + consumer
|
|
|
|
|
|
|
|
## 数据接收来源
|
|
|
|
<!--来源于kafka还是归集的文件、topic的group？-->
|
|
|
|
|
|
|
|
|
|
|
|
## 数据存储表地址
|
|
|
|
|
|
|
|
* 数据库地址：
|
|
|
|
* 表名： |
|
|
|
\ No newline at end of file |
