Skip to content

GitLab

search_ads · Changes

Page history
update: index authored by 袁波's avatar 袁波
Hide whitespace changes
Inline Side-by-side
Showing with 78 additions and 47 deletions
data_stream/search_ads.md
View page @ bf37c0d5
......@@ -85,11 +85,11 @@ ad_toutiao_h5 头条 移动端
## 代码地址
```buildoutcfg
项目地址:http://192.168.109.199/granite/project-gravel/tree/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders
项目地址:http://192.168.109.110/granite/project-gravel/tree/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders
百度移动端:
项目入口脚本: http://192.168.109.199/granite/project-gravel/blob/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/ad_baidu_h5.py
代码具体实现板块: http://192.168.109.199/granite/project-gravel/tree/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/search_ads_reqs/baidu
项目入口脚本: http://192.168.109.110/granite/project-gravel/blob/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/ad_baidu_h5.py
代码具体实现板块: http://192.168.109.110/granite/project-gravel/tree/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/search_ads_reqs/baidu
(说明: 同理，其它爬虫都根据爬虫名称在同级目录或板块下)
```
......@@ -190,40 +190,39 @@ recommend: 搜索推荐的关键词
<!--包含所有字段的json数据，每个value都要有样例值-->
```buildoutcfg
{
"data": [
{
"find_time": "2021-04-14 21:40:46.358",
"match_word": "牡丹江,招聘信息,招聘,招聘,招聘信息,人才招聘,职位信息",
"land_url": "https://mkt.51job.com/tg/sem/lp2020_H5_jc/index.php?partner=sem_baidu20_126156",
"ad_content": "【前程无忧51job】高薪招聘信息,专业人才招聘网站,海量职位信息,快速投递简历.手机轻松找工作!",
"title": "牡丹江_招聘信息_热门招聘_高薪招聘【51Job】招聘网"
}
],
"http_code": 200,
"error_msg": "",
"task_result": 1000,
"data_type": "detail",
"spider_start_time": "2021-04-14 21:42:39.901",
"spider_end_time": "2021-04-14 21:42:41.688",
"task_params": {
"se_platform": "baidu_h5",
"word": "0453牡丹江最新招聘信息",
"area": "全国",
"outbound": "ad_all_platform",
"routed_count": 1,
"submitter": "taskhub",
"group_retry_times": 0,
"submit_time": "2021-04-14 21:19:45",
"token_scope": "ad",
"retry_limits": 2,
"rt": false,
"priority": null,
"task_uuid": "a2bd1b18-e671-4b77-81ee-3e636372f385",
"retry_times": 0
},
"metadata": { },
"spider_name": "ad_baidu_h5",
"spider_ip": "192.168.108.155"
"data": [{
"find_time": "2021-06-21 10:38:57",
"index": 1,
"match_word": "周口师范学院,专业,院校",
"land_url": "https://kaoyan.gaodun.com/2022ky/m/?x_field_1=yMz2eyQ7zuueZFnmue&userid=3054421673&planid=3351335152&unitid=921677877&kwdid=39873900866&adid=5571563434&data_industry=ocpc_ms_convert&fromPlatform=qihoo360&qhclickid=014dbd21e67eacd2",
"title": "周口师范学院考研_报名条件及考试方式_点击进入",
"ad_content": "周口师范学院考研 高顿考研全新学习模式,定向辅导班,专业课名校师资授课,全程.精准考点答疑,实力的名校师资辅导,实时的院校招录信息,考研上名校高顿更专业!"
}],
"http_code": 200,
"error_msg": "",
"task_result": 1000,
"data_type": "detail",
"spider_start_time": "2021-06-21 10:39:24.062",
"spider_end_time": "2021-06-21 10:39:34.150",
"task_params": {
"se_platform": "360_h5",
"word": "周口师范学院专业排名",
"area": "全国",
"outbound": "ad_all_platform",
"routed_count": 1,
"submitter": "taskhub",
"group_retry_times": 0,
"submit_time": "2021-06-19 16:21:02",
"token_scope": "ad",
"retry_limits": 2,
"rt": false,
"priority": null,
"task_uuid": "5110a910-0e42-4df5-957a-cf8c2a3cf79e",
"retry_times": 0
},
"metadata": {},
"spider_name": "ad_360_h5",
"spider_ip": "10.8.6.57"
}
```
......@@ -231,9 +230,41 @@ recommend: 搜索推荐的关键词
## 实际爬虫结果的数据结构
<!--可能与超级数据一致，可能不同的data_type的爬虫结果结构不同，超级数据是把所有data_type的结果组合在一起-->
```buildoutcfg
baidu_h5 样例链接: http://192.168.109.199/granite/project-gravel/wikis/ad-baidu-h5-data-example
360_pc 样例链接: http://192.168.109.199/granite/project-gravel/wikis/ad-360-pc-data-example
360_h5 样例链接: http://192.168.109.199/granite/project-gravel/wikis/ad-360-h5-data-example
{
"data": [{
"find_time": "2021-06-21 10:38:57",
"index": 1,
"match_word": "周口师范学院,专业,院校",
"land_url": "https://kaoyan.gaodun.com/2022ky/m/?x_field_1=yMz2eyQ7zuueZFnmue&userid=3054421673&planid=3351335152&unitid=921677877&kwdid=39873900866&adid=5571563434&data_industry=ocpc_ms_convert&fromPlatform=qihoo360&qhclickid=014dbd21e67eacd2",
"title": "周口师范学院考研_报名条件及考试方式_点击进入",
"ad_content": "周口师范学院考研 高顿考研全新学习模式,定向辅导班,专业课名校师资授课,全程.精准考点答疑,实力的名校师资辅导,实时的院校招录信息,考研上名校高顿更专业!"
}],
"http_code": 200,
"error_msg": "",
"task_result": 1000,
"data_type": "detail",
"spider_start_time": "2021-06-21 10:39:24.062",
"spider_end_time": "2021-06-21 10:39:34.150",
"task_params": {
"se_platform": "360_h5",
"word": "周口师范学院专业排名",
"area": "全国",
"outbound": "ad_all_platform",
"routed_count": 1,
"submitter": "taskhub",
"group_retry_times": 0,
"submit_time": "2021-06-19 16:21:02",
"token_scope": "ad",
"retry_limits": 2,
"rt": false,
"priority": null,
"task_uuid": "5110a910-0e42-4df5-957a-cf8c2a3cf79e",
"retry_times": 0
},
"metadata": {},
"spider_name": "ad_360_h5",
"spider_ip": "10.8.6.57"
}
```
## 爬虫运行环境
......@@ -248,19 +279,19 @@ scrapy
```buildoutcfg
10.8.6.62
baidu_pc 10个进程
baidu_h5 15个进程
360_pc 15个进程
360_h5 20个进程
sougou_pc 20个进程
sougou_h5 10个进程
shenma 20个进程
baidu_h5 5个进程
360_pc 10个进程
360_h5 5个进程
sougou_pc 10个进程
sougou_h5 5个进程
shenma 5个进程
```
## Taskhub地址
```buildoutcfg
提交任务地址: http://10.8.6.222:18518/task/
代码编写地址: http://192.168.109.199/granite/project-taskhub/blob/master/taskhub/config/ad/config_ad.d/01_task.yaml
代码编写地址: http://192.168.109.110/granite/project-taskhub/blob/master/taskhub/config/ad/config_ad.d/01_task.yaml
```
## Taskhub调度规则说明
......