yuanbo基本信息
baidu_planner 百度规划师
baidu_index 百度指数
数据名称（中文）
搜索词优化数据英文名称
baidu_planner
baidu_index采集网站（采集入口）
百度规划师文档页: https://dev2.baidu.com/content?sceneType=0&pageId=100222&nodeId=null&subhead=
百度规划师python API调用页面: https://dev2.baidu.com/debugtool?url=/json/sms/service/KRService/getEstimatedData
百度规划师流量查询页面: https://fengchao.baidu.com/fc/manage/new/user/29416528/kr
百度指数入口页: https://index.baidu.com/v2/index.html#/
具体调用的接口: https://m.baidu.com/sugrec?prod=open_index&wd=章
采集频率及采集策略
存量更新策略
百度规划师: 目前启动5个进程，每次请求携带500个关键词，目前24小时可以将se_word中的关键词3400w左右数据全部获取一遍
百度指数: 目前启动了100个进程，速度非常慢，一天预计只能跑完30w条关键词,有待加资源，加进程增量采集策略
暂无爬虫名称以及平台
spider_name 平台名称
baidu_planner 百度规划师
baidu_index 百度指数责任人
袁波代码地址
项目地址:http://192.168.109.110/granite/project-gravel/-/tree/develop_adsearch_20210517/app_search_ads
百度指数是通过udms方式实现的:
http://192.168.109.110/granite/project-gravel/-/tree/develop_adsearch_20210517/app_search_ads/data_pump
http://192.168.109.110/granite/project-gravel/-/tree/develop_adsearch_20210517/app_search_ads/udms队列名称及队列地址
- redis host: redis://:utn@0818@bdp-mq-001.redis.rds.aliyuncs.com:6379/7
- redis port: 6379
- redis db: 7
- redis key:
- baidu_planner
- baidu_index
优先级队列说明
baidu_planner
baidu_index
说明:无特别处理，都是默认优先级10任务来源
db_host: bdp-rds-103.mysql.rds.aliyuncs.com
db_name: aicha_ad
db_user: aicha
db_password: aicha@0611
se_word数据表中任务作为初始任务任务输入参数（样例）
{
"id": 306,
"word": "000018股票走势"
}任务样例
百度规划师返回结果:
{
"header": {
"desc": "success",
"failures": [],
"oprs": 1,
"succ": 1,
"oprtime": 0,
"quota": 1,
"rquota": 200000000,
"status": 0
},
"body": {
"data": [{
"all": [{
"bid": 5.22, # 指导价
"show": 534, # 预估月均展现
"click": 83.0, # 预估月均点击
"charge": 0.0, # ?
"pv": 15842, # 月均搜索量
"competition": 1 # 竞争激烈程度
}, {
"bid": 1.2,
"show": 6607,
"click": 1034.0,
"charge": 0.0,
"pv": 190317,
"competition": 1
}, {
"bid": 6.35,
"show": 54162,
"click": 7832.0,
"charge": 0.0,
"pv": 1129693,
"competition": 1
}],
"pc": [{
"bid": 5.22, # 计算机指导价
"show": 260,
"click": 35.0,
"charge": 0.0,
"pv": 6354, # 月均计算机搜索量
"competition": 2
}, {
"bid": 0.77,
"show": 3368,
"click": 455.0,
"charge": 0.0,
"pv": 34995,
"competition": 2
}, {
"bid": 6.35,
"show": 42538,
"click": 5750.0,
"charge": 0.0,
"pv": 286344,
"competition": 2
}],
"mobile": [{
"bid": 2.5, # 移动指导价
"show": 274,
"click": 48.0,
"charge": 0.0,
"pv": 9488, # 月均移动搜索量
"competition": 1 #
}, {
"bid": 1.2,
"show": 3239,
"click": 579.0,
"charge": 0.0,
"pv": 155322,
"competition": 1
}, {
"bid": 4.35,
"show": 11624,
"click": 2081.0,
"charge": 0.0,
"pv": 843349,
"competition": 1
}],
"words": ["水箱", "奶茶", "二手车"],
"showReasons": [
["高频热搜词"], # 特色
[],
["黑马词"]
]
}]
}
}
百度指数返回结果:
{
"word": "000540",
"suggest": "09对战平台", # 推荐词
"not_found": false # false表示此关键词找到了推荐词
}任务参数说明
见上面样例说明data_type说明
爬虫结果的超级数据
{
"data": [{
"find_time": "2021-06-21 10:38:57",
"index": 1,
"match_word": "周口师范学院,专业,院校",
"land_url": "https://kaoyan.gaodun.com/2022ky/m/?x_field_1=yMz2eyQ7zuueZFnmue&userid=3054421673&planid=3351335152&unitid=921677877&kwdid=39873900866&adid=5571563434&data_industry=ocpc_ms_convert&fromPlatform=qihoo360&qhclickid=014dbd21e67eacd2",
"title": "周口师范学院考研_报名条件及考试方式_点击进入",
"ad_content": "周口师范学院考研 高顿考研全新学习模式,定向辅导班,专业课名校师资授课,全程.精准考点答疑,实力的名校师资辅导,实时的院校招录信息,考研上名校高顿更专业!"
}],
"http_code": 200,
"error_msg": "",
"task_result": 1000,
"data_type": "detail",
"spider_start_time": "2021-06-21 10:39:24.062",
"spider_end_time": "2021-06-21 10:39:34.150",
"task_params": {
"se_platform": "360_h5",
"word": "周口师范学院专业排名",
"area": "全国",
"outbound": "ad_all_platform",
"routed_count": 1,
"submitter": "taskhub",
"group_retry_times": 0,
"submit_time": "2021-06-19 16:21:02",
"token_scope": "ad",
"retry_limits": 2,
"rt": false,
"priority": null,
"task_uuid": "5110a910-0e42-4df5-957a-cf8c2a3cf79e",
"retry_times": 0
},
"metadata": {},
"spider_name": "ad_360_h5",
"spider_ip": "10.8.6.57"
}
实际爬虫结果的数据结构
{
"header": {
"desc": "success",
"failures": [],
"oprs": 1,
"succ": 1,
"oprtime": 0,
"quota": 1,
"rquota": 200000000,
"status": 0
},
"body": {
"data": [{
"all": [{
"bid": 5.22, # 指导价
"show": 534, # 预估月均展现
"click": 83.0, # 预估月均点击
"charge": 0.0, # ?
"pv": 15842, # 月均搜索量
"competition": 1 # 竞争激烈程度
}, {
"bid": 1.2,
"show": 6607,
"click": 1034.0,
"charge": 0.0,
"pv": 190317,
"competition": 1
}, {
"bid": 6.35,
"show": 54162,
"click": 7832.0,
"charge": 0.0,
"pv": 1129693,
"competition": 1
}],
"pc": [{
"bid": 5.22, # 计算机指导价
"show": 260,
"click": 35.0,
"charge": 0.0,
"pv": 6354, # 月均计算机搜索量
"competition": 2
}, {
"bid": 0.77,
"show": 3368,
"click": 455.0,
"charge": 0.0,
"pv": 34995,
"competition": 2
}, {
"bid": 6.35,
"show": 42538,
"click": 5750.0,
"charge": 0.0,
"pv": 286344,
"competition": 2
}],
"mobile": [{
"bid": 2.5, # 移动指导价
"show": 274,
"click": 48.0,
"charge": 0.0,
"pv": 9488, # 月均移动搜索量
"competition": 1 #
}, {
"bid": 1.2,
"show": 3239,
"click": 579.0,
"charge": 0.0,
"pv": 155322,
"competition": 1
}, {
"bid": 4.35,
"show": 11624,
"click": 2081.0,
"charge": 0.0,
"pv": 843349,
"competition": 1
}],
"words": ["水箱", "奶茶", "二手车"],
"showReasons": [
["高频热搜词"], # 特色
[],
["黑马词"]
]
}]
}
}爬虫运行环境
udm爬虫部署信息
10.8.6.21 baidu_index 100个进程
10.8.6.23 baidu_planner 5个进程Taskhub地址
无Taskhub调度规则说明
无爬虫监控指标设计
无爬虫待采集结果目录
/data/baidu_planner
/data/baidu_index数据归集
责任人
范召贤数据归集方式
-
爬虫直接写kafka
-
爬虫写文件logstash采集
归集后存放目录
无logstash配置文件名称
无logstash文件采集type
无数据归集的topic
无ES日志索引及筛选条件
无监控指标看板
数据保留策略
数据清洗
责任人
代码地址
部署地址
部署方法及说明
- crontab + data_pump
- supervisor + data_pump
- supervisor + consumer
数据接收来源
数据存储表地址
- 数据库地址：
- 表名：