Skip to content

GitLab

Last edited by yuanbo
Page history
This is an old version of this page. You can view the most recent version or browse the history.

search_ads_optimization

基本信息 

baidu_planner   百度规划师
baidu_index     百度指数

数据名称（中文）

搜索词优化

数据英文名称

baidu_planner
baidu_index

采集网站（采集入口）

百度规划师文档页: https://dev2.baidu.com/content?sceneType=0&pageId=100222&nodeId=null&subhead=
百度规划师python API调用页面: https://dev2.baidu.com/debugtool?url=/json/sms/service/KRService/getEstimatedData
百度规划师流量查询页面: https://fengchao.baidu.com/fc/manage/new/user/29416528/kr

百度指数入口页: https://index.baidu.com/v2/index.html#/
具体调用的接口: https://m.baidu.com/sugrec?prod=open_index&wd=章

采集频率及采集策略

存量更新策略

百度规划师: 目前启动5个进程，每次请求携带500个关键词，目前24小时可以将se_word中的关键词3400w左右数据全部获取一遍
百度指数: 目前启动了100个进程，速度非常慢，一天预计只能跑完30w条关键词,有待加资源，加进程

增量采集策略

暂无

爬虫名称以及平台 

spider_name   平台名称
baidu_planner   百度规划师
baidu_index     百度指数

责任人

袁波

代码地址

项目地址:http://192.168.109.110/granite/project-gravel/-/tree/develop_adsearch_20210517/app_search_ads

百度指数是通过udms方式实现的:
http://192.168.109.110/granite/project-gravel/-/tree/develop_adsearch_20210517/app_search_ads/data_pump
http://192.168.109.110/granite/project-gravel/-/tree/develop_adsearch_20210517/app_search_ads/udms

队列名称及队列地址

优先级队列说明

baidu_planner
baidu_index
说明:无特别处理，都是默认优先级10

任务来源

db_host: bdp-rds-103.mysql.rds.aliyuncs.com
db_name: aicha_ad
db_user: aicha
db_password: aicha@0611
se_word数据表中任务作为初始任务

任务输入参数（样例）

{
  "id": 306,
  "word": "000018股票走势"
}

任务样例

百度规划师返回结果:

{
	"header": {
		"desc": "success",
		"failures": [],
		"oprs": 1,
		"succ": 1,
		"oprtime": 0,
		"quota": 1,
		"rquota": 200000000,
		"status": 0
	},
	"body": {
		"data": [{
			"all": [{
				"bid": 5.22, # 指导价
				"show": 534,  # 预估月均展现
				"click": 83.0,  # 预估月均点击
				"charge": 0.0,  # ?
				"pv": 15842,  # 月均搜索量
				"competition": 1  # 竞争激烈程度
			}, {
				"bid": 1.2,
				"show": 6607,
				"click": 1034.0,
				"charge": 0.0,
				"pv": 190317,
				"competition": 1
			}, {
				"bid": 6.35,
				"show": 54162,
				"click": 7832.0,
				"charge": 0.0,
				"pv": 1129693,
				"competition": 1
			}],
			"pc": [{
				"bid": 5.22,  # 计算机指导价
				"show": 260,
				"click": 35.0,
				"charge": 0.0,
				"pv": 6354,   # 月均计算机搜索量
				"competition": 2
			}, {
				"bid": 0.77,
				"show": 3368,
				"click": 455.0,
				"charge": 0.0,
				"pv": 34995,
				"competition": 2
			}, {
				"bid": 6.35,
				"show": 42538,
				"click": 5750.0,
				"charge": 0.0,
				"pv": 286344,
				"competition": 2
			}],
			"mobile": [{
				"bid": 2.5, # 移动指导价
				"show": 274,
				"click": 48.0,
				"charge": 0.0,
				"pv": 9488,   # 月均移动搜索量
				"competition": 1   # 
			}, {
				"bid": 1.2,
				"show": 3239,
				"click": 579.0,
				"charge": 0.0,
				"pv": 155322,
				"competition": 1
			}, {
				"bid": 4.35,
				"show": 11624,
				"click": 2081.0,
				"charge": 0.0,
				"pv": 843349,
				"competition": 1
			}],
			"words": ["水箱", "奶茶", "二手车"],
			"showReasons": [
				["高频热搜词"],  # 特色
				[],
				["黑马词"]
			]
		}]
	}
}

百度指数返回结果:

{
	"word": "000540",
	"suggest": "09对战平台",    # 推荐词
	"not_found": false          # false表示此关键词找到了推荐词
}

任务参数说明

见上面样例说明

data_type说明

爬虫结果的超级数据

百度规划师返回结果:

{
	"header": {
		"desc": "success",
		"failures": [],
		"oprs": 1,
		"succ": 1,
		"oprtime": 0,
		"quota": 1,
		"rquota": 200000000,
		"status": 0
	},
	"body": {
		"data": [{
			"all": [{
				"bid": 5.22, # 指导价
				"show": 534,  # 预估月均展现
				"click": 83.0,  # 预估月均点击
				"charge": 0.0,  # ?
				"pv": 15842,  # 月均搜索量
				"competition": 1  # 竞争激烈程度
			}, {
				"bid": 1.2,
				"show": 6607,
				"click": 1034.0,
				"charge": 0.0,
				"pv": 190317,
				"competition": 1
			}, {
				"bid": 6.35,
				"show": 54162,
				"click": 7832.0,
				"charge": 0.0,
				"pv": 1129693,
				"competition": 1
			}],
			"pc": [{
				"bid": 5.22,  # 计算机指导价
				"show": 260,
				"click": 35.0,
				"charge": 0.0,
				"pv": 6354,   # 月均计算机搜索量
				"competition": 2
			}, {
				"bid": 0.77,
				"show": 3368,
				"click": 455.0,
				"charge": 0.0,
				"pv": 34995,
				"competition": 2
			}, {
				"bid": 6.35,
				"show": 42538,
				"click": 5750.0,
				"charge": 0.0,
				"pv": 286344,
				"competition": 2
			}],
			"mobile": [{
				"bid": 2.5, # 移动指导价
				"show": 274,
				"click": 48.0,
				"charge": 0.0,
				"pv": 9488,   # 月均移动搜索量
				"competition": 1   # 
			}, {
				"bid": 1.2,
				"show": 3239,
				"click": 579.0,
				"charge": 0.0,
				"pv": 155322,
				"competition": 1
			}, {
				"bid": 4.35,
				"show": 11624,
				"click": 2081.0,
				"charge": 0.0,
				"pv": 843349,
				"competition": 1
			}],
			"words": ["水箱", "奶茶", "二手车"],
			"showReasons": [
				["高频热搜词"],  # 特色
				[],
				["黑马词"]
			]
		}]
	}
}

实际爬虫结果的数据结构

{
	"header": {
		"desc": "success",
		"failures": [],
		"oprs": 1,
		"succ": 1,
		"oprtime": 0,
		"quota": 1,
		"rquota": 200000000,
		"status": 0
	},
	"body": {
		"data": [{
			"all": [{
				"bid": 5.22, # 指导价
				"show": 534,  # 预估月均展现
				"click": 83.0,  # 预估月均点击
				"charge": 0.0,  # ?
				"pv": 15842,  # 月均搜索量
				"competition": 1  # 竞争激烈程度
			}, {
				"bid": 1.2,
				"show": 6607,
				"click": 1034.0,
				"charge": 0.0,
				"pv": 190317,
				"competition": 1
			}, {
				"bid": 6.35,
				"show": 54162,
				"click": 7832.0,
				"charge": 0.0,
				"pv": 1129693,
				"competition": 1
			}],
			"pc": [{
				"bid": 5.22,  # 计算机指导价
				"show": 260,
				"click": 35.0,
				"charge": 0.0,
				"pv": 6354,   # 月均计算机搜索量
				"competition": 2
			}, {
				"bid": 0.77,
				"show": 3368,
				"click": 455.0,
				"charge": 0.0,
				"pv": 34995,
				"competition": 2
			}, {
				"bid": 6.35,
				"show": 42538,
				"click": 5750.0,
				"charge": 0.0,
				"pv": 286344,
				"competition": 2
			}],
			"mobile": [{
				"bid": 2.5, # 移动指导价
				"show": 274,
				"click": 48.0,
				"charge": 0.0,
				"pv": 9488,   # 月均移动搜索量
				"competition": 1   # 
			}, {
				"bid": 1.2,
				"show": 3239,
				"click": 579.0,
				"charge": 0.0,
				"pv": 155322,
				"competition": 1
			}, {
				"bid": 4.35,
				"show": 11624,
				"click": 2081.0,
				"charge": 0.0,
				"pv": 843349,
				"competition": 1
			}],
			"words": ["水箱", "奶茶", "二手车"],
			"showReasons": [
				["高频热搜词"],  # 特色
				[],
				["黑马词"]
			]
		}]
	}
}

爬虫运行环境

udm

爬虫部署信息

10.8.6.21 baidu_index   100个进程
10.8.6.23 baidu_planner  5个进程

Taskhub地址

无

Taskhub调度规则说明

无

爬虫监控指标设计

无

爬虫待采集结果目录

/data/baidu_planner
/data/baidu_index

数据归集

责任人

范召贤

数据归集方式

  • 爬虫直接写kafka

  • 爬虫写文件logstash采集

归集后存放目录

无

logstash配置文件名称

无

logstash文件采集type

无

数据归集的topic

无

ES日志索引及筛选条件

无

监控指标看板

数据保留策略

数据清洗

责任人

代码地址

部署地址

部署方法及说明

  • crontab + data_pump
  • supervisor + data_pump
  • supervisor + consumer

数据接收来源

数据存储表地址

  • 数据库地址：
  • 表名：