Skip to content

GitLab

Last edited by yuanbo
Page history
This is an old version of this page. You can view the most recent version or browse the history.

search_ads

基本信息 

se_platform
baidu_pc 百度搜索
baidu_h5 百度移动
360_pc 360搜索
360_h5 360移动
sougou_pc 搜狗搜索
sougou_h5 搜狗移动
shenma 神马搜索

数据名称（中文）

搜索广告

数据英文名称

search_ads

采集网站（采集入口）

浏览器端h5入口:
https://www.baidu.com/from=844b/s?word=python&ts=8478142&t_kt=0&ie=utf-8&fm_kl=021394be2f&rsv_iqid=2183853714&rsv_t=376a8VEY8%252FDOKsDMkKlCpa2B%252FASIiibUR4bHEz4HRUJlSfO8rYgnUcCdQw&sa=ih_2&ms=1&rsv_pq=2183853714&rsv_sug4=2159&ss=011&tj=1

采集频率及采集策略

存量更新策略

db_host: bdp-rds-103.mysql.rds.aliyuncs.com
db_name: aicha_ad
db_user: aicha
db_password: aicha@0611
se_word数据表中任务作为初始任务
逐条更新
目前2周更新一轮

增量采集策略

暂无

爬虫 

百度移动端 ad_baidu_h5

责任人

袁波

爬虫名称

ad_baidu_h5

代码地址

项目地址:http://192.168.109.199/granite/project-gravel/tree/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders

百度移动端: 
项目入口脚本: http://192.168.109.199/granite/project-gravel/blob/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/ad_baidu_h5.py
代码具体实现板块: http://192.168.109.199/granite/project-gravel/tree/develop_app_10jqka_20210121/scrapy_spiders/gravel_spiders/spiders/search_ads_reqs/baidu

队列名称及队列地址

优先级队列说明

  • ad_baidu_h5

任务来源

db_host: bdp-rds-103.mysql.rds.aliyuncs.com
db_name: aicha_ad
db_user: aicha
db_password: aicha@0611
se_word数据表中任务作为初始任务

任务输入参数（样例）

{
    "se_platform": "baidu_h5", 
    "word": "1200目普通硅微粉", 
    "area": "全国"
}

任务样例

{
    "se_platform": "baidu_h5", 
    "word": "1200目普通硅微粉", 
    "area": "全国", 
    "outbound": "ad_all_platform", 
    "routed_count": 1, 
    "submitter": "taskhub", 
    "group_retry_times": 0, 
    "submit_time": "2021-04-15 15:11:02", 
    "token_scope": "ad", 
    "retry_limits": 2, 
    "rt": false, 
    "priority": null, 
    "task_uuid": "c1577311-f58b-4235-8778-ce08d54df889", 
    "retry_times": 0
}

任务参数说明

{
    "se_platform": "baidu_h5", # 平台名称
    "word": "1200目普通硅微粉",  # 搜索关键词
    "area": "全国"              # 广告范围
}

data_type说明

list: 暂未使用
detail: 暂未使用

爬虫结果的超级数据

{
    "data": [
        {
            "find_time": "2021-04-14 21:40:46.358", 
            "match_word": "牡丹江,招聘信息,招聘,招聘,招聘信息,人才招聘,职位信息", 
            "land_url": "https://mkt.51job.com/tg/sem/lp2020_H5_jc/index.php?partner=sem_baidu20_126156", 
            "ad_content": "【前程无忧51job】高薪招聘信息,专业人才招聘网站,海量职位信息,快速投递简历.手机轻松找工作!", 
            "title": "牡丹江_招聘信息_热门招聘_高薪招聘【51Job】招聘网"
        }
    ], 
    "http_code": 200, 
    "error_msg": "", 
    "task_result": 1000, 
    "data_type": "detail", 
    "spider_start_time": "2021-04-14 21:42:39.901", 
    "spider_end_time": "2021-04-14 21:42:41.688", 
    "task_params": {
        "se_platform": "baidu_h5", 
        "word": "0453牡丹江最新招聘信息", 
        "area": "全国", 
        "outbound": "ad_all_platform", 
        "routed_count": 1, 
        "submitter": "taskhub", 
        "group_retry_times": 0, 
        "submit_time": "2021-04-14 21:19:45", 
        "token_scope": "ad", 
        "retry_limits": 2, 
        "rt": false, 
        "priority": null, 
        "task_uuid": "a2bd1b18-e671-4b77-81ee-3e636372f385", 
        "retry_times": 0
    }, 
    "metadata": { }, 
    "spider_name": "ad_baidu_h5", 
    "spider_ip": "192.168.108.155"
}

实际爬虫结果的数据结构

baidu_h5
{
    "data": [
        {
            "find_time": "2021-04-14 21:40:46.358", 
            "match_word": "牡丹江,招聘信息,招聘,招聘,招聘信息,人才招聘,职位信息", 
            "land_url": "https://mkt.51job.com/tg/sem/lp2020_H5_jc/index.php?partner=sem_baidu20_126156", 
            "ad_content": "【前程无忧51job】高薪招聘信息,专业人才招聘网站,海量职位信息,快速投递简历.手机轻松找工作!", 
            "title": "牡丹江_招聘信息_热门招聘_高薪招聘【51Job】招聘网"
        }
    ], 
    "http_code": 200, 
    "error_msg": "", 
    "task_result": 1000, 
    "data_type": "detail", 
    "spider_start_time": "2021-04-14 21:42:39.901", 
    "spider_end_time": "2021-04-14 21:42:41.688", 
    "task_params": {
        "se_platform": "baidu_h5", 
        "word": "0453牡丹江最新招聘信息", 
        "area": "全国", 
        "outbound": "ad_all_platform", 
        "routed_count": 1, 
        "submitter": "taskhub", 
        "group_retry_times": 0, 
        "submit_time": "2021-04-14 21:19:45", 
        "token_scope": "ad", 
        "retry_limits": 2, 
        "rt": false, 
        "priority": null, 
        "task_uuid": "a2bd1b18-e671-4b77-81ee-3e636372f385", 
        "retry_times": 0
    }, 
    "metadata": { }, 
    "spider_name": "ad_baidu_h5", 
    "spider_ip": "192.168.108.155"
}

爬虫运行环境

scrapy

爬虫部署信息

baidu_h5: 10.8.6.57 10个进程

Taskhub地址

提交任务地址: http://10.8.6.222:18518/task/
代码编写地址: http://192.168.109.199/granite/project-taskhub/blob/master/taskhub/config/ad/config_ad.d/01_task.yaml

Taskhub调度规则说明

task_result=1000    # 正常获取到详情任务
task_result=9101    # 超时错误，需要进行重试，目前重试5次
task_result=8000    # 参数错误

爬虫监控指标设计

(先观察，待补充)
索引: 
监控频率: 
监控起止时间: 
报警条件: 
报警群:  
报警内容:

数据归集

责任人

数据归集方式

  • 爬虫直接写kafka

  • 爬虫写文件logstash采集

爬虫结果目录

归集后存放目录

logstash配置文件名称

logstash文件采集type

数据归集的topic

ES日志索引及筛选条件

监控指标看板

数据保留策略

数据清洗

责任人

代码地址

部署地址

部署方法及说明

  • crontab + data_pump
  • supervisor + data_pump
  • supervisor + consumer

数据接收来源

数据存储表地址

  • 数据库地址：
  • 表名：