Skip to content

GitLab

Last edited by yuanbo
Page history

ad_baidu_v

基本信息 

baidu_v

数据名称（中文）

百度v

数据英文名称

baidu_v

采集网站（采集入口）

https://v123.baidu.com/normal/?from=zixinAId&cn=云南石林旅游集团有限公司

采集频率及采集策略

每日持续采集(目前不确定crontab在哪台机器上，任务来源不清楚)

爬虫名称以及平台 

baidu_v   百度v

责任人

袁波

爬虫名称

baidu_v

代码地址

项目地址:http://192.168.109.110/granite/project-collie-app/-/tree/baiduv_20211124/app_1688/spiders/b2b/b2b/spiders/baidu_v

队列名称及队列地址

优先级队列说明

任务来源

crontab对应机器: 10.8.6.21
*/1 * * * *  cd /home/collie/product/app_ad/jobs/data_pump && /home/collie/.conda/envs/collie/bin/python data_pump.py pump-data -c ../../data_pump/commit_baiduv_person_task.yml > /dev/null 2>&1
*/1 * * * *  cd /home/collie/product/app_ad/jobs/data_pump && /home/collie/.conda/envs/collie/bin/python data_pump.py pump-data -c ../../data_pump/commit_baiduv_company_task.yml > /dev/null 2>&1

http://192.168.109.110/granite/project-collie-app/-/blob/baiduv_20211124/app_ad/data_pump/commit_baiduv_company_task.yml
http://192.168.109.110/granite/project-collie-app/-/blob/baiduv_20211124/app_ad/data_pump/commit_baiduv_person_task.yml

任务输入参数（样例）

任务样例

{
	"platform_name": "baiduv",
	"company_name": "云南石林旅游集团有限公司",
	"data_type": "company"
}

任务参数说明

data_type: company
data_type: person

data_type说明

data_type: company
      db:
        host: bdp-rds-103.mysql.rds.aliyuncs.com
        port: 3306
        database: taskhub
        user: taskhub_ro
        password: thro@0818

      query:
        table: task_scheduler_ic
        limit: 2500
        columns:
          - task_id
          - task_params


data_type: person
      db:
        host: bdp-rds-103.mysql.rds.aliyuncs.com
        port: 3306
        database: taskhub
        user: taskhub_ro
        password: thro@0818

      query:
        table: task_scheduler_iich
        limit: 2500
        columns:
          - task_id
          - task_params

爬虫结果的超级数据

实际爬虫结果的数据结构

{
	"spider_name": "baidu_v",
	"task_params": {
		"company_name": "上海衡艳机械设备租赁有限公司",
		"platform_name": "baiduv",
		"data_type": "company",
		"outbound": "baiduv",
		"routed_count": 1,
		"submitter": "taskhub",
		"group_retry_times": 0,
		"submit_time": "2021-11-24 16:31:05",
		"token_scope": "baiduv",
		"retry_limits": 5,
		"rt": false,
		"priority": null,
		"task_uuid": "b9b1be9b-1174-43a9-951c-c761f62ea06f",
		"retry_times": 0
	},
	"platform_name": "baiduv",
	"found_time": "2021-11-24 16:31:19",
	"http_code": 200,
	"data": {
		"is_baidu_v": 1,
		"top_domain": "qianhu.wejianzhan.com",
		"sponsor": "上海衡艳机械设备租赁有限公司",
		"detail": {
			"name": "上海衡艳机械设备租赁有限公司",
			"is_v": "true",
			"x1": "false",
			"x2": "false",
			"x3": "false",
			"x4": "false",
			"x5": "false",
			"link_info": {}
		},
		"ctime": "2021-11-24 16:31:19",
		"mtime": "2021-11-24 16:31:19",
		"x1": 0,
		"x2": 0,
		"x3": 0,
		"x4": 0,
		"x5": 0,
		"contact": 0
	},
	"task_result": 1000
}

爬虫运行环境

scrapy

爬虫部署信息

10.8.6.57 20个进程

Taskhub地址

http://192.168.109.110/granite/project-taskhub/-/blob/master/taskhub/config/ad/config_ad.d/01_task.yaml

Taskhub调度规则说明

outbound:
  - name: 'baiduv'
    selector:
      - "platform_name == 'baiduv'"
    retry_limits: 5
    token_per_second: 4
    class: 'redis_queue.RedisListOutbound'
    token_scope: 'baiduv'
    init:
      url: '{redis_url}'
      key: "baiduv_company_name"
      task_params:
        - company_name
        - platform_name
        - data_type
      task_params_wrapped: False

爬虫监控指标设计

爬虫待采集结果目录

/data/b2b_spider_data/baidu_v

数据归集

责任人

范召贤

数据归集方式

  • 爬虫直接写kafka

  • 爬虫写文件logstash采集

归集后存放目录

/data2/ad_spider_result/baidu_v

logstash配置文件名称

logstash文件采集type

数据归集的topic

ES日志索引及筛选条件

无

监控指标看板

数据保留策略

数据清洗

责任人

代码地址

部署地址

部署方法及说明

  • crontab + data_pump
  • supervisor + data_pump
  • supervisor + consumer

数据接收来源

数据存储表地址

  • 数据库地址：
  • 表名：