|
|
# 新增说明20221208
|
|
|
```
|
|
|
日志查看: http://10.8.6.23:5000/2/servers/
|
|
|
目前线上爬虫:
|
|
|
ad_baidu_pc/ad_baidu_h5/ad_360_pc/ad_360_h5/ad_sougou_pc/baidu_pc_live/baidu_h5_live
|
|
|
快速运行本地代码
|
|
|
scrapy_spiders/scrapy.cfg
|
|
|
用default = gravel_spiders.settings_dev配置
|
|
|
scrapy_spiders/test.py 入口启动代码
|
|
|
爬虫任务提交:
|
|
|
10.8.6.23 切换到collie用户
|
|
|
crontab -l查看任务提交
|
|
|
history | grep deploy 查看相关历史命令
|
|
|
相关任务提交的配置文件:
|
|
|
app_search_ads/data_pump/commit_ad_all_task.yml
|
|
|
|
|
|
部署机器: 10.8.6.27
|
|
|
python3 spider_admin.py -a ad_search -n ad_baidu_pc -s deploy
|
|
|
python3 spider_admin.py -a ad_search -n ad_baidu_pc -s list
|
|
|
python3 spider_admin.py -a ad_search -n ad_baidu_pc -s stop
|
|
|
python3 spider_admin.py -a ad_search -n ad_baidu_pc -s start -m 40
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
# **基本信息**
|
|
|
```buildoutcfg
|
|
|
se_platform spider_name 平台名称
|
... | ... | |