... | ... | @@ -211,6 +211,9 @@ task_result=8000 # 参数错误 |
|
|
# **数据归集**
|
|
|
|
|
|
## 责任人
|
|
|
```html
|
|
|
范召贤
|
|
|
```
|
|
|
|
|
|
## 数据归集方式
|
|
|
|
... | ... | @@ -219,22 +222,41 @@ task_result=8000 # 参数错误 |
|
|
- [x] 爬虫写文件logstash采集
|
|
|
|
|
|
## 爬虫结果目录
|
|
|
|
|
|
```html
|
|
|
/data/gravel_spiders/public_company_eastmoney
|
|
|
/data/gravel_spiders/public_company_notice
|
|
|
/data/gravel_spiders/public_company_jqka
|
|
|
```
|
|
|
|
|
|
## 归集后存放目录
|
|
|
|
|
|
```html
|
|
|
/data2_227/grvael_spider_result/public_company_eastmoney
|
|
|
/data2_227/grvael_spider_result/public_company_jqka
|
|
|
/data2_227/grvael_spider_result/public_company_notice
|
|
|
```
|
|
|
|
|
|
## logstash配置文件名称
|
|
|
|
|
|
```html
|
|
|
project-deploy/logstash/10.8.6.246/conf.d/collie_spider_data_to_kfk.conf(入topic)
|
|
|
project-deploy/logstash/10.8.6.229/conf.d/grvael_spider_to_es.conf(入es)
|
|
|
```
|
|
|
|
|
|
## logstash文件采集type
|
|
|
|
|
|
```html
|
|
|
type=>"eastmoney-spider-data"
|
|
|
type=>"eastmoney-notice-spider-data"
|
|
|
type=>"jqka-spider-data"
|
|
|
```
|
|
|
|
|
|
## 数据归集的topic
|
|
|
|
|
|
```html
|
|
|
topic_id => "public-company-spider-data"
|
|
|
```
|
|
|
|
|
|
## ES日志索引及筛选条件
|
|
|
|
|
|
```html
|
|
|
index => "public-company-spider-data-%{log_date}"
|
|
|
```
|
|
|
|
|
|
## 监控指标看板
|
|
|
|
... | ... | |