| ... | ... | @@ -301,7 +301,7 @@ task_result=8000 # 参数错误 |
|
|
|
|
|
|
|
- [ ] 爬虫直接写kafka
|
|
|
|
|
|
|
|
- [ ] 爬虫写文件logstash采集
|
|
|
|
- [X] 爬虫写文件logstash采集
|
|
|
|
|
|
|
|
## 爬虫结果目录
|
|
|
|
|
| ... | ... | @@ -312,25 +312,25 @@ task_result=8000 # 参数错误 |
|
|
|
## 归集后存放目录
|
|
|
|
|
|
|
|
```html
|
|
|
|
|
|
|
|
/data2_227/grvael_spider_result/wenshu_spider
|
|
|
|
```
|
|
|
|
|
|
|
|
## logstash配置文件名称
|
|
|
|
|
|
|
|
```html
|
|
|
|
|
|
|
|
project-deploy/logstash/10.8.6.246/conf.d/collie_spider_data_to_kfk.conf
|
|
|
|
```
|
|
|
|
|
|
|
|
## logstash文件采集type
|
|
|
|
|
|
|
|
```html
|
|
|
|
|
|
|
|
wenshu_spider
|
|
|
|
```
|
|
|
|
|
|
|
|
## 数据归集的topic
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
general-taxpayer
|
|
|
|
```
|
|
|
|
|
|
|
|
## ES日志索引及筛选条件
|
| ... | ... | |
