... | @@ -4,6 +4,8 @@ |
... | @@ -4,6 +4,8 @@ |
|
equity_penetration_qcc,通过scrapy部署
|
|
equity_penetration_qcc,通过scrapy部署
|
|
项目名称:project-gravel
|
|
项目名称:project-gravel
|
|
分支:develop_equity_penetration
|
|
分支:develop_equity_penetration
|
|
|
|
|
|
|
|
当前仅运行apph5的非登录爬虫
|
|
```
|
|
```
|
|
|
|
|
|
|
|
|
... | @@ -95,9 +97,11 @@ equity_penetration_qcc_login (登录) |
... | @@ -95,9 +97,11 @@ equity_penetration_qcc_login (登录) |
|
```json
|
|
```json
|
|
# 地域列表任务
|
|
# 地域列表任务
|
|
{"area_code": "AH_340100", "page": "1"}
|
|
{"area_code": "AH_340100", "page": "1"}
|
|
|
|
{"area_code": "AH_340100", "page": "1", "direct_flag": true}
|
|
|
|
|
|
# 搜索列表任务
|
|
# 搜索列表任务
|
|
{"search_key": "北京出国邦出入境服务有限公司"}
|
|
{"search_key": "北京出国邦出入境服务有限公司"}
|
|
|
|
{"search_key": "北京出国邦出入境服务有限公司", "direct_flag": true}
|
|
|
|
|
|
# 公司详情页信息
|
|
# 公司详情页信息
|
|
{"fid": "0727d5d1a4f95d791ff4b7ce5d6e975a"}
|
|
{"fid": "0727d5d1a4f95d791ff4b7ce5d6e975a"}
|
... | @@ -130,6 +134,7 @@ equity_penetration_qcc_login (登录) |
... | @@ -130,6 +134,7 @@ equity_penetration_qcc_login (登录) |
|
+ search_key: 搜索框输入内容
|
|
+ search_key: 搜索框输入内容
|
|
+ fid: QCC企业id
|
|
+ fid: QCC企业id
|
|
+ pid: QCC个人id
|
|
+ pid: QCC个人id
|
|
|
|
+ direct_flag: 直接跳转详情请求(不会生成列表item)
|
|
|
|
|
|
|
|
|
|
## data_type说明
|
|
## data_type说明
|
... | @@ -154,9 +159,9 @@ equity_penetration_qcc_login (登录) |
... | @@ -154,9 +159,9 @@ equity_penetration_qcc_login (登录) |
|
> [列表任务结果](http://tech.pingansec.com/granite/project-gravel/-/tree/develop_equity_penetration/scrapy_spiders/gravel_spiders/spiders/example/list) <br>
|
|
> [列表任务结果](http://tech.pingansec.com/granite/project-gravel/-/tree/develop_equity_penetration/scrapy_spiders/gravel_spiders/spiders/example/list) <br>
|
|
> 分为地域列表,搜索列表,详见data_type说明
|
|
> 分为地域列表,搜索列表,详见data_type说明
|
|
|
|
|
|
> [公司页详情结果](http://tech.pingansec.com/granite/project-gravel/-/tree/develop_equity_penetration/scrapy_spiders/gravel_spiders/spiders/example/company) <br>
|
|
> [公司页详情结果](http://tech.pingansec.com/granite/project-gravel/-/tree/develop_equity_penetration/scrapy_spiders/gravel_spiders/spiders/example/no_login/company) <br>
|
|
|
|
|
|
> [个人页详情结果](http://tech.pingansec.com/granite/project-gravel/-/tree/develop_equity_penetration/scrapy_spiders/gravel_spiders/spiders/example/person) <br>
|
|
> [个人页详情结果](http://tech.pingansec.com/granite/project-gravel/-/tree/develop_equity_penetration/scrapy_spiders/gravel_spiders/spiders/example/login/person) <br>
|
|
|
|
|
|
|
|
|
|
## 爬虫运行环境
|
|
## 爬虫运行环境
|
... | @@ -169,7 +174,7 @@ scrapy |
... | @@ -169,7 +174,7 @@ scrapy |
|
## 爬虫部署信息
|
|
## 爬虫部署信息
|
|
<!--部署在哪些机器?每个机器多少进程?项目名称是什么?-->
|
|
<!--部署在哪些机器?每个机器多少进程?项目名称是什么?-->
|
|
```buildoutcfg
|
|
```buildoutcfg
|
|
target: node_43,node_42,node_32,node_33,node_29
|
|
target: node_43,node_42,node_32,node_33,node_29,node_28
|
|
project: equity_penetration
|
|
project: equity_penetration
|
|
spider_name: equity_penetration_qcc,equity_penetration_qcc_login
|
|
spider_name: equity_penetration_qcc,equity_penetration_qcc_login
|
|
```
|
|
```
|
... | | ... | |