Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
project-collie
project-collie
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 5
    • Issues 5
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge requests 2
    • Merge requests 2
  • Operations
    • Operations
    • Incidents
  • Analytics
    • Analytics
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • granite
  • project-collieproject-collie
  • Wiki
    • Data_pump
    • Readers
  • sql

sql · Changes

Page history
创建 authored Apr 23, 2020 by 吴一博's avatar 吴一博
Hide whitespace changes
Inline Side-by-side
Showing with 44 additions and 0 deletions
+44 -0
  • data_pump/readers/sql.md data_pump/readers/sql.md +44 -0
  • No files found.
data_pump/readers/sql.md 0 → 100644
View page @ cd37d5dd
> 从mysql数据库中读取
**class**参数配置为```sql.SqlDocReader```
示例:
```yaml
sql_qichacha_push:
class: sql.SqlDocReader
init:
db:
host: 192.168.109.220
port: 3306
database: collie
user: collie
password: eill
query:
table:
a: company_lawsuit
b: company_lawsuit_parsed_info
columns:
a: "*"
b:
- id as bid
- jd_case_reason
- plaintiffs
- defendants
- thirdParties
condition: a.uuid = b.lawsuit_uuid
offset:
field: id
store: "file:///home/collie/project-collie/offset_company_lawsuit.txt"
```
* **db**: mysql数据库连接配置
* **query**: 查询条件,支持直接完整sql语句、指定表名和列名两种方式
1. 完整sql语句
* **query_sql**: select查询语句,其中where子句必须包含**$CONDITIONS**关键字
2. 指定表名和列名
* **query.table**: 数据表,支持单表,多表join查询
* **query.columns**: select查询的列名列表,逗号(,)分割。默认是所有列
* **query.condition**: select查询的where条件
* **offset**: 记录上次查询结束时记录在表中的offset
1. **offset.field**: 记录offset的字段名称
2. **offset.store**: offset保存的路径,支持本地文件和mysql数据库,前缀分别是```file://```、```mysql://```
Clone repository
  • README
  • data_pump
    • data_pump
    • filters
    • filters
      • bloom
    • readers
    • readers
      • file
      • kafka
      • mongodb
      • sql
    • writers
    • writers
      • file
  • dev_guide
  • dev_manual
  • Home
  • ops
    • ansible
View All Pages