Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
project-collie
project-collie
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 5
    • Issues 5
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge requests 2
    • Merge requests 2
  • Operations
    • Operations
    • Incidents
  • Analytics
    • Analytics
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • granite
  • project-collieproject-collie
  • Wiki
    • Data_pump
    • Readers
  • file

file · Changes

Page history
增加数据类型说明 authored Apr 29, 2020 by 宋志鹏's avatar 宋志鹏
Hide whitespace changes
Inline Side-by-side
Showing with 3 additions and 0 deletions
+3 -0
  • data_pump/readers/file.md data_pump/readers/file.md +3 -0
  • No files found.
data_pump/readers/file.md
View page @ 2a53aa41
......@@ -11,7 +11,10 @@ company_name: # 名称(自定义)
path: "hdfs://hdp-nn-001:8020/user/data/digest_company_name/"
formater: company_name_digest
pattern: "*.gz"
offset:
path: "/home/collie/product/offset_store"
```
* **path**:文件路径,支持hdfs、ftp和本地文件
* **formater**: 格式化处理器
* **pattern**: 文件名匹配模式。比如```*.py```匹配.py结尾的文件,```*.gz```匹配.gz格式压缩文件
* **offset**: 读取起始的偏移量。如果读取本地文件,将offset设为True, offset文件放在path(init)目录下面。如果读取集群文件,要指定offset文件的存放目录,如示例所示。如果读取的是单个文件,一定要指定offset文件的存放目录。offset文件的命名格式为'.{offset}.db'.format(offset=name), 如示例,则offset文件的路径为:/home/collie/product/offset_store/.company_name.db
Clone repository
  • README
  • data_pump
    • data_pump
    • filters
    • filters
      • bloom
    • readers
    • readers
      • file
      • kafka
      • mongodb
      • sql
    • writers
    • writers
      • file
  • dev_guide
  • dev_manual
  • Home
  • ops
    • ansible
View All Pages