Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
project-collie
project-collie
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 5
    • Issues 5
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge requests 2
    • Merge requests 2
  • Operations
    • Operations
    • Incidents
  • Analytics
    • Analytics
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • granite
  • project-collieproject-collie
  • Wiki
    • Data_pump
    • Writers
  • file

file · Changes

Page history
file.FileWriter的切分文件存储功能上线 authored Oct 13, 2020 by fanzx's avatar fanzx
Hide whitespace changes
Inline Side-by-side
Showing with 6 additions and 0 deletions
+6 -0
  • data_pump/writers/file.md data_pump/writers/file.md +6 -0
  • No files found.
data_pump/writers/file.md
View page @ cf1827b9
...@@ -19,6 +19,12 @@ ...@@ -19,6 +19,12 @@
|escaped_sep| | output_form 为csv时, 将字段内容中的分隔符替换为指定的字符,如csv_sep='\t',escaped_sep='\\t'时,将字段内容中的'\t' 替换为'\\t' |escaped_sep| | output_form 为csv时, 将字段内容中的分隔符替换为指定的字符,如csv_sep='\t',escaped_sep='\\t'时,将字段内容中的'\t' 替换为'\\t'
|is_none|None| output_form 为csv时,如果is_none is True, 会将None写为\N, 否则将None写为'' |is_none|None| output_form 为csv时,如果is_none is True, 会将None写为\N, 否则将None写为''
|compress|不压缩| 是否压缩文件,,如果需要压缩,可设为'gz','tar' |compress|不压缩| 是否压缩文件,,如果需要压缩,可设为'gz','tar'
|data_split| False |是否根据data中的某一列值为文件夹名来存储数据,如果需要,设为True
|data_key| None | 按照此列值为文件夹名来存储数据,与data_split参数一起使用,与data_split=True时,这是必填参数
|date_split| False |是否根据日期来存储数据,如果需要,设为True(当data_split,date_split同时启用时,切分效果同时生效,但是data_split优先切分,即先按字段切分再按日期切分)
|data_key| None | 按照此数据中列值为日期单位存储数据,不填写的话,按照系统的时间来存储数据
|custom_time_pattern | None | 按照给的正则表达式提供的格式确定数据最小时间切分单位,比如按照小时存存储,custom_time_pattern='%Y%m%d/%H'
### 设置保存文件的目录 ### 设置保存文件的目录
* path * path
......
Clone repository
  • README
  • data_pump
    • data_pump
    • filters
    • filters
      • bloom
    • readers
    • readers
      • file
      • kafka
      • mongodb
      • sql
    • writers
    • writers
      • file
  • dev_guide
  • dev_manual
  • Home
  • ops
    • ansible
View All Pages