{{/* Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */}}

Key | Default | Type | Description
--- | --- | --- | ---

##### auto-create

false

Boolean

Whether to create underlying storage when reading and writing the table.

##### bucket

1

Integer

Bucket number for file store.

##### bucket-key

(none)

String

Specify the paimon distribution policy. Data is assigned to each bucket according to the hash value of bucket-key.
If you specify multiple fields, delimiter is ','.
If not specified, the primary key will be used; if there is no primary key, the full row will be used.

##### changelog-producer

none

Enum

Whether to double write to a changelog file. This changelog file keeps the details of data changes, it can be read directly during stream reads. This can be applied to tables with primary keys.
Possible values:

- "none": No changelog file.
- "input": Double write to a changelog file when flushing memory table, the changelog is from input.
- "full-compaction": Generate changelog files with each full compaction.
- "lookup": Generate changelog files through 'lookup' before committing the data writing.

##### changelog-producer.row-deduplicate

false

Boolean

Whether to generate -U, +U changelog for the same record. This configuration is only valid for the changelog-producer is lookup or full-compaction.

##### commit.callback.#.param

(none)

String

Parameter string for the constructor of class #. Callback class should parse the parameter by itself.

##### commit.callbacks

(none)

String

A list of commit callback classes to be called after a successful commit. Class names are connected with comma (example: com.test.CallbackA,com.sample.CallbackB).

##### commit.force-compact

false

Boolean

Whether to force a compaction before commit.

##### compaction.max-size-amplification-percent

200

Integer

The size amplification is defined as the amount (in percentage) of additional storage needed to store a single byte of data in the merge tree for changelog mode table.

##### compaction.max.file-num

50

Integer

For file set [f_0,...,f_N], the maximum file number to trigger a compaction for append-only table, even if sum(size(f_i)) < targetFileSize. This value avoids pending too much small files, which slows down the performance.

##### compaction.min.file-num

5

Integer

For file set [f_0,...,f_N], the minimum file number which satisfies sum(size(f_i)) >= targetFileSize to trigger a compaction for append-only table. This value avoids almost-full-file to be compacted, which is not cost-effective.

##### compaction.size-ratio

1

Integer

Percentage flexibility while comparing sorted run size for changelog mode table. If the candidate sorted run(s) size is 1% smaller than the next sorted run's size, then include next sorted run into this candidate set.

##### consumer-id

(none)

String

Consumer id for recording the offset of consumption in the storage.

##### consumer.expiration-time

(none)

Duration

The expiration interval of consumer files. A consumer file will be expired if it's lifetime after last modification is over this value.

##### continuous.discovery-interval

10 s

Duration

The discovery interval of continuous reading.

##### dynamic-bucket.assigner-parallelism

(none)

Integer

Parallelism of assigner operator for dynamic bucket mode, it is related to the number of initialized bucket, too small will lead to insufficient processing speed of assigner.

##### dynamic-bucket.target-row-num

2000000

Long

If the bucket is -1, for primary key table, is dynamic bucket mode, this option controls the target row number for one bucket.

##### dynamic-partition-overwrite

true

Boolean

Whether only overwrite dynamic partition when overwriting a partitioned table with dynamic partition columns. Works only when the table has partition keys.

##### file.compression

(none)

String

Default file compression format, can be overridden by file.compression.per.level

##### file.compression.per.level

Map

Define different compression policies for different level, you can add the conf like this: 'file.compression.per.level' = '0:lz4,1:zlib', for orc file format, the compression value could be NONE, ZLIB, SNAPPY, LZO, LZ4, for parquet file format, the compression value could be UNCOMPRESSED, SNAPPY, GZIP, LZO, BROTLI, LZ4, ZSTD.

##### file.format

orc

Enum

Specify the message format of data files, currently orc, parquet and avro are supported.
Possible values:

- "orc": ORC file format.
- "parquet": Parquet file format.
- "avro": Avro file format.

##### file.format.per.level

Map

Define different file format for different level, you can add the conf like this: 'file.format.per.level' = '0:avro,3:parquet', if the file format for level is not provided, the default format which set by `file.format` will be used.

##### full-compaction.delta-commits

(none)

Integer

Full compaction will be constantly triggered after delta commits.

##### incremental-between

(none)

String

Read incremental changes between start snapshot (exclusive) and end snapshot, for example, '5,10' means changes between snapshot 5 and snapshot 10.

##### incremental-between-timestamp

(none)

String

Read incremental changes between start timestamp (exclusive) and end timestamp, for example, 't1,t2' means changes between timestamp t1 and timestamp t2.

##### local-merge-buffer-size

(none)

MemorySize

Local merge will buffer and merge input records before they're shuffled by bucket and written into sink. The buffer will be flushed when it is full. Mainly to resolve data skew on primary keys. We recommend starting with 64 mb when trying out this feature.

##### local-sort.max-num-file-handles

128

Integer

The maximal fan-in for external merge sort. It limits the number of file handles. If it is too small, may cause intermediate merging. But if it is too large, it will cause too many files opened at the same time, consume memory and lead to random reading.

##### log.changelog-mode

auto

Enum

Specify the log changelog mode for table.
Possible values:

- "auto": Upsert for table with primary key, all for table without primary key.
- "all": The log system stores all changes including UPDATE_BEFORE.
- "upsert": The log system does not store the UPDATE_BEFORE changes, the log consumed job will automatically add the normalized node, relying on the state to generate the required update_before.

##### log.consistency

transactional

Enum

Specify the log consistency mode for table.
Possible values:

- "transactional": Only the data after the checkpoint can be seen by readers, the latency depends on checkpoint interval.
- "eventual": Immediate data visibility, you may see some intermediate states, but eventually the right results will be produced, only works for table with primary key.

##### log.format

"debezium-json"

String

Specify the message format of log system.

##### log.key.format

"json"

String

Specify the key message format of log system with primary key.

##### log.scan.remove-normalize

false

Boolean

Whether to force the removal of the normalize node when streaming read. Note: This is dangerous and is likely to cause data errors if downstream is used to calculate aggregation and the input is not complete changelog.

##### lookup.cache-file-retention

1 h

Duration

The cached files retention time for lookup. After the file expires, if there is a need for access, it will be re-read from the DFS to build an index on the local disk.

##### lookup.cache-max-disk-size

9223372036854775807 bytes

MemorySize

Max disk size for lookup cache, you can use this option to limit the use of local disks.

##### lookup.cache-max-memory-size

256 mb

MemorySize

Max memory size for lookup cache.

##### lookup.hash-load-factor

0.75

Float

The index load factor for lookup.

##### manifest.format

avro

Enum

Specify the message format of manifest files.
Possible values:

- "orc": ORC file format.
- "parquet": Parquet file format.
- "avro": Avro file format.

##### manifest.full-compaction-threshold-size

16 mb

MemorySize

The size threshold for triggering full compaction of manifest.

##### manifest.merge-min-count

30

Integer

To avoid frequent manifest merges, this parameter specifies the minimum number of ManifestFileMeta to merge.

##### manifest.target-file-size

8 mb

MemorySize

Suggested file size of a manifest file.

##### merge-engine

deduplicate

Enum

Specify the merge engine for table with primary key.
Possible values:

- "deduplicate": De-duplicate and keep the last row.
- "partial-update": Partial update non-null fields.
- "aggregation": Aggregate fields with same primary key.
- "first-row": De-duplicate and keep the first row.

##### metadata.stats-mode

"truncate(16)"

String

The mode of metadata stats collection. none, counts, truncate(16), full is available.

- "none": means disable the metadata stats collection.

- "counts" means only collect the null count.

- "full": means collect the null count, min/max value.

- "truncate(16)": means collect the null count, min/max value with truncated length of 16.

- Field level stats mode can be specified by fields.{field_name}.stats-mode

##### metastore.partitioned-table

false

Boolean

Whether to create this table as a partitioned table in metastore. For example, if you want to list all partitions of a Paimon table in Hive, you need to create this table as a partitioned table in Hive metastore. This config option does not affect the default filesystem metastore.

##### num-levels

(none)

Integer

Total level number, for example, there are 3 levels, including 0,1,2 levels.

##### num-sorted-run.compaction-trigger

5

Integer

The sorted run number to trigger compaction. Includes level0 files (one file one sorted run) and high-level runs (one level one sorted run).

##### num-sorted-run.stop-trigger

(none)

Integer

The number of sorted runs that trigger the stopping of writes, the default value is 'num-sorted-run.compaction-trigger' + 1.

##### orc.bloom.filter.columns

(none)

String

A comma-separated list of columns for which to create a bloom filter when writing.

##### orc.bloom.filter.fpp

0.05

Double

Define the default false positive probability for bloom filters.

##### orc.column.encoding.direct

(none)

Integer

Comma-separated list of fields for which dictionary encoding is to be skipped in orc.

##### orc.dictionary.key.threshold

(none)

Integer

If the number of distinct keys in a dictionary is greater than this fraction of the total number of non-null rows, turn off dictionary encoding in orc. Use 1 to always use dictionary encoding.

##### orc.write.batch-size

1024

Integer

write batch size for orc.

##### page-size

64 kb

MemorySize

Memory page size.

##### parquet.enable.dictionary

(none)

Integer

Turn off the dictionary encoding for all fields in parquet.

##### partial-update.ignore-delete

false

Boolean

Whether to ignore delete records in partial-update mode.

##### partition

(none)

String

Define partition by table options, cannot define partition on DDL and table options at the same time.

##### partition.default-name

"__DEFAULT_PARTITION__"

String

The default partition name in case the dynamic partition column value is null/empty string.

##### partition.expiration-check-interval

1 h

Duration

The check interval of partition expiration.

##### partition.expiration-time

(none)

Duration

The expiration interval of a partition. A partition will be expired if it‘s lifetime is over this value. Partition time is extracted from the partition value.

##### partition.timestamp-formatter

(none)

String

The formatter to format timestamp from string. It can be used with 'partition.timestamp-pattern' to create a formatter using the specified value.

- Default formatter is 'yyyy-MM-dd HH:mm:ss' and 'yyyy-MM-dd'.
- Supports multiple partition fields like '$year-$month-$day $hour:00:00'.
- The timestamp-formatter is compatible with Java's DateTimeFormatter.

##### partition.timestamp-pattern

(none)

String

You can specify a pattern to get a timestamp from partitions. The formatter pattern is defined by 'partition.timestamp-formatter'.

- By default, read from the first field.
- If the timestamp in the partition is a single field called 'dt', you can use '$dt'.
- If it is spread across multiple fields for year, month, day, and hour, you can use '$year-$month-$day $hour:00:00'.
- If the timestamp is in fields dt and hour, you can use '$dt $hour:00:00'.

##### primary-key

(none)

String

Define primary key by table options, cannot define primary key on DDL and table options at the same time.

##### read.batch-size

1024

Integer

Read batch size for orc and parquet.

##### scan.bounded.watermark

(none)

Long

End condition "watermark" for bounded streaming mode. Stream reading will end when a larger watermark snapshot is encountered.

##### scan.manifest.parallelism

(none)

Integer

The parallelism of scanning manifest files, default value is the size of cpu processor. Note: Scale-up this parameter will increase memory usage while scanning manifest files. We can consider downsize it when we encounter an out of memory exception while scanning

##### scan.mode

default

Enum

Specify the scanning behavior of the source.
Possible values:

- "default": Determines actual startup mode according to other table properties. If "scan.timestamp-millis" is set the actual startup mode will be "from-timestamp", and if "scan.snapshot-id" or "scan.tag-name" is set the actual startup mode will be "from-snapshot". Otherwise the actual startup mode will be "latest-full".
- "latest-full": For streaming sources, produces the latest snapshot on the table upon first startup, and continue to read the latest changes. For batch sources, just produce the latest snapshot but does not read new changes.
- "full": Deprecated. Same as "latest-full".
- "latest": For streaming sources, continuously reads latest changes without producing a snapshot at the beginning. For batch sources, behaves the same as the "latest-full" startup mode.
- "compacted-full": For streaming sources, produces a snapshot after the latest compaction on the table upon first startup, and continue to read the latest changes. For batch sources, just produce a snapshot after the latest compaction but does not read new changes. Snapshots of full compaction are picked when scheduled full-compaction is enabled.
- "from-timestamp": For streaming sources, continuously reads changes starting from timestamp specified by "scan.timestamp-millis", without producing a snapshot at the beginning. For batch sources, produces a snapshot at timestamp specified by "scan.timestamp-millis" but does not read new changes.
- "from-snapshot": For streaming sources, continuously reads changes starting from snapshot specified by "scan.snapshot-id", without producing a snapshot at the beginning. For batch sources, produces a snapshot specified by "scan.snapshot-id" or "scan.tag-name" but does not read new changes.
- "from-snapshot-full": For streaming sources, produces from snapshot specified by "scan.snapshot-id" on the table upon first startup, and continuously reads changes. For batch sources, produces a snapshot specified by "scan.snapshot-id" but does not read new changes.
- "incremental": Read incremental changes between start and end snapshot or timestamp.

##### scan.plan-sort-partition

false

Boolean

Whether to sort plan files by partition fields, this allows you to read according to the partition order, even if your partition writes are out of order.
It is recommended that you use this for streaming read of the 'append-only' table. By default, streaming read will read the full snapshot first. In order to avoid the disorder reading for partitions, you can open this option.

##### scan.snapshot-id

(none)

Long

Optional snapshot id used in case of "from-snapshot" or "from-snapshot-full" scan mode

##### scan.tag-name

(none)

String

Optional tag name used in case of "from-snapshot" scan mode.

##### scan.timestamp-millis

(none)

Long

Optional timestamp used in case of "from-timestamp" scan mode.

##### sequence.auto-padding

(none)

String

Specify the way of padding precision, if the provided sequence field is used to indicate "time" but doesn't meet the precise.

- You can specific:
- 1. "row-kind-flag": Pads a bit flag to indicate whether it is retract (0) or add (1) message.
- 2. "second-to-micro": Pads the sequence field that indicates time with precision of seconds to micro-second.
- 3. "millis-to-micro": Pads the sequence field that indicates time with precision of milli-second to micro-second.
- 4. Composite pattern: for example, "second-to-micro,row-kind-flag".

##### sequence.field

(none)

String

The field that generates the sequence number for primary key table, the sequence number determines which data is the most recent.

##### sink.watermark-time-zone

"UTC"

String

The time zone to parse the long watermark value to TIMESTAMP value. The default value is 'UTC', which means the watermark is defined on TIMESTAMP column or not defined. If the watermark is defined on TIMESTAMP_LTZ column, the time zone of watermark is user configured time zone, the value should be the user configured local time zone. The option value is either a full name such as 'America/Los_Angeles', or a custom timezone id such as 'GMT-08:00'.

##### snapshot.num-retained.max

2147483647

Integer

The maximum number of completed snapshots to retain. Should be greater than or equal to the minimum number.

##### snapshot.num-retained.min

10

Integer

The minimum number of completed snapshots to retain. Should be greater than or equal to 1.

##### snapshot.time-retained

1 h

Duration

The maximum time of completed snapshots to retain.

##### sort-engine

loser-tree

Enum

Specify the sort engine for table with primary key.
Possible values:

- "min-heap": Use min-heap for multiway sorting.
- "loser-tree": Use loser-tree for multiway sorting. Compared with heapsort, loser-tree has fewer comparisons and is more efficient.

##### sort-spill-buffer-size

64 mb

MemorySize

Amount of data to spill records to disk in spilled sort.

##### sort-spill-threshold

(none)

Integer

If the maximum number of sort readers exceeds this value, a spill will be attempted. This prevents too many readers from consuming too much memory and causing OOM.

##### source.split.open-file-cost

4 mb

MemorySize

Open file cost of a source file. It is used to avoid reading too many files with a source split, which can be very slow.

##### source.split.target-size

128 mb

MemorySize

Target size of a source split when scanning a bucket.

##### streaming-read-mode

(none)

Enum

The mode of streaming read that specifies to read the data of table file or log
Possible values:

- file: Reads from the data of table file store.

- log: Read from the data of table log store.

Possible values:

- "log": Reads from the log store.
- "file": Reads from the file store.

##### streaming-read-overwrite

false

Boolean

Whether to read the changes from overwrite in streaming mode.

##### tag.automatic-creation

none

Enum

Whether to create tag automatically. And how to generate tags.
Possible values:

- "none": No automatically created tags.
- "process-time": Based on the time of the machine, create TAG once the processing time passes period time plus delay.
- "watermark": Based on the watermark of the input, create TAG once the watermark passes period time plus delay.

##### tag.creation-delay

0 ms

Duration

How long is the delay after the period ends before creating a tag. This can allow some late data to enter the Tag.

##### tag.creation-period

daily

Enum

What frequency is used to generate tags.
Possible values:

- "daily": Generate a tag every day.
- "hourly": Generate a tag every hour.
- "two-hours": Generate a tag every two hours.

##### tag.num-retained-max

(none)

Integer

The maximum number of tags to retain.

##### target-file-size

128 mb

MemorySize

Target size of a file.

##### write-buffer-size

256 mb

MemorySize

Amount of data to build up in memory before converting to a sorted on-disk file.

##### write-buffer-spillable

(none)

Boolean

Whether the write buffer can be spillable. Enabled by default when using object storage.

##### write-manifest-cache

0 bytes

MemorySize

Cache size for reading manifest files for write initialization.

##### write-mode

auto

Enum

Specify the write mode for table.
Possible values:

- "auto": The change-log for table with primary key, append-only for table without primary key.
- "append-only": The table can only accept append-only insert operations. Neither data deduplication nor any primary key constraints will be done when inserting rows into paimon.
- "change-log": The table can accept insert/delete/update operations.

##### write-only

false

Boolean

If set to true, compactions and snapshot expiration will be skipped. This option is used along with dedicated compact jobs.