java/tools/README.md - tsfile - Git at Google

 <!--

     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at

         http://www.apache.org/licenses/LICENSE-2.0

     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.

 -->

 [English](./README.md) | [Chinese](./README-zh.md)
 # TsFile Tools Manual
 ## Introduction

 ## Development

 ### Prerequisites

 To build the Java version of TsFile Tools, you must have the following dependencies installed:

 1. Java >= 1.8 (1.8, 11 to 17 are verified. Make sure the environment variable is set).
 2. Maven >= 3.6 (if you are compiling TsFile from source).

 ### Build with Maven

 ```sh
 mvn clean package -P with-java -DskipTests
 ```

 ### Install to local machine

 ```
 mvn install -P with-java -DskipTests
 ```

 ## Hybrid CSV Import

 Combine one main time-series CSV (with a real time column) and multiple supplement CSVs (same TAG/FIELD columns, **no** time column) into a **single** TsFile. Supplement rows receive synthetic timestamps `1, 2, …, N` per file; each file is isolated with a virtual TAG `batch_id` (one ChunkGroup per file per business TAG combination).

 Example config (`hybrid.conf`):

 ```
 output_tsfile=combined.tsfile
 shared_schema=main.schema
 main_csv=timeseries.csv
 main_batch_id=main
 batch_id_tag=batch_id
 validate_uniform_tags=true
 supplement_sort_by_variance=true
 supplement_csv=experiment_1.csv
 supplement_batch_id=experiment_1
 supplement_csv=experiment_2.csv
 supplement_batch_id=experiment_2
 ```

 Run:

 ```sh
 java -jar tsfile-tools.jar --hybrid_config hybrid.conf
 ```

 Supplement CSV headers must list all business TAG and FIELD columns from `shared_schema`, excluding the time column (e.g. `Region,DeviceId,Temperature,Pressure`).

 For each supplement CSV separately (`supplement_sort_by_variance=true` by default):

 1. Compute variance of each **FIELD** column **within that CSV only**.
 2. Order columns by variance descending (higher variance = higher sort priority).
 3. Sort rows in that CSV ascending (multi-key comparator).
 4. Write one ChunkGroup per CSV; timestamps are **consecutive** inside the group (`startId`, `startId+1`, …).
 5. The next supplement CSV continues ids from `maxId + 1` (file1: `1..n1`, file2: `n1+1..n1+n2`, …).

 Programmatic API: `HybridCsvTsFileAssembler.execute(HybridImportConfig)`.

 ## Schema Definition

 ### Parameters

 | Parameter | Description | Required | Default |
 |-----------|------------|----------|---------|
 | table_name | Table name | Yes | |
 | time_precision | Time precision (ms / us / ns / s) | No | ms |
 | has_header | Whether CSV contains a header (true / false). Ignored for Parquet / Arrow. | No | true |
 | separator | CSV delimiter (, / tab / ;). Ignored for Parquet / Arrow. | No | , |
 | null_format | String value treated as null in CSV. Ignored for Parquet / Arrow (native null). | No | |
 | tag_columns | Tag columns (device identifiers / primary key). Supports virtual columns with DEFAULT value. | No | |
 | time_column | Time column name | Yes | |
 | source_columns | Column definitions mapping to source file columns | Yes | |

 > **Backward compatibility**: `id_columns` and `csv_columns` are still accepted as aliases for `tag_columns` and `source_columns`.

 ### Column Concepts

 - **time_column**: Exactly one per table. Written as `time` column with type `TIMESTAMP` in TsFile.
 - **tag_columns**: Device identifiers (composite primary key), 0 or more. Supports virtual columns not present in the source file via `DEFAULT` keyword.
 - **source_columns**: Maps every column in the source file by position (CSV) or by name (Parquet / Arrow). Use `SKIP` to ignore a column.
 - **FIELD** (derived, not configured): All columns in `source_columns` that are not `time_column`, not in `tag_columns`, and not `SKIP`. These are the measurement columns whose values change over time.

 ### Schema Example

 CSV file content:
 ```
 Region,FactoryNumber,DeviceNumber,Model,MaintenanceCycle,Time,Temperature,Emission
 hebei,1001,1,10,1,1,80.0,1000.0
 hebei,1001,1,10,1,4,80.0,1000.0
 hebei,1002,7,5,2,1,90.0,1200.0
 ```

 Schema file (`import.schema`):
 ```
 table_name=root.db1
 time_precision=ms
 has_header=true
 separator=,
 null_format=\N

 tag_columns
 Group DEFAULT Datang
 Region
 FactoryNumber
 DeviceNumber

 time_column=Time

 source_columns
 Region TEXT,
 FactoryNumber TEXT,
 DeviceNumber TEXT,
 SKIP,
 SKIP,
 Time INT64,
 Temperature FLOAT,
 Emission DOUBLE,
 ```

 In this example:
 - `Group` is a virtual tag column (not in CSV) with default value `Datang`
 - `Region`, `FactoryNumber`, `DeviceNumber` are tag columns read from CSV
 - `Model` and `MaintenanceCycle` are skipped via `SKIP`
 - `Temperature` and `Emission` are automatically derived as FIELD columns

 For Parquet / Arrow in schema mode, `source_columns` matches by column **name** instead of position. Named SKIP is also supported:
 ```
 source_columns
 Time INT64,
 unused_col SKIP,
 Temperature FLOAT,
 Emission DOUBLE,
 ```

 ## CLI Parameters

 | Parameter | Description | Required | Default |
 |-----------|------------|----------|---------|
 | -s, --source | Input file or directory | Yes | |
 | -t, --target | Output directory | Yes | |
 | --schema | Schema file path. Omit for auto mode. | No | |
 | --fail_dir | Directory for failed source files | No | failed |
 | --format | Source format: csv / parquet / arrow. Auto-detected by file extension if omitted. | No | auto-detect |
 | --table_name | Table name override (auto mode) | No | derived from filename |
 | --time_precision | Time precision override (auto mode): ms / us / ns / s | No | ms |
 | --separator | CSV delimiter (auto mode): , / tab / ; | No | , |
 | -b, --block_size | CSV chunk size (e.g. 256M, 1G) | No | 256M |
 | -tn, --thread_num | Thread count for parallel processing | No | 8 |

 ## Modes

 ### Schema Mode

 Provide a `--schema` file to explicitly define column mapping, types, tags, and time column.

 ```sh
 # CSV
 csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed --schema ./schema/import.schema
 csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed --schema .\schema\import.schema

 # Parquet
 parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed --schema ./schema/import.schema
 parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed --schema .\schema\import.schema

 # Arrow
 arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed --schema ./schema/import.schema
 arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed --schema .\schema\import.schema
 ```

 ### Auto Mode

 Omit `--schema` to automatically infer column types and detect the time column.

 **Auto mode rules:**
 - Time column: must be named exactly `time` or `TIME` (case-sensitive, strict match)
 - All other columns become FIELD (no tag inference)
 - CSV type inference uses a 100-row sampling window. Promotion rules: INT64 and DOUBLE promote to DOUBLE; any other mixed pair (including BOOLEAN with numeric) promotes to STRING.
 - Parquet / Arrow use native schema types directly
 - Default table name: derived from source filename (e.g. `sensor.csv` → table `sensor`)
 - Default null tokens (CSV only): empty cell and `\N`

 **Auto mode example:**

 CSV file (`sensor.csv`):
 ```
 time,temperature,humidity,status
 1000,25.5,60.0,true
 2000,26.1,55.3,false
 3000,27.0,58.1,true
 ```

 Auto mode infers:
 ```
 table name:  sensor        (from filename)
 time column: time
 fields:      temperature DOUBLE, humidity DOUBLE, status BOOLEAN
 tags:        (none)
 ```

 **Commands:**
 ```sh
 # CSV
 csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed
 csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed

 # CSV with options
 csv2tsfile.sh --source ./data/csv --target ./output --table_name my_table --separator tab --time_precision us

 # Parquet
 parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed
 parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed

 # Arrow (.arrow / .ipc / .feather)
 arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed
 arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed
 ```

 ### Output File Naming

 - Single batch: `{source_basename}.tsfile`
 - Multiple batches: `{source_basename}_1.tsfile`, `{source_basename}_2.tsfile`, ...
 - Table name and output filename are independent — table name comes from schema or `--table_name`, filename comes from source file.
	<!--

	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.

	-->

	[English](./README.md) \| [Chinese](./README-zh.md)
	# TsFile Tools Manual
	## Introduction

	## Development

	### Prerequisites

	To build the Java version of TsFile Tools, you must have the following dependencies installed:

	1. Java >= 1.8 (1.8, 11 to 17 are verified. Make sure the environment variable is set).
	2. Maven >= 3.6 (if you are compiling TsFile from source).

	### Build with Maven

	```sh
	mvn clean package -P with-java -DskipTests
	```

	### Install to local machine

	```
	mvn install -P with-java -DskipTests
	```

	## Hybrid CSV Import

	Combine one main time-series CSV (with a real time column) and multiple supplement CSVs (same TAG/FIELD columns, no time column) into a single TsFile. Supplement rows receive synthetic timestamps `1, 2, …, N` per file; each file is isolated with a virtual TAG `batch_id` (one ChunkGroup per file per business TAG combination).

	Example config (`hybrid.conf`):

	```
	output_tsfile=combined.tsfile
	shared_schema=main.schema
	main_csv=timeseries.csv
	main_batch_id=main
	batch_id_tag=batch_id
	validate_uniform_tags=true
	supplement_sort_by_variance=true
	supplement_csv=experiment_1.csv
	supplement_batch_id=experiment_1
	supplement_csv=experiment_2.csv
	supplement_batch_id=experiment_2
	```

	Run:

	```sh
	java -jar tsfile-tools.jar --hybrid_config hybrid.conf
	```

	Supplement CSV headers must list all business TAG and FIELD columns from `shared_schema`, excluding the time column (e.g. `Region,DeviceId,Temperature,Pressure`).

	For each supplement CSV separately (`supplement_sort_by_variance=true` by default):

	1. Compute variance of each FIELD column within that CSV only.
	2. Order columns by variance descending (higher variance = higher sort priority).
	3. Sort rows in that CSV ascending (multi-key comparator).
	4. Write one ChunkGroup per CSV; timestamps are consecutive inside the group (`startId`, `startId+1`, …).
	5. The next supplement CSV continues ids from `maxId + 1` (file1: `1..n1`, file2: `n1+1..n1+n2`, …).

	Programmatic API: `HybridCsvTsFileAssembler.execute(HybridImportConfig)`.

	## Schema Definition

	### Parameters

	\| Parameter \| Description \| Required \| Default \|
	\|-----------\|------------\|----------\|---------\|
	\| table_name \| Table name \| Yes \| \|
	\| time_precision \| Time precision (ms / us / ns / s) \| No \| ms \|
	\| has_header \| Whether CSV contains a header (true / false). Ignored for Parquet / Arrow. \| No \| true \|
	\| separator \| CSV delimiter (, / tab / ;). Ignored for Parquet / Arrow. \| No \| , \|
	\| null_format \| String value treated as null in CSV. Ignored for Parquet / Arrow (native null). \| No \| \|
	\| tag_columns \| Tag columns (device identifiers / primary key). Supports virtual columns with DEFAULT value. \| No \| \|
	\| time_column \| Time column name \| Yes \| \|
	\| source_columns \| Column definitions mapping to source file columns \| Yes \| \|

	> Backward compatibility: `id_columns` and `csv_columns` are still accepted as aliases for `tag_columns` and `source_columns`.

	### Column Concepts

	- time_column: Exactly one per table. Written as `time` column with type `TIMESTAMP` in TsFile.
	- tag_columns: Device identifiers (composite primary key), 0 or more. Supports virtual columns not present in the source file via `DEFAULT` keyword.
	- source_columns: Maps every column in the source file by position (CSV) or by name (Parquet / Arrow). Use `SKIP` to ignore a column.
	- FIELD (derived, not configured): All columns in `source_columns` that are not `time_column`, not in `tag_columns`, and not `SKIP`. These are the measurement columns whose values change over time.

	### Schema Example

	CSV file content:
	```
	Region,FactoryNumber,DeviceNumber,Model,MaintenanceCycle,Time,Temperature,Emission
	hebei,1001,1,10,1,1,80.0,1000.0
	hebei,1001,1,10,1,4,80.0,1000.0
	hebei,1002,7,5,2,1,90.0,1200.0
	```

	Schema file (`import.schema`):
	```
	table_name=root.db1
	time_precision=ms
	has_header=true
	separator=,
	null_format=\N

	tag_columns
	Group DEFAULT Datang
	Region
	FactoryNumber
	DeviceNumber

	time_column=Time

	source_columns
	Region TEXT,
	FactoryNumber TEXT,
	DeviceNumber TEXT,
	SKIP,
	SKIP,
	Time INT64,
	Temperature FLOAT,
	Emission DOUBLE,
	```

	In this example:
	- `Group` is a virtual tag column (not in CSV) with default value `Datang`
	- `Region`, `FactoryNumber`, `DeviceNumber` are tag columns read from CSV
	- `Model` and `MaintenanceCycle` are skipped via `SKIP`
	- `Temperature` and `Emission` are automatically derived as FIELD columns

	For Parquet / Arrow in schema mode, `source_columns` matches by column name instead of position. Named SKIP is also supported:
	```
	source_columns
	Time INT64,
	unused_col SKIP,
	Temperature FLOAT,
	Emission DOUBLE,
	```

	## CLI Parameters

	\| Parameter \| Description \| Required \| Default \|
	\|-----------\|------------\|----------\|---------\|
	\| -s, --source \| Input file or directory \| Yes \| \|
	\| -t, --target \| Output directory \| Yes \| \|
	\| --schema \| Schema file path. Omit for auto mode. \| No \| \|
	\| --fail_dir \| Directory for failed source files \| No \| failed \|
	\| --format \| Source format: csv / parquet / arrow. Auto-detected by file extension if omitted. \| No \| auto-detect \|
	\| --table_name \| Table name override (auto mode) \| No \| derived from filename \|
	\| --time_precision \| Time precision override (auto mode): ms / us / ns / s \| No \| ms \|
	\| --separator \| CSV delimiter (auto mode): , / tab / ; \| No \| , \|
	\| -b, --block_size \| CSV chunk size (e.g. 256M, 1G) \| No \| 256M \|
	\| -tn, --thread_num \| Thread count for parallel processing \| No \| 8 \|

	## Modes

	### Schema Mode

	Provide a `--schema` file to explicitly define column mapping, types, tags, and time column.

	```sh
	# CSV
	csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed --schema ./schema/import.schema
	csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed --schema .\schema\import.schema

	# Parquet
	parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed --schema ./schema/import.schema
	parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed --schema .\schema\import.schema

	# Arrow
	arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed --schema ./schema/import.schema
	arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed --schema .\schema\import.schema
	```

	### Auto Mode

	Omit `--schema` to automatically infer column types and detect the time column.

	Auto mode rules:
	- Time column: must be named exactly `time` or `TIME` (case-sensitive, strict match)
	- All other columns become FIELD (no tag inference)
	- CSV type inference uses a 100-row sampling window. Promotion rules: INT64 and DOUBLE promote to DOUBLE; any other mixed pair (including BOOLEAN with numeric) promotes to STRING.
	- Parquet / Arrow use native schema types directly
	- Default table name: derived from source filename (e.g. `sensor.csv` → table `sensor`)
	- Default null tokens (CSV only): empty cell and `\N`

	Auto mode example:

	CSV file (`sensor.csv`):
	```
	time,temperature,humidity,status
	1000,25.5,60.0,true
	2000,26.1,55.3,false
	3000,27.0,58.1,true
	```

	Auto mode infers:
	```
	table name: sensor (from filename)
	time column: time
	fields: temperature DOUBLE, humidity DOUBLE, status BOOLEAN
	tags: (none)
	```

	Commands:
	```sh
	# CSV
	csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed
	csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed

	# CSV with options
	csv2tsfile.sh --source ./data/csv --target ./output --table_name my_table --separator tab --time_precision us

	# Parquet
	parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed
	parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed

	# Arrow (.arrow / .ipc / .feather)
	arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed
	arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed
	```

	### Output File Naming

	- Single batch: `{source_basename}.tsfile`
	- Multiple batches: `{source_basename}_1.tsfile`, `{source_basename}_2.tsfile`, ...
	- Table name and output filename are independent — table name comes from schema or `--table_name`, filename comes from source file.