java/tools/README.md - tsfile - Git at Google

 <!--

     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at

         http://www.apache.org/licenses/LICENSE-2.0

     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.

 -->

 [English](./README.md) | [Chinese](./README-zh.md)
 # TsFile Tools Manual
 ## Introduction

 ## Development

 ### Prerequisites

 To build the Java version of TsFile Tools, you must have the following dependencies installed:

 1. Java >= 1.8 (1.8, 11 to 17 are verified. Make sure the environment variable is set).
 2. Maven >= 3.6 (if you are compiling TsFile from source).

 ### Build with Maven

 ```sh
 mvn clean package -P with-java -DskipTests
 ```

 ### Install to local machine

 ```
 mvn install -P with-java -DskipTests
 ```

 ## Schema Definition

 ### Parameters

 | Parameter | Description | Required | Default |
 |-----------|------------|----------|---------|
 | table_name | Table name | Yes | |
 | time_precision | Time precision (ms / us / ns / s) | No | ms |
 | has_header | Whether CSV contains a header (true / false). Ignored for Parquet / Arrow. | No | true |
 | separator | CSV delimiter (, / tab / ;). Ignored for Parquet / Arrow. | No | , |
 | null_format | String value treated as null in CSV. Ignored for Parquet / Arrow (native null). | No | |
 | tag_columns | Tag columns (device identifiers / primary key). Supports virtual columns with DEFAULT value. | No | |
 | time_column | Time column name | Yes | |
 | source_columns | Column definitions mapping to source file columns | Yes | |

 > **Backward compatibility**: `id_columns` and `csv_columns` are still accepted as aliases for `tag_columns` and `source_columns`.

 ### Column Concepts

 - **time_column**: Exactly one per table. Written as `time` column with type `TIMESTAMP` in TsFile.
 - **tag_columns**: Device identifiers (composite primary key), 0 or more. Supports virtual columns not present in the source file via `DEFAULT` keyword.
 - **source_columns**: Maps every column in the source file by position (CSV) or by name (Parquet / Arrow). Use `SKIP` to ignore a column.
 - **FIELD** (derived, not configured): All columns in `source_columns` that are not `time_column`, not in `tag_columns`, and not `SKIP`. These are the measurement columns whose values change over time.

 ### Schema Example

 CSV file content:
 ```
 Region,FactoryNumber,DeviceNumber,Model,MaintenanceCycle,Time,Temperature,Emission
 hebei,1001,1,10,1,1,80.0,1000.0
 hebei,1001,1,10,1,4,80.0,1000.0
 hebei,1002,7,5,2,1,90.0,1200.0
 ```

 Schema file (`import.schema`):
 ```
 table_name=root.db1
 time_precision=ms
 has_header=true
 separator=,
 null_format=\N

 tag_columns
 Group DEFAULT Datang
 Region
 FactoryNumber
 DeviceNumber

 time_column=Time

 source_columns
 Region TEXT,
 FactoryNumber TEXT,
 DeviceNumber TEXT,
 SKIP,
 SKIP,
 Time INT64,
 Temperature FLOAT,
 Emission DOUBLE,
 ```

 In this example:
 - `Group` is a virtual tag column (not in CSV) with default value `Datang`
 - `Region`, `FactoryNumber`, `DeviceNumber` are tag columns read from CSV
 - `Model` and `MaintenanceCycle` are skipped via `SKIP`
 - `Temperature` and `Emission` are automatically derived as FIELD columns

 For Parquet / Arrow in schema mode, `source_columns` matches by column **name** instead of position. Named SKIP is also supported:
 ```
 source_columns
 Time INT64,
 unused_col SKIP,
 Temperature FLOAT,
 Emission DOUBLE,
 ```

 ## CLI Parameters

 | Parameter | Description | Required | Default |
 |-----------|------------|----------|---------|
 | -s, --source | Input file or directory | Yes | |
 | -t, --target | Output directory | Yes | |
 | --schema | Schema file path. Omit for auto mode. | No | |
 | --fail_dir | Directory for failed source files | No | failed |
 | --format | Source format: csv / parquet / arrow. Auto-detected by file extension if omitted. | No | auto-detect |
 | --table_name | Table name override (auto mode) | No | derived from filename |
 | --time_precision | Time precision override (auto mode): ms / us / ns / s | No | ms |
 | --separator | CSV delimiter (auto mode): , / tab / ; | No | , |
 | -b, --block_size | CSV chunk size (e.g. 256M, 1G) | No | 256M |
 | -tn, --thread_num | Thread count for parallel processing | No | 8 |

 ## Modes

 ### Schema Mode

 Provide a `--schema` file to explicitly define column mapping, types, tags, and time column.

 ```sh
 # CSV
 csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed --schema ./schema/import.schema
 csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed --schema .\schema\import.schema

 # Parquet
 parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed --schema ./schema/import.schema
 parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed --schema .\schema\import.schema

 # Arrow
 arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed --schema ./schema/import.schema
 arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed --schema .\schema\import.schema
 ```

 ### Auto Mode

 Omit `--schema` to automatically infer column types and detect the time column.

 **Auto mode rules:**
 - Time column: must be named exactly `time` or `TIME` (case-sensitive, strict match)
 - All other columns become FIELD (no tag inference)
 - CSV type inference uses a 100-row sampling window. Promotion rules: INT64 and DOUBLE promote to DOUBLE; any other mixed pair (including BOOLEAN with numeric) promotes to STRING.
 - Parquet / Arrow use native schema types directly
 - Default table name: derived from source filename (e.g. `sensor.csv` → table `sensor`)
 - Default null tokens (CSV only): empty cell and `\N`

 **Auto mode example:**

 CSV file (`sensor.csv`):
 ```
 time,temperature,humidity,status
 1000,25.5,60.0,true
 2000,26.1,55.3,false
 3000,27.0,58.1,true
 ```

 Auto mode infers:
 ```
 table name:  sensor        (from filename)
 time column: time
 fields:      temperature DOUBLE, humidity DOUBLE, status BOOLEAN
 tags:        (none)
 ```

 **Commands:**
 ```sh
 # CSV
 csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed
 csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed

 # CSV with options
 csv2tsfile.sh --source ./data/csv --target ./output --table_name my_table --separator tab --time_precision us

 # Parquet
 parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed
 parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed

 # Arrow (.arrow / .ipc / .feather)
 arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed
 arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed
 ```

 ### Output File Naming

 - Single batch: `{source_basename}.tsfile`
 - Multiple batches: `{source_basename}_1.tsfile`, `{source_basename}_2.tsfile`, ...
 - Table name and output filename are independent — table name comes from schema or `--table_name`, filename comes from source file.
	<!--

	Licensed to the Apache Software Foundation (ASF) under one
	or more contributor license agreements. See the NOTICE file
	distributed with this work for additional information
	regarding copyright ownership. The ASF licenses this file
	to you under the Apache License, Version 2.0 (the
	"License"); you may not use this file except in compliance
	with the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	Unless required by applicable law or agreed to in writing,
	software distributed under the License is distributed on an
	"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
	KIND, either express or implied. See the License for the
	specific language governing permissions and limitations
	under the License.

	-->

	[English](./README.md) \| [Chinese](./README-zh.md)
	# TsFile Tools Manual
	## Introduction

	## Development

	### Prerequisites

	To build the Java version of TsFile Tools, you must have the following dependencies installed:

	1. Java >= 1.8 (1.8, 11 to 17 are verified. Make sure the environment variable is set).
	2. Maven >= 3.6 (if you are compiling TsFile from source).

	### Build with Maven

	```sh
	mvn clean package -P with-java -DskipTests
	```

	### Install to local machine

	```
	mvn install -P with-java -DskipTests
	```

	## Schema Definition

	### Parameters

	\| Parameter \| Description \| Required \| Default \|
	\|-----------\|------------\|----------\|---------\|
	\| table_name \| Table name \| Yes \| \|
	\| time_precision \| Time precision (ms / us / ns / s) \| No \| ms \|
	\| has_header \| Whether CSV contains a header (true / false). Ignored for Parquet / Arrow. \| No \| true \|
	\| separator \| CSV delimiter (, / tab / ;). Ignored for Parquet / Arrow. \| No \| , \|
	\| null_format \| String value treated as null in CSV. Ignored for Parquet / Arrow (native null). \| No \| \|
	\| tag_columns \| Tag columns (device identifiers / primary key). Supports virtual columns with DEFAULT value. \| No \| \|
	\| time_column \| Time column name \| Yes \| \|
	\| source_columns \| Column definitions mapping to source file columns \| Yes \| \|

	> Backward compatibility: `id_columns` and `csv_columns` are still accepted as aliases for `tag_columns` and `source_columns`.

	### Column Concepts

	- time_column: Exactly one per table. Written as `time` column with type `TIMESTAMP` in TsFile.
	- tag_columns: Device identifiers (composite primary key), 0 or more. Supports virtual columns not present in the source file via `DEFAULT` keyword.
	- source_columns: Maps every column in the source file by position (CSV) or by name (Parquet / Arrow). Use `SKIP` to ignore a column.
	- FIELD (derived, not configured): All columns in `source_columns` that are not `time_column`, not in `tag_columns`, and not `SKIP`. These are the measurement columns whose values change over time.

	### Schema Example

	CSV file content:
	```
	Region,FactoryNumber,DeviceNumber,Model,MaintenanceCycle,Time,Temperature,Emission
	hebei,1001,1,10,1,1,80.0,1000.0
	hebei,1001,1,10,1,4,80.0,1000.0
	hebei,1002,7,5,2,1,90.0,1200.0
	```

	Schema file (`import.schema`):
	```
	table_name=root.db1
	time_precision=ms
	has_header=true
	separator=,
	null_format=\N

	tag_columns
	Group DEFAULT Datang
	Region
	FactoryNumber
	DeviceNumber

	time_column=Time

	source_columns
	Region TEXT,
	FactoryNumber TEXT,
	DeviceNumber TEXT,
	SKIP,
	SKIP,
	Time INT64,
	Temperature FLOAT,
	Emission DOUBLE,
	```

	In this example:
	- `Group` is a virtual tag column (not in CSV) with default value `Datang`
	- `Region`, `FactoryNumber`, `DeviceNumber` are tag columns read from CSV
	- `Model` and `MaintenanceCycle` are skipped via `SKIP`
	- `Temperature` and `Emission` are automatically derived as FIELD columns

	For Parquet / Arrow in schema mode, `source_columns` matches by column name instead of position. Named SKIP is also supported:
	```
	source_columns
	Time INT64,
	unused_col SKIP,
	Temperature FLOAT,
	Emission DOUBLE,
	```

	## CLI Parameters

	\| Parameter \| Description \| Required \| Default \|
	\|-----------\|------------\|----------\|---------\|
	\| -s, --source \| Input file or directory \| Yes \| \|
	\| -t, --target \| Output directory \| Yes \| \|
	\| --schema \| Schema file path. Omit for auto mode. \| No \| \|
	\| --fail_dir \| Directory for failed source files \| No \| failed \|
	\| --format \| Source format: csv / parquet / arrow. Auto-detected by file extension if omitted. \| No \| auto-detect \|
	\| --table_name \| Table name override (auto mode) \| No \| derived from filename \|
	\| --time_precision \| Time precision override (auto mode): ms / us / ns / s \| No \| ms \|
	\| --separator \| CSV delimiter (auto mode): , / tab / ; \| No \| , \|
	\| -b, --block_size \| CSV chunk size (e.g. 256M, 1G) \| No \| 256M \|
	\| -tn, --thread_num \| Thread count for parallel processing \| No \| 8 \|

	## Modes

	### Schema Mode

	Provide a `--schema` file to explicitly define column mapping, types, tags, and time column.

	```sh
	# CSV
	csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed --schema ./schema/import.schema
	csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed --schema .\schema\import.schema

	# Parquet
	parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed --schema ./schema/import.schema
	parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed --schema .\schema\import.schema

	# Arrow
	arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed --schema ./schema/import.schema
	arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed --schema .\schema\import.schema
	```

	### Auto Mode

	Omit `--schema` to automatically infer column types and detect the time column.

	Auto mode rules:
	- Time column: must be named exactly `time` or `TIME` (case-sensitive, strict match)
	- All other columns become FIELD (no tag inference)
	- CSV type inference uses a 100-row sampling window. Promotion rules: INT64 and DOUBLE promote to DOUBLE; any other mixed pair (including BOOLEAN with numeric) promotes to STRING.
	- Parquet / Arrow use native schema types directly
	- Default table name: derived from source filename (e.g. `sensor.csv` → table `sensor`)
	- Default null tokens (CSV only): empty cell and `\N`

	Auto mode example:

	CSV file (`sensor.csv`):
	```
	time,temperature,humidity,status
	1000,25.5,60.0,true
	2000,26.1,55.3,false
	3000,27.0,58.1,true
	```

	Auto mode infers:
	```
	table name: sensor (from filename)
	time column: time
	fields: temperature DOUBLE, humidity DOUBLE, status BOOLEAN
	tags: (none)
	```

	Commands:
	```sh
	# CSV
	csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed
	csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed

	# CSV with options
	csv2tsfile.sh --source ./data/csv --target ./output --table_name my_table --separator tab --time_precision us

	# Parquet
	parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed
	parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed

	# Arrow (.arrow / .ipc / .feather)
	arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed
	arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed
	```

	### Output File Naming

	- Single batch: `{source_basename}.tsfile`
	- Multiple batches: `{source_basename}_1.tsfile`, `{source_basename}_2.tsfile`, ...
	- Table name and output filename are independent — table name comes from schema or `--table_name`, filename comes from source file.