English | Chinese

TsFile Tools Manual

Introduction

Development

Prerequisites

To build the Java version of TsFile Tools, you must have the following dependencies installed:

1. Java >= 1.8 (1.8, 11 to 17 are verified. Make sure the environment variable is set).
2. Maven >= 3.6 (if you are compiling TsFile from source).

Build with Maven

mvn clean package -P with-java -DskipTests

Install to local machine

mvn install -P with-java -DskipTests

Hybrid CSV Import

Combine one main time-series CSV (with a real time column) and multiple supplement CSVs (same TAG/FIELD columns, no time column) into a single TsFile. Supplement rows receive synthetic timestamps 1, 2, …, N per file; each file is isolated with a virtual TAG batch_id (one ChunkGroup per file per business TAG combination).

Example config (hybrid.conf):

output_tsfile=combined.tsfile
shared_schema=main.schema
main_csv=timeseries.csv
main_batch_id=main
batch_id_tag=batch_id
validate_uniform_tags=true
supplement_sort_by_variance=true
supplement_csv=experiment_1.csv
supplement_batch_id=experiment_1
supplement_csv=experiment_2.csv
supplement_batch_id=experiment_2

Run:

java -jar tsfile-tools.jar --hybrid_config hybrid.conf

Supplement CSV headers must list all business TAG and FIELD columns from shared_schema, excluding the time column (e.g. Region,DeviceId,Temperature,Pressure).

For each supplement CSV separately (supplement_sort_by_variance=true by default):

1. Compute variance of each FIELD column within that CSV only.
2. Order columns by variance descending (higher variance = higher sort priority).
3. Sort rows in that CSV ascending (multi-key comparator).
4. Write one ChunkGroup per CSV; timestamps are consecutive inside the group (startId, startId+1, …).
5. The next supplement CSV continues ids from maxId + 1 (file1: 1..n1, file2: n1+1..n1+n2, …).

Programmatic API: HybridCsvTsFileAssembler.execute(HybridImportConfig).

Schema Definition

Parameters

Backward compatibility: id_columns and csv_columns are still accepted as aliases for tag_columns and source_columns.

Column Concepts

• time_column: Exactly one per table. Written as time column with type TIMESTAMP in TsFile.
• tag_columns: Device identifiers (composite primary key), 0 or more. Supports virtual columns not present in the source file via DEFAULT keyword.
• source_columns: Maps every column in the source file by position (CSV) or by name (Parquet / Arrow). Use SKIP to ignore a column.
• FIELD (derived, not configured): All columns in source_columns that are not time_column, not in tag_columns, and not SKIP. These are the measurement columns whose values change over time.

Schema Example

CSV file content:

Region,FactoryNumber,DeviceNumber,Model,MaintenanceCycle,Time,Temperature,Emission
hebei,1001,1,10,1,1,80.0,1000.0
hebei,1001,1,10,1,4,80.0,1000.0
hebei,1002,7,5,2,1,90.0,1200.0

Schema file (import.schema):

table_name=root.db1
time_precision=ms
has_header=true
separator=,
null_format=\N

tag_columns
Group DEFAULT Datang
Region
FactoryNumber
DeviceNumber

time_column=Time

source_columns
Region TEXT,
FactoryNumber TEXT,
DeviceNumber TEXT,
SKIP,
SKIP,
Time INT64,
Temperature FLOAT,
Emission DOUBLE,

In this example:

• Group is a virtual tag column (not in CSV) with default value Datang
• Region, FactoryNumber, DeviceNumber are tag columns read from CSV
• Model and MaintenanceCycle are skipped via SKIP
• Temperature and Emission are automatically derived as FIELD columns

For Parquet / Arrow in schema mode, source_columns matches by column name instead of position. Named SKIP is also supported:

source_columns
Time INT64,
unused_col SKIP,
Temperature FLOAT,
Emission DOUBLE,

CLI Parameters

Modes

Schema Mode

Provide a --schema file to explicitly define column mapping, types, tags, and time column.

# CSV
csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed --schema ./schema/import.schema
csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed --schema .\schema\import.schema

# Parquet
parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed --schema ./schema/import.schema
parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed --schema .\schema\import.schema

# Arrow
arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed --schema ./schema/import.schema
arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed --schema .\schema\import.schema

Auto Mode

Omit --schema to automatically infer column types and detect the time column.

Auto mode rules:

• Time column: must be named exactly time or TIME (case-sensitive, strict match)
• All other columns become FIELD (no tag inference)
• CSV type inference uses a 100-row sampling window. Promotion rules: INT64 and DOUBLE promote to DOUBLE; any other mixed pair (including BOOLEAN with numeric) promotes to STRING.
• Parquet / Arrow use native schema types directly
• Default table name: derived from source filename (e.g. sensor.csv → table sensor)
• Default null tokens (CSV only): empty cell and \N

Auto mode example:

CSV file (sensor.csv):

time,temperature,humidity,status
1000,25.5,60.0,true
2000,26.1,55.3,false
3000,27.0,58.1,true

Auto mode infers:

table name:  sensor        (from filename)
time column: time
fields:      temperature DOUBLE, humidity DOUBLE, status BOOLEAN
tags:        (none)

Commands:

# CSV
csv2tsfile.sh --source ./data/csv --target ./output --fail_dir ./failed
csv2tsfile.bat --source .\data\csv --target .\output --fail_dir .\failed

# CSV with options
csv2tsfile.sh --source ./data/csv --target ./output --table_name my_table --separator tab --time_precision us

# Parquet
parquet2tsfile.sh --source ./data/parquet --target ./output --fail_dir ./failed
parquet2tsfile.bat --source .\data\parquet --target .\output --fail_dir .\failed

# Arrow (.arrow / .ipc / .feather)
arrow2tsfile.sh --source ./data/arrow --target ./output --fail_dir ./failed
arrow2tsfile.bat --source .\data\arrow --target .\output --fail_dir .\failed

Output File Naming

• Single batch: {source_basename}.tsfile
• Multiple batches: {source_basename}_1.tsfile, {source_basename}_2.tsfile, ...
• Table name and output filename are independent — table name comes from schema or --table_name, filename comes from source file.