[Python] Support tree-model TsFiles in TsFileDataFrame (#816)

* [Python] Support tree-model TsFiles in TsFileDataFrame

Squash of the tsdf-tree branch: tree-model support plus dataset catalog memory and indexing improvements.

- support tree-model TsFiles in TsFileDataFrame
- shrink the dataframe catalog memory footprint
- drop table-model phantom cells
- inline iter_owned_series_refs
- rename _LogicalIndex to _DataFrameCatalog
- dedup repeated series specifiers in tsdf.loc

* Fix tree-model metadata segments

* [Python] Dedupe dataset model constants and harden tree-model reads

- metadata: define MODEL_TABLE/MODEL_TREE once and import them in reader
  and dataframe instead of redefining the same literals in both modules,
  removing the drift risk between duplicate definitions.
- metadata: MetadataCatalog.series_count now returns the count of
  physically-present series (len(series_stats_by_ref)) instead of the
  device x declared-field cross-product, so it matches len(tsdf) and the
  reader property for sparse schemas; TsFileSeriesReader.series_count
  delegates to it as the single source of truth.
- reader: document why _metadata_field_stats gates on the value
  statistic's non-null row_count (not the timeline) and why the
  timeline_statistic reads below cannot raise.
- reader: mark the tree-model full-scan reads as O(total_rows) per device
  (cwrapper per-device pushdown limitation) so the hot path is easy to
  locate when profiling.
- reader: fail fast with an assertion if the cwrapper stale col_ leak
  pattern changes, instead of silently slicing the wrong path columns and
  returning wrong-device data.
- tests: add a sparse-schema regression assertion locking series_count to
  the physically-present count.

* [Python] Support unioning multiple tree-model TsFiles in TsFileDataFrame

- Tree TsFiles synthesize their schema per file, so loading several into one
  TsFileDataFrame previously raised "Incompatible schema" whenever the files
  differed in measurement subset or device depth. Widen the dataframe-level
  synthetic table to the union instead: _merge_tree_table_entries unions the
  field columns (existing order first) and grows the tag columns to the
  deepest file's depth so shallower devices pad with nulls.
- Key each logical series by the global field index resolved by measurement
  name against the merged table, while each shard keeps its reader-local
  field index for the read path. This keeps per-(device, field) ownership
  exact and avoids mis-mapping measurements across files. For table model the
  global and local indices always coincide, so behavior is unchanged
  (incompatible table schemas are still rejected).
- tests: add multi-file tree coverage -- identical-structure merge (shards and
  time bounds), field union across files, and differing-depth union with tag
  padding.

* [Python] Filter non-numeric measurements in tree-model TsFileDataFrame

The dataset surface is numeric (values are read as float64), and table mode
already drops non-numeric (STRING/TEXT) fields from its schema. Tree mode did
not, so a string/text measurement was surfaced as a series that then crashed on
read (float() on a string). Apply the numeric-type filter once in
_metadata_field_stats so both models behave consistently -- this also makes the
existing tree-model "measurements that pass the numeric filter" comment accurate.

- reader: skip measurements whose data_type is not numeric in
  _metadata_field_stats (shared by table and tree paths); table mode is
  unaffected since it already filters at the schema level.
- tests: add a tree-model regression test asserting a STRING measurement is
  dropped (not listed, raises KeyError) while the numeric one reads back.
5 files changed
tree: 953810d5835423d15326ad37789182d0ffa2e04d
  1. .github/
  2. .mvn/
  3. cpp/
  4. docs/
  5. java/
  6. python/
  7. .asf.yaml
  8. .clang-format
  9. .gitattributes
  10. .gitignore
  11. checkstyle.xml
  12. CLAUDE.md
  13. codecov.yml
  14. doap_tsfile.rdf
  15. jenkins.pom
  16. Jenkinsfile
  17. LICENSE
  18. mvnw
  19. mvnw.cmd
  20. NOTICE
  21. pom.xml
  22. README-zh.md
  23. README.md
  24. RELEASE_NOTES.md
README.md

English | δΈ­ζ–‡

TsFile Document

codecov Maven Central PyPI

Introduction

TsFile is a columnar storage file format designed for time series data, which supports efficient compression, high throughput of read and write, and compatibility with various frameworks, such as Spark and Flink. It is easy to integrate TsFile into IoT big data processing frameworks.

Time series data is becoming increasingly important in a wide range of applications, including IoT, intelligent control, finance, log analysis, and monitoring systems.

TsFile is the first existing standard file format for time series data. Despite the widespread presence and significance of temporal data, there has been a longstanding absence of standardized file formats for its management. The advent of TsFile introduces a unified file format to facilitate users in managing temporal data.

Click for More Information

TsFile Features

TsFile offers several distinctive features and benefits:

  • Multi Language Independent Use: Multiple language SDK can be used to directly read and write TsFile, making it possible for some lightweight data reading and writing scenarios.

  • Efficient Writing and Compression: A column storage format tailored for time series, organizing data by device and ensuring continuous storage of data for each sequence, minimizing storage space. Compared to CSV, the compression ratio can be increased by more than 90%.

  • High Query Performance: By indexing devices, measurement, and time dimensions, TsFile implements fast filtering and querying of temporal data based on specific time ranges. Compared to general file formats, query throughput can be increased by 2-10 times.

  • Open Integration: TsFile is the underlying storage file format of the temporal database IoTDB, which can form a pluggable storage computing separation architecture with IoTDB. TsFile supports compatibility with Spark Flink and other big data software establish seamless ecosystem integration to ensure compatibility and interoperability across different data processing environments, and achieve deep analysis of temporal data across ecosystems.

TsFile Basic Concepts

TsFile can manage the time series data of multiple devices. Each device can have different measurement.

Each measurement of each device corresponds to a time series.

The TsFile Scheme defines a set of measurement for all devices, as shown in the table below (m1~m5)

TimedeviceIdm1m2m3m4m5
1device1123
2device1123
3device21345
4device21345
5device312345

Among them, Time and deviceId are built-in fields that do not need to be defined and can be written directly.

TsFile Design

File Structure

TsFile adopts a columnar storage design, similar to other file formats, primarily to optimize time-series data's storage efficiency and query performance. This design aligns with the nature of time series data, which often involves large volumes of similar data types recorded over time. However, TsFile was developed particularly with a structure of page, chunk, chunk group, and index:

  • Page: The basic unit for storing time series data, sorted by time in ascending order with separate columns for timestamps and values.

  • Chunk: Comprising metadata headers and several pages, each chunk belongs to one time series, with variable sizes allowing for different compression and encoding methods.

  • Chunk Group: Multiple chunks within a chunk group belong to one or multiple series of a device written in the same period, facilitating efficient query processing.

  • Index: The file metadata at the end of TsFile contains a chunk-level index and file-level statistics for efficient data access.

TsFile Architecture

Encoding and Compression

TsFile employs advanced encoding and compression techniques to optimize storage and access for time series data. It uses methods like run-length encoding (RLE), bit-packing, and Snappy for efficient compression, allowing separate encoding of timestamp and value columns for better data processing. Its unique encoding algorithms are designed specifically for the characteristics of time series data in IoT scenarios, focusing on regular time intervals and the correlation among series.

Its uniqueness lies in the encoding algorithm designed specifically for time series data characteristics, focusing on the correlation between time attributes and data.

The table below compares 3 file formats in different dimensions.

TsFile, CSV and Parquet in Comparison

DimensionTsFileCSVParquet
Data ModelIoTPlainNested
Write ModeTablet, LineLineLine
CompressionYesNoYes
Read ModeQuery, ScanScanQuery
Index on SeriesYesNoNo
Index on TimeYesNoNo

Its development facilitates efficient data encoding, compression, and access, reflecting a deep understanding of industry needs, pioneering a path toward efficient, scalable, and flexible data analytics platforms.

Data TypeRecommended EncodingRecommended Compression
INT32TS_2DIFFLZ4
INT64TS_2DIFFLZ4
FLOATGORILLALZ4
DOUBLEGORILLALZ4
BOOLEANRLELZ4
TEXTDICTIONARYLZ4

more see Docs

Build and Use TsFile

Java

C++

Python

Command-Line Tool (tsfile-cli)

Apache TsFile ships tsfile-cli, a single, pipe-friendly command-line tool for inspecting and importing .tsfile files directly from the shell. Read commands (ls, meta, schema, stats, count, head, cat, sample) print to stdout and diagnostics to stderr, so they compose with awk, jq, sort, and friends; the write command imports CSV/TSV into a new .tsfile. Output formats: csv, tsv, json (NDJSON), and table.

Commands

CommandWhat it does
lsList the tables (table model) or devices (tree model), one name per line
metaFile summary: data model, table/device/series counts, time range, and file size
schemaPer-series data type, encoding, and compression
statsPer-series statistics: count, time range, min/max, first/last, and sum
countPer-series row counts plus a total β€” read from statistics, without scanning pages
headPrint the first N rows (default 10; -n to change)
catStream every matching row
sampleTake a reproducible reservoir sample of rows (-n, --seed)
writeImport CSV/TSV into a new table-model .tsfile

The metadata commands (ls, meta, schema, stats, count) answer most questions without decoding data pages, while head, cat, and sample read the actual rows.

Examples

tsfile-cli ls data.tsfile                          # list tables / devices
tsfile-cli meta data.tsfile                        # file overview (model, counts, time range, size)
tsfile-cli head -n 20 data.tsfile                  # first 20 rows
tsfile-cli cat -m temp,humidity -f csv data.tsfile # stream selected columns as CSV

# import CSV/TSV into a new table-model .tsfile
printf 'time,id1,s1\n0,dev,0\n1,dev,10\n' \
  | tsfile-cli write --table t1 --columns "id1:STRING:tag,s1:INT64:field" -o out.tsfile -

Building

Platform support. Building tsfile-cli from source is currently supported on Linux and macOS only. Standalone, pre-built releases of the tool are planned for a later date.

tsfile-cli is built together with the C++ module, so building that module with Maven from the repository root includes it in the build output:

./mvnw clean package -P with-cpp

This produces the executable at cpp/target/build/bin/tsfile-cli, alongside the shared library it depends on, libtsfile, under cpp/target/build/lib/ (libtsfile.so on Linux, libtsfile.dylib on macOS). tsfile-cli loads libtsfile at runtime, so to use it the library must sit where the dynamic linker can find it β€” keep it under cpp/target/build/lib and put that directory on the library search path, or copy libtsfile next to the binary (or into a system library directory):

# Linux
export LD_LIBRARY_PATH=cpp/target/build/lib:$LD_LIBRARY_PATH
# macOS
export DYLD_LIBRARY_PATH=cpp/target/build/lib:$DYLD_LIBRARY_PATH

cpp/target/build/bin/tsfile-cli --version
cpp/target/build/bin/tsfile-cli --help

See cpp/tools/README.md for the full command and option reference.