| <!-- | |
| Licensed to the Apache Software Foundation (ASF) under one | |
| or more contributor license agreements. See the NOTICE file | |
| distributed with this work for additional information | |
| regarding copyright ownership. The ASF licenses this file | |
| to you under the Apache License, Version 2.0 (the | |
| "License"); you may not use this file except in compliance | |
| with the License. You may obtain a copy of the License at | |
| http://www.apache.org/licenses/LICENSE-2.0 | |
| Unless required by applicable law or agreed to in writing, | |
| software distributed under the License is distributed on an | |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | |
| KIND, either express or implied. See the License for the | |
| specific language governing permissions and limitations | |
| under the License. | |
| --> | |
| # Interface Definitions - Go | |
| The Go package `github.com/apache/tsfile/go/tsfile` exposes the table model. Tablet rows and columns are | |
| zero-based; result-set columns are one-based, with the timestamp at column 1. Table and column identifiers are | |
| case-insensitive and stored normalized to lower case. | |
| ## Data Types and Schema | |
| ```go | |
| // DataType identifies the data type of a table column. | |
| type DataType int32 | |
| const ( | |
| DataTypeBoolean DataType = 0 | |
| DataTypeInt32 DataType = 1 | |
| DataTypeInt64 DataType = 2 | |
| DataTypeFloat DataType = 3 | |
| DataTypeDouble DataType = 4 | |
| DataTypeText DataType = 5 | |
| DataTypeTimestamp DataType = 8 | |
| DataTypeDate DataType = 9 | |
| DataTypeBlob DataType = 10 | |
| DataTypeString DataType = 11 | |
| ) | |
| // ColumnCategory identifies the category of a schema entry. | |
| type ColumnCategory int32 | |
| const ( | |
| ColumnCategoryTag ColumnCategory = 0 | |
| ColumnCategoryField ColumnCategory = 1 | |
| ) | |
| ``` | |
| Only TAG and FIELD columns are accepted: a TAG column is part of the device identifier and must use | |
| `DataTypeString`; a FIELD column holds the measured value and may use any type above. | |
| ```go | |
| // ColumnSchema describes one table-model column. | |
| type ColumnSchema struct { | |
| Name string | |
| DataType DataType | |
| Category ColumnCategory | |
| } | |
| // TableSchema describes a table-model table. | |
| type TableSchema struct { | |
| Table string | |
| Columns []ColumnSchema | |
| } | |
| // TabletColumn fixes the name and type of a Tablet column. | |
| type TabletColumn struct { | |
| Name string | |
| DataType DataType | |
| } | |
| // ColumnMetadata describes one column in a query result. The timestamp is the | |
| // first column. | |
| type ColumnMetadata struct { | |
| Name string | |
| DataType DataType | |
| } | |
| ``` | |
| `NewWriter` copies and validates the schema, so later changes to the caller's `Columns` slice do not affect the | |
| writer. Validation rejects an empty table or column name, duplicate column names, an unsupported data type, and a | |
| TAG column that is not `DataTypeString`. | |
| ## Write Interface | |
| ### Writer | |
| ```go | |
| // WriterOption customizes a writer constructor. | |
| type WriterOption func(*writerOptions) error | |
| // WithMemoryThreshold sets the native writer's in-memory threshold in bytes. | |
| // The default is 128 MB. | |
| func WithMemoryThreshold(bytes uint64) WriterOption | |
| // NewWriter creates or truncates a TsFile and binds it to schema. | |
| func NewWriter(path string, schema TableSchema, options ...WriterOption) (*Writer, error) | |
| // WriteTableTablet writes a table-model tablet without consuming it. | |
| func (w *Writer) WriteTableTablet(tablet *Tablet) error | |
| // WriteArrowBatch writes an Arrow record batch or table to the bound table. | |
| // The input remains owned by the caller and may be released after this method | |
| // returns. | |
| func (w *Writer) WriteArrowBatch(data any) error | |
| // AddProperty adds a binary TsFile property. Empty keys and empty values are | |
| // rejected. | |
| func (w *Writer) AddProperty(key string, value []byte) error | |
| // Flush writes buffered data without closing the writer. | |
| func (w *Writer) Flush() error | |
| // Close flushes and releases the writer. It is idempotent. | |
| func (w *Writer) Close() error | |
| ``` | |
| A `Writer` serializes access to the native writer, so its methods may be called from multiple goroutines. | |
| `WriteTableTablet` requires the tablet columns to match the writer schema by name and type; a missing column | |
| returns `ErrColumnNotExist` and a type difference returns `ErrTypeMismatch`. | |
| ### Tablet | |
| ```go | |
| // NewTablet allocates a tablet with fixed columns and row capacity. | |
| // maxRows must be in [1, 1<<30). | |
| func NewTablet(columns []TabletColumn, maxRows int) (*Tablet, error) | |
| // AddTimestamp assigns the timestamp for a row. | |
| func (t *Tablet) AddTimestamp(row int, timestamp int64) error | |
| // SetBool assigns a BOOLEAN value by zero-based row and column. | |
| func (t *Tablet) SetBool(row, column int, value bool) error | |
| // SetInt32 assigns an INT32 or DATE value. | |
| func (t *Tablet) SetInt32(row, column int, value int32) error | |
| // SetInt64 assigns an INT64 or TIMESTAMP value. | |
| func (t *Tablet) SetInt64(row, column int, value int64) error | |
| // SetFloat32 assigns a FLOAT value. | |
| func (t *Tablet) SetFloat32(row, column int, value float32) error | |
| // SetFloat64 assigns a DOUBLE value. | |
| func (t *Tablet) SetFloat64(row, column int, value float64) error | |
| // SetString assigns a TEXT or STRING value. | |
| func (t *Tablet) SetString(row, column int, value string) error | |
| // SetBytes assigns a BLOB value without interpreting its contents. | |
| func (t *Tablet) SetBytes(row, column int, value []byte) error | |
| // Rows returns the number of rows currently present in the tablet. | |
| func (t *Tablet) Rows() int | |
| // Close releases the native tablet. It is safe to call more than once. | |
| func (t *Tablet) Close() error | |
| ``` | |
| | Setter | Accepted column types | | |
| |---|---| | |
| | `SetBool` | `BOOLEAN` | | |
| | `SetInt32` | `INT32`, `DATE` | | |
| | `SetInt64` | `INT64`, `TIMESTAMP` | | |
| | `SetFloat32` | `FLOAT` | | |
| | `SetFloat64` | `DOUBLE` | | |
| | `SetString` | `TEXT`, `STRING` | | |
| | `SetBytes` | `BLOB` | | |
| Each setter returns `ErrOutOfRange` for a row or column outside the tablet, `ErrTypeMismatch` for a value that does | |
| not match the column type, and `ErrClosed` after the tablet has been closed. Strings must not contain an embedded | |
| NUL byte. A tablet is not safe for concurrent use and must not be modified or closed while | |
| `Writer.WriteTableTablet` is using it. | |
| ### Arrow batch schema | |
| `WriteArrowBatch` accepts an `arrow.Record` or an `arrow.Table`. The schema must have exactly one more field than | |
| the writer schema: a `time` column plus every table column. The `time` column may be `int64` or `timestamp[ns]` | |
| without a time zone and must not contain nulls; any other null column is written as a null value. | |
| | `DataType` | Arrow type | | |
| |---|---| | |
| | `DataTypeBoolean` | `arrow.FixedWidthTypes.Boolean` | | |
| | `DataTypeInt32` | `arrow.PrimitiveTypes.Int32` | | |
| | `DataTypeInt64` | `arrow.PrimitiveTypes.Int64` | | |
| | `DataTypeFloat` | `arrow.PrimitiveTypes.Float32` | | |
| | `DataTypeDouble` | `arrow.PrimitiveTypes.Float64` | | |
| | `DataTypeText`, `DataTypeString` | `arrow.BinaryTypes.String` | | |
| | `DataTypeTimestamp` | `arrow.TimestampType{Unit: arrow.Nanosecond}` | | |
| | `DataTypeDate` | `arrow.FixedWidthTypes.Date32` | | |
| | `DataTypeBlob` | `arrow.BinaryTypes.Binary` | | |
| A batch whose column count differs from the writer schema plus the time column, that repeats or omits a column, or | |
| that uses a type other than the ones listed returns `ErrInvalidSchema`, `ErrColumnNotExist`, or | |
| `ErrTypeMismatch`. A batch must have fewer than `1<<30` rows. The input | |
| remains owned by the caller and may be released as soon as the call returns. | |
| ## Read Interface | |
| ### Reader | |
| ```go | |
| // NewReader opens an existing TsFile for queries. | |
| func NewReader(path string) (*Reader, error) | |
| // Query executes a table query. Omitting options scans the full time range in | |
| // row mode. | |
| func (r *Reader) Query(table string, columns []string, options ...QueryOption) (*ResultSet, error) | |
| // GetTableSchema returns an independent copy of one table schema. | |
| func (r *Reader) GetTableSchema(table string) (TableSchema, error) | |
| // GetAllTableSchemas returns independent copies of every table schema. | |
| func (r *Reader) GetAllTableSchemas() ([]TableSchema, error) | |
| // Close releases all active result sets and then the reader. It is idempotent. | |
| func (r *Reader) Close() error | |
| ``` | |
| `columns` selects the data columns to return and must not be empty; the timestamp column is always added by the | |
| reader. An unknown table returns `ErrTableNotExist` and an unknown column returns `ErrColumnNotExist`. | |
| ### Query options | |
| ```go | |
| // WithTimeRange limits the query to the inclusive [start, end] range. | |
| func WithTimeRange(start, end int64) QueryOption | |
| // WithTagFilter restricts the query with a TAG predicate. | |
| func WithTagFilter(filter TagFilter) QueryOption | |
| // WithOffset skips the leading rows. | |
| func WithOffset(offset int) QueryOption | |
| // WithLimit caps the returned rows; a negative value means unlimited. | |
| func WithLimit(limit int) QueryOption | |
| // WithBatchSize selects batch mode; a positive value returns Arrow batches | |
| // of at most that many rows, a value <= 0 returns rows one by one. | |
| func WithBatchSize(batchSize int) QueryOption | |
| ``` | |
| Without `WithTimeRange` the query covers the full time range. When `start` is greater than `end` the range is | |
| empty and the query returns no rows. Offset, limit, time range, and tag filter are pushed down to the native | |
| reader. | |
| ### TagFilter | |
| ```go | |
| // TagFilter is an immutable predicate over table TAG columns. | |
| type TagFilter interface { tagFilter() } | |
| func Eq(column, value string) TagFilter | |
| func Neq(column, value string) TagFilter | |
| func Lt(column, value string) TagFilter | |
| func Lte(column, value string) TagFilter | |
| func Gt(column, value string) TagFilter | |
| func Gte(column, value string) TagFilter | |
| func Between(column, lower, upper string) TagFilter | |
| func And(left, right TagFilter) TagFilter | |
| func Or(left, right TagFilter) TagFilter | |
| func Not(filter TagFilter) TagFilter | |
| ``` | |
| Tag values are compared as strings, matching the `DataTypeString` requirement for TAG columns. `Between` requires | |
| `lower <= upper`. Passing an invalid filter to `WithTagFilter` returns `ErrInvalidArgument`. | |
| ### ResultSet | |
| ```go | |
| // Next advances to the next row. It returns false with a nil error at EOF. | |
| func (rs *ResultSet) Next() (bool, error) | |
| // Metadata returns a copy of the result columns, including timestamp first. | |
| func (rs *ResultSet) Metadata() []ColumnMetadata | |
| // IsNull reports whether the one-based column is null in the current row. | |
| func (rs *ResultSet) IsNull(column int) (bool, error) | |
| // Typed getters for the current row; column is one-based. | |
| func (rs *ResultSet) Bool(column int) (bool, error) | |
| func (rs *ResultSet) Int32(column int) (int32, error) | |
| func (rs *ResultSet) Int64(column int) (int64, error) | |
| func (rs *ResultSet) Float32(column int) (float32, error) | |
| func (rs *ResultSet) Float64(column int) (float64, error) | |
| func (rs *ResultSet) String(column int) (string, error) | |
| func (rs *ResultSet) Bytes(column int) ([]byte, error) | |
| // ReadArrowRecordBatch returns the next query batch. The caller owns the | |
| // returned record and must call Release. It returns io.EOF after the last | |
| // batch. | |
| func (rs *ResultSet) ReadArrowRecordBatch() (arrow.Record, error) | |
| // ReadArrowBatch returns the next query batch as a one-batch Arrow table. | |
| func (rs *ResultSet) ReadArrowBatch() (arrow.Table, error) | |
| // Close releases the native result. It is idempotent. | |
| func (rs *ResultSet) Close() error | |
| ``` | |
| A query built with `WithBatchSize(n)` where `n > 0` runs in batch mode, and only `ReadArrowRecordBatch`, | |
| `ReadArrowBatch`, `Metadata`, and `Close` are usable; `Next` and the scalar getters return | |
| `ErrWrongResultMode`. Every other query runs in row mode, where only `Next`, the scalar getters, `Metadata`, and | |
| `Close` are usable and the Arrow readers return `ErrWrongResultMode`. | |
| In row mode, `Next` must return `true` before a value can be read; otherwise the getters return | |
| `ErrNoCurrentRow`. Reading a null column with a typed getter returns `ErrNullValue`, so call `IsNull` first. A | |
| getter whose type does not match the column returns `ErrTypeMismatch`. | |
| The typed getters accept the same column types as their Tablet counterparts (`Int32` also reads `DATE`, `Int64` | |
| also reads `TIMESTAMP`, `String` also reads `TEXT`). | |
| A result set is owned by its reader: closing the reader closes every result set created from it. A result set is | |
| forward-only; call `Close` when done to release the native resources before the reader is closed. | |
| ## Error Handling | |
| Native error codes are returned as `*tsfile.Error`, which compares by code, so `errors.Is` works with the exported | |
| sentinels: | |
| ```go | |
| if err := writer.WriteTableTablet(tablet); errors.Is(err, tsfile.ErrTypeMismatch) { | |
| // The tablet column type does not match the writer schema. | |
| } | |
| ``` | |
| | Sentinel | Native code | | |
| |---|---| | |
| | `ErrInvalidArgument` | invalid argument | | |
| | `ErrOutOfRange` | out of range | | |
| | `ErrOverflow` | overflow | | |
| | `ErrInvalidSchema` | invalid schema | | |
| | `ErrTypeNotSupported` | type not supported | | |
| | `ErrTypeMismatch` | type mismatch | | |
| | `ErrFileOpen` | file open failed | | |
| | `ErrFileClose` | file close failed | | |
| | `ErrFileWrite` | file write failed | | |
| | `ErrFileRead` | file read failed | | |
| | `ErrInvalidPath` | invalid path | | |
| | `ErrTableNotExist` | table does not exist | | |
| | `ErrColumnNotExist` | column does not exist | | |
| The package also returns `ErrClosed`, `ErrNullValue`, `ErrWrongResultMode`, and `ErrNoCurrentRow`, which are | |
| plain Go errors rather than native codes. The `*tsfile.Error` value carries the numeric `Code` and the operation | |
| `Op` that produced it. |