blob: c58cde785313d719c63227ef5400cc6db63143e9 [file] [view]
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Quick Start - Go
## Dependencies and Installation
The Go package `github.com/apache/tsfile/go/tsfile` is a cgo binding over the C++ implementation, so a C/C++
build of TsFile must be available before Go code can be built. There are two ways to get it:
- Install the published C/C++ packages or SDK. No C++ build is needed; this works on Linux, macOS, and Windows.
- Build the C++ library from source. Use this to change TsFile or on a platform without a published package.
The Go module itself is fetched from GitHub in both cases; it is not published to the module proxy yet.
### Dependencies
- Go >= 1.22 with `CGO_ENABLED=1`
- A C/C++ toolchain and CMake, needed only when building the native library from source
- JDK >= 17 and Maven, needed only when building the native library from source
- On Windows, MinGW-w64 is the supported cgo toolchain
### Option 1: Use the published C/C++ packages (Linux, macOS, Windows)
Install the runtime and development files as described on the
[download and installation page](../../../Download/README.md#cpp-packages), for example the SDK archive, the DEB/RPM
packages, the Homebrew formula, or the Windows ZIP. Headers are installed under an `include/tsfile` subdirectory,
so point cgo at that directory and at the library directory of the install prefix:
```bash
export TSFILE_HOME=/path/to/tsfile-sdk # or: $(brew --prefix local/tsfile250/tsfile)
export CGO_CFLAGS="-I$TSFILE_HOME/include/tsfile"
export CGO_LDFLAGS="-L$TSFILE_HOME/lib -Wl,-rpath,$TSFILE_HOME/lib"
```
The `-rpath` entry lets the compiled program find `libtsfile` at run time; drop it when the shared library is
installed in a standard system directory. For a system installation under `/usr`, the compiler finds the library
without `-L`, but the headers still need `-I/usr/include/tsfile`. The environment variables are added to the
package's own flags, whose paths point at a source checkout under `cpp/target/build`; when you use the installed
packages those paths do not exist, and the linker prints a search-path warning that can be ignored.
On Windows, extract the ZIP and set the same variables in PowerShell. The ZIP ships `bin\tsfile.dll` and an import
library in `lib\tsfile.lib` that the MinGW linker accepts, so `-ltsfile` resolves through `-L` as usual:
```powershell
$env:TSFILE_HOME = "C:\path\to\tsfile-2.5.0-windows-x86_64"
$env:CGO_CFLAGS = "-I$env:TSFILE_HOME\include\tsfile"
$env:CGO_LDFLAGS = "-L$env:TSFILE_HOME\lib -ltsfile"
$env:Path = "$env:TSFILE_HOME\bin;$env:Path"
```
Windows has no run-time library search path, so the compiled program finds `tsfile.dll` through `PATH`; keep
`bin` on `PATH`, or copy `tsfile.dll` next to the executable. The DLL links the C runtime statically and depends
only on the system libraries, so no Visual C++ redistributable is required.
cgo reads these variables when it compiles the package, so set them in every shell where you build a program that
links against the installed library, or record them with `go env -w CGO_CFLAGS=... CGO_LDFLAGS=...`. They are not
needed when running an already built program.
### Option 2: Build the native library from source
Clone the source code and build the C++ library in the repository root:
```bash
git clone https://github.com/apache/tsfile.git
cd tsfile
./mvnw -P with-cpp -DskipTests package
```
This produces the headers under `cpp/target/build/include` and the library under `cpp/target/build/lib` that the
Go package links against by default.
On Windows, build with MinGW. MinGW-w64 is this project's default C++ generator on Windows, and it is the compiler
that cgo uses for the Go package, so `-Dcpp.toolchain=mingw` makes the build produce a DLL and an import library that
match the Go toolchain directly. The build writes `tsfile.dll` to `cpp\target\build\lib` instead of a package's
`bin` directory, so point `PATH` there (or copy the DLL next to the executable) in addition to the MinGW-w64 `bin`
directory that provides the compiler:
```powershell
.\mvnw.cmd -P with-cpp -Dcpp.toolchain=mingw -DskipTests package
$env:Path = "$PWD\cpp\target\build\lib;C:\mingw64\bin;$env:Path"
```
Run that `PATH` assignment in every new terminal before building or running Go programs.
### Use the module in your project
With the installed packages (option 1), fetch the module directly in your Go project:
```bash
go get github.com/apache/tsfile/go@develop
go mod tidy
```
The module has no tagged release yet, so `@develop` resolves to a pseudo-version of the branch head. Record that
pseudo-version in `go.mod`, or pin a specific commit, for reproducible builds.
With a source build (option 2), replace the module with the `go` directory of your checkout; the package then
finds the headers and the library built next to it:
```bash
go mod edit -replace github.com/apache/tsfile/go=/path/to/tsfile/go
go get github.com/apache/tsfile/go
```
To run the bundled example instead, from the `go` directory of the checkout:
```bash
cd go
go run ./examples/table_read_write
```
## Writing Process
### Construct Writer
```go
schema := tsfile.TableSchema{
Table: "table1",
Columns: []tsfile.ColumnSchema{
{Name: "id1", DataType: tsfile.DataTypeString, Category: tsfile.ColumnCategoryTag},
{Name: "id2", DataType: tsfile.DataTypeString, Category: tsfile.ColumnCategoryTag},
{Name: "s1", DataType: tsfile.DataTypeInt32, Category: tsfile.ColumnCategoryField},
{Name: "s2", DataType: tsfile.DataTypeBoolean, Category: tsfile.ColumnCategoryField},
},
}
writer, err := tsfile.NewWriter("test.tsfile", schema, tsfile.WithMemoryThreshold(10*1024*1024))
if err != nil {
return err
}
defer writer.Close()
```
`NewWriter` creates or truncates the file. `WithMemoryThreshold` is optional and defaults to 128 MB.
### Write Data
```go
tablet, err := tsfile.NewTablet([]tsfile.TabletColumn{
{Name: "id1", DataType: tsfile.DataTypeString},
{Name: "id2", DataType: tsfile.DataTypeString},
{Name: "s1", DataType: tsfile.DataTypeInt32},
{Name: "s2", DataType: tsfile.DataTypeBoolean},
}, 5)
if err != nil {
return err
}
defer tablet.Close()
for row := 0; row < 5; row++ {
if err := tablet.AddTimestamp(row, int64(row)); err != nil {
return err
}
tablet.SetString(row, 0, "id1_filed_1")
tablet.SetString(row, 1, "id2_filed_1")
tablet.SetInt32(row, 2, int32(row))
tablet.SetBool(row, 3, true)
}
if err := writer.WriteTableTablet(tablet); err != nil {
return err
}
```
Tablet rows and columns are zero-based. Every setter validates the index and the column type, and returns an error
when the row or column is out of range or the value does not match the declared type; check those errors in the same
way as `AddTimestamp`.
Data can also be written from an Arrow record batch or table. The schema must contain a `time` column of type
`int64` or `timestamp[ns]` without a time zone, followed by exactly the columns of the table schema:
```go
arrowSchema := arrow.NewSchema([]arrow.Field{
{Name: "time", Type: arrow.PrimitiveTypes.Int64},
{Name: "id1", Type: arrow.BinaryTypes.String},
{Name: "id2", Type: arrow.BinaryTypes.String},
{Name: "s1", Type: arrow.PrimitiveTypes.Int32},
{Name: "s2", Type: arrow.FixedWidthTypes.Boolean},
}, nil)
builder := array.NewRecordBuilder(memory.DefaultAllocator, arrowSchema)
defer builder.Release()
timeBuilder := builder.Field(0).(*array.Int64Builder)
id1Builder := builder.Field(1).(*array.StringBuilder)
id2Builder := builder.Field(2).(*array.StringBuilder)
s1Builder := builder.Field(3).(*array.Int32Builder)
s2Builder := builder.Field(4).(*array.BooleanBuilder)
for row := 5; row < 10; row++ {
timeBuilder.Append(int64(row))
id1Builder.Append("id1_filed_1")
id2Builder.Append("id2_filed_1")
s1Builder.Append(int32(row))
s2Builder.Append(true)
}
record := builder.NewRecord()
defer record.Release()
if err := writer.WriteArrowBatch(record); err != nil {
return err
}
```
### Close File
```go
writer.Close()
```
### Sample Code
The sample code of using these interfaces is in
<https://github.com/apache/tsfile/blob/develop/go/examples/table_read_write/main.go>
## Query Process
### Construct Reader
```go
reader, err := tsfile.NewReader("test.tsfile")
if err != nil {
return err
}
defer reader.Close()
```
### Construct Query Request
```go
result, err := reader.Query(
"table1",
[]string{"id1", "id2", "s1", "s2"},
tsfile.WithTimeRange(2, 8),
tsfile.WithTagFilter(tsfile.And(tsfile.Eq("id1", "id1_filed_1"), tsfile.Eq("id2", "id2_filed_1"))),
tsfile.WithLimit(100),
)
if err != nil {
return err
}
defer result.Close()
```
Time range, tag filter, offset, limit, and batch size are all optional and are pushed down to the native reader.
### Query Data
```go
metadata := result.Metadata()
for _, column := range metadata {
fmt.Printf("%s(%d) ", column.Name, column.DataType)
}
fmt.Println()
for {
hasNext, err := result.Next()
if err != nil {
return err
}
if !hasNext {
break
}
for column := 1; column <= len(metadata); column++ {
isNull, err := result.IsNull(column)
if err != nil {
return err
}
if isNull {
fmt.Print("null ")
continue
}
switch metadata[column-1].DataType {
case tsfile.DataTypeBoolean:
value, err := result.Bool(column)
if err != nil {
return err
}
fmt.Print(value, " ")
case tsfile.DataTypeInt32, tsfile.DataTypeDate:
value, err := result.Int32(column)
if err != nil {
return err
}
fmt.Print(value, " ")
case tsfile.DataTypeInt64, tsfile.DataTypeTimestamp:
value, err := result.Int64(column)
if err != nil {
return err
}
fmt.Print(value, " ")
case tsfile.DataTypeFloat:
value, err := result.Float32(column)
if err != nil {
return err
}
fmt.Print(value, " ")
case tsfile.DataTypeDouble:
value, err := result.Float64(column)
if err != nil {
return err
}
fmt.Print(value, " ")
case tsfile.DataTypeString, tsfile.DataTypeText:
value, err := result.String(column)
if err != nil {
return err
}
fmt.Print(value, " ")
}
}
fmt.Println()
}
```
Result-set columns are one-based; column 1 is the timestamp and the selected columns follow in query order. The
typed getters (`Bool`, `Int32`, `Int64`, `Float32`, `Float64`, `String`, `Bytes`) reject a column whose data type
does not match, so dispatch on the metadata as shown above.
### Query Data as Arrow Batches
When `WithBatchSize` is positive the query runs in batch mode and rows are returned as Arrow record batches:
```go
result, err := reader.Query("table1", []string{"id1", "id2", "s1", "s2"}, tsfile.WithBatchSize(1024))
if err != nil {
return err
}
defer result.Close()
for {
record, err := result.ReadArrowRecordBatch()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
return err
}
fmt.Println(record.NumRows())
record.Release()
}
```
### Close File
```go
result.Close()
reader.Close()
```
Closing the reader also closes every result set created from it.
### Sample Code
The sample code of using these interfaces is in
<https://github.com/apache/tsfile/blob/develop/go/examples/table_read_write/main.go>
> Note: The above read/write examples are all based on the table model interface. For details about the interface
definition, please refer to [Go Interface Definition](./InterfaceDefinition/InterfaceDefinition-Go.md). If you need
information regarding the tree model, please contact us.