| commit | f3813937d149b503052dcd1b7820d273f396417a | [log] [tgz] |
|---|---|---|
| author | Joseph Daniel <godwhoa@users.noreply.github.com> | Mon May 19 20:05:07 2025 +0530 |
| committer | GitHub <noreply@github.com> | Mon May 19 10:35:07 2025 -0400 |
| tree | 8c9d06d10350d6f3c578f53f087481ae43d26955 | |
| parent | fb22d01def6708234b2524af65f3ac10c20c1417 [diff] |
fix(table): apply transform before comparing lower and upper partition values (#432) While trying to to use `table.Transaction.AddFiles` API, I was getting this error, which was odd since I have a day transform. Digging into the code I found that we were comparing without applying the transform. ``` failed to add file to transaction: error encountered during parquet file conversion: cannot infer partition value from parquet metadata as there is more than one value for partition field: date_partition. (low: 1733183899000000, high: 1733183942000000) ``` This PR ensures that we first apply the transform and then do the comparison. Ref: https://apache-iceberg.slack.com/archives/C05J3MJ42BD/p1747319038338949
iceberg is a Golang implementation of the Iceberg table spec.
$ git clone https://github.com/apache/iceberg-go.git $ cd iceberg-go/cmd/iceberg && go build .
| Filesystem Type | Supported |
|---|---|
| S3 | X |
| Google Cloud Storage | X |
| Azure Blob Storage | X |
| Local Filesystem | X |
| Operation | Supported |
|---|---|
| Get Schema | X |
| Get Snapshots | X |
| Get Sort Orders | X |
| Get Partition Specs | X |
| Get Manifests | X |
| Create New Manifests | X |
| Plan Scan | x |
| Plan Scan for Snapshot | x |
| Operation | REST | Hive | DynamoDB | Glue | SQL |
|---|---|---|---|---|---|
| Load Table | X | X | X | ||
| List Tables | X | X | X | ||
| Create Table | X | X | X | ||
| Register Table | X | X | |||
| Update Current Snapshot | X | X | |||
| Create New Snapshot | X | X | |||
| Rename Table | X | X | X | ||
| Drop Table | X | X | X | ||
| Alter Table | X | X | X | ||
| Check Table Exists | X | X | X | ||
| Set Table Properties | X | X | X | ||
| List Namespaces | X | X | X | ||
| Create Namespace | X | X | X | ||
| Check Namespace Exists | X | X | X | ||
| Drop Namespace | X | X | X | ||
| Update Namespace Properties | X | X | X | ||
| Create View | X | ||||
| List View | X | ||||
| Drop View | X | ||||
| Check View Exists | X |
As long as the FileSystem is supported and the Catalog supports altering the table, the following tracks the current write support:
| Operation | Supported |
|---|---|
| Append Stream | X |
| Append Data Files | X |
| Rewrite Files | |
| Rewrite manifests | |
| Overwrite Files | |
| Write Pos Delete | |
| Write Eq Delete | |
| Row Delta |
Run go build ./cmd/iceberg from the root of this repository to build the CLI executable, alternately you can run go install github.com/apache/iceberg-go/cmd/iceberg to install it to the bin directory of your GOPATH.
The iceberg CLI usage is very similar to pyiceberg CLI
You can pass the catalog URI with --uri argument.
Example: You can start the Iceberg REST API docker image which runs on default in port 8181
docker pull apache/iceberg-rest-fixture:latest docker run -p 8181:8181 apache/iceberg-rest-fixture:latest
and run the iceberg CLI pointing to the REST API server.
./iceberg --uri http://0.0.0.0:8181 list ┌─────┐ | IDs | | --- | └─────┘
Create Namespace
./iceberg --uri http://0.0.0.0:8181 create namespace taxitrips
List Namespace
./iceberg --uri http://0.0.0.0:8181 list ┌───────────┐ | IDs | | --------- | | taxitrips | └───────────┘