feat(catalog): add hadoop namespace operations (#963) [3: Namespace operations](https://github.com/apache/iceberg-go/issues/798#issuecomment-4320784323) Implement all six namespace interface methods. CreateNamespace uses os.MkdirAll and errors if the directory already exists or if non-empty properties are passed. DropNamespace checks that there are no tables or child namespaces before calling os.Remove. CheckNamespaceExists uses os.Stat. ListNamespaces reads directory entries and filters out table dirs via isTableDir. LoadNamespaceProperties returns a synthetic {"location": path} (not persisted). UpdateNamespaceProperties returns an unsupported error. Tests cover create, create duplicate, create nested (a.b.c), create with properties (error), drop, drop non-existent, drop non-empty with tables and child namespaces, list empty and mixed entries, check exists, load properties, and update properties error. Relates to #798
iceberg is a Golang implementation of the Iceberg table spec.
$ git clone https://github.com/apache/iceberg-go.git $ cd iceberg-go/cmd/iceberg && go build .
Use the Makefile so commands stay in sync with CI (e.g. golangci-lint version).
make test
make lint
Install the linter first
make lint-install # or: go install github.com/golangci/golangci-lint/cmd/golangci-lint@v2.8.0
Prerequisites: Docker, Docker Compose
Start the Docker containers using docker compose:
make integration-setup
Export the required environment variables:
export AWS_S3_ENDPOINT=http://$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' minio):9000 export AWS_REGION=us-east-1 export SPARK_CONTAINER_ID=$(docker ps -qf 'name=spark-iceberg') export DOCKER_API_VER=$(docker version -f '{{.Server.APIVersion}}')
Run the integration tests:
make integration-test
Or run a single suite: make integration-scanner, make integration-io, make integration-rest, make integration-spark.
| Filesystem Type | Supported |
|---|---|
| S3 | X |
| Google Cloud Storage | X |
| Azure Blob Storage | X |
| Local Filesystem | X |
| Operation | Supported |
|---|---|
| Get Schema | X |
| Get Snapshots | X |
| Get Sort Orders | X |
| Get Partition Specs | X |
| Get Manifests | X |
| Create New Manifests | X |
| Plan Scan | x |
| Plan Scan for Snapshot | x |
| Operation | REST | Hive | Glue | SQL |
|---|---|---|---|---|
| Load Table | X | X | X | X |
| List Tables | X | X | X | X |
| Create Table | X | X | X | X |
| Register Table | X | X | X | |
| Update Current Snapshot | X | X | X | X |
| Create New Snapshot | X | X | X | X |
| Rename Table | X | X | X | X |
| Drop Table | X | X | X | X |
| Alter Table | X | X | X | X |
| Check Table Exists | X | X | X | X |
| Set Table Properties | X | X | X | X |
| List Namespaces | X | X | X | X |
| Create Namespace | X | X | X | X |
| Check Namespace Exists | X | X | X | X |
| Drop Namespace | X | X | X | X |
| Update Namespace Properties | X | X | X | X |
| Create View | X | X | X | |
| Load View | X | X | ||
| List View | X | X | X | |
| Drop View | X | X | X | |
| Check View Exists | X | X | X |
As long as the FileSystem is supported and the Catalog supports altering the table, the following tracks the current write support:
| Operation | Supported |
|---|---|
| Append Stream | X |
| Append Data Files | X |
| Rewrite Files | |
| Rewrite manifests | |
| Overwrite Files | X |
| Copy-On-Write Delete | X |
| Write Pos Delete | X |
| Write Eq Delete | |
| Row Delta |
Run go build ./cmd/iceberg from the root of this repository to build the CLI executable, alternately you can run go install github.com/apache/iceberg-go/cmd/iceberg@latest to install it to the bin directory of your GOPATH.
The iceberg CLI usage is very similar to pyiceberg CLI
You can pass the catalog URI with --uri argument.
Example: You can start the Iceberg REST API docker image which runs on default in port 8181
docker pull apache/iceberg-rest-fixture:latest docker run -p 8181:8181 apache/iceberg-rest-fixture:latest
and run the iceberg CLI pointing to the REST API server.
./iceberg --uri http://0.0.0.0:8181 list ┌─────┐ | IDs | | --- | └─────┘
Create Namespace
./iceberg --uri http://0.0.0.0:8181 create namespace taxitrips
List Namespace
./iceberg --uri http://0.0.0.0:8181 list ┌───────────┐ | IDs | | --------- | | taxitrips | └───────────┘