---

sidebar_position: 4

import Tabs from ‘@theme/Tabs’; import TabItem from ‘@theme/TabItem’;

BigLake Metastore

This document walks through the steps to register a Onetable synced Iceberg table in BigLake Metastore on GCP.

Pre-requisites

1. Source (Hudi/Delta) table(s) already written to Google Cloud Storage. If you don't have the source table written in GCS, you can follow the steps in this tutorial to set it up.
2. To ensure that the BigLake API's caller (your service account used by Onetable) has the necessary permissions to create a BigLake table, ask your administrator to grant BigLake Admin (roles/bigquery.admin) access to the service account.
3. To ensure that the Storage Account API's caller (your service account used by Onetable) has the necessary permissions to write log/metadata files in GCS, ask your administrator to grant Storage Object User (roles/storage.objectUser) access to the service account.
4. If you're running Onetable outside of GCP, you need to provide the machine access to interact with BigLake and GCS. To do so, store the permissions key for your service account in your machine using

   export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account_key.json
   

5. Clone the Onetable repository and create the utilities-0.1.0-SNAPSHOT-bundled.jar by following the steps on the Installation page

Steps

:::danger Important: Currently BigLake Metastore is only accessible through Google's BigLake Rest APIs, and as such Onetable requires you to setup the below items prior to running sync on your source dataset.

• BigLake Catalog
• BigLake Database :::

Create BigLake Catalog

Use the Try this method on Google's REST reference docs for projects.locations.catalogs.create method to create a catalog.

In this tutorial we'll use us-west1 region.

projects/<yourProjectName>/locations/us-west1/catalogs

onetable

Create BigLake Database

Use the Try this method on Google's REST reference docs for projects.locations.catalogs.databases.create method to create a database.

projects/<yourProjectName>/locations/us-west1/catalogs/onetable/databases

onetable_synced_db

:::danger Limitation: The current implementation of Onetable requires you to set the below options in the Java object.

• catalogImpl
• catalogName
• gcp_project
• gcp_location
• warehouse

Then proceed to creating the utilities-0.1.0-SNAPSHOT-bundled.jar by following the steps here. :::

:::tip The improvement to let users pass these options through the my_config.yaml is being tracked under this github issue. :::

Running sync

<Tabs groupId=“table-format” defaultValue=“hudi” values={[ { label: ‘sourceFormat: HUDI’, value: ‘hudi’, }, { label: ‘sourceFormat: DELTA’, value: ‘delta’, }, ]}

sourceFormat: HUDI
targetFormats:
  - ICEBERG
datasets:
  -
    tableBasePath: gs://path/to/source/data
    tableName: table_name
    partitionSpec: partitionpath:VALUE

sourceFormat: DELTA
targetFormats:
  - ICEBERG
datasets:
  -
    tableBasePath: gs://path/to/source/data
    tableName: table_name
    partitionSpec: partitionpath:VALUE

:::danger Note: Replace with appropriate values for tableBasePath and tableName fields. :::

From your terminal under the cloned Onetable directory, run the sync process using the below command.

java -jar utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar -datasetConfig my_config.yaml

:::tip Note: At this point, if you check your bucket path, you will be able to see the metadata directory with metadata files which contains the information that helps query engines to interpret the data as an Iceberg table. :::

Validating the results

Once the sync succeeds, Onetable would have written the table directly to BigLake Metastore. We can use Try this method option on Google's REST reference docs for projects.locations.catalogs.databases.tables.get method to view the created table.

projects/<yourProjectName>/locations/us-west1/catalogs/onetable/databases/onetable_synced_db/tables/table_name

Conclusion

In this guide we saw how to,

1. sync a source table to create Iceberg metadata with Onetable
2. catalog the data as an Iceberg table in BigLake Metastore
3. validate the table creation using projects.locations.catalogs.databases.tables.get method