Apache DataSketches library functions for Google BigQuery

User-Defined Aggregate Functions (UDAFs) and non-aggregate (scalar) functions (UDFs) for BigQuery SQL engine.

DataSketches are probabilistic data structures that can process massive amounts of data and return very accurate results with a small memory footprint. Because of this, DataSketches are particularly useful for “big data” use cases such as streaming analytics and data warehousing.

Please visit the main Apache DataSketches website for more information about DataSketches library.

If you are interested in making contributions to this project please see our Community page for how to contact us.

Requirements

Requires Emscripten (emcc compiler)
Requires a link to datasketches-cpp in this repository
Requires make utility
Requires Google Cloud CLI
Requires npm and @dataform/cli package npm install -g @dataform/cli
Requires setting the following environment variables to your own values:
- JS_BUCKET: to hold compiled artifacts (must include gs://)
- BQ_PROJECT: location of stored SQL functions (routines)
- BQ_DATASET: location of stored SQL functions (routines)
- BQ_LOCATION: location of BQ_DATASET

Building, Installing, and Testing

Install All DataSketches

Run the following steps in this repo's root directory to install everything:

gcloud auth application-default login # for authentication
make          # performs compilation
make install  # upload to $JS_BUCKET & create functions in $BQ_PROJECT.$BQ_DATASET
make test     # runs predefined tests in BQ

Install Specific DataSketches

To install a specific sketch, change into an individual sketch directory and run the following:

gcloud auth application-default login # for authentication
make          # performs compilation
make install  # upload to $JS_BUCKET 
make create   # create functions in $BQ_PROJECT.$BQ_DATASET
make test     # runs predefined tests in BQ