Apache DataSketches library functions for Google BigQuery

User-Defined Aggregate Functions (UDAFs) and non-aggregate (scalar) functions (UDFs) for BigQuery SQL engine.

DataSketches are probabilistic data structures that can process massive amounts of data and return very accurate results with a small memory footprint. Because of this, DataSketches are particularly useful for “big data” use cases such as streaming analytics and data warehousing.

Please visit the main Apache DataSketches website for more information about DataSketches library.

If you are interested in making contributions to this project please see our Community page for how to contact us.

Requirements

Requires Emscripten (emcc compiler)

git clone https://github.com/emscripten-core/emsdk.git \
&& cd emsdk \
&& ./emsdk install latest \
&& ./emsdk activate latest \
&& source ./emsdk_env.sh \
&& cd ..

This can be installed using ‘brew install emscripten’ on MacOS.

Requires a link to datasketches-cpp in this repository
```
make datasketches-cpp
```
This target is a part of the default target ‘all’. This requires wget and unzip.
Requires make utility

Requires Google Cloud CLI

curl https://sdk.cloud.google.com | bash

Requires npm and @dataform/cli package
```
npm install -g @dataform/cli
```

Requires setting the following environment variables to your own values:

export JS_BUCKET=    # GCS bucket to hold compiled artifacts (must include gs://)
export BQ_PROJECT=   # location of stored SQL functions (routines)
export BQ_DATASET=   # location of stored SQL functions (routines)
export BQ_LOCATION=  # location of BQ_DATASET

Building, Installing, and Testing

Install All DataSketches

Run the following steps in this repo's root directory to install everything via Cloud Build:

gcloud builds submit \ 
  --project=$BQ_PROJECT \
  --substitutions=_BQ_LOCATION=$BQ_LOCATION,_BQ_DATASET=$BQ_DATASET,_JS_BUCKET=$JS_BUCKET \
  .

Install All DataSketches

Run the following steps in this repo's root directory to install everything:

gcloud auth application-default login # for authentication
make          # performs compilation
make install  # upload to $JS_BUCKET & create functions in $BQ_PROJECT.$BQ_DATASET
make test     # runs predefined tests in BQ

Install Specific DataSketches

To install a specific sketch, change into an individual sketch directory and run the following:

gcloud auth application-default login # for authentication
make          # performs compilation
make install  # upload to $JS_BUCKET 
make create   # create functions in $BQ_PROJECT.$BQ_DATASET