| --- |
| layout: doc_page |
| --- |
| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| # Sketches by [Component Repository](https://github.com/apache?utf8=%E2%9C%93&q=datasketches) |
| |
| The DataSketches Library is organized into the following repository groups: |
| |
| ## Java |
| |
| ### datasketches-java |
| This repository has the core-java sketching classes, which are leveraged by some of the other repositories. |
| This repository has no external dependencies outside of the DataSketches/memory repository, Java and TestNG for unit tests. |
| This code is versioned and the latest release can be obtained from |
| [Downloads](https://datasketches.apache.org/docs/Community/Downloads.html). |
| |
| <b>High-level Repositories Structure</b> |
| |
| Sketches-core Packages. | Package Description |
| -------------------------------|--------------------- |
| org.apache.datasketches | Common functions and utilities |
| org.apache.datasketches.cpc | New Unique Counting Sketch with better accuracy per size than HLL |
| org.apache.datasketches.fdt | Frequent Distinct Tuples Sketch. |
| org.apache.datasketches.frequencies | Frequent Item Sketches, for both longs and generics |
| org.apache.datasketches.hash | The 128-bit MurmurHash3 and adaptors |
| org.apache.datasketches.hll | Unique counting HLL sketches for both heap and off-heap. |
| org.apache.datasketches.hllmap | The (HLL) Unique Count Map Sketch |
| org.apache.datasketches.kll | Quantiles sketch with better accuracy per size than the standard quantiles sketch. Includes PMF, CDF functions, for floats, doubles. On-heap & off-heap. |
| org.apache.datasketches.quantiles | Standard Quantiles sketch, plus PMF and CDF functions, for doubles and generics. On-heap & off-heap. |
| org.apache.datasketches.req | Relative Error Quantiles (REQ) sketch, plus PMF and CDF functions for floats, on-heap. Extremely high accuracy for very high ranks (e.g., 99.999%ile), or very low ranks (e.g., .00001%ile. |
| org.apache.datasketches.sampling | Weighted and uniform reservoir sampling with generics |
| org.apache.datasketches.theta | Unique counting Theta Sketches for both on-heap & off-heap |
| org.apache.datasketches.tuple | Tuple sketches for both primitives and generics |
| org.apache.datasketches.tuple.adouble | A Tuple sketch with a Summary of a single double |
| org.apache.datasketches.tuple.aninteger | A Tuple sketch with a Summary of a single integer |
| org.apache.datasketches.tuple.Strings | A Tuple sketch with a Summary of an array of Strings |
| |
| ### datasketches-memory |
| This code is versioned and the latest release can be obtained from |
| [Downloads](https://datasketches.apache.org/docs/Community/Downloads.html). |
| |
| Memory Packages | Package Description |
| -------------------------------|--------------------- |
| org.apache.datasketches.memory | Low level, high-performance Memory data-structure management primarily for off-heap. |
| |
| |
| ### datasketches-hive |
| This repository contains Hive UDFs and UDAFs for use within Hadoop grid enviornments. |
| This code has dependencies on sketches-core as well as Hadoop and Hive. |
| Users of this code are advised to use Maven to bring in all the required dependencies. |
| This code is versioned and the latest release can be obtained from |
| [Downloads](https://datasketches.apache.org/docs/Community/Downloads.html). |
| |
| Sketches-hive Packages | Package Description |
| -------------------------------------|--------------------- |
| org.apache.datasketches.hive.cpc | Hive UDF and UDAFs for CPC sketches |
| org.apache.datasketches.hive.frequencies | Hive UDF and UDAFs for Frequent Items sketches |
| org.apache.datasketches.hive.hll | Hive UDF and UDAFs for HLL sketches |
| org.apache.datasketches.hive.kll | Hive UDF and UDAFs for KLL sketches |
| org.apache.datasketches.hive.quantiles | Hive UDF and UDAFs for Quantiles sketches |
| org.apache.datasketches.hive.theta | Hive UDF and UDAFs for Theta sketches |
| org.apache.datasketches.hive.tuple | Hive UDF and UDAFs for Tuple sketches |
| |
| ### datasketches-pig |
| This repository contains Pig User Defined Functions (UDF) for use within Hadoop grid environments. |
| This code has dependencies on sketches-core as well as Hadoop and Pig. |
| Users of this code are advised to use Maven to bring in all the required dependencies. |
| This code is versioned and the latest release can be obtained from |
| [Downloads](https://datasketches.apache.org/docs/Community/Downloads.html). |
| |
| Sketches-pig Packages | Package Description |
| -----------------------------------|--------------------- |
| org.apache.datasketches.pig.cpc | Pig UDFs for CPC sketches |
| org.apache.datasketches.pig.frequencies | Pig UDFs for Frequent Items sketches |
| org.apache.datasketches.pig.hash | Pig UDFs for MurmerHash3 |
| org.apache.datasketches.pig.hll | Pig UDFs for HLL sketches |
| org.apache.datasketches.pig.kll | Pig UDFs for KLL sketches |
| org.apache.datasketches.pig.quantiles | Pig UDFs for Quantiles sketches |
| org.apache.datasketches.pig.sampling. | Pig UDFs for Sampling sketches |
| org.apache.datasketches.pig.theta | Pig UDFs for Theta sketches |
| org.apache.datasketches.pig.tuple | Pig UDFs for Tuple sketches |
| |
| |
| ### datasketches-characterization |
| This relatively new repository is for Java and C++ code that we use to characterize the accuracy and speed performance of the sketches in |
| the library and is constantly being updated. Examples of the job command files used for various tests can be found in the src/main/resources directory. |
| Some of these tests can run for hours depending on its configuration. This component is not formally released and code must be obtained from |
| the [GitHub site](https://github.com/apache/datasketches-characterization). |
| |
| Characterization Packages | Package Description |
| ------------------------------------------------|--------------------- |
| org.apache.datasketches.characterization | Common functions and utilities |
| org.apache.datasketches.characterization.concurrent | Concurrent Theta Sketch |
| org.apache.datasketches.characterization.cpc | Compressed Probabilistic Counting Sketch |
| org.apache.datasketches.characterization.fdt | Frequent Distinct Tuples Sketch |
| org.apache.datasketches.characterization.frequencies | Frequent Items Sketches |
| org.apache.datasketches.characterization.hash | Hash function performance |
| org.apache.datasketches.characterization.hll | HyperLogLog Sketcch |
| org.apache.datasketches.characterization.memory | Memory performance |
| org.apache.datasketches.characterization.quantiles | Quantiles performance |
| org.apache.datasketches.characterization.theta | Theta Sketch |
| org.apache.datasketches.characterization.uniquecount | Base Profiles for Unique Counting Sketches |
| |
| ### datasketches-server |
| This is a new repository for our experimental docker/container server that enables easy access to the core sketches in the library via HTTP. |
| This component is not formally released and code must be obtained from |
| the [GitHub site](https://github.com/apache/datasketches-server). |
| |
| #### C++ Characterizations |
| * CPC |
| * Frequent Items |
| * HLL |
| * KLL |
| * Theta |
| |
| |
| ### datasketches-vector |
| This component implements the [Frequent Directions Algorithm](/docs/Community/Research.html) [GLP16]. It is still experimental in that the theoretical work has not yet supplied a suitable measure of error for production work. It can be used as is, but it will not go through a formal Apache Release until we can find a way to provide better error properties. It has a dependence on the Memory component. |
| This component is not formally released and code must be obtained from |
| the [GitHub site](https://github.com/apache/datasketches-vector). |
| |
| |
| ## C++ and Python |
| |
| ### datasketches-cpp |
| This is the evolving C++ implementations of the same sketches that are available in Java. |
| These implementations are *binary compatible* with their counterparts in Java. |
| In other words, a sketch created and stored in C++ can be opened and read in Java and visa-versa. |
| This code is versioned and the latest release can be obtained from |
| [Downloads](https://datasketches.apache.org/docs/Community/Downloads.html). |
| |
| This site also has our [Python adaptors](https://github.com/apache/datasketches-cpp/tree/master/python) that basically wrap the C++ implementations, making the high performance C++ implementations available from Python. |
| |
| ### datasketches-postgresql |
| This site provides the postgres-specific adaptors that wrap the C++ implementations making |
| them available to the PostgreSQL database users. PostgreSQL users should download the PostgreSQL extension from [pgxn.org](https://pgxn.org/dist/datasketches/). For examples refer to the README on the component site. This code is versioned and the latest release can be obtained from |
| [Downloads](https://datasketches.apache.org/docs/Community/Downloads.html). |
| |
| |
| |
| |
| |
| |
| |
