Estimates the number of distinct elements in a data stream using the HyperLogLog++ algorithm. The respective transforms to create and merge sketches, and to extract from them, are:
HllCount.Init
aggregates inputs into HLL++ sketches.HllCount.MergePartial
merges HLL++ sketches into a new sketch.HllCount.Extract
extracts the estimated count of distinct elements from HLL++ sketches.You can read more about what a sketch is at https://github.com/google/zetasketch.
Example 1: creates a long-type sketch for a PCollection<Long>
with a custom precision:
PCollection<Long> input = ...; int p = ...; PCollection<byte[]> sketch = input.apply(HllCount.Init.forLongs().withPrecision(p).globally());
Example 2: creates a bytes-type sketch for a PCollection<KV<String, byte[]>>
:
PCollection<KV<String, byte[]>> input = ...; PCollection<KV<String, byte[]>> sketch = input.apply(HllCount.Init.forBytes().perKey());
Example 3: merges existing sketches in a PCollection<byte[]>
into a new sketch, which summarizes the union of the inputs that were aggregated in the merged sketches:
PCollection<byte[]> sketches = ...; PCollection<byte[]> mergedSketch = sketches.apply(HllCount.MergePartial.globally());
Example 4: estimates the count of distinct elements in a PCollection<String>
:
PCollection<String> input = ...; PCollection<Long> countDistinct = input.apply(HllCount.Init.forStrings().globally()).apply(HllCount.Extract.globally());
Example 5: extracts the count distinct estimate from an existing sketch:
PCollection<byte[]> sketch = ...; PCollection<Long> countDistinct = sketch.apply(HllCount.Extract.globally());
HllCount
).