commit | 8d2feef927ec472b444412fa32daa9b22860ffca | [log] [tgz] |
---|---|---|
author | Knut Nesheim <knutin@gmail.com> | Fri Jan 10 17:41:26 2014 +0100 |
committer | Knut Nesheim <knutin@gmail.com> | Fri Jan 10 17:41:26 2014 +0100 |
tree | 863eaad948b6f0640e3a11861759b3c945762a75 | |
parent | d78ca2f5bcbef57a644591ad7805bc5e28d3ab91 [diff] |
Serialization tests.
hyper
is an implementation of the HyperLogLog algorithm with bias correction from HLL++ as (described by Google)[http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/40671.pdf].
Using hyper
you can estimate the cardinality of large data sets using constant memory. The union of two HLL filters is lossless, which allows perfect parallellization.
1> hyper:insert(<<"foobar">>, hyper:insert(<<"quux">>, hyper:new(4))). {hyper,4,{hyper_gb,{2,{10,1,{8,1,nil,nil},nil}}}} 2> hyper:card(v(-1)). 2.136502281992361
The error from estimations can be seen in this example:
10> Run = fun (P, Card) -> hyper:card(lists:foldl(fun (_, H) -> hyper:insert(crypto:rand_bytes(32), H) end, hyper:new(P), lists:seq(1, Card))) end. #Fun<erl_eval.12.80484245> 11> Run(12, 10000). 9984.777312630407 12> Run(14, 10000). 9994.719695068114 13> Run(16, 10000). 9991.34646427657