commit | 077669abeb626e788f0926440328d949192ad5e1 | [log] [tgz] |
---|---|---|
author | Knut Nesheim <knutin@gmail.com> | Tue Jan 14 18:59:48 2014 +0100 |
committer | Knut Nesheim <knutin@gmail.com> | Tue Jan 14 18:59:48 2014 +0100 |
tree | b031960c39b6c46d64ec1a1226e5326648412522 | |
parent | 34c78d50710b48652794f6f56e8e42750cb7d219 [diff] |
Better quickcheck properties. Added first version of hyper_binary_rle backend.
hyper
is an implementation of the HyperLogLog algorithm with bias correction from HLL++ as (described by Google)[http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/40671.pdf].
Using hyper
you can estimate the cardinality of large data sets using constant memory. The union of two HLL filters is lossless, which allows perfect parallellization.
1> hyper:insert(<<"foobar">>, hyper:insert(<<"quux">>, hyper:new(4))). {hyper,4,{hyper_gb,{2,{10,1,{8,1,nil,nil},nil}}}} 2> hyper:card(v(-1)). 2.136502281992361
The error from estimations can be seen in this example:
10> Run = fun (P, Card) -> hyper:card(lists:foldl(fun (_, H) -> hyper:insert(crypto:rand_bytes(32), H) end, hyper:new(P), lists:seq(1, Card))) end. #Fun<erl_eval.12.80484245> 11> Run(12, 10000). 9984.777312630407 12> Run(14, 10000). 9994.719695068114 13> Run(16, 10000). 9991.34646427657