Simpler and Better Cardinality Estimators for HyperLogLog and PCSA
share
Cardinality Estimation (aka Distinct Elements) is a classic problem in sketching with many industrial applications. Although sketching algorithms are fairly simple, analyzing the cardinality estimators is notoriously difficult, and even today the state-of-the-art sketches such as HyperLogLog and (compressed) are not covered in graduate level Big Data courses. In this paper we define a class of generalized remaining area () estimators, and observe that HyperLogLog, LogLog, and some estimators for PCSA are merely instantiations of for various integral values of τ. We then analyze the limiting relative variance of estimators. It turns out that the standard estimators for HyperLogLog and PCSA can be improved by choosing a fractional value of τ. The resulting estimators come very close to the Cramér-Rao lower bounds for HyperLogLog and PCSA derived from their Fisher information. Although the Cramér-Rao lower bound can be achieved with the Maximum Likelihood Estimator (MLE), the MLE is cumbersome to compute and dynamically update. In contrast, estimators are trivial to update in constant time. Our presentation assumes only basic calculus and probability, not any complex analysis <cit.>.
READ FULL TEXT