Simple and Efficient Cardinality Estimation in Data Streams
share
We study sketching schemes for the cardinality estimation problem in data streams, and advocate for measuring the efficiency of such a scheme in terms of its MVP: Memory-Variance Product, i.e., the product of its space, in bits, and the relative variance of its estimates. Under this natural metric, the celebrated HyperLogLog sketch of Flajolet et al. (2007) has an MVP approaching 6(3ln 2-1)≈ 6.48 for estimating cardinalities up to 2^64. Applying the Cohen/Ting (2014) martingale transformation results in a sketch Martingale HyperLogLog with MVP ≈ 4.16, though it is not composable. Recently Pettie and Wang (2020) proved that it is possible to achieve MVP approaching ≈ 1.98 with a composable sketch called Fishmonger, though the time required to update this sketch is not constant. Our aim in this paper is to strike a nice balance between extreme simplicity (exemplified by (Martingale) (Hyper)LogLog) and extreme information-theoretic efficiency exemplified by Fishmonger). We develop a new class of "curtain" sketches that are a bit more complex than Martingale LogLog but with substantially better s, e.g., Martingale Curtain has MVP ≈ 2.31. We also prove that Martingale Fishmonger has an MVP of around 1.63, and conjecture this to be an information-theoretic lower bound on the problem, independent of update time.
READ FULL TEXT