A Theory of Dynamic Benchmarks

by   Ali Shirali, et al.

Dynamic benchmarks interweave model fitting and data collection in an attempt to mitigate the limitations of static benchmarks. In contrast to an extensive theoretical and empirical study of the static setting, the dynamic counterpart lags behind due to limited empirical studies and no apparent theoretical foundation to date. Responding to this deficit, we initiate a theoretical study of dynamic benchmarking. We examine two realizations, one capturing current practice and the other modeling more complex settings. In the first model, where data collection and model fitting alternate sequentially, we prove that model performance improves initially but can stall after only three rounds. Label noise arising from, for instance, annotator disagreement leads to even stronger negative results. Our second model generalizes the first to the case where data collection and model fitting have a hierarchical dependency structure. We show that this design guarantees strictly more progress than the first, albeit at a significant increase in complexity. We support our theoretical analysis by simulating dynamic benchmarks on two popular datasets. These results illuminate the benefits and practical limitations of dynamic benchmarking, providing both a theoretical foundation and a causal explanation for observed bottlenecks in empirical work.


page 1

page 2

page 3

page 4


From Monolith to Microservices: Static and Dynamic Analysis Comparison

One of the most challenging problems in the migration of a monolith to a...

Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants

In Dynamic Adversarial Data Collection (DADC), human annotators are task...

DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation

Recently, one critical issue looms large in the field of recommender sys...

Dynamic Matching Algorithms in Practice

In recent years, significant advances have been made in the design and a...

Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise

Supervised learning under label noise has seen numerous advances recentl...

Autotuning Benchmarking Techniques: A Roofline Model Case Study

Peak performance metrics published by vendors often do not correspond to...

Adaptive data collection for intra-individual studies affected by adherence

Recently the use of mobile technologies in Ecological Momentary Assessme...

Please sign up or login with your details

Forgot password? Click here to reset