Competitions in AI – Robustly Ranking Solvers Using Statistical Resampling

by   Chris Fawcett, et al.

Solver competitions play a prominent role in assessing and advancing the state of the art for solving many problems in AI and beyond. Notably, in many areas of AI, competitions have had substantial impact in guiding research and applications for many years, and for a solver to be ranked highly in a competition carries considerable weight. But to which extent can we expect competition results to generalise to sets of problem instances different from those used in a particular competition? This is the question we investigate here, using statistical resampling techniques. We show that the rankings resulting from the standard interpretation of competition results can be very sensitive to even minor changes in the benchmark instance set used as the basis for assessment and can therefore not be expected to carry over to other samples from the same underlying instance distribution. To address this problem, we introduce a novel approach to statistically meaningful analysis of competition results based on resampling performance data. Our approach produces confidence intervals of competition scores as well as statistically robust solver rankings with bounded error. Applied to recent SAT, AI planning and computer vision competitions, our analysis reveals frequent statistical ties in solver performance as well as some inversions of ranks compared to the official results based on simple scoring.


page 1

page 2

page 3

page 4


A portfolio-based analysis method for competition results

Competitions such as the MiniZinc Challenges or the SAT competitions hav...

The Configurable SAT Solver Challenge (CSSC)

It is well known that different solution strategies work well for differ...

SATzilla: Portfolio-based Algorithm Selection for SAT

It has been widely observed that there is no single "dominant" SAT solve...

Towards Miss Universe Automatic Prediction: The Evening Gown Competition

Can we predict the winner of Miss Universe after watching how they strid...

How to Host a Data Competition: Statistical Advice for Design and Analysis of a Data Competition

Data competitions rely on real-time leaderboards to rank competitor entr...

Simulation leagues: Analysis of competition formats

The selection of an appropriate competition format is critical for both ...

A Framework for Generating Informative Benchmark Instances

Benchmarking is an important tool for assessing the relative performance...

Please sign up or login with your details

Forgot password? Click here to reset