Fair Comparison: Quantifying Variance in Resultsfor Fine-grained Visual Categorization

by   Matthew Gwilliam, et al.

For the task of image classification, researchers work arduously to develop the next state-of-the-art (SOTA) model, each bench-marking their own performance against that of their predecessors and of their peers. Unfortunately, the metric used most frequently to describe a model's performance, average categorization accuracy, is often used in isolation. As the number of classes increases, such as in fine-grained visual categorization (FGVC), the amount of information conveyed by average accuracy alone dwindles. While its most glaring weakness is its failure to describe the model's performance on a class-by-class basis, average accuracy also fails to describe how performance may vary from one trained model of the same architecture, on the same dataset, to another (both averaged across all categories and at the per-class level). We first demonstrate the magnitude of these variations across models and across class distributions based on attributes of the data, comparing results on different visual domains and different per-class image distributions, including long-tailed distributions and few-shot subsets. We then analyze the impact various FGVC methods have on overall and per-class variance. From this analysis, we both highlight the importance of reporting and comparing methods based on information beyond overall accuracy, as well as point out techniques that mitigate variance in FGVC results.


page 2

page 3

page 6

page 7


Classification-Specific Parts for Improving Fine-Grained Visual Categorization

Fine-grained visual categorization is a classification task for distingu...

CDLT: A Dataset with Concept Drift and Long-Tailed Distribution for Fine-Grained Visual Categorization

Data is the foundation for the development of computer vision, and the e...

Facing the Hard Problems in FGVC

In fine-grained visual categorization (FGVC), there is a near-singular f...

Fine-graind Image Classification via Combining Vision and Language

Fine-grained image classification is a challenging task due to the large...

Fine-grained Categorization -- Short Summary of our Entry for the ImageNet Challenge 2012

In this paper, we tackle the problem of visual categorization of dog bre...

The Devil is in the Tails: Fine-grained Classification in the Wild

The world is long-tailed. What does this mean for computer vision and vi...

Coping with Change: Learning Invariant and Minimum Sufficient Representations for Fine-Grained Visual Categorization

Fine-grained visual categorization (FGVC) is a challenging task due to s...

Please sign up or login with your details

Forgot password? Click here to reset