The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain

by   Arseny Moskvichev, et al.

The abilities to form and abstract concepts is key to human intelligence, but such abilities remain lacking in state-of-the-art AI systems. There has been substantial research on conceptual abstraction in AI, particularly using idealized domains such as Raven's Progressive Matrices and Bongard problems, but even when AI systems succeed on such problems, the systems are rarely evaluated in depth to see if they have actually grasped the concepts they are meant to capture. In this paper we describe an in-depth evaluation benchmark for the Abstraction and Reasoning Corpus (ARC), a collection of few-shot abstraction and analogy problems developed by Chollet [2019]. In particular, we describe ConceptARC, a new, publicly available benchmark in the ARC domain that systematically assesses abstraction and generalization abilities on a number of basic spatial and semantic concepts. ConceptARC differs from the original ARC dataset in that it is specifically organized around "concept groups" – sets of problems that focus on specific concepts and that are vary in complexity and level of abstraction. We report results on testing humans on this benchmark as well as three machine solvers: the top two programs from a 2021 ARC competition and OpenAI's GPT-4. Our results show that humans substantially outperform the machine solvers on this benchmark, showing abilities to abstract and generalize concepts that are not yet captured by AI systems. We believe that this benchmark will spur improvements in the development of AI systems for conceptual abstraction and in the effective evaluation of such systems.


page 4

page 6

page 13

page 20

page 21

page 22


Evaluating Understanding on Conceptual Abstraction Benchmarks

A long-held objective in AI is to build systems that understand concepts...

Abstraction and Analogy-Making in Artificial Intelligence

Conceptual abstraction and analogy-making are key abilities underlying h...

Applying human-centered AI in developing effective human-AI teaming: A perspective of human-AI joint cognitive systems

Research and application have used human-AI teaming (HAT) as a new parad...

Human-Level Intelligence or Animal-Like Abilities?

The vision systems of the eagle and the snake outperform everything that...

Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning

Humans have an inherent ability to learn novel concepts from only a few ...

The Measure of Intelligence

To make deliberate progress towards more intelligent and more human-like...

Abstraction for Zooming-In to Unsolvability Reasons of Grid-Cell Problems

Humans are capable of abstracting away irrelevant details when studying ...

Please sign up or login with your details

Forgot password? Click here to reset