A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categories

by   Arijit Dasgupta, et al.

Recent work in computer vision and cognitive reasoning has given rise to an increasing adoption of the Violation-of-Expectation (VoE) paradigm in synthetic datasets. Inspired by infant psychology, researchers are now evaluating a model's ability to label scenes as either expected or surprising with knowledge of only expected scenes. However, existing VoE-based 3D datasets in physical reasoning provide mainly vision data with little to no heuristics or inductive biases. Cognitive models of physical reasoning reveal infants create high-level abstract representations of objects and interactions. Capitalizing on this knowledge, we established a benchmark to study physical reasoning by curating a novel large-scale synthetic 3D VoE dataset armed with ground-truth heuristic labels of causally relevant features and rules. To validate our dataset in five event categories of physical reasoning, we benchmarked and analyzed human performance. We also proposed the Object File Physical Reasoning Network (OFPR-Net) which exploits the dataset's novel heuristics to outperform our baseline and ablation models. The OFPR-Net is also flexible in learning an alternate physical reality, showcasing its ability to learn universal causal relationships in physical reasoning to create systems with better interpretability.


page 3

page 5

page 6


AVoE: A Synthetic 3D Dataset on Understanding Violation of Expectation for Artificial Cognition

Recent work in cognitive reasoning and computer vision has engendered an...

SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments

Recent advancements in deep learning, computer vision, and embodied AI h...

PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning

A critical aspect of human visual perception is the ability to parse vis...

Forward Prediction for Physical Reasoning

Physical reasoning requires forward prediction: the ability to forecast ...

Blocksworld Revisited: Learning and Reasoning to Generate Event-Sequences from Image Pairs

The process of identifying changes or transformations in a scene along w...

The Scope and Limits of Simulation in Cognitive Models

It has been proposed that human physical reasoning consists largely of r...

On the Learning Mechanisms in Physical Reasoning

Is dynamics prediction indispensable for physical reasoning? If so, what...

Please sign up or login with your details

Forgot password? Click here to reset