Knowledge Guided Learning: Towards Open Domain Egocentric Action Recognition with Zero Supervision

Advances in deep learning have enabled the development of models that have exhibited a remarkable tendency to recognize and even localize actions in videos. However, they tend to experience errors when faced with scenes or examples beyond their initial training environment. Hence, they fail to adapt to new domains without significant retraining with large amounts of annotated data. Current algorithms are trained in an inductive learning environment where they use data-driven models to learn associations between input observations with a fixed set of known classes. In this paper, we propose to overcome these limitations by moving to an open world setting by decoupling the ideas of recognition and reasoning. Building upon the compositional representation offered by Grenander's Pattern Theory formalism, we show that attention and commonsense knowledge can be used to enable the self-supervised discovery of novel actions in egocentric videos in an open-world setting, a considerably more difficult task than zero-shot learning and (un)supervised domain adaptation tasks where target domain data (both labeled and unlabeled) are available during training. We show that our approach can be used to infer and learn novel classes for open vocabulary classification in egocentric videos and novel object detection with zero supervision. Extensive experiments show that it performs competitively with fully supervised baselines on publicly available datasets under open-world conditions. This is one of the first works to address the problem of open-world action recognition in egocentric videos with zero human supervision to the best of our knowledge.


page 1

page 7


Discovering Novel Actions in an Open World with Object-Grounded Visual Commonsense Reasoning

Learning to infer labels in an open world, i.e., in an environment where...

Generalized Zero-Shot Learning for Action Recognition with Web-Scale Video Data

Action recognition in surveillance video makes our life safer by detecti...

All About Knowledge Graphs for Actions

Current action recognition systems require large amounts of training dat...

Guided weak supervision for action recognition with scarce data to assess skills of children with autism

Diagnostic and intervention methodologies for skill assessment of autism...

Abductive Reasoning as Self-Supervision for Common Sense Question Answering

Question answering has seen significant advances in recent times, especi...

Few-Shot Drum Transcription in Polyphonic Music

Data-driven approaches to automatic drum transcription (ADT) are often l...

Learning Actor-centered Representations for Action Localization in Streaming Videos using Predictive Learning

Event perception tasks such as recognizing and localizing actions in str...

Please sign up or login with your details

Forgot password? Click here to reset