Hallmarks of Human-Machine Collaboration: A framework for assessment in the DARPA Communicating with Computers Program

by   Robyn Kozierok, et al.

There is a growing desire to create computer systems that can communicate effectively to collaborate with humans on complex, open-ended activities. Assessing these systems presents significant challenges. We describe a framework for evaluating systems engaged in open-ended complex scenarios where evaluators do not have the luxury of comparing performance to a single right answer. This framework has been used to evaluate human-machine creative collaborations across story and music generation, interactive block building, and exploration of molecular mechanisms in cancer. These activities are fundamentally different from the more constrained tasks performed by most contemporary personal assistants as they are generally open-ended, with no single correct solution, and often no obvious completion criteria. We identified the Key Properties that must be exhibited by successful systems. From there we identified "Hallmarks" of success – capabilities and features that evaluators can observe that would be indicative of progress toward achieving a Key Property. In addition to being a framework for assessment, the Key Properties and Hallmarks are intended to serve as goals in guiding research direction.


page 1

page 7

page 11

page 15

page 16


Plan, Write, and Revise: an Interactive System for Open-Domain Story Generation

Story composition is a challenging problem for machines and even for hum...

OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics

Automatic metrics are essential for developing natural language generati...

Towards Stroke Patients' Upper-limb Automatic Motor Assessment Using Smartwatches

Assessing the physical condition in rehabilitation scenarios is a challe...

A Brief Guide to Designing and Evaluating Human-Centered Interactive Machine Learning

Interactive machine learning (IML) is a field of research that explores ...

StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement

Despite its benefits for children's skill development and parent-child b...

crea.blender: A Neural Network-Based Image Generation Game to Assess Creativity

We present a pilot study on crea.blender, a novel co-creative game desig...

Please sign up or login with your details

Forgot password? Click here to reset