Rewriting the Script: Adapting Text Instructions for Voice Interaction

by   Alyssa Hwang, et al.

Voice assistants have sharply risen in popularity in recent years, but their use has been limited mostly to simple applications like music, hands-free search, or control of internet-of-things devices. What would it take for voice assistants to guide people through more complex tasks? In our work, we study the limitations of the dominant approach voice assistants take to complex task guidance: reading aloud written instructions. Using recipes as an example, we observe twelve participants cook at home with a state-of-the-art voice assistant. We learn that the current approach leads to nine challenges, including obscuring the bigger picture, overwhelming users with too much information, and failing to communicate affordances. Instructions delivered by a voice assistant are especially difficult because they cannot be skimmed as easily as written instructions. Alexa in particular did not surface crucial details to the user or answer questions well. We draw on our observations to propose eight ways in which voice assistants can “rewrite the script” – summarizing, signposting, splitting, elaborating, volunteering, reordering, redistributing, and visualizing – to transform written sources into forms that are readily communicated through spoken conversation. We conclude with a vision of how modern advancements in natural language processing can be leveraged for intelligent agents to guide users effectively through complex tasks.


page 3

page 5

page 11


A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures

Despite huge gains in performance in natural language understanding via ...

Challenges in Supporting Exploratory Search through Voice Assistants

Voice assistants have been successfully adopted for simple, routine task...

Incorporating Voice Instructions in Model-Based Reinforcement Learning for Self-Driving Cars

This paper presents a novel approach that supports natural language voic...

Voice Over Body? Older Adults' Reactions to Robot and Voice Assistant Facilitators of Group Conversation

Intelligent agents have great potential as facilitators of group convers...

Pardon? An Overview of the Current State and Requirements of Voice User Interfaces for Blind and Visually Impaired Users

People with special needs like blind and visually impaired (BVI) people ...

VOICE: Visual Oracle for Interaction, Conversation, and Explanation

We present VOICE, a novel approach for connecting large language models'...

Smart Home Appliances: Chat with Your Fridge

Current home appliances are capable to execute a limited number of voice...

Please sign up or login with your details

Forgot password? Click here to reset