Smart Home Appliances: Chat with Your Fridge

by   Denis Gudovskiy, et al.

Current home appliances are capable to execute a limited number of voice commands such as turning devices on or off, adjusting music volume or light conditions. Recent progress in machine reasoning gives an opportunity to develop new types of conversational user interfaces for home appliances. In this paper, we apply state-of-the-art visual reasoning model and demonstrate that it is feasible to ask a smart fridge about its contents and various properties of the food with close-to-natural conversation experience. Our visual reasoning model answers user questions about existence, count, category and freshness of each product by analyzing photos made by the image sensor inside the smart fridge. Users may chat with their fridge using off-the-shelf phone messenger while being away from home, for example, when shopping in the supermarket. We generate a visually realistic synthetic dataset to train machine learning reasoning model that achieves 95 data. We present the results of initial user tests and discuss how we modify distribution of generated questions for model training based on human-in-the-loop guidance. We open source code for the whole system including dataset generation, reasoning model and demonstration scripts.


page 5

page 10

page 11


Design Challenges for the Implementation of Smart Homes

Home automation for many years had faced challenges that limit its sprea...

Arcades: A deep model for adaptive decision making in voice controlled smart-home

In a voice-controlled smart-home, a controller must respond not only to ...

Conversation-Based Complex Event Management in Smart-Spaces

Smart space management can be done in many ways. On one hand, there are ...

Sasha: creative goal-oriented reasoning in smart homes with large language models

Every smart home user interaction has an explicit or implicit goal. Exis...

Product Title Generation for Conversational Systems using BERT

Through recent advancements in speech technology and introduction of sma...

Online Guest Detection in a Smart Home using Pervasive Sensors and Probabilistic Reasoning

Smart home environments equipped with distributed sensor networks are ca...

Rewriting the Script: Adapting Text Instructions for Voice Interaction

Voice assistants have sharply risen in popularity in recent years, but t...

Please sign up or login with your details

Forgot password? Click here to reset