Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents

by   Zihao Wang, et al.

In this paper, we study the problem of planning in Minecraft, a popular, democratized yet challenging open-ended environment for developing multi-task embodied agents. We've found two primary challenges of empowering such agents with planning: 1) planning in an open-ended world like Minecraft requires precise and multi-step reasoning due to the long-term nature of the tasks, and 2) as vanilla planners do not consider the proximity to the current agent when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient. To this end, we propose "Describe, Explain, Plan and Select" (DEPS), an interactive planning approach based on Large Language Models (LLMs). Our approach helps with better error correction from the feedback during the long-haul planning, while also bringing the sense of proximity via goal Selector, a learnable module that ranks parallel sub-goals based on the estimated steps of completion and improves the original plan accordingly. Our experiments mark the milestone of the first multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly doubles the overall performances. Finally, the ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the grand challenge with our approach. The code is released at


page 2

page 5

page 14

page 16

page 17

page 18

page 19

page 20


Distilling Script Knowledge from Large Language Models for Constrained Language Planning

In everyday life, humans often plan their actions by following step-by-s...

Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction

We study the problem of learning goal-conditioned policies in Minecraft,...

AdaPlanner: Adaptive Planning from Feedback with Language Models

Large language models (LLMs) have recently demonstrated the potential in...

RoCo: Dialectic Multi-Robot Collaboration with Large Language Models

We propose a novel approach to multi-robot collaboration that harnesses ...

Using Sub-Optimal Plan Detection to Identify Commitment Abandonment in Discrete Environments

Assessing whether an agent has abandoned a goal or is actively pursuing ...

Assumption-Based Planning

The purpose of the paper is to introduce a new approach of planning call...

The Introspective Agent: Interdependence of Strategy, Physiology, and Sensing for Embodied Agents

The last few years have witnessed substantial progress in the field of e...

Please sign up or login with your details

Forgot password? Click here to reset