Show Your Work: Scratchpads for Intermediate Computation with Language Models

by   Maxwell Nye, et al.

Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text or synthesizing computer programs. However, they struggle with tasks that require unbounded multi-step computation, such as adding integers or executing programs. Surprisingly, we find that these same models are able to perform complex multi-step computations – even in the few-shot regime – when asked to perform the operation "step by step", showing the results of intermediate computations. In particular, we train transformers to perform multi-step computations by asking them to emit intermediate computation steps into a "scratchpad". On a series of increasingly complex tasks ranging from long addition to the execution of arbitrary programs, we show that scratchpads dramatically improve the ability of language models to perform multi-step computations.


Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

Recently, there has been significant progress in teaching language model...

ART: Automatic multi-step reasoning and tool-use for large language models

Large language models (LLMs) can perform complex reasoning in few- and z...

Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning

Large language models (LLMs) have been shown to be capable of impressive...

Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach

While Pre-trained Language Models (PLMs) internalize a great amount of w...

Why think step-by-step? Reasoning emerges from the locality of experience

Humans have a powerful and mysterious capacity to reason. By working thr...

Limitations of Language Models in Arithmetic and Symbolic Induction

Recent work has shown that large pretrained Language Models (LMs) can no...

Making Bubbling Practical

Bubbling is a run-time graph transformation studied for the execution of...

Please sign up or login with your details

Forgot password? Click here to reset