Satisfiability-Aided Language Models Using Declarative Prompting

by   Xi Ye, et al.

Prior work has combined chain-of-thought prompting in large language models (LLMs) with programmatic representations to perform effective and transparent reasoning. While such an approach works very well for tasks that only require forward reasoning (e.g., straightforward arithmetic), it is less effective for constraint solving problems that require more sophisticated planning and search. In this paper, we propose a new satisfiability-aided language modeling (SATLM) approach for improving the reasoning capabilities of LLMs. We use an LLM to generate a declarative task specification rather than an imperative program and leverage an off-the-shelf automated theorem prover to derive the final answer. This approach has two key advantages. The declarative specification is closer to the problem description than the reasoning steps are, so the LLM can parse it out of the description more accurately. Furthermore, by offloading the actual reasoning task to an automated theorem prover, our approach can guarantee the correctness of the answer with respect to the parsed specification and avoid planning errors in the solving process. We evaluate SATLM on 6 different datasets and show that it consistently outperforms program-aided LMs in an imperative paradigm. In particular, SATLM outperforms program-aided LMs by 23 arithmetic reasoning dataset; SATLM also achieves a new SoTA on LSAT, surpassing previous models that are trained on the full training set.


page 1

page 2

page 3

page 4


PAL: Program-aided Language Models

Large language models (LLMs) have recently demonstrated an impressive ab...

Automatic Model Selection with Large Language Models for Reasoning

Chain-of-Thought and Program-Aided Language Models represent two distinc...

Cumulative Reasoning with Large Language Models

While language models are powerful and versatile, they often fail to add...

A Conversational Paradigm for Program Synthesis

Program synthesis strives to generate a computer program as a solution t...

Generalized Planning in PDDL Domains with Pretrained Large Language Models

Recent work has considered whether large language models (LLMs) can func...

Solving math word problems with process- and outcome-based feedback

Recent work has shown that asking language models to generate reasoning ...

Please sign up or login with your details

Forgot password? Click here to reset