Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning

by   Zhanming Jie, et al.

Chain-of-thought (CoT) prompting with large language models has proven effective in numerous natural language processing tasks, but designing prompts that generalize well to diverse problem types can be challenging, especially in the context of math word problem (MWP) solving. Additionally, it is common to have a large amount of training data that have a better diversity coverage but CoT annotations are not available, which limits the use of supervised learning techniques. To address these issues, we investigate two approaches to leverage the training data in a few-shot prompting scenario: dynamic program prompting and program distillation. Our approach is largely inspired by Gao et al., (2022), where they proposed to replace the CoT with the programs as the intermediate reasoning step. Such a prompting strategy allows us to accurately verify the answer correctness through program execution in MWP solving. Our dynamic program prompting involves annotating the training data by sampling correct programs from a large language model, while program distillation involves adapting a smaller model to the program-annotated training data. Our experiments on three standard MWP datasets demonstrate the effectiveness of these approaches, yielding significant improvements over previous baselines for prompting and fine-tuning. Our results suggest that leveraging a large amount of training data can improve the generalization ability of prompts and boost the performance of fine-tuned small models in MWP solving


page 1

page 2

page 3

page 4


Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

Recently, there has been significant progress in teaching language model...

Exploring an LM to generate Prolog Predicates from Mathematics Questions

Recently, there has been a surge in interest in NLP driven by ChatGPT. C...

Learning Performance-Improving Code Edits

The waning of Moore's Law has shifted the focus of the tech industry tow...

Program Synthesis with Large Language Models

This paper explores the limits of the current generation of large langua...

Design of Chain-of-Thought in Math Problem Solving

Chain-of-Thought (CoT) plays a crucial role in reasoning for math proble...

Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuning

Recently, the release of INSTRUCTEVAL has provided valuable insights int...

Strategic Reasoning with Language Models

Strategic reasoning enables agents to cooperate, communicate, and compet...

Please sign up or login with your details

Forgot password? Click here to reset