On the Depth between Beam Search and Exhaustive Search for Text Generation

by   Yuu Jinnai, et al.

Beam search and exhaustive search are two extreme ends of text decoding algorithms with respect to the search depth. Beam search is limited in both search width and depth, whereas exhaustive search is a global search that has no such limitations. Surprisingly, beam search is not only computationally cheaper but also performs better than exhaustive search despite its higher search error. Plenty of research has investigated a range of beam widths, from small to large, and reported that a beam width that is neither too large nor too small is desirable. However, in terms of search depth, only the two extreme ends, beam search and exhaustive search are studied intensively. In this paper, we examine a range of search depths between the two extremes to discover the desirable search depth. To this end, we introduce Lookahead Beam Search (LBS), a multi-step lookahead search that optimizes the objective considering a fixed number of future steps. Beam search and exhaustive search are special cases of LBS where the lookahead depth is set to 0 and ∞, respectively. We empirically evaluate the performance of LBS and find that it outperforms beam search overall on machine translation tasks. The result suggests there is room for improvement in beam search by searching deeper. Inspired by the analysis, we propose Lookbehind Heuristic Beam Search, a computationally feasible search algorithm that heuristically simulates LBS with 1-step lookahead. The empirical results show that the proposed method outperforms vanilla beam search on machine translation and text summarization tasks.


When to Finish? Optimal Beam Search for Neural Text Generation (modulo beam size)

In neural text generation such as neural machine translation, summarizat...

What Do You Get When You Cross Beam Search with Nucleus Sampling?

We combine beam search with the probabilistic pruning technique of nucle...

Rethinking the Evaluation of Neural Machine Translation

The evaluation of neural machine translation systems is usually built up...

Conformal Autoregressive Generation: Beam Search with Coverage Guarantees

We introduce two new extensions to the beam search algorithm based on co...

A Quantum Search Decoder for Natural Language Processing

Probabilistic language models, e.g. those based on an LSTM, often face t...

Narrowband Transmit Beam Pattern in medical ultrasound: a stochastic approach to delays optimization

Transmit Beam Pattern (TBP) optimization is an important task in medical...

Learning to Accelerate Decomposition for Multi-Directional 3D Printing

Multi-directional 3D printing has the capability of decreasing or elimin...

Please sign up or login with your details

Forgot password? Click here to reset