Leveraging Large Language Models for Automated Dialogue Analysis

by   Sarah E. Finch, et al.

Developing high-performing dialogue systems benefits from the automatic identification of undesirable behaviors in system responses. However, detecting such behaviors remains challenging, as it draws on a breadth of general knowledge and understanding of conversational practices. Although recent research has focused on building specialized classifiers for detecting specific dialogue behaviors, the behavior coverage is still incomplete and there is a lack of testing on real-world human-bot interactions. This paper investigates the ability of a state-of-the-art large language model (LLM), ChatGPT-3.5, to perform dialogue behavior detection for nine categories in real human-bot dialogues. We aim to assess whether ChatGPT can match specialized models and approximate human performance, thereby reducing the cost of behavior detection tasks. Our findings reveal that neither specialized models nor ChatGPT have yet achieved satisfactory results for this task, falling short of human performance. Nevertheless, ChatGPT shows promising potential and often outperforms specialized detection models. We conclude with an in-depth examination of the prevalent shortcomings of ChatGPT, offering guidance for future research to enhance LLM capabilities.


page 1

page 2

page 3

page 4


A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding

Zero-shot dialogue understanding aims to enable dialogue to track the us...

Refine and Imitate: Reducing Repetition and Inconsistency in Persuasion Dialogues via Reinforcement Learning and Human Demonstration

Despite the recent success of large-scale language models on various dow...

Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?

Coaxing out desired behavior from pretrained models, while avoiding unde...

A Critical Review of Large Language Models: Sensitivity, Bias, and the Path Toward Specialized AI

This paper examines the comparative effectiveness of a specialized compi...

Is ChatGPT Equipped with Emotional Dialogue Capabilities?

This report presents a study on the emotional dialogue capability of Cha...

SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation

Dialogue segmentation is a crucial task for dialogue systems allowing a ...

Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling

Enhancing AI systems with efficient communication skills that align with...

Please sign up or login with your details

Forgot password? Click here to reset