Evaluating the Efficacy of Supervised Learning vs Large Language Models for Identifying Cognitive Distortions and Suicidal Risks in Chinese Social Media

by   Hongzhi Qi, et al.

Large language models, particularly those akin to the rapidly progressing GPT series, are gaining traction for their expansive influence. While there is keen interest in their applicability within medical domains such as psychology, tangible explorations on real-world data remain scant. Concurrently, users on social media platforms are increasingly vocalizing personal sentiments; under specific thematic umbrellas, these sentiments often manifest as negative emotions, sometimes escalating to suicidal inclinations. Timely discernment of such cognitive distortions and suicidal risks is crucial to effectively intervene and potentially avert dire circumstances. Our study ventured into this realm by experimenting on two pivotal tasks: suicidal risk and cognitive distortion identification on Chinese social media platforms. Using supervised learning as a baseline, we examined and contrasted the efficacy of large language models via three distinct strategies: zero-shot, few-shot, and fine-tuning. Our findings revealed a discernible performance gap between the large language models and traditional supervised learning approaches, primarily attributed to the models' inability to fully grasp subtle categories. Notably, while GPT-4 outperforms its counterparts in multiple scenarios, GPT-3.5 shows significant enhancement in suicide risk classification after fine-tuning. To our knowledge, this investigation stands as the maiden attempt at gauging large language models on Chinese social media tasks. This study underscores the forward-looking and transformative implications of using large language models in the field of psychology. It lays the groundwork for future applications in psychological research and practice.


page 1

page 2

page 3

page 4


Transfer Learning for Power Outage Detection Task with Limited Training Data

Early detection of power outages is crucial for maintaining a reliable p...

Clickbait Detection via Large Language Models

Clickbait, which aims to induce users with some surprising and even thri...

SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding

We study the ability of transformer-based language models (LMs) to under...

A Comparative Study of Pretrained Language Models on Thai Social Text Categorization

The ever-growing volume of data of user-generated content on social medi...

Incorporating Emotions into Health Mention Classification Task on Social Media

The health mention classification (HMC) task is the process of identifyi...

Toxicity Detection with Generative Prompt-based Inference

Due to the subtleness, implicity, and different possible interpretations...

Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media

Stance detection predicts attitudes towards targets in texts and has gai...

Please sign up or login with your details

Forgot password? Click here to reset