An Assessment of ChatGPT on Log Data

by   Priyanka Mudgal, et al.

Recent development of large language models (LLMs), such as ChatGPT has been widely applied to a wide range of software engineering tasks. Many papers have reported their analysis on the potential advantages and limitations of ChatGPT for writing code, summarization, text generation, etc. However, the analysis of the current state of ChatGPT for log processing has received little attention. Logs generated by large-scale software systems are complex and hard to understand. Despite their complexity, they provide crucial information for subject matter experts to understand the system status and diagnose problems of the systems. In this paper, we investigate the current capabilities of ChatGPT to perform several interesting tasks on log data, while also trying to identify its main shortcomings. Our findings show that the performance of the current version of ChatGPT for log processing is limited, with a lack of consistency in responses and scalability issues. We also outline our views on how we perceive the role of LLMs in the log processing discipline and possible next steps to improve the current capabilities of ChatGPT and the future LLMs in this area. We believe our work can contribute to future academic research to address the identified issues.


page 1

page 2

page 3

page 4


An Evaluation of Log Parsing with ChatGPT

Software logs play an essential role in ensuring the reliability and mai...

Towards an Understanding of Large Language Models in Software Engineering Tasks

Large Language Models (LLMs) have drawn widespread attention and researc...

The Sense of Logging in the Linux Kernel

Logging plays a crucial role in software engineering because it is key t...

ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation

Recent advancements in large language models (LLMs) have demonstrated ex...

A Prompt Log Analysis of Text-to-Image Generation Systems

Recent developments in large language models (LLM) and generative AI hav...

Large Language Models in Fault Localisation

Large Language Models (LLMs) have shown promise in multiple software eng...

Please sign up or login with your details

Forgot password? Click here to reset