Comparing Software Developers with ChatGPT: An Empirical Investigation

by   Nathalia Nascimento, et al.
University of Waterloo

The advent of automation in particular Software Engineering (SE) tasks has transitioned from theory to reality. Numerous scholarly articles have documented the successful application of Artificial Intelligence to address issues in areas such as project management, modeling, testing, and development. A recent innovation is the introduction of ChatGPT, an ML-infused chatbot, touted as a resource proficient in generating programming codes and formulating software testing strategies for developers and testers respectively. Although there is speculation that AI-based computation can increase productivity and even substitute software engineers in software development, there is currently a lack of empirical evidence to verify this. Moreover, despite the primary focus on enhancing the accuracy of AI systems, non-functional requirements including energy efficiency, vulnerability, fairness (i.e., human bias), and safety frequently receive insufficient attention. This paper posits that a comprehensive comparison of software engineers and AI-based solutions, considering various evaluation criteria, is pivotal in fostering human-machine collaboration, enhancing the reliability of AI-based methods, and understanding task suitability for humans or AI. Furthermore, it facilitates the effective implementation of cooperative work structures and human-in-the-loop processes. This paper conducts an empirical investigation, contrasting the performance of software engineers and AI systems, like ChatGPT, across different evaluation metrics. The empirical study includes a case of assessing ChatGPT-generated code versus code produced by developers and uploaded in Leetcode.


Software Engineers vs. Machine Learning Algorithms: An Empirical Study Assessing Performance and Reuse Tasks

Several papers have recently contained reports on applying machine learn...

What is Software Quality for AI Engineers? Towards a Thinning of the Fog

It is often overseen that AI-enabled systems are also software systems a...

GitHub Copilot AI pair programmer: Asset or Liability?

Automatic program synthesis is a long-lasting dream in software engineer...

Optimizing Workflow for Elite Developers: Perspectives on Leveraging SE Bots

Small-scale automation services in Software Engineering, known as SE Bot...

The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

Generative AI tools hold promise to increase human productivity. This pa...

AI and Non AI Assessments for Dementia

Current progress in the artificial intelligence domain has led to the de...

Please sign up or login with your details

Forgot password? Click here to reset