How Much Data Analytics is Enough? The ROI of Machine Learning Classification and its Application to Requirements Dependency Classification

by   Gouri Deshpande, et al.

Machine Learning (ML) can substantially improve the efficiency and effectiveness of organizations and is widely used for different purposes within Software Engineering. However, the selection and implementation of ML techniques rely almost exclusively on accuracy criteria. Thus, for organizations wishing to realize the benefits of ML investments, this narrow approach ignores crucial considerations around the anticipated costs of the ML activities across the ML lifecycle, while failing to account for the benefits that are likely to accrue from the proposed activity. We present findings for an approach that addresses this gap by enhancing the accuracy criterion with return on investment (ROI) considerations. Specifically, we analyze the performance of the two state-of-the-art ML techniques: Random Forest and Bidirectional Encoder Representations from Transformers (BERT), based on accuracy and ROI for two publicly available data sets. Specifically, we compare decision-making on requirements dependency extraction (i) exclusively based on accuracy and (ii) extended to include ROI analysis. As a result, we propose recommendations for selecting ML classification techniques based on the degree of training data used. Our findings indicate that considering ROI as additional criteria can drastically influence ML selection when compared to decisions based on accuracy as the sole criterion


User-centric Composable Services: A New Generation of Personal Data Analytics

Machine Learning (ML) techniques, such as Neural Network, are widely use...

Beyond Accuracy: ROI-driven Data Analytics of Empirical Data

This vision paper demonstrates that it is crucial to consider Return-on-...

Approaches to Improving the Accuracy of Machine Learning Models in Requirements Elicitation Techniques Selection

Selecting techniques is a crucial element of the business analysis appro...

ML-AQP: Query-Driven Approximate Query Processing based on Machine Learning

As more and more organizations rely on data-driven decision making, larg...

Unraveling the Interconnected Axes of Heterogeneity in Machine Learning for Democratic and Inclusive Advancements

The growing utilization of machine learning (ML) in decision-making proc...

A Machine Learning Approach for Hierarchical Classification of Software Requirements

Context: Classification of software requirements into different categori...

Explainable AI for tool wear prediction in turning

This research aims develop an Explainable Artificial Intelligence (XAI) ...

Please sign up or login with your details

Forgot password? Click here to reset