PUMiner: Mining Security Posts from Developer Question and Answer Websites with PU Learning

by   Triet H. M. Le, et al.

Security is an increasing concern in software development. Developer Question and Answer (Q A) websites provide a large amount of security discussion. Existing studies have used human-defined rules to mine security discussions, but these works still miss many posts, which may lead to an incomplete analysis of the security practices reported on Q A websites. Traditional supervised Machine Learning methods can automate the mining process; however, the required negative (non-security) class is too expensive to obtain. We propose a novel learning framework, PUMiner, to automatically mine security posts from Q A websites. PUMiner builds a context-aware embedding model to extract features of the posts, and then develops a two-stage PU model to identify security content using the labelled Positive and Unlabelled posts. We evaluate PUMiner on more than 17.2 million posts on Stack Overflow and 52,611 posts on Security StackExchange. We show that PUMiner is effective with the validation performance of at least 0.85 across all model configurations. Moreover, Matthews Correlation Coefficient (MCC) of PUMiner is 0.906, 0.534 and 0.084 points higher than one-class SVM, positive-similarity filtering, and one-stage PU models on unseen testing posts, respectively. PUMiner also performs well with an MCC of 0.745 for scenarios where string matching totally fails. Even when the ratio of the labelled positive posts to the unlabelled ones is only 1:100, PUMiner still achieves a strong MCC of 0.65, which is 160 fully-supervised learning. Using PUMiner, we provide the largest and up-to-date security content on Q A websites for practitioners and researchers.


Broken External Links on Stack Overflow

Stack Overflow hosts valuable programming-related knowledge with 11,926,...

Demystifying the Mysteries of Security Vulnerability Discussions on Developer Q A Sites

Detection and mitigation of Security Vulnerabilities (SVs) are integral ...

Sentiment Overflow in the Testing Stack: Analysing Software Testing Posts on Stack Overflow

Software testing is an integral part of modern software engineering prac...

How Reliable is the Crowdsourced Knowledge of Security Implementation?

Stack Overflow (SO) is the most popular online Q&A site for developers t...

Opinion mining of text documents written in Macedonian language

The ability to extract public opinion from web portals such as review si...

A Puff of Steem: Security Analysis of Decentralized Content Curation

Decentralized content curation is the process through which uploaded pos...

Please sign up or login with your details

Forgot password? Click here to reset