A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature

by   Sarah Heckman, et al.

Computing Education Research (CER) is critical for supporting the increasing number of students who need to learn computing skills. To systematically advance knowledge, publications must be clear enough to support replications, meta-analyses, and theory-building. The goal of this study is to characterize the reporting of empiricism in CER literature by identifying whether publications include information to support replications, meta-analyses, and theory building. The research questions are: RQ1) What percentage of papers in CER venues have empirical evaluation? RQ2) What are the characteristics of the empirical evaluation? RQ3) Do the papers with empirical evaluation follow reporting norms (both for inclusion and for labeling of key information)? We conducted an SLR of 427 papers published during 2014 and 2015 in five CER venues: SIGCSE TS, ICER, ITiCSE, TOCE, and CSE. We developed and applied the CER Empiricism Assessment Rubric. Over 80 evaluation. Quantitative evaluation methods were the most frequent. Papers most frequently reported results on interventions around pedagogical techniques, curriculum, community, or tools. There was a split in papers that had some type of comparison between an intervention and some other data set or baseline. Many papers lacked properly reported research objectives, goals, research questions, or hypotheses, description of participants, study design, data collection, and threats to validity. CER authors are contributing empirical results to the literature; however, not all norms for reporting are met. We encourage authors to provide clear, labeled details about their work so readers can use the methodologies and results for replications and meta-analyses. As our community grows, our reporting of CER should mature to help establish computing education theory to support the next generation of computing learners.


page 1

page 2

page 3

page 4


What Are Cybersecurity Education Papers About? A Systematic Literature Review of SIGCSE and ITiCSE Conferences

Cybersecurity is now more important than ever, and so is education in th...

Towards Machine-assisted Meta-Studies: The Hubble Constant

We present an approach for automatic extraction of measured values from ...

What and How Are We Reporting in HRI? A Review and Recommendations for Reporting Recruitment, Compensation, and Gender

Study reproducibility and generalizability of results to broadly inclusi...

Survey of ETA prediction methods in public transport networks

The majority of public transport vehicles are fitted with Automatic Vehi...

Reporting Results in High Energy Physics Papers: a Manifesto

The complexity of collider data analyses has dramatically increased from...

Evaluation Methodologies in Software Protection Research

Man-at-the-end (MATE) attackers have full control over the system on whi...

Incorporating Ethics in Computing Courses: Perspectives from Educators

Incorporating ethics into computing education has become a priority for ...

Please sign up or login with your details

Forgot password? Click here to reset