Systematic Review in Software Engineering
📜 Abstract
Systematic review is a scientific methodology that can be used to integrate empirical research on SE. Though its importance, conducting systematic reviews is not a simple task. It evolves performing complex actives and understanding specific concepts and terms that may be unknown to researches. The difficulties found during the systematic reviews execution pointed the need to investing research efforts in developing systematic reviews planning and execution methodologies. Therefore, we defined a systematic review conduction process. This process aims at guiding researches when performing systematic reviews. To facilitate the execution of this process, we developed a review protocol template. This template was based on systematic review protocols developed in the medical area, on the guidelines proposed by [Kitchenham, 2004] and the protocol example found in [Mendes and Kitchenham, 2004]. Its objective is to serve as a guideline to SE researchers when conducting the systematic reviews. The template leads researchers through each step of the systematic review process, defining clearly the content of each protocol section. Another issue is that systematic reviews require an additional conduction effort. The review must be planned before execution and the whole process must be documented, including, intermediary results. However, our experience on conducting systematic reviews shows that most part of the efforts is concentrated at the execution phase of the process. Searching and evaluating studies still represent a bottleneck in the literature review process.
✨ Summary
Paper summary
This technical report introduces systematic reviews as a formal research method for collecting, evaluating, comparing, and synthesizing evidence about focused software-engineering questions. It distinguishes systematic reviews from conventional literature reviews by emphasizing an explicit protocol, predefined questions and eligibility criteria, reproducible search procedures, quality evaluation, standardized data extraction, and documented synthesis. Meta-analysis is presented as one possible quantitative synthesis technique, but the report also recognizes narrative and comparative forms of synthesis.
The proposed process comprises review planning, execution, and result analysis, with continuous documentation and checkpoints for evaluating the protocol and search execution. Its protocol template structures the work around question formulation; source selection; study selection; information extraction; and results summarization. The template includes practical elements such as keywords and synonyms, search strings, inclusion and exclusion criteria, study-quality evaluation, extraction forms, reviewer-disagreement resolution, sensitivity analysis, publication-bias assessment, and recommendations for applying results.
The report argues that software engineering cannot simply replicate medical systematic-review practice. Software-engineering interventions are often performed by practitioners who know which technique they are using, making blinding difficult; techniques interact with other lifecycle activities; and causal effects may be difficult to isolate. In addition, the field had fragmented empirical evidence, limited replication, and no equivalent of a centralized evidence-synthesis infrastructure.
A pilot involving four master’s students found that the template could guide inexperienced researchers through the process. The pilot also indicated that searching for and evaluating studies consumed most of the effort. In the included example, a search found 125 studies but only one satisfied the selection criteria, illustrating both the value and cost of explicit screening.
Influence
The report’s documented influence is primarily methodological. A subsequent 2007 publication by overlapping authors extended the template-oriented approach by presenting a scientific research ontology for describing systematic-review and experimental-software-engineering knowledge. (alarcos.esi.uclm.es) Later software-engineering review research explicitly treated Biolchini et al. (2005) as one of the foundational guideline sources and examined how systematic-review practice evolved in the field. (sciencedirect.com) Individual later studies have also reported using the Biolchini protocol or guidelines, including a 2020 systematic literature review in model-driven engineering. (sciencedirect.com) The available evidence supports influence on software-engineering research methods; no specific industrial deployment or product adoption attributable to this report was identified in the search.