• Methodology
  • Open access
  • Published: 26 March 2019

SANRA—a scale for the quality assessment of narrative review articles

  • Christopher Baethge   ORCID: orcid.org/0000-0001-6246-3674 1 , 2 ,
  • Sandra Goldbeck-Wood 1 , 3 &
  • Stephan Mertens 1  

Research Integrity and Peer Review volume  4 , Article number:  5 ( 2019 ) Cite this article

128k Accesses

574 Citations

38 Altmetric

Metrics details

Narrative reviews are the commonest type of articles in the medical literature. However, unlike systematic reviews and randomized controlled trials (RCT) articles, for which formal instruments exist to evaluate quality, there is currently no instrument available to assess the quality of narrative reviews. In response to this gap, we developed SANRA, the Scale for the Assessment of Narrative Review Articles.

A team of three experienced journal editors modified or deleted items in an earlier SANRA version based on face validity, item-total correlations, and reliability scores from previous tests. We deleted an item which addressed a manuscript’s writing and accessibility due to poor inter-rater reliability. The six items which form the revised scale are rated from 0 (low standard) to 2 (high standard) and cover the following topics: explanation of (1) the importance and (2) the aims of the review, (3) literature search and (4) referencing and presentation of (5) evidence level and (6) relevant endpoint data. For all items, we developed anchor definitions and examples to guide users in filling out the form. The revised scale was tested by the same editors (blinded to each other’s ratings) in a group of 30 consecutive non-systematic review manuscripts submitted to a general medical journal.

Raters confirmed that completing the scale is feasible in everyday editorial work. The mean sum score across all 30 manuscripts was 6.0 out of 12 possible points (SD 2.6, range 1–12). Corrected item-total correlations ranged from 0.33 (item 3) to 0.58 (item 6), and Cronbach’s alpha was 0.68 (internal consistency). The intra-class correlation coefficient (average measure) was 0.77 [95% CI 0.57, 0.88] (inter-rater reliability). Raters often disagreed on items 1 and 4.

Conclusions

SANRA’s feasibility, inter-rater reliability, homogeneity of items, and internal consistency are sufficient for a scale of six items. Further field testing, particularly of validity, is desirable. We recommend rater training based on the “explanations and instructions” document provided with SANRA. In editorial decision-making, SANRA may complement journal-specific evaluation of manuscripts—pertaining to, e.g., audience, originality or difficulty—and may contribute to improving the standard of non-systematic reviews.

Peer Review reports

Narrative review articles are common in the medical literature. Bastian et al. found that they constitute the largest share of all text types in medicine and they concluded that they “remain the staple of medical literature” [ 1 ]. Narrative reviews also appear popular among both authors and readers, and it is plausible to assume that they exercise an enormous influence among doctors in clinical practice and research. However, because their quality varies widely, they have frequently been compared in blanket, negative terms with systematic reviews.

We use the term narrative review to refer to an attempt to summarize the literature in a way which is not explicitly systematic, where the minimum requirement for the term systematic relates to the method of the literature search, but in a wider sense includes a specific research question and a comprehensive summary of all studies [ 2 ].

While systematic reviews are not per se superior articles and while certain systematic reviews have been criticized lately [ 3 ], non-systematic reviews or narrative reviews have been widely criticized as unreliable [ 1 , 4 ]. Hence, the hierarchy of evidence-based medicine places systematic reviews much higher than non-systematic ones. However, it is likely—and even desirable—that good quality narrative reviews will continue to play an important role in medicine: while systematic reviews are superior to narrative reviews in answering specific questions (for example, whether it is advisable to switch an antidepressant among antidepressant non-responders in patients with major depressive disorder [ 5 ]), narrative reviews are better suited to addressing a topic in wider ways (for example, outlining the general principles of diagnosing and treating depression [ 6 ]).

Critical appraisal tools have been developed for systematic reviews (e.g., AMSTAR 2 [A MeaSurement Tool to Assess Systematic Reviews] [ 7 ]) and papers on RCTs (e.g., the CASP [Critical Appraisal Skills Program] checklist for randomized trials [ 8 ]) and other types of medical studies. For narrative reviews, in contrast, no critical appraisal, or quality assessment tool is available. Such a tool, however, if simple and brief enough for day-to-day use, may support editors in choosing or improving manuscripts, help reviewers and readers in assessing the quality of a paper, and aid authors in preparing narrative reviews. It may improve the general quality of narrative reviews.

As a consequence, we have developed SANRA, the Scale for the Assessment of Narrative Review Articles, a brief critical appraisal tool for the assessment of non-systematic articles. Here, we present the revised scale and the results of a field test regarding its feasibility, item-total correlation, internal consistency, reliability, and criterion validity.

SANRA was developed between 2010 and 2017 by three experienced editors (CB, SGW, and SM) working at a general medical journal, Deutsches Ärzteblatt , the journal of the German Medical Association and the National Association of Statutory Health Insurance Physicians . It is intended to be a simple and brief quality assessment instrument not only to assist editors in their decisions about manuscripts, but also to help reviewers and readers in their assessment of papers and authors in writing narrative reviews.

Two earlier, seven-item versions of SANRA have been developed and tested by the authors, the first in 10 narrative reviews from the field of neurology as retrieved through a PubMed search, the second among 12 consecutive narrative reviews submitted to Deutsches Ärzteblatt —both showing satisfactory internal consistency and inter-rater reliability [ 9 ].

The current version of SANRA [ 10 ] has been revised by the authors in 2014 in order to simplify the scale and make it more robust. We simplified the wording of the items, and we deleted an item addressing a manuscript’s writing and accessibility because ratings of that item differed considerably. The six items that form the revised scale are rated in integers from 0 (low standard) to 2 (high standard), with 1 as an intermediate score. The maximal sum score is 12.

The sum score of the scale is intended to measure the construct “quality of a narrative review article” and covers the following topics: explanation of the review’s importance (item 1) and statement of the aims (item 2) of the review, description of the literature search (item 3), referencing (item 4), scientific reasoning (item 5), and presentation of relevant and appropriate endpoint data (item 6) (Fig.  1 ). For all items, we developed anchor definitions and examples to guide users in filling out the instrument, provided in the document “explanations and instructions,” accompanying the scale. This document was also edited to improve clarity (Fig.  2 ).

figure 1

SANRA - Scale

figure 2

SANRA—explanations and instructions document

In 2015, one rater (CB) screened all submissions to Deutsches Ärzteblatt in 2015, and the first 30 consecutive review manuscripts without systematic literature searches were selected for inclusion in the present study. All three raters (CB, SGW, and SM) are editors, with, in 2015, at least 10 years of experience each. They scored the manuscripts independently and blinded to each other’s ratings.

Statistical analysis

Descriptive data are shown as means or medians, as appropriate, and as ranges, standard deviations, or confidence intervals. This study aimed at testing SANRA’s internal consistency (Cronbach’s alpha) and the item-total correlation—indicating whether the items measure the same phenomenon, here different aspects of review paper quality—as well as SANRA’s inter-rater reliability with regard to its sum score. Inter-rater reliability, as a measure of the consistency among different raters, was expressed as the average measure intra-class correlation, ICC, using a two-way random effects model (consistency definition). As an approximation of SANRA’s criterion validity (Is the score predictive of other indicators of paper quality, e.g., acceptance and rejection or citations?), we analyzed post hoc whether average sum scores of SANRA were associated with the decision to accept or reject the 30 manuscripts under study (point biserial correlation for the association between a dichotomous and a continuous variable). All calculations were carried out using SPSS. Where possible, the presentation follows the recommendations of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS) [ 11 ].

All 90 ratings (3 raters × 30 manuscripts) were used for statistical analysis. The mean sum score across all 30 manuscripts ( N  = 90) was 6.0 out of 12 possible points (SD 2.6, range 1–12, median 6). Highest scores were rated for item 4 (mean 1.25; SD 0.70), item 2 (mean 1.14; SD 0.84), and item 1 (mean 1.1; SD 0.69) whereas items 6, 5, and 3 had the lowest scores (means of 0.81 (SD 0.65), 0.83 (SD 0.67), and 0.84 (SD 0.60), respectively) (all single-item medians: 1).

The scale’s internal consistency, measured as Cronbach’s alpha, was 0.68. Corrected item-total correlations ranged from 0.33 to 0.58 (Table  1 ). Tentative deletions of each item to assess the effect of these on consistency showed reduced internal consistency with every deleted item (0.58–0.67) (as shown by the alpha values in Table  1 ).

Across 180 single-item ratings (6 items × 30 manuscripts), the maximum difference among the 3 raters was 2 in 12.8% ( n  = 23; most often in items 1, 2, and 4), in 56.7% ( n  = 102), the raters differed by no more than 1 point, and in 30.6% ( n  = 55), they entirely agreed (most often in items 2 and 3). The intra-class correlation coefficient (average measure) amounted to 0.77 [95% CI 0.57, 0.88; F 4.3; df 29, 58]. Disagreements most often occurred with regard to items 1 and 4.

Average SANRA sum scores of the 30 manuscripts were modestly associated with the editorial decision of acceptance (mean score 6.6, SD 1.9; n  = 17) or rejection (mean score 5.1, SD 2.1; n  = 13): point biserial correlation of 0.37 ( t  = 2.09, df 28; two-sided p  = 0.046).

All raters confirmed that completing the scale is feasible in everyday editorial work.

This study yielded three important findings: (1) SANRA can be applied to manuscripts in everyday editorial work. (2) SANRA’s internal consistency and item-total correlation are sufficient. (3) SANRA’s inter-rater reliability is satisfactory.

Feasibility

It is our experience with the current and earlier SANRA versions that editors, once accustomed to the scale, can integrate the scale into their everyday routine. It is important, however, to learn how to fill out SANRA. To this end, together with SANRA, we provide definitions and examples in the explanations and instructions document, and we recommend that new users train filling out SANRA using this resource. Editorial teams or teams of scientists and/or clinicians may prefer to learn using SANRA in group sessions.

Consistency and homogeneity

With Cronbach’s alpha of 0.68 and corrected item-total correlations between 0.33 and 0.58, we consider the scale’s consistency and item homogeneity sufficient for widespread application. It should be noted that because coefficient alpha increases with the number of items [ 12 ], simplifying a scale by reducing the number of items—as we did—may decrease internal consistency. However, this needs to be balanced against the practical need for brevity. In fact, the earlier seven-item versions of SANRA had higher values of alpha: 0.80 and 0.84, respectively [ 9 ]. Still, the number of items is not necessarily the only explanation for differences in alpha values. For example, the manuscripts included in the two earlier studies may have been easier to rate.

Inter-rater reliability

The scale’s intra-class correlation (0.77 after 0.76 in [ 9 ]) indicates that SANRA can be used reliably by different raters—an important property of a scale that may be applied for manuscript preparation and review, in editorial decision-making, or even in research on narrative reviews. Like internal consistency, reliability increases with the number of items [ 12 ], and there is a trade-off between simplicity (e.g., a small number of items) and reliability. While the ICC suggests sufficient reliability, however, the lower confidence limit (0.57) does not preclude a level of reliability normally deemed unacceptable in most applications of critical appraisal tools. This finding underscores the importance of rater training. Raters more often disagreed on items 1 and 4. After the study, we have therefore slightly edited these items, along with items 5 and 6 which we edited for clarity. In the same vein, we revised our explanations and instructions document.

It is important to bear in mind that testing of a scale always relates only to the setting of a given study. Thus, in the strict sense, the results presented here are not a general feature of SANRA but of SANRA filled out by certain raters with regard to a particular sample of manuscripts. However, from our experience, we trust that our setting is similar to that of many journals, and our sample of manuscripts represents an average group of papers. As a consequence, we are confident SANRA can be applied by other editors, reviewers, readers, and authors.

In a post hoc analysis, we found a modest, but statistically significant correlation of SANRA sum scores with manuscript acceptance. We interpret this as a sign of criterion validity, but emphasize that this is both a post hoc result and only a weak correlation. The latter, however, points to the fact that, at the level of submitted papers, other aspects than quality alone influence editorial decision-making: for example, whether the topic has been covered in the journal recently or whether editors believe that authors or topics of manuscripts have potential, even with low initial SANRA scores. SANRA will therefore often be used as one, and not the only, decision aid. Also, the decision to accept a paper has been made after the papers had been revised.

Moreover, additional results on criterion validity are needed, as are results on SANRA’s construct validity. On the other hand, SANRA’s content validity, defined as a scale’s ability to completely cover all aspects of a construct, will be restricted because we decided to limit the scale to six items, too few to encompass all facets of review article quality—SANRA is a critical appraisal tool and not a reporting guideline. For example, we deleted an item on the accessibility of the manuscript. Other possible domains that are not part of SANRA are, for example, originality of the manuscript or quality of tables and figures. These features are important, but we believe the six items forming SANRA are a core set that sufficiently indicates the quality of a review manuscript and, at the same time, is short enough to be applied without too much time and effort. SANRA’s brevity is also in contrast to other tools to assess articles, such as AMSTAR 2, for systematic reviews, or, to a lesser extent, CASP for RCTs, with its 16 and 11 items, respectively.

Throughout this paper we have referred to the current version of SANRA as the revision of earlier forms. This is technically true. However, because it is normal that scales go through different versions before publication and because this paper is first widespread publication of SANRA, we propose to call the present version simpy SANRA.

While medicine has achieved a great deal in the formalization and improvement of the presentation of randomized trials and systematic review articles, and also a number of other text types in medicine, much less work have been done with regard to the most frequent form of medical publications, the narrative review. There are exceptions: Gasparyan et al. [ 13 ], for example, have provided guidance for writing narrative reviews, and Byrne [ 14 ] as well as Pautasso [ 15 ] has written, from different angles, thoughtful editorials on improving narrative reviews and presented lists of key features of writing a good review—lists that naturally overlap with SANRA items (e.g., on referencing). These lists, however, are not tested scales and not intended for comparing different manuscripts. SANRA can be used in comparisons of manuscripts the way we used it in our editorial office, that is, in one setting. At the present time, however, it seems unwise to compare manuscripts across different settings because, so far, there are no established cut-offs for different grades of quality (e.g., poor-fair-moderate-good-very good). Still, in our experience, a score of 4 or below indicates very poor quality.

Limitations

The main limitation of this study is its sample size. While, in our experience, a group of 30 is not unusual in testing scales, it represents a compromise between the aims of representativeness for our journal and adequate power and feasibility; it took us about 6 months to sample 30 consecutive narrative reviews. Also, in this study, the authors of the scale were also the test-raters, and it is possible that inter-rater reliability is lower in groups less familiar with the scale. As for most scales, this underscores the importance of using the instructions that belong to the scale, in the present case the explanations and instructions document. It is also advisable to train using the scale before applying SANRA for manuscript rating. In addition, by design, this is not a study of test-retest reliability, another important feature of a scale. Finally, as previously acknowledged, although we believe in the representativeness of our setting for medical journals, the present results refer to the setting of this study, and consistency and reliability measures are study-specific.

We present SANRA, a brief scale for the quality assessment of narrative review articles, the most widespread form of article in the medical literature. We suggest SANRA can be integrated into the work of editors, reviewers, and authors. We encourage readers to consider using SANRA as an aid to critically appraising articles, and authors to consider its use on preparing narrative reviews, with a view to improving the quality of submitted and published manuscripts.

SANRA and its explanations and instructions document are available (open access) at: https://www.aerzteblatt.de/down.asp?id=22862 , https://www.aerzteblatt.de/down.asp?id=22861 .

Abbreviations

A MeaSurement Tool to Assess Systematic Reviews

Critical Appraisal Skills Program

Guidelines for Reporting Reliability and Agreement Studies

Intra-class correlation

Randomized controlled trial

Scale for the Assessment of Narrative Review Articles

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326. https://doi.org/10.1371/journal.pmed.1000326 .

Article   Google Scholar  

Higgins JPT, Green S. (eds.). Cochrane Handbook for Systematic Reviews and Interventions. Version 5.1.0. The Cochrane Collaboration 2011, section 1.2.2., www.handbook.cochrane.org , retrieved on Oct 31, 2018.

Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514. https://doi.org/10.1111/1468-0009.12210 .

Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987;106:485–8.

Bschor T, Kern H, Henssler J, Baethge C. Switching the antidepressant after nonresponse in adults with major depression: a systematic literature search and meta-analysis. J Clin Psychiatry. 2018;79(1):16r10749. https://doi.org/10.4088/JCP.16r10749 .

Bschor T, Adli M. Treatment of depressive disorders. Dtsch Arztebl Int. 2008;105(45):782–92.

Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:J4008.

Critical Appraisal Skills Programme 2018. CASP Randomised controlled trial checklist available at: https://casp-uk.net/casp-tools-checklists/ . Accessed: 31 Oct 2018.

Baethge C, Mertens S, Goldbeck-Wood S. Development of a quality score for the assessment of non-systematic review articles (SANRA). In: Poster, Seventh International Congress on Peer Review and Biomedical Publication. USA: Chicago, Illinois; 2013.

Baethge C, Goldbeck-Wood S, Mertens S. A scale for the assessment of non-systematic review articles (SANRA). In: Poster, Eighth International Congress on Peer and Biomedical Publication. USA: Chicago, Illinois; 2017.

Kottner J, Audige L, Brorson S, Donner A, Gajewski BJ, Hrobjartsson A, Roberts C, Shoukri M, Streiner DL. Guidelines for reporting reliability and agreement studies (GRASS) were proposed. J Clin Epidemiol. 2011;64:96–106.

Streiner DL, Norman GR. Health measurment scales—a practical guide to their development and use. Fourth edition. New York: Oxford University Press; 2008.

Book   Google Scholar  

Gasparyan AY, Ayvazyan L, Blackmore H, Kitas GD. Writing a narrative biomedical review: considerations for authors, peer reviewers, and editors. Rheumatol Int. 2011;31:1409–17.

Byrne JA. Improving the peer review of narrative literature reviews. Research Integrity and Peer Review. 2016;1(12). https://doi.org/10.1186/s41073-016-0019-2 .

Pautasso M. Ten simple rules for writing a literature review. PLoS Comput Biol. 2013;9(7):e1003149. https://doi.org/10.1371/journal.pcbi.1003149 .

Download references

Acknowledgements

This work has been presented at the Eighth International Congress on Peer Review and Scientific Publication in Chicago, Illinois, USA. (September 10-12, 2017) and at the 14th EASE Conference in Bucharest, Romania (June 8-10, 2018).

This work has not been externally funded.

Availability of data and materials

The dataset generated during the course of this study is available from the authors upon request.

Author information

Authors and affiliations.

Deutsches Ärzteblatt and Deutsches Ärzteblatt International, Dieselstraße 2, D-50859, Cologne, Germany

Christopher Baethge, Sandra Goldbeck-Wood & Stephan Mertens

Department of Psychiatry and Psychotherapy, University of Cologne Medical School, Cologne, Germany

Christopher Baethge

BMJ Sexual and Reproductive Health, London, UK

Sandra Goldbeck-Wood

You can also search for this author in PubMed   Google Scholar

Contributions

All authors (CB, SM, and SGW) made substantial contributions to the conception of the study and to the acquisition and interpretation of data. CB analyzed the data and drafted the manuscript. SM and SGW revised the draft critically for important intellectual content. All authors sufficiently participated in this work to take public responsibility for its content, all finally approved the manuscript, and all are accountable for every aspect of this project.

Corresponding author

Correspondence to Christopher Baethge .

Ethics declarations

Ethics approval and consent to participate.

Not applicable

Consent for publication

Competing interests.

Non-financial competing interest: all authors (CB, SM, and SGW) had their part in the development of the scale under study. The authors declare that they have no financial competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Baethge, C., Goldbeck-Wood, S. & Mertens, S. SANRA—a scale for the quality assessment of narrative review articles. Res Integr Peer Rev 4 , 5 (2019). https://doi.org/10.1186/s41073-019-0064-8

Download citation

Received : 02 December 2018

Accepted : 26 February 2019

Published : 26 March 2019

DOI : https://doi.org/10.1186/s41073-019-0064-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Periodicals as topic
  • Narrative review articles
  • Non-systematic review articles
  • Reliability
  • Item-total correlation
  • Internal consistency
  • Cronbach’s alpha
  • Intra-class correlation coefficient

Research Integrity and Peer Review

ISSN: 2058-8615

narrative review research paper

Journal of Graduate Medical Education

  • Previous Article
  • Next Article

Introduction

Foundations, why choose a narrative review, process and rigor, distinctive methods and subtypes, strengths and weaknesses, conclusions, narrative reviews: flexible, rigorous, and practical.

  • Split-Screen
  • Article contents
  • Figures & tables
  • Supplementary Data
  • Peer Review
  • Open the PDF for in another window
  • Guest Access
  • Get Permissions
  • Cite Icon Cite
  • Search Site

Javeed Sukhera; Narrative Reviews: Flexible, Rigorous, and Practical. J Grad Med Educ 1 August 2022; 14 (4): 414–417. doi: https://doi.org/10.4300/JGME-D-22-00480.1

Download citation file:

  • Ris (Zotero)
  • Reference Manager

Narrative reviews are a type of knowledge synthesis grounded in a distinct research tradition. They are often framed as non-systematic, which implies that there is a hierarchy of evidence placing narrative reviews below other review forms. 1   However, narrative reviews are highly useful to medical educators and researchers. While a systematic review often focuses on a narrow question in a specific context, with a prespecified method to synthesize findings from similar studies, a narrative review can include a wide variety of studies and provide an overall summary, with interpretation and critique. 1   Examples of narrative review types include state-of-the-art, critical, and integrative reviews, among many others.

Narrative reviews are situated within diverse disciplines in the social sciences and humanities. Most forms of narrative reviews align with subjectivist and interpretivist paradigms. These worldviews emphasize that reality is subjective and dynamic. They contrast with the positivist and post-positivist worldviews that are the foundations of systematic reviews: a single reality can be known through experimental research. Unlike systematic reviews, narrative reviews offer researchers the ability to synthesize multiple points of view and harness unique review team perspectives, which will shape the analysis. Therefore, insights gained from a narrative review will vary depending on the individual, organizational, or historical contexts in which the review was conducted. 1 - 5  

Narrative reviews allow researchers to describe what is known on a topic while conducting a subjective examination and critique of an entire body of literature. Authors can describe the topic's current status while providing insights on advancing the field, new theories, or current evidence viewed from different or unusual perspectives. 3   Therefore, such reviews can be useful by exploring topics that are under-researched as well as for new insights or ways of thinking regarding well-developed, robustly researched fields.

Narrative reviews are often useful for topics that require a meaningful synthesis of research evidence that may be complex or broad and that require detailed, nuanced description and interpretation. 1   See Boxes 1 and 2 for resources on writing a narrative review as well as a case example of a program director's use of a narrative review for an interprofessional education experience. This Journal of Graduate Medical Education (JGME) special review series will continue to use the Case of Dr. Smith to consider the same question using different review methodologies.

Dr. Smith, a program director, has been tasked to develop an interprofessional education (IPE) experience for the residency program. Dr. Smith decides that conducting a literature review would be a savvy way to examine the existing evidence and generate a publication useful to others. Using PubMed and a general subject search with “interprofessional education,” Dr. Smith identifies 24 000 matches. Dr. Smith begins to randomly sample the papers and notes the huge diversity of types and approaches: randomized trials, qualitative investigations, critical perspectives, and more.

Dr. Smith decides to do a meta-narrative review, because she notes that there are tensions and contradictions in the ways in which IPE is discussed by different health professions education communities, such as in nursing literature vs in medical journals.

Ferrari R. Writing narrative style literature reviews. Med Writing . 2015;24(4):230-235. doi: https://doi.org/10.1179/2047480615Z.000000000329

Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med . 2006;5(3):101-117. doi: https://doi.org/10.1016/S0899-3467(07)60142-6

Gregory AT, Denniss AR. An introduction to writing narrative and systematic reviews—tasks, tips and traps for aspiring authors. Heart Lung Circ . 2018;27(7):893-898. doi: https://doi.org/10.1016/j.hlc.2018.03.027

Murphy CM. Writing an effective review article. J Med Toxicol . 2012;8(2):89-90. doi: https://doi.org/10.1007/s13181-012-0234-2

While each type of narrative review has its own associated markers of rigor, the following guidelines are broadly applicable to narrative reviews and can help readers critically appraise their quality. These principles may also guide researchers who wish to conduct narrative reviews. When engaging with a narrative review as a reader or a researcher, scholars are advised to be conversant with the following 5 foundational elements of narrative reviews.

Rationale for a Narrative Review

First, scholars should consider the framing of the research question. Does the topic being studied align with the type of knowledge synthesis performed through a narrative review? Authors should have a clear research question and a specific audience target. Authors should also provide a rationale for why a narrative review method was chosen. 6   The manuscript should include the initial research question as well as details about any iterative refinements to the question.

Clarity of Boundaries, Scope, and Definitions

Second, although narrative reviews do not typically involve strict predetermined inclusion or exclusion criteria, scholars should explicitly demarcate the boundaries and scope of their topic. They should also clearly define key terms related to the topic and research question and any definitions used. Authors should elaborate why they chose a particular definition if others were available. As narrative reviews are flexible, the initial scope may change through the review process. In such circumstances, authors should provide reasonable justification for the evolution of inclusion and exclusion criteria and a description of how this affected the literature search.

Justification for Inclusion and Exclusion Criteria

Third, authors of narrative reviews should explain which search terms and databases were included in the synthesis and why. For example, did authors include research studies from a particular database, time frame, or study design? Did they include commentaries or empirical articles? Did they include grey literature such as trade publications, reports, or digital media? Each of the authors' choices should be outlined with appropriate reasoning. 7   Narrative reviews tend to be iterative and involve multiple cycles of searching, analysis, and interpretation. High-quality narrative reviews usually include pivotal or seminal papers that address the phenomenon of interest and other manuscripts that are relevant to the research question.

Reflexivity and a Saturation/Sufficiency Statement

Fourth, narrative reviews should clearly specify any factors that may have shaped the authors' interpretations and analysis. One fundamental distinction between narrative and non-narrative reviews is that narrative reviews explicitly recognize that they may not include all relevant literature on a topic. Since narrative reviews do not aim to be inclusive of all literature addressing the phenomenon of interest, a justification for the selection of manuscripts must be included. Authors should carefully outline how researchers conducted analyses and how they determined that sufficient analysis and interpretation was achieved. This latter concept is similar to considerations of saturation or thematic sufficiency in primary qualitative research. 8  

Details on Analysis and Interpretation

Lastly, since several different categories of reviews fall under the narrative review umbrella, the analysis conducted in a narrative review varies by type. Regardless of the type of narrative review carried out, authors should clearly describe how analyses were conducted and provide justification for their approach. Narrative reviews are enhanced when researchers are explicit about how their perspectives and experiences informed problem identification, interpretation, and analysis. Given that authors' unique perspectives shape the selection of literature and its interpretation, narrative reviews may be reproduced, but different authors will likely yield different insights and interpretations.

The narrative review has been commonly framed as an umbrella term that includes several different subtypes of reviews. These narrative medicine subtypes share the goals of deepening an understanding of a topic, while describing why researchers chose to explore and analyze the topic in a specific way.

There are several subtypes of narrative reviews with distinctive methodologies; each offers a unique way of approaching the research question and analyzing and interpreting the literature. This article will describe some common narrative review types that will also be discussed in upcoming JGME special articles on reviews: state-of-the-art , meta-ethnographic , critical , and theory integration reviews.

A state-of-the-art review attempts to summarize the research concerning a specific topic along a timeline of significant changes in understanding or research orientations. By focusing on such turning points in the history of evolving understandings of a phenomenon, state-of-the-art reviews offer a summary of the current state of understanding, how such an understanding was developed, and an idea of future directions. A state-of-the art review seeks to offer a 3-part description: where are we now in our understanding, how did we get here, and where should we go next?

A meta-ethnographic review involves choosing and interpreting qualitative research evidence about a specific topic. Working exclusively with qualitative data, this type of knowledge synthesis aims to generate new insights or new conclusions about a topic. It draws together insights and analyses from existing publications of qualitative research to construct new knowledge that spans across these individual, and often small scale, studies.

A meta-narrative review seeks to explore and make sense of contradictions and tensions within the literature. A meta-narrative review maps how a certain topic is understood in distinct ways, conducts a focused search to describe and compare narratives, and then seeks to make sense of how such narratives are interpreted across different disciplines or historical contexts, as part of the analysis. 9  

A critical review is a narrative synthesis of literature that brings an interpretative lens: the review is shaped by a theory, a critical point of view, or perspectives from other domains to inform the literature analysis. Critical reviews involve an interpretative process that combines the reviewer's theoretical premise with existing theories and models to allow for synthesis and interpretation of diverse studies. First, reviewers develop and outline their interpretive theoretical position, which is informed by individual knowledge and experience. Next, a noncomprehensive search is completed to capture and identify dominant themes focused on a research question. 8 , 10  

An integrative review typically has 1 of 2 different orientations. Empirical integrative reviews analyze and synthesize publications of evidence-based studies with diverse methodologies. In contrast, theoretical integrative reviews conduct an analysis of the available theories addressing a phenomenon, critically appraise those theories, and propose an advancement in the development of those theories. Both types of integrative reviews follow a multistage approach including problem identification, searching, evaluation, analysis, and presentation. 11  

Narrative reviews have many strengths. They are flexible and practical, and ideally provide a readable, relevant synthesis of a diverse literature. Narrative reviews are often helpful for teaching or learning about a topic because they deliver a general overview. They are also useful for setting the stage for future research, as they offer an interpretation of the literature, note gaps, and critique research to date.

Such reviews may be useful for providing general background; however, a more comprehensive form of review may be necessary. Narrative reviews do not offer an evidence-based synthesis for focused questions, nor do they offer definitive guideline statements. All types of narrative reviews offer interpretations that are open to critique and will vary depending on the author team or context of the review.

Well-done narrative reviews provide a readable, thoughtful, and practical synthesis on a topic. They allow review authors to advance new ideas while describing and interpreting literature in the field. Narrative reviews do not aim to be systematic syntheses that answer a specific, highly focused question; instead, they offer carefully thought out and rigorous interpretations of a body of knowledge. Such reviews will not provide an exhaustive, comprehensive review of the literature; however, they are useful for a rich and meaningful summary of a topic.

Recipient(s) will receive an email with a link to 'Narrative Reviews: Flexible, Rigorous, and Practical' and will not need an account to access the content.

Subject: Narrative Reviews: Flexible, Rigorous, and Practical

(Optional message may have a maximum of 1000 characters.)

Citing articles via

Never miss an issue, get new journal tables of contents sent right to your email inbox.

  • Submit an Article

Affiliations

Journal of Graduate Medical Education Logo

  • eISSN 1949-8357
  • ISSN 1949-8349
  • Privacy Policy
  • Get Adobe Acrobat Reader

This Feature Is Available To Subscribers Only

Sign In or Create an Account

Subject Guides

Literature Review and Evidence Synthesis

  • Reviews as Assignments
  • Annotated Bibliography

What is a Narrative Literature Review

Literature review process.

  • Integrative Review
  • Scoping Review This link opens in a new window
  • Systematic Review This link opens in a new window
  • Other Review Types
  • Subject Librarian Assistance with Reviews
  • Grey Literature This link opens in a new window

Subject Librarians

Find your Subject Librarian Here

narrative review research paper

A narrative literature review is an integrated analysis of the existing literature used to summarize a body of literature, draw conclusions about a topic, and identify research gaps.  By understanding the current state of the literature, you can show how new research fits into the larger research landscape.  

A narrative literature review is NOT:  

  • Just a summary of sources
  • A review of  everything  written on a particular topic
  • A research paper arguing for a specific viewpoint - a lit review should avoid bias and highlight areas of disagreements
  • A systematic review

Purposes of a narrative literature review:

  • Explain the background of research on a topic
  • Demonstrate the importance of a topic
  • Suggest new areas of research
  • Identify major themes, concepts, and researchers in a topic
  • Identify critical gaps, points of disagreement, or flawed approaches for a research topic

1. Choose a topic & create a research question

  • Use a narrow research question for more focused search results
  • Use a question framework such as PICO to develop your research question
  • Breakdown your research question into searchable concepts and keywords
  • Research skills tutorials : How to choose a topic
  • Ask a librarian for assistance

2. Select the sources for searching & develop a search strategy

  • Identify databases to search for articles relevant to your topic
  • Ask a librarian for recommended databases
  • Develop a comprehensive search strategy using keywords, controlled vocabularies and Boolean operators
  • Research skills tutorials: How to develop a search strategy

3. Conduct the search

  • Use a consistent search strategy between databases
  • Document the strategies employed to keep track of which are more successful
  • Use a citation manager to organize your search results
  • Ask a librarian for help or refer to the Research skills tutorials

4. Review the references

  • Review the search results for relevant articles that answer your research question
  • Review the bibliography of all relevant articles for additional sources
  • Consider developing subfolders in the citation manager to organize sources by topic
  • Use interlibrary loan for any articles without full text access

5. Summarize findings

  • Synthesize the findings from the articles into a final paper
  • The final paper should cover the themes identified in the research, explain any conflicts or disagreements, identify research gaps and potential future research areas, explain how this narrative review fits within the existing research and answer the research question . 

For additional information : 

Hempel. (2020). Conducting your literature review. American Psychological Association .

  • Buchholz, & Dickins, K. A. (2023). Literature review and synthesis : a guide for nurses and other healthcare professionals . Springer Publishing Company, LLC.
  • Coughlan, Michael, and Patricia Cronin.  Doing a Literature Review in Nursing, Health and Social Care . 2nd edition., SAGE, 2017.
  • << Previous: Annotated Bibliography
  • Next: Integrative Review >>
  • Last Updated: Feb 6, 2024 2:50 PM
  • URL: https://libraryguides.binghamton.edu/literaturereview
  • share facebook
  • share twitter
  • share pinterest
  • share linkedin
  • share email
  • Systematic review
  • Open access
  • Published: 19 February 2024

‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice

  • Annette Boaz   ORCID: orcid.org/0000-0003-0557-1294 1 ,
  • Juan Baeza 2 ,
  • Alec Fraser   ORCID: orcid.org/0000-0003-1121-1551 2 &
  • Erik Persson 3  

Implementation Science volume  19 , Article number:  15 ( 2024 ) Cite this article

239 Accesses

30 Altmetric

Metrics details

The gap between research findings and clinical practice is well documented and a range of strategies have been developed to support the implementation of research into clinical practice. The objective of this study was to update and extend two previous reviews of systematic reviews of strategies designed to implement research evidence into clinical practice.

We developed a comprehensive systematic literature search strategy based on the terms used in the previous reviews to identify studies that looked explicitly at interventions designed to turn research evidence into practice. The search was performed in June 2022 in four electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched from January 2010 up to June 2022 and applied no language restrictions. Two independent reviewers appraised the quality of included studies using a quality assessment checklist. To reduce the risk of bias, papers were excluded following discussion between all members of the team. Data were synthesised using descriptive and narrative techniques to identify themes and patterns linked to intervention strategies, targeted behaviours, study settings and study outcomes.

We identified 32 reviews conducted between 2010 and 2022. The reviews are mainly of multi-faceted interventions ( n  = 20) although there are reviews focusing on single strategies (ICT, educational, reminders, local opinion leaders, audit and feedback, social media and toolkits). The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Furthermore, a lot of nuance lies behind these headline findings, and this is increasingly commented upon in the reviews themselves.

Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been identified. We need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of research perspectives (including social science) in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed.

Peer Review reports

Contribution to the literature

Considerable time and money is invested in implementing and evaluating strategies to increase the implementation of research into clinical practice.

The growing body of evidence is not providing the anticipated clear lessons to support improved implementation.

Instead what is needed is better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice.

This would involve a more central role in implementation science for a wider range of perspectives, especially from the social, economic, political and behavioural sciences and for greater use of different types of synthesis, such as realist synthesis.

Introduction

The gap between research findings and clinical practice is well documented and a range of interventions has been developed to increase the implementation of research into clinical practice [ 1 , 2 ]. In recent years researchers have worked to improve the consistency in the ways in which these interventions (often called strategies) are described to support their evaluation. One notable development has been the emergence of Implementation Science as a field focusing explicitly on “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice” ([ 3 ] p. 1). The work of implementation science focuses on closing, or at least narrowing, the gap between research and practice. One contribution has been to map existing interventions, identifying 73 discreet strategies to support research implementation [ 4 ] which have been grouped into 9 clusters [ 5 ]. The authors note that they have not considered the evidence of effectiveness of the individual strategies and that a next step is to understand better which strategies perform best in which combinations and for what purposes [ 4 ]. Other authors have noted that there is also scope to learn more from other related fields of study such as policy implementation [ 6 ] and to draw on methods designed to support the evaluation of complex interventions [ 7 ].

The increase in activity designed to support the implementation of research into practice and improvements in reporting provided the impetus for an update of a review of systematic reviews of the effectiveness of interventions designed to support the use of research in clinical practice [ 8 ] which was itself an update of the review conducted by Grimshaw and colleagues in 2001. The 2001 review [ 9 ] identified 41 reviews considering a range of strategies including educational interventions, audit and feedback, computerised decision support to financial incentives and combined interventions. The authors concluded that all the interventions had the potential to promote the uptake of evidence in practice, although no one intervention seemed to be more effective than the others in all settings. They concluded that combined interventions were more likely to be effective than single interventions. The 2011 review identified a further 13 systematic reviews containing 313 discrete primary studies. Consistent with the previous review, four main strategy types were identified: audit and feedback; computerised decision support; opinion leaders; and multi-faceted interventions (MFIs). Nine of the reviews reported on MFIs. The review highlighted the small effects of single interventions such as audit and feedback, computerised decision support and opinion leaders. MFIs claimed an improvement in effectiveness over single interventions, although effect sizes remained small to moderate and this improvement in effectiveness relating to MFIs has been questioned in a subsequent review [ 10 ]. In updating the review, we anticipated a larger pool of reviews and an opportunity to consolidate learning from more recent systematic reviews of interventions.

This review updates and extends our previous review of systematic reviews of interventions designed to implement research evidence into clinical practice. To identify potentially relevant peer-reviewed research papers, we developed a comprehensive systematic literature search strategy based on the terms used in the Grimshaw et al. [ 9 ] and Boaz, Baeza and Fraser [ 8 ] overview articles. To ensure optimal retrieval, our search strategy was refined with support from an expert university librarian, considering the ongoing improvements in the development of search filters for systematic reviews since our first review [ 11 ]. We also wanted to include technology-related terms (e.g. apps, algorithms, machine learning, artificial intelligence) to find studies that explored interventions based on the use of technological innovations as mechanistic tools for increasing the use of evidence into practice (see Additional file 1 : Appendix A for full search strategy).

The search was performed in June 2022 in the following electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched for articles published since the 2011 review. We searched from January 2010 up to June 2022 and applied no language restrictions. Reference lists of relevant papers were also examined.

We uploaded the results using EPPI-Reviewer, a web-based tool that facilitated semi-automation of the screening process and removal of duplicate studies. We made particular use of a priority screening function to reduce screening workload and avoid ‘data deluge’ [ 12 ]. Through machine learning, one reviewer screened a smaller number of records ( n  = 1200) to train the software to predict whether a given record was more likely to be relevant or irrelevant, thus pulling the relevant studies towards the beginning of the screening process. This automation did not replace manual work but helped the reviewer to identify eligible studies more quickly. During the selection process, we included studies that looked explicitly at interventions designed to turn research evidence into practice. Studies were included if they met the following pre-determined inclusion criteria:

The study was a systematic review

Search terms were included

Focused on the implementation of research evidence into practice

The methodological quality of the included studies was assessed as part of the review

Study populations included healthcare providers and patients. The EPOC taxonomy [ 13 ] was used to categorise the strategies. The EPOC taxonomy has four domains: delivery arrangements, financial arrangements, governance arrangements and implementation strategies. The implementation strategies domain includes 20 strategies targeted at healthcare workers. Numerous EPOC strategies were assessed in the review including educational strategies, local opinion leaders, reminders, ICT-focused approaches and audit and feedback. Some strategies that did not fit easily within the EPOC categories were also included. These were social media strategies and toolkits, and multi-faceted interventions (MFIs) (see Table  2 ). Some systematic reviews included comparisons of different interventions while other reviews compared one type of intervention against a control group. Outcomes related to improvements in health care processes or patient well-being. Numerous individual study types (RCT, CCT, BA, ITS) were included within the systematic reviews.

We excluded papers that:

Focused on changing patient rather than provider behaviour

Had no demonstrable outcomes

Made unclear or no reference to research evidence

The last of these criteria was sometimes difficult to judge, and there was considerable discussion amongst the research team as to whether the link between research evidence and practice was sufficiently explicit in the interventions analysed. As we discussed in the previous review [ 8 ] in the field of healthcare, the principle of evidence-based practice is widely acknowledged and tools to change behaviour such as guidelines are often seen to be an implicit codification of evidence, despite the fact that this is not always the case.

Reviewers employed a two-stage process to select papers for inclusion. First, all titles and abstracts were screened by one reviewer to determine whether the study met the inclusion criteria. Two papers [ 14 , 15 ] were identified that fell just before the 2010 cut-off. As they were not identified in the searches for the first review [ 8 ] they were included and progressed to assessment. Each paper was rated as include, exclude or maybe. The full texts of 111 relevant papers were assessed independently by at least two authors. To reduce the risk of bias, papers were excluded following discussion between all members of the team. 32 papers met the inclusion criteria and proceeded to data extraction. The study selection procedure is documented in a PRISMA literature flow diagram (see Fig.  1 ). We were able to include French, Spanish and Portuguese papers in the selection reflecting the language skills in the study team, but none of the papers identified met the inclusion criteria. Other non- English language papers were excluded.

figure 1

PRISMA flow diagram. Source: authors

One reviewer extracted data on strategy type, number of included studies, local, target population, effectiveness and scope of impact from the included studies. Two reviewers then independently read each paper and noted key findings and broad themes of interest which were then discussed amongst the wider authorial team. Two independent reviewers appraised the quality of included studies using a Quality Assessment Checklist based on Oxman and Guyatt [ 16 ] and Francke et al. [ 17 ]. Each study was rated a quality score ranging from 1 (extensive flaws) to 7 (minimal flaws) (see Additional file 2 : Appendix B). All disagreements were resolved through discussion. Studies were not excluded in this updated overview based on methodological quality as we aimed to reflect the full extent of current research into this topic.

The extracted data were synthesised using descriptive and narrative techniques to identify themes and patterns in the data linked to intervention strategies, targeted behaviours, study settings and study outcomes.

Thirty-two studies were included in the systematic review. Table 1. provides a detailed overview of the included systematic reviews comprising reference, strategy type, quality score, number of included studies, local, target population, effectiveness and scope of impact (see Table  1. at the end of the manuscript). Overall, the quality of the studies was high. Twenty-three studies scored 7, six studies scored 6, one study scored 5, one study scored 4 and one study scored 3. The primary focus of the review was on reviews of effectiveness studies, but a small number of reviews did include data from a wider range of methods including qualitative studies which added to the analysis in the papers [ 18 , 19 , 20 , 21 ]. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. In this section, we discuss the different EPOC-defined implementation strategies in turn. Interestingly, we found only two ‘new’ approaches in this review that did not fit into the existing EPOC approaches. These are a review focused on the use of social media and a review considering toolkits. In addition to single interventions, we also discuss multi-faceted interventions. These were the most common intervention approach overall. A summary is provided in Table  2 .

Educational strategies

The overview identified three systematic reviews focusing on educational strategies. Grudniewicz et al. [ 22 ] explored the effectiveness of printed educational materials on primary care physician knowledge, behaviour and patient outcomes and concluded they were not effective in any of these aspects. Koota, Kääriäinen and Melender [ 23 ] focused on educational interventions promoting evidence-based practice among emergency room/accident and emergency nurses and found that interventions involving face-to-face contact led to significant or highly significant effects on patient benefits and emergency nurses’ knowledge, skills and behaviour. Interventions using written self-directed learning materials also led to significant improvements in nurses’ knowledge of evidence-based practice. Although the quality of the studies was high, the review primarily included small studies with low response rates, and many of them relied on self-assessed outcomes; consequently, the strength of the evidence for these outcomes is modest. Wu et al. [ 20 ] questioned if educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes. Although based on evaluation projects and qualitative data, their results also suggest that positive changes on patient outcomes can be made following the implementation of specific evidence-based approaches (or projects). The differing positive outcomes for educational strategies aimed at nurses might indicate that the target audience is important.

Local opinion leaders

Flodgren et al. [ 24 ] was the only systemic review focusing solely on opinion leaders. The review found that local opinion leaders alone, or in combination with other interventions, can be effective in promoting evidence‐based practice, but this varies both within and between studies and the effect on patient outcomes is uncertain. The review found that, overall, any intervention involving opinion leaders probably improves healthcare professionals’ compliance with evidence-based practice but varies within and across studies. However, how opinion leaders had an impact could not be determined because of insufficient details were provided, illustrating that reporting specific details in published studies is important if diffusion of effective methods of increasing evidence-based practice is to be spread across a system. The usefulness of this review is questionable because it cannot provide evidence of what is an effective opinion leader, whether teams of opinion leaders or a single opinion leader are most effective, or the most effective methods used by opinion leaders.

Pantoja et al. [ 26 ] was the only systemic review focusing solely on manually generated reminders delivered on paper included in the overview. The review explored how these affected professional practice and patient outcomes. The review concluded that manually generated reminders delivered on paper as a single intervention probably led to small to moderate increases in adherence to clinical recommendations, and they could be used as a single quality improvement intervention. However, the authors indicated that this intervention would make little or no difference to patient outcomes. The authors state that such a low-tech intervention may be useful in low- and middle-income countries where paper records are more likely to be the norm.

ICT-focused approaches

The three ICT-focused reviews [ 14 , 27 , 28 ] showed mixed results. Jamal, McKenzie and Clark [ 14 ] explored the impact of health information technology on the quality of medical and health care. They examined the impact of electronic health record, computerised provider order-entry, or decision support system. This showed a positive improvement in adherence to evidence-based guidelines but not to patient outcomes. The number of studies included in the review was low and so a conclusive recommendation could not be reached based on this review. Similarly, Brown et al. [ 28 ] found that technology-enabled knowledge translation interventions may improve knowledge of health professionals, but all eight studies raised concerns of bias. The De Angelis et al. [ 27 ] review was more promising, reporting that ICT can be a good way of disseminating clinical practice guidelines but conclude that it is unclear which type of ICT method is the most effective.

Audit and feedback

Sykes, McAnuff and Kolehmainen [ 29 ] examined whether audit and feedback were effective in dementia care and concluded that it remains unclear which ingredients of audit and feedback are successful as the reviewed papers illustrated large variations in the effectiveness of interventions using audit and feedback.

Non-EPOC listed strategies: social media, toolkits

There were two new (non-EPOC listed) intervention types identified in this review compared to the 2011 review — fewer than anticipated. We categorised a third — ‘care bundles’ [ 36 ] as a multi-faceted intervention due to its description in practice and a fourth — ‘Technology Enhanced Knowledge Transfer’ [ 28 ] was classified as an ICT-focused approach. The first new strategy was identified in Bhatt et al.’s [ 30 ] systematic review of the use of social media for the dissemination of clinical practice guidelines. They reported that the use of social media resulted in a significant improvement in knowledge and compliance with evidence-based guidelines compared with more traditional methods. They noted that a wide selection of different healthcare professionals and patients engaged with this type of social media and its global reach may be significant for low- and middle-income countries. This review was also noteworthy for developing a simple stepwise method for using social media for the dissemination of clinical practice guidelines. However, it is debatable whether social media can be classified as an intervention or just a different way of delivering an intervention. For example, the review discussed involving opinion leaders and patient advocates through social media. However, this was a small review that included only five studies, so further research in this new area is needed. Yamada et al. [ 31 ] draw on 39 studies to explore the application of toolkits, 18 of which had toolkits embedded within larger KT interventions, and 21 of which evaluated toolkits as standalone interventions. The individual component strategies of the toolkits were highly variable though the authors suggest that they align most closely with educational strategies. The authors conclude that toolkits as either standalone strategies or as part of MFIs hold some promise for facilitating evidence use in practice but caution that the quality of many of the primary studies included is considered weak limiting these findings.

Multi-faceted interventions

The majority of the systematic reviews ( n  = 20) reported on more than one intervention type. Some of these systematic reviews focus exclusively on multi-faceted interventions, whilst others compare different single or combined interventions aimed at achieving similar outcomes in particular settings. While these two approaches are often described in a similar way, they are actually quite distinct from each other as the former report how multiple strategies may be strategically combined in pursuance of an agreed goal, whilst the latter report how different strategies may be incidentally used in sometimes contrasting settings in the pursuance of similar goals. Ariyo et al. [ 35 ] helpfully summarise five key elements often found in effective MFI strategies in LMICs — but which may also be transferrable to HICs. First, effective MFIs encourage a multi-disciplinary approach acknowledging the roles played by different professional groups to collectively incorporate evidence-informed practice. Second, they utilise leadership drawing on a wide set of clinical and non-clinical actors including managers and even government officials. Third, multiple types of educational practices are utilised — including input from patients as stakeholders in some cases. Fourth, protocols, checklists and bundles are used — most effectively when local ownership is encouraged. Finally, most MFIs included an emphasis on monitoring and evaluation [ 35 ]. In contrast, other studies offer little information about the nature of the different MFI components of included studies which makes it difficult to extrapolate much learning from them in relation to why or how MFIs might affect practice (e.g. [ 28 , 38 ]). Ultimately, context matters, which some review authors argue makes it difficult to say with real certainty whether single or MFI strategies are superior (e.g. [ 21 , 27 ]). Taking all the systematic reviews together we may conclude that MFIs appear to be more likely to generate positive results than single interventions (e.g. [ 34 , 45 ]) though other reviews should make us cautious (e.g. [ 32 , 43 ]).

While multi-faceted interventions still seem to be more effective than single-strategy interventions, there were important distinctions between how the results of reviews of MFIs are interpreted in this review as compared to the previous reviews [ 8 , 9 ], reflecting greater nuance and debate in the literature. This was particularly noticeable where the effectiveness of MFIs was compared to single strategies, reflecting developments widely discussed in previous studies [ 10 ]. We found that most systematic reviews are bounded by their clinical, professional, spatial, system, or setting criteria and often seek to draw out implications for the implementation of evidence in their areas of specific interest (such as nursing or acute care). Frequently this means combining all relevant studies to explore the respective foci of each systematic review. Therefore, most reviews we categorised as MFIs actually include highly variable numbers and combinations of intervention strategies and highly heterogeneous original study designs. This makes statistical analyses of the type used by Squires et al. [ 10 ] on the three reviews in their paper not possible. Further, it also makes extrapolating findings and commenting on broad themes complex and difficult. This may suggest that future research should shift its focus from merely examining ‘what works’ to ‘what works where and what works for whom’ — perhaps pointing to the value of realist approaches to these complex review topics [ 48 , 49 ] and other more theory-informed approaches [ 50 ].

Some reviews have a relatively small number of studies (i.e. fewer than 10) and the authors are often understandably reluctant to engage with wider debates about the implications of their findings. Other larger studies do engage in deeper discussions about internal comparisons of findings across included studies and also contextualise these in wider debates. Some of the most informative studies (e.g. [ 35 , 40 ]) move beyond EPOC categories and contextualise MFIs within wider systems thinking and implementation theory. This distinction between MFIs and single interventions can actually be very useful as it offers lessons about the contexts in which individual interventions might have bounded effectiveness (i.e. educational interventions for individual change). Taken as a whole, this may also then help in terms of how and when to conjoin single interventions into effective MFIs.

In the two previous reviews, a consistent finding was that MFIs were more effective than single interventions [ 8 , 9 ]. However, like Squires et al. [ 10 ] this overview is more equivocal on this important issue. There are four points which may help account for the differences in findings in this regard. Firstly, the diversity of the systematic reviews in terms of clinical topic or setting is an important factor. Secondly, there is heterogeneity of the studies within the included systematic reviews themselves. Thirdly, there is a lack of consistency with regards to the definition and strategies included within of MFIs. Finally, there are epistemological differences across the papers and the reviews. This means that the results that are presented depend on the methods used to measure, report, and synthesise them. For instance, some reviews highlight that education strategies can be useful to improve provider understanding — but without wider organisational or system-level change, they may struggle to deliver sustained transformation [ 19 , 44 ].

It is also worth highlighting the importance of the theory of change underlying the different interventions. Where authors of the systematic reviews draw on theory, there is space to discuss/explain findings. We note a distinction between theoretical and atheoretical systematic review discussion sections. Atheoretical reviews tend to present acontextual findings (for instance, one study found very positive results for one intervention, and this gets highlighted in the abstract) whilst theoretically informed reviews attempt to contextualise and explain patterns within the included studies. Theory-informed systematic reviews seem more likely to offer more profound and useful insights (see [ 19 , 35 , 40 , 43 , 45 ]). We find that the most insightful systematic reviews of MFIs engage in theoretical generalisation — they attempt to go beyond the data of individual studies and discuss the wider implications of the findings of the studies within their reviews drawing on implementation theory. At the same time, they highlight the active role of context and the wider relational and system-wide issues linked to implementation. It is these types of investigations that can help providers further develop evidence-based practice.

This overview has identified a small, but insightful set of papers that interrogate and help theorise why, how, for whom, and in which circumstances it might be the case that MFIs are superior (see [ 19 , 35 , 40 ] once more). At the level of this overview — and in most of the systematic reviews included — it appears to be the case that MFIs struggle with the question of attribution. In addition, there are other important elements that are often unmeasured, or unreported (e.g. costs of the intervention — see [ 40 ]). Finally, the stronger systematic reviews [ 19 , 35 , 40 , 43 , 45 ] engage with systems issues, human agency and context [ 18 ] in a way that was not evident in the systematic reviews identified in the previous reviews [ 8 , 9 ]. The earlier reviews lacked any theory of change that might explain why MFIs might be more effective than single ones — whereas now some systematic reviews do this, which enables them to conclude that sometimes single interventions can still be more effective.

As Nilsen et al. ([ 6 ] p. 7) note ‘Study findings concerning the effectiveness of various approaches are continuously synthesized and assembled in systematic reviews’. We may have gone as far as we can in understanding the implementation of evidence through systematic reviews of single and multi-faceted interventions and the next step would be to conduct more research exploring the complex and situated nature of evidence used in clinical practice and by particular professional groups. This would further build on the nuanced discussion and conclusion sections in a subset of the papers we reviewed. This might also support the field to move away from isolating individual implementation strategies [ 6 ] to explore the complex processes involving a range of actors with differing capacities [ 51 ] working in diverse organisational cultures. Taxonomies of implementation strategies do not fully account for the complex process of implementation, which involves a range of different actors with different capacities and skills across multiple system levels. There is plenty of work to build on, particularly in the social sciences, which currently sits at the margins of debates about evidence implementation (see for example, Normalisation Process Theory [ 52 ]).

There are several changes that we have identified in this overview of systematic reviews in comparison to the review we published in 2011 [ 8 ]. A consistent and welcome finding is that the overall quality of the systematic reviews themselves appears to have improved between the two reviews, although this is not reflected upon in the papers. This is exhibited through better, clearer reporting mechanisms in relation to the mechanics of the reviews, alongside a greater attention to, and deeper description of, how potential biases in included papers are discussed. Additionally, there is an increased, but still limited, inclusion of original studies conducted in low- and middle-income countries as opposed to just high-income countries. Importantly, we found that many of these systematic reviews are attuned to, and comment upon the contextual distinctions of pursuing evidence-informed interventions in health care settings in different economic settings. Furthermore, systematic reviews included in this updated article cover a wider set of clinical specialities (both within and beyond hospital settings) and have a focus on a wider set of healthcare professions — discussing both similarities, differences and inter-professional challenges faced therein, compared to the earlier reviews. These wider ranges of studies highlight that a particular intervention or group of interventions may work well for one professional group but be ineffective for another. This diversity of study settings allows us to consider the important role context (in its many forms) plays on implementing evidence into practice. Examining the complex and varied context of health care will help us address what Nilsen et al. ([ 6 ] p. 1) described as, ‘society’s health problems [that] require research-based knowledge acted on by healthcare practitioners together with implementation of political measures from governmental agencies’. This will help us shift implementation science to move, ‘beyond a success or failure perspective towards improved analysis of variables that could explain the impact of the implementation process’ ([ 6 ] p. 2).

This review brings together 32 papers considering individual and multi-faceted interventions designed to support the use of evidence in clinical practice. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been conducted. As a whole, this substantial body of knowledge struggles to tell us more about the use of individual and MFIs than: ‘it depends’. To really move forwards in addressing the gap between research evidence and practice, we may need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of perspectives, especially from the social, economic, political and behavioural sciences in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed. Harvey et al. [ 53 ] suggest that when context is likely to be critical to implementation success there are a range of primary research approaches (participatory research, realist evaluation, developmental evaluation, ethnography, quality/ rapid cycle improvement) that are likely to be appropriate and insightful. While these approaches often form part of implementation studies in the form of process evaluations, they are usually relatively small scale in relation to implementation research as a whole. As a result, the findings often do not make it into the subsequent systematic reviews. This review provides further evidence that we need to bring qualitative approaches in from the periphery to play a central role in many implementation studies and subsequent evidence syntheses. It would be helpful for systematic reviews, at the very least, to include more detail about the interventions and their implementation in terms of how and why they worked.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Before and after study

Controlled clinical trial

Effective Practice and Organisation of Care

High-income countries

Information and Communications Technology

Interrupted time series

Knowledge translation

Low- and middle-income countries

Randomised controlled trial

Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30. https://doi.org/10.1016/S0140-6736(03)14546-1 .

Article   PubMed   Google Scholar  

Green LA, Seifert CM. Translation of research into practice: why we can’t “just do it.” J Am Board Fam Pract. 2005;18:541–5. https://doi.org/10.3122/jabfm.18.6.541 .

Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1:1–3. https://doi.org/10.1186/1748-5908-1-1 .

Article   PubMed Central   Google Scholar  

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:2–14. https://doi.org/10.1186/s13012-015-0209-1 .

Article   Google Scholar  

Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:1–8. https://doi.org/10.1186/s13012-015-0295-0 .

Nilsen P, Ståhl C, Roback K, et al. Never the twain shall meet? - a comparison of implementation science and policy implementation research. Implementation Sci. 2013;8:2–12. https://doi.org/10.1186/1748-5908-8-63 .

Rycroft-Malone J, Seers K, Eldh AC, et al. A realist process evaluation within the Facilitating Implementation of Research Evidence (FIRE) cluster randomised controlled international trial: an exemplar. Implementation Sci. 2018;13:1–15. https://doi.org/10.1186/s13012-018-0811-0 .

Boaz A, Baeza J, Fraser A, European Implementation Score Collaborative Group (EIS). Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011;4:212. https://doi.org/10.1186/1756-0500-4-212 .

Article   PubMed   PubMed Central   Google Scholar  

Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior – an overview of systematic reviews of interventions. Med Care. 2001;39 8Suppl 2:II2–45.

Google Scholar  

Squires JE, Sullivan K, Eccles MP, et al. Are multifaceted interventions more effective than single-component interventions in changing health-care professionals’ behaviours? An overview of systematic reviews. Implement Sci. 2014;9:1–22. https://doi.org/10.1186/s13012-014-0152-6 .

Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R. Development of an efficient search filter to retrieve systematic reviews from PubMed. J Med Libr Assoc. 2021;109:561–74. https://doi.org/10.5195/jmla.2021.1223 .

Thomas JM. Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation? OA Evid Based Med. 2013;1:1–6.

Effective Practice and Organisation of Care (EPOC). The EPOC taxonomy of health systems interventions. EPOC Resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services; 2016. epoc.cochrane.org/epoc-taxonomy . Accessed 9 Oct 2023.

Jamal A, McKenzie K, Clark M. The impact of health information technology on the quality of medical and health care: a systematic review. Health Inf Manag. 2009;38:26–37. https://doi.org/10.1177/183335830903800305 .

Menon A, Korner-Bitensky N, Kastner M, et al. Strategies for rehabilitation professionals to move evidence-based knowledge into practice: a systematic review. J Rehabil Med. 2009;41:1024–32. https://doi.org/10.2340/16501977-0451 .

Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44:1271–8. https://doi.org/10.1016/0895-4356(91)90160-b .

Article   CAS   PubMed   Google Scholar  

Francke AL, Smit MC, de Veer AJ, et al. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;8:1–11. https://doi.org/10.1186/1472-6947-8-38 .

Jones CA, Roop SC, Pohar SL, et al. Translating knowledge in rehabilitation: systematic review. Phys Ther. 2015;95:663–77. https://doi.org/10.2522/ptj.20130512 .

Scott D, Albrecht L, O’Leary K, Ball GDC, et al. Systematic review of knowledge translation strategies in the allied health professions. Implement Sci. 2012;7:1–17. https://doi.org/10.1186/1748-5908-7-70 .

Wu Y, Brettle A, Zhou C, Ou J, et al. Do educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes? A systematic review. Nurse Educ Today. 2018;70:109–14. https://doi.org/10.1016/j.nedt.2018.08.026 .

Yost J, Ganann R, Thompson D, Aloweni F, et al. The effectiveness of knowledge translation interventions for promoting evidence-informed decision-making among nurses in tertiary care: a systematic review and meta-analysis. Implement Sci. 2015;10:1–15. https://doi.org/10.1186/s13012-015-0286-1 .

Grudniewicz A, Kealy R, Rodseth RN, Hamid J, et al. What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses. Implement Sci. 2015;10:2–12. https://doi.org/10.1186/s13012-015-0347-5 .

Koota E, Kääriäinen M, Melender HL. Educational interventions promoting evidence-based practice among emergency nurses: a systematic review. Int Emerg Nurs. 2018;41:51–8. https://doi.org/10.1016/j.ienj.2018.06.004 .

Flodgren G, O’Brien MA, Parmelli E, et al. Local opinion leaders: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD000125.pub5 .

Arditi C, Rège-Walther M, Durieux P, et al. Computer-generated reminders delivered on paper to healthcare professionals: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2017. https://doi.org/10.1002/14651858.CD001175.pub4 .

Pantoja T, Grimshaw JM, Colomer N, et al. Manually-generated reminders delivered on paper: effects on professional practice and patient outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD001174.pub4 .

De Angelis G, Davies B, King J, McEwan J, et al. Information and communication technologies for the dissemination of clinical practice guidelines to health professionals: a systematic review. JMIR Med Educ. 2016;2:e16. https://doi.org/10.2196/mededu.6288 .

Brown A, Barnes C, Byaruhanga J, McLaughlin M, et al. Effectiveness of technology-enabled knowledge translation strategies in improving the use of research in public health: systematic review. J Med Internet Res. 2020;22:e17274. https://doi.org/10.2196/17274 .

Sykes MJ, McAnuff J, Kolehmainen N. When is audit and feedback effective in dementia care? A systematic review. Int J Nurs Stud. 2018;79:27–35. https://doi.org/10.1016/j.ijnurstu.2017.10.013 .

Bhatt NR, Czarniecki SW, Borgmann H, et al. A systematic review of the use of social media for dissemination of clinical practice guidelines. Eur Urol Focus. 2021;7:1195–204. https://doi.org/10.1016/j.euf.2020.10.008 .

Yamada J, Shorkey A, Barwick M, Widger K, et al. The effectiveness of toolkits as knowledge translation strategies for integrating evidence into clinical care: a systematic review. BMJ Open. 2015;5:e006808. https://doi.org/10.1136/bmjopen-2014-006808 .

Afari-Asiedu S, Abdulai MA, Tostmann A, et al. Interventions to improve dispensing of antibiotics at the community level in low and middle income countries: a systematic review. J Glob Antimicrob Resist. 2022;29:259–74. https://doi.org/10.1016/j.jgar.2022.03.009 .

Boonacker CW, Hoes AW, Dikhoff MJ, Schilder AG, et al. Interventions in health care professionals to improve treatment in children with upper respiratory tract infections. Int J Pediatr Otorhinolaryngol. 2010;74:1113–21. https://doi.org/10.1016/j.ijporl.2010.07.008 .

Al Zoubi FM, Menon A, Mayo NE, et al. The effectiveness of interventions designed to increase the uptake of clinical practice guidelines and best practices among musculoskeletal professionals: a systematic review. BMC Health Serv Res. 2018;18:2–11. https://doi.org/10.1186/s12913-018-3253-0 .

Ariyo P, Zayed B, Riese V, Anton B, et al. Implementation strategies to reduce surgical site infections: a systematic review. Infect Control Hosp Epidemiol. 2019;3:287–300. https://doi.org/10.1017/ice.2018.355 .

Borgert MJ, Goossens A, Dongelmans DA. What are effective strategies for the implementation of care bundles on ICUs: a systematic review. Implement Sci. 2015;10:1–11. https://doi.org/10.1186/s13012-015-0306-1 .

Cahill LS, Carey LM, Lannin NA, et al. Implementation interventions to promote the uptake of evidence-based practices in stroke rehabilitation. Cochrane Database Syst Rev. 2020. https://doi.org/10.1002/14651858.CD012575.pub2 .

Pedersen ER, Rubenstein L, Kandrack R, Danz M, et al. Elusive search for effective provider interventions: a systematic review of provider interventions to increase adherence to evidence-based treatment for depression. Implement Sci. 2018;13:1–30. https://doi.org/10.1186/s13012-018-0788-8 .

Jenkins HJ, Hancock MJ, French SD, Maher CG, et al. Effectiveness of interventions designed to reduce the use of imaging for low-back pain: a systematic review. CMAJ. 2015;187:401–8. https://doi.org/10.1503/cmaj.141183 .

Bennett S, Laver K, MacAndrew M, Beattie E, et al. Implementation of evidence-based, non-pharmacological interventions addressing behavior and psychological symptoms of dementia: a systematic review focused on implementation strategies. Int Psychogeriatr. 2021;33:947–75. https://doi.org/10.1017/S1041610220001702 .

Noonan VK, Wolfe DL, Thorogood NP, et al. Knowledge translation and implementation in spinal cord injury: a systematic review. Spinal Cord. 2014;52:578–87. https://doi.org/10.1038/sc.2014.62 .

Albrecht L, Archibald M, Snelgrove-Clarke E, et al. Systematic review of knowledge translation strategies to promote research uptake in child health settings. J Pediatr Nurs. 2016;31:235–54. https://doi.org/10.1016/j.pedn.2015.12.002 .

Campbell A, Louie-Poon S, Slater L, et al. Knowledge translation strategies used by healthcare professionals in child health settings: an updated systematic review. J Pediatr Nurs. 2019;47:114–20. https://doi.org/10.1016/j.pedn.2019.04.026 .

Bird ML, Miller T, Connell LA, et al. Moving stroke rehabilitation evidence into practice: a systematic review of randomized controlled trials. Clin Rehabil. 2019;33:1586–95. https://doi.org/10.1177/0269215519847253 .

Goorts K, Dizon J, Milanese S. The effectiveness of implementation strategies for promoting evidence informed interventions in allied healthcare: a systematic review. BMC Health Serv Res. 2021;21:1–11. https://doi.org/10.1186/s12913-021-06190-0 .

Zadro JR, O’Keeffe M, Allison JL, Lembke KA, et al. Effectiveness of implementation strategies to improve adherence of physical therapist treatment choices to clinical practice guidelines for musculoskeletal conditions: systematic review. Phys Ther. 2020;100:1516–41. https://doi.org/10.1093/ptj/pzaa101 .

Van der Veer SN, Jager KJ, Nache AM, et al. Translating knowledge on best practice into improving quality of RRT care: a systematic review of implementation strategies. Kidney Int. 2011;80:1021–34. https://doi.org/10.1038/ki.2011.222 .

Pawson R, Greenhalgh T, Harvey G, et al. Realist review–a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10Suppl 1:21–34. https://doi.org/10.1258/1355819054308530 .

Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research. Implementation Sci. 2012;7:1–10. https://doi.org/10.1186/1748-5908-7-33 .

Johnson MJ, May CR. Promoting professional behaviour change in healthcare: what interventions work, and why? A theory-led overview of systematic reviews. BMJ Open. 2015;5:e008592. https://doi.org/10.1136/bmjopen-2015-008592 .

Metz A, Jensen T, Farley A, Boaz A, et al. Is implementation research out of step with implementation practice? Pathways to effective implementation support over the last decade. Implement Res Pract. 2022;3:1–11. https://doi.org/10.1177/26334895221105585 .

May CR, Finch TL, Cornford J, Exley C, et al. Integrating telecare for chronic disease management in the community: What needs to be done? BMC Health Serv Res. 2011;11:1–11. https://doi.org/10.1186/1472-6963-11-131 .

Harvey G, Rycroft-Malone J, Seers K, Wilson P, et al. Connecting the science and practice of implementation – applying the lens of context to inform study design in implementation research. Front Health Serv. 2023;3:1–15. https://doi.org/10.3389/frhs.2023.1162762 .

Download references

Acknowledgements

The authors would like to thank Professor Kathryn Oliver for her support in the planning the review, Professor Steve Hanney for reading and commenting on the final manuscript and the staff at LSHTM library for their support in planning and conducting the literature search.

This study was supported by LSHTM’s Research England QR strategic priorities funding allocation and the National Institute for Health and Care Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust. Grant number NIHR200152. The views expressed are those of the author(s) and not necessarily those of the NIHR, the Department of Health and Social Care or Research England.

Author information

Authors and affiliations.

Health and Social Care Workforce Research Unit, The Policy Institute, King’s College London, Virginia Woolf Building, 22 Kingsway, London, WC2B 6LE, UK

Annette Boaz

King’s Business School, King’s College London, 30 Aldwych, London, WC2B 4BG, UK

Juan Baeza & Alec Fraser

Federal University of Santa Catarina (UFSC), Campus Universitário Reitor João Davi Ferreira Lima, Florianópolis, SC, 88.040-900, Brazil

Erik Persson

You can also search for this author in PubMed   Google Scholar

Contributions

AB led the conceptual development and structure of the manuscript. EP conducted the searches and data extraction. All authors contributed to screening and quality appraisal. EP and AF wrote the first draft of the methods section. AB, JB and AF performed result synthesis and contributed to the analyses. AB wrote the first draft of the manuscript and incorporated feedback and revisions from all other authors. All authors revised and approved the final manuscript.

Corresponding author

Correspondence to Annette Boaz .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: appendix a., additional file 2: appendix b., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Boaz, A., Baeza, J., Fraser, A. et al. ‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice. Implementation Sci 19 , 15 (2024). https://doi.org/10.1186/s13012-024-01337-z

Download citation

Received : 01 November 2023

Accepted : 05 January 2024

Published : 19 February 2024

DOI : https://doi.org/10.1186/s13012-024-01337-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Implementation
  • Interventions
  • Clinical practice
  • Research evidence
  • Multi-faceted

Implementation Science

ISSN: 1748-5908

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

narrative review research paper

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Epidemiol Health

Narrative reviews

Associated data.

Although qualitative researches (QR) are invaluable in understanding complex healthcare situations, the quantitative systematic reviews could not treat them. To improve quality of healthcare services, results of QR should be considered in healthcare decision-making processes. Several methods and theories for synthesizing evidences of QR have been developed. In order to activate the narrative reviews and mixed methods reviews in Korean healthcare academies, I arranged the related nomenclatures and suggested some issues to conduct them.

INTRODUCTION

In providing healthcare services, the evidence-based medicine (EBM) paradigm [ 1 - 3 ]—the idea that healthcare-related decisions should be made on the basis of the best evidence—is expanding to healthcare-related policy-making in nursing and health science as well as medicine [ 4 - 6 ]. Amidst this trend, quantitative systematic reviews is a new research approach. It is a term used to designate the approach to examining the effects of a treatment or an intervention adopting a statistical method called meta-analysis [ 7 ]. However, the results of qualitative research which cannot obtain quantitative outcomes, such as relative risk, odds ratio, and number needed to treat (NNT), tend to be ignored or excluded in the application of the systematic reviews process [ 8 ].

Not only the definition of qualitative research, but also the characteristic comparisons between qualitative and quantitative research are well documented in the papers by Murphy et al. [ 9 ] and Draper [ 10 ]. In short, qualitative research can be a useful guide for understanding complicated situations in the real world and can serve as groundwork for new hypotheses [ 11 - 15 ]. Consequently, efforts to use the synthesis of qualitative research results as the grounds for decision-making were already underway in 1990s when EBM emerged [ 16 - 23 ]. However, recently, in line with the trend toward improving medical treatment quality through patient-centered and evidence-based diagnostic and therapeutic services [ 24 , 25 ], efforts for complementary utilization of qualitative research results are experiencing a revival [ 13 , 15 , 26 - 34 ]. In particular, the claim that both quantitative and qualitative research should be utilized for proper understanding of overall healthcare problems is gaining influence. [ 18 , 35 , 36 ].

Meanwhile, because the synthesis of qualitative research results is inevitably different from a quantitative systematic reviews, there have been many attempts to overcome this gap [ 8 , 37 - 40 ]. For this reason, I would like to systemize the attempts to synthesize qualitative research results undertaken so far. This work is expected to show the framework of the discussions related to qualitative research and lay a cornerstone for the vitalization of systematic reviews on qualitative research in South Korea.

Terms related to the synthesis of qualitative research

Among the theories about qualitative research presented so far, works of Draper [ 10 ], Barnett-Page & Thomas [ 41 ], Dixon-Woods et al. [ 42 ], and Thorne et al. [ 43 ] may be representative. I arranged various terms related to qualitative research, grouping them according to the emphasis intended. Appendix 1 is the overview of the healthcare research applying this nomenclature. At a glance, we can see many terms are suggested with the emphasis placed on the meaning of the term “synthesis” of qualitative research results in contrast to the application of the meta-analytic statistical method of quantitative research results. In particular, we can see the term “mixed methods research” has often been applied recently in the attempt to include both quantitative and qualitative research [ 44 ].

Process of synthesis of qualitative research

As can be seen in Appendix 1 , the high number of related terms implies that the establishment of the relevant research methodology is a difficult task [ 45 ]. Sinuff et al. [ 46 ] shows the difference between quantitative and qualitative research processes in a diagram. However, a look at the suggested processes in relation to qualitative research synthesis [ 34 , 45 , 47 - 54 ] reveals that they stick to the big frame of “Ask - Acquire - Appraise - Apply - Assess,” 5A of the evidence cycle, although they show a certain diversity [ 55 ].

Searches of qualitative research to collect a body of related literature are more difficult than quantitative research [ 56 - 58 ]. This is because of different database services and the need to search for gray literature that has been issued but not officially published and made available in market, such as reports published by institutions or academic narrative reports [ 8 ]. It is also attributable to the need to resort, in addition to securing lists through search formulae, to hand searching which involves bibliography browsing in search of related papers and snowballing searching which traces one paper after another in chronological order [ 14 , 59 ].

Because of the diversity of research methods and fields of application, qualitative research does not easily lend itself to standardizing the items of qualitative evaluation in the literature of interest [ 8 , 59 - 64 ]. Nevertheless, the following achievements harvested so far deserve to be listed: (1) Thomas et al. [ 65 ] suggested evaluation items which matched quantitative-qualitative research. (2) Clark [ 66 ] developed ‘RATS’ evaluation tool, which is an acronym for Relevance, Appropriateness, Transparency, and Soundness. (3) Daly et al. [ 67 ] suggested a stratified structure by the contents of qualitative research. (4) Rodgers et al. [ 34 ] applied the EPPI approach that evaluates the persuasive power of evidence. And (5) Dixon-Woods et al. [ 68 ] suggested a tool called CASP (Critical Appraisal Skills Programme tool) which is composed of 10 items.

For the synthesis of evaluation results, a best fit frameworks is established [ 65 , 69 - 73 ] or a simulations model is selected [ 74 ]. The commercial program called NVivo has been developed [ 52 , 75 , 76 ]. Feasibility research examining the applicability of this program in Korean society needs to be accumulated.

Also, a reporting guideline named RAMESES (Realist And MEta-narrative Evidence Synthesis: Evolving Standards) has been developed for application when the research results of systematic reviews of qualitative research are to be reported in papers [ 77 , 78 ]. Making a flow chart is also suggested in cases where the mixed methods reviews approach is adopted which involves both quantitative and qualitative research [ 79 , 80 ].

CONCLUSION AND PROPOSAL

There are few systematic reviews on qualitative research in healthcare-related scholarship in South Korea. For the qualitative improvement of healthcare in this scarce situation, research in various healthcare fields should be conducted. Therefore, I suggest three things as below.

First, establishing the nomenclature of systematic reviews on qualitative research is an urgent task, because systematic and coherent use of well-established terms is important when a multitude of suggested terms are in use, as shown in Appendix 1 . The synthesis of quantitative research through the meta-analytic statistical method can be termed ‘quantitative systematic reviews’ and that of qualitative research, ‘qualitative systematic reviews’ [ 81 ]. However, given the current situation that systematic reviews have been established as the major research methodology of the synthesis of the evidences of quantitative research and the meta-analysis applied to this is recognized as statistical methodology [ 81 ], systematic reviews in the narrow sense mean quantitative systematic reviews [ 82 ]. On the contrary, qualitative systematic reviews is also called narrative systematic reviews and recently in a more abbreviated form called “narrative reviews” [ 29 , 83 - 85 ]. However, the term ‘meta-narrative reviews’ [ 77 , 78 ] does not fit and its use should be avoided because the meta-analysis corresponding to this is a statistical method, not a research method [ 81 ]. Therefore, I suggest a terminological differentiation between (quantitative) systematic reviews and (qualitative) narrative reviews, whereby the term “mixed methods reviews” may be used when both quantitative and qualitative research are involved. It was in consideration of this point that I titled this study “Narrative Reviews.”

Second, the practical way to revitalize narrative reviews research when Korean researchers do not have much experience conducting narrative reviews is the critical reading of good research cases from various academic fields and their application to practical use. To facilitate this process, I organized useful research cases by academic field and presented them in Appendix 2 .

Third, there is a need to establish a research-supporting organization and expand research human resources for qualitative research in the process of planning and conducting clinical studies [ 15 ]. The basic prior condition of proper narrative reviews is good results from qualitative research. First and foremost, given the fact that multidisciplinary cooperation is of vital importance for qualitative research, an organizational reshuffling appears necessary to facilitate efficient cooperation.

Acknowledgments

This study was conducted by a 2014 academic promotion research funding project grant of Jeju National University.

Appendix 1.Summary tables of nomenclatures about methodologies for synthesis of qualitative researches

Appendix 2.some articles related to qualitative researches about health care services.

The author has no conflicts of interest to declare for this study

SUPPLEMENTARY MATERIAL

Supplementary material is available at http://www.e-epih.org/ .

IMAGES

  1. Writing narrative literature reviews by Remy THEMVAC

    narrative review research paper

  2. Free Narrative Essay Examples

    narrative review research paper

  3. Narrative Report For Research Paper Essay Example

    narrative review research paper

  4. Narrative Research

    narrative review research paper

  5. Narrative research paper outline. Learn these easy steps to create a

    narrative review research paper

  6. Narrative Analysis

    narrative review research paper

VIDEO

  1. Writing Systematic and Narrative Review

  2. Disadvantages of Compact Academic Writing Programs

  3. Narrative Analysis

  4. Want To Finish Your PhD Or Publish Papers FASTER? Do This

  5. Your Key To Publishing Success

  6. Research Paper Methodology

COMMENTS

  1. Narrative Reviews: Flexible, Rigorous, and Practical

    Introduction. Narrative reviews are a type of knowledge synthesis grounded in a distinct research tradition. They are often framed as non-systematic, which implies that there is a hierarchy of evidence placing narrative reviews below other review forms. 1 However, narrative reviews are highly useful to medical educators and researchers. While a systematic review often focuses on a narrow ...

  2. An Introduction to Writing Narrative and Systematic Reviews

    The type of review needed may be influenced by the number of already published research papers and reviews. Adapted from Reference [9] . Translating Pautasso's picture into words, we need to know what literature is already "out there" — both original research and literature reviews, before working out what kind of review, if any, would ...

  3. PDF Formatting Guide for Narrative Reviews

    Narrative reviews are evidence-based summaries on a particular, defined topic, often covering a range of specific questions from pathophysiology to treatment. The content may be clinical, ethical, policy or legal review. The scope of the narrative review should be defined in the work. Though the standards of

  4. How to Write a Systematic Review: A Narrative Review

    A systematic review, as its name suggests, is a systematic way of collecting, evaluating, integrating, and presenting findings from several studies on a specific question or topic. [ 1] A systematic review is a research that, by identifying and combining evidence, is tailored to and answers the research question, based on an assessment of all ...

  5. Narrative Review

    A narrative review is the type first-year college students often learn as a general approach. Its purpose is to identify a few studies that describe a problem of interest. Narrative reviews have no predetermined research question or specified search strategy, only a topic of interest. They are not systematic and follow no specified protocol.

  6. Writing and appraising narrative reviews : Journal of Clinical ...

    ive reviews in today's era of sophisticated research methodologies and to describe in detail the process involved in writing a narrative review. Narrative reviews provide a concise review of literature with the liberty to raise questions and stimulate further research. The process of writing a good narrative review involves framing a research question, deciding the most suitable methodology ...

  7. PDF Writing narrative literature reviews for peer-reviewed journals

    Objective: To describe and discuss the process used to write a narrative review of the literature for publication in a peer-reviewed journal. Publication of narrative overviews of the literature should be standardized to increase their objectivity. Background: In the past decade numerous changes in research methodology pertaining to reviews of ...

  8. How to Conduct a Systematic Review: A Narrative Literature Review

    Writing a research question is the first step in conducting a systematic review and is of paramount importance as it outlines both the need and validity of systematic reviews (Nguyen, et al., unpublished data). It also increases the efficiency of the review by limiting the time and cost of identifying and obtaining relevant literature [ 11 ].

  9. How to write a superb literature review

    One of my favourite review-style articles 3 presents a plot bringing together data from multiple research papers (many of which directly contradict each other). This is then used to identify broad ...

  10. The Art and Science of Writing Narrative Reviews

    al bias in the appraisal of retrieved articles, and interpretation of findings, they serve as sources of quick up-to-date reference for specific areas of interest of readers. Well-conducted reviews could inform readers about gaps in existing literature and areas that need new primary research. Crafting a narrative review requires a blend of good scientific approach and the skillful art of ...

  11. An Introduction to Writing Narrative and Systematic Reviews

    A narrative review is the "older" format of the two, presenting a (non-systematic) summation and analysis of available literature on a specific topic of interest. ... The type of review needed may be influenced by the number of already published research papers and reviews. Adapted from Reference [Pautasso M. Ten simple rules for writing a ...

  12. (PDF) Writing narrative style literature reviews

    As a research method, the authors used narrative review of literature. The research material was 12 previous research papers report on nurses' opinions and attitudes regarding patient wills.

  13. Narrative Review

    A research paper arguing for a specific viewpoint - a lit review should avoid bias and highlight areas of disagreements; A systematic review; Purposes of a Literature Review. ... The authors performed a narrative review on Severe Acute Respiratory Syndrome- CoronaVirus-2 (SARS-CoV-2) and all infectious agents with the primary endpoints to ...

  14. Improving the peer review of narrative literature reviews

    As the size of the published scientific literature has increased exponentially over the past 30 years, review articles play an increasingly important role in helping researchers to make sense of original research results. Literature reviews can be broadly classified as either "systematic" or "narrative". Narrative reviews may be broader in scope than systematic reviews, but have been ...

  15. Narrative Review

    The labels Narrative Review and Literature Review are often describing the same type of review. For scientific purposes, the term Literature Review is the one used most often. For more information on the Literature Review, click on that link under the Review By Type tab.

  16. Writing narrative literature reviews for peer-reviewed journals

    New and experienced authors wishing to write a narrative overview should find this article useful in constructing such a paper and carrying out the research process. It is hoped that this article will stimulate scholarly dialog amongst colleagues about this research design and other complex literature review methods.

  17. How to write a narrative review

    A narrative review is a synthesis of information and existing literature in relation to a focused topic. They are generally non-quantitative, thematic, educational and, where appropriate, opinionated. ... We do not usually ask the same content experts to write a full systematic review in the same way because this is a research paper that uses ...

  18. SANRA—a scale for the quality assessment of narrative review articles

    Background Narrative reviews are the commonest type of articles in the medical literature. However, unlike systematic reviews and randomized controlled trials (RCT) articles, for which formal instruments exist to evaluate quality, there is currently no instrument available to assess the quality of narrative reviews. In response to this gap, we developed SANRA, the Scale for the Assessment of ...

  19. Narrative Reviews: Flexible, Rigorous, and Practical

    Narrative reviews are a type of knowledge synthesis grounded in a distinct research tradition. They are often framed as non-systematic, which implies that there is a hierarchy of evidence placing narrative reviews below other review forms. 1 However, narrative reviews are highly useful to medical educators and researchers. While a systematic review often focuses on a narrow question in a ...

  20. Narrative Literature Review

    The final paper should cover the themes identified in the research, explain any conflicts or disagreements, identify research gaps and potential future research areas, explain how this narrative review fits within the existing research and answer the research question. For additional information: Hempel. (2020). Conducting your literature review.

  21. (PDF) Narrative Research

    Abstract. Narrative research aims to unravel consequential stories of people's lives as told by them in their own words and worlds. In the context of the health, social sciences, and education ...

  22. Mental Health Social Work Practitioner Research: A Narrative Review of

    Purpose: This narrative review explores papers published in peer-reviewed journals reporting research from a practice research module of a qualifying program to examine their potential contribution to knowledge in mental health social work.Methods: A narrative review was undertaken according to the PRISMA extension for scoping reviews of papers published by the first three cohorts of a ...

  23. Narrative Reviews in Medical Education: Key Steps for Researchers

    The first step in conducting a narrative review requires researchers to describe the rationale and justification for the review. Narrative reviews are useful for research questions across many different topics. For example, researchers may be seeking clarity on a topic where there is limited knowledge, or to synthesize and analyze an existing ...

  24. 'It depends': what 86 systematic reviews tell us about what strategies

    This review updates and extends our previous review of systematic reviews of interventions designed to implement research evidence into clinical practice. To identify potentially relevant peer-reviewed research papers, we developed a comprehensive systematic literature search strategy based on the terms used in the Grimshaw et al. [ 9 ] and ...

  25. PDF Alcohol Research Current Reviews Instructions for Reviewers

    2: Figures and tables contribute to better understanding of the article. Scoping reviews only: The flow diagram is clear, and the reasons for excluding full-text articles are described. 1: Figures and tables somewhat help to better understand the article. Scoping reviews only: The flow diagram is unclear or confusing.

  26. Narrative reviews

    Process of synthesis of qualitative research. As can be seen in Appendix 1, the high number of related terms implies that the establishment of the relevant research methodology is a difficult task [].Sinuff et al. [] shows the difference between quantitative and qualitative research processes in a diagram.However, a look at the suggested processes in relation to qualitative research synthesis ...