Writing Center Home Page

OASIS: Writing Center

Reference list: common reference list examples, article (with doi).

Alvarez, E., & Tippins, S. (2019). Socialization agents that Puerto Rican college students use to make financial decisions. Journal of Social Change , 11 (1), 75–85. https://doi.org/10.5590/JOSC.2019.11.1.07

Laplante, J. P., & Nolin, C. (2014). Consultas and socially responsible investing in Guatemala: A case study examining Maya perspectives on the Indigenous right to free, prior, and informed consent. Society & Natural Resources , 27 , 231–248. https://doi.org/10.1080/08941920.2013.861554

Use the DOI number for the source whenever one is available. DOI stands for "digital object identifier," a number specific to the article that can help others locate the source. In APA 7, format the DOI as a web address. Active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list. Also see our Quick Answer FAQ, "Can I use the DOI format provided by library databases?"

Jerrentrup, A., Mueller, T., Glowalla, U., Herder, M., Henrichs, N., Neubauer, A., & Schaefer, J. R. (2018). Teaching medicine with the help of “Dr. House.” PLoS ONE , 13 (3), Article e0193972. https://doi.org/10.1371/journal.pone.0193972

For journal articles that are assigned article numbers rather than page ranges, include the article number in place of the page range.
For more on citing electronic resources, see  Electronic Sources References .

YouTube

Article (Without DOI)

Found in a common academic research database or in print.

Casler , T. (2020). Improving the graduate nursing experience through support on a social media platform. MEDSURG Nursing , 29 (2), 83–87.

If an article does not have a DOI and you retrieved it from a common academic research database through the university library, there is no need to include any additional electronic retrieval information. The reference list entry looks like the entry for a print copy of the article. (This format differs from APA 6 guidelines that recommended including the URL of a journal's homepage when the DOI was not available.) Note that APA 7 has additional guidance on reference list entries for articles found only in specific databases or archives such as Cochrane Database of Systematic Reviews, UpToDate, ProQuest Dissertations and Theses Global, and university archives. See APA 7, Section 9.30 for more information.

Found on an Open Access Website

Eaton, T. V., & Akers, M. D. (2007). Whistleblowing and good governance. CPA Journal , 77 (6), 66–71. http://archives.cpajournal.com/2007/607/essentials/p58.htm

Provide the direct web address/URL to a journal article found on the open web, often on an open access journal's website. In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list.

Weinstein, J. A. (2010).  Social change  (3rd ed.). Rowman & Littlefield.

If the book has an edition number, include it in parentheses after the title of the book. If the book does not list any edition information, do not include an edition number. The edition number is not italicized.

American Nurses Association. (2015). Nursing: Scope and standards of practice (3rd ed.).

If the author and publisher are the same, only include the author in its regular place and omit the publisher.

Lencioni, P. (2012). The advantage: Why organizational health trumps everything else in business . Jossey-Bass. https://amzn.to/343XPSJ

As a change from APA 6 to APA 7, it is no longer necessary to include the ebook format in the title. However, if you listened to an audiobook and the content differs from the text version (e.g., abridged content) or your discussion highlights elements of the audiobook (e.g., narrator's performance), then note that it is an audiobook in the title element in brackets. For ebooks and online audiobooks, also include the DOI number (if available) or nondatabase URL but leave out the electronic retrieval element if the ebook was found in a common academic research database, as with journal articles. APA 7 allows for the shortening of long DOIs and URLs, as shown in this example. See APA 7, Section 9.36 for more information.

Chapter in an Edited Book

Poe, M. (2017). Reframing race in teaching writing across the curriculum. In F. Condon & V. A. Young (Eds.), Performing antiracist pedagogy in rhetoric, writing, and communication (pp. 87–105). University Press of Colorado.

Include the page numbers of the chapter in parentheses after the book title.

Christensen, L. (2001). For my people: Celebrating community through poetry. In B. Bigelow, B. Harvey, S. Karp, & L. Miller (Eds.), Rethinking our classrooms: Teaching for equity and justice (Vol. 2, pp. 16–17). Rethinking Schools.

Also include the volume number or edition number in the parenthetical information after the book title when relevant.

Freud, S. (1961). The ego and the id. In J. Strachey (Ed.),  The standard edition of the complete psychological works of Sigmund Freud  (Vol. 19, pp. 3-66). Hogarth Press. (Original work published 1923)

When a text has been republished as part of an anthology collection, after the author’s name include the date of the version that was read. At the end of the entry, place the date of the original publication inside parenthesis along with the note “original work published.” For in-text citations of republished work, use both dates in the parenthetical citation, original date first with a slash separating the years, as in this example: Freud (1923/1961). For more information on reprinted or republished works, see APA 7, Sections 9.40-9.41.

Classroom Resources

Citing classroom resources.

If you need to cite content found in your online classroom, use the author (if there is one listed), the year of publication (if available), the title of the document, and the main URL of Walden classrooms. For example, you are citing study notes titled "Health Effects of Exposure to Forest Fires," but you do not know the author's name, your reference entry will look like this:

Health effects of exposure to forest fires [Lecture notes]. (2005). Walden University Canvas. https://waldenu.instructure.com

If you do know the author of the document, your reference will look like this:

Smith, A. (2005). Health effects of exposure to forest fires [PowerPoint slides]. Walden University Canvas. https://waldenu.instructure.com  

A few notes on citing course materials:

  • [Lecture notes]
  • [Course handout]
  • [Study notes]
  • It can be difficult to determine authorship of classroom documents. If an author is listed on the document, use that. If the resource is clearly a product of Walden (such as the course-based videos), use Walden University as the author. If you are unsure or if no author is indicated, place the title in the author spot, as above.
  • If you cannot determine a date of publication, you can use n.d. (for "no date") in place of the year.

Note:  The web location for Walden course materials is not directly retrievable without a password, and therefore, following APA guidelines, use the main URL for the class sites: https://class.waldenu.edu.

Citing Tempo Classroom Resources

Clear author: 

Smith, A. (2005). Health effects of exposure to forest fires [PowerPoint slides]. Walden University Brightspace. https://mytempo.waldenu.edu

Unclear author:

Health effects of exposure to forest fires [Lecture notes]. (2005). Walden University Brightspace. https://mytempo.waldenu.edu

Conference Sessions and Presentations

Feinman, Y. (2018, July 27). Alternative to proctoring in introductory statistics community college courses [Poster presentation]. Walden University Research Symposium, Minneapolis, MN, United States. https://scholarworks.waldenu.edu/symposium2018/23/

Torgerson, K., Parrill, J., & Haas, A. (2019, April 5-9). Tutoring strategies for online students [Conference session]. The Higher Learning Commission Annual Conference, Chicago, IL, United States. http://onlinewritingcenters.org/scholarship/torgerson-parrill-haas-2019/

Dictionary Entry

Merriam-Webster. (n.d.). Leadership. In Merriam-Webster.com dictionary . Retrieved May 28, 2020, from https://www.merriam-webster.com/dictionary/leadership

When constructing a reference for an entry in a dictionary or other reference work that has no byline (i.e., no named individual authors), use the name of the group—the institution, company, or organization—as author (e.g., Merriam Webster, American Psychological Association, etc.). The name of the entry goes in the title position, followed by "In" and the italicized name of the reference work (e.g., Merriam-Webster.com dictionary , APA dictionary of psychology ). In this instance, APA 7 recommends including a retrieval date as well for this online source since the contents of the page change over time. End the reference entry with the specific URL for the defined word.

Discussion Board Post

Osborne, C. S. (2010, June 29). Re: Environmental responsibility [Discussion post]. Walden University Canvas.  https://waldenu.instructure.com  

Dissertations or Theses

Retrieved From a Database

Nalumango, K. (2019). Perceptions about the asylum-seeking process in the United States after 9/11 (Publication No. 13879844) [Doctoral dissertation, Walden University]. ProQuest Dissertations and Theses.

Retrieved From an Institutional or Personal Website

Evener. J. (2018). Organizational learning in libraries at for-profit colleges and universities [Doctoral dissertation, Walden University]. ScholarWorks. https://scholarworks.waldenu.edu/cgi/viewcontent.cgi?article=6606&context=dissertations

Unpublished Dissertation or Thesis

Kirwan, J. G. (2005). An experimental study of the effects of small-group, face-to-face facilitated dialogues on the development of self-actualization levels: A movement towards fully functional persons [Unpublished doctoral dissertation]. Saybrook Graduate School and Research Center.

For further examples and information, see APA 7, Section 10.6.

Legal Material

For legal references, APA follows the recommendations of The Bluebook: A Uniform System of Citation , so if you have any questions beyond the examples provided in APA, seek out that resource as well.

Court Decisions

Reference format:

Name v. Name, Volume Reporter Page (Court Date). URL

Sample reference entry:

Brown v. Board of Education, 347 U.S. 483 (1954). https://www.oyez.org/cases/1940-1955/347us483

Sample citation:

In Brown v. Board of Education (1954), the Supreme Court ruled racial segregation in schools unconstitutional.

Note: Italicize the case name when it appears in the text of your paper.

Name of Act, Title Source § Section Number (Year). URL

Sample reference entry for a federal statute:

Individuals With Disabilities Education Act, 20 U.S.C. § 1400 et seq. (2004). https://www.congress.gov/108/plaws/publ446/PLAW-108publ446.pdf

Sample reference entry for a state statute:

Minnesota Nurse Practice Act, Minn. Stat. §§ 148.171 et seq. (2019). https://www.revisor.mn.gov/statutes/cite/148.171

Sample citation: Minnesota nurses must maintain current registration in order to practice (Minnesota Nurse Practice Act, 2010).

Note: The § symbol stands for "section." Use §§ for sections (plural). To find this symbol in Microsoft Word, go to "Insert" and click on Symbol." Look in the Latin 1-Supplement subset. Note: U.S.C. stands for "United States Code." Note: The Latin abbreviation " et seq. " means "and what follows" and is used when the act includes the cited section and ones that follow. Note: List the chapter first followed by the section or range of sections.

Unenacted Bills and Resolutions

(Those that did not pass and become law)

Title [if there is one], bill or resolution number, xxx Cong. (year). URL

Sample reference entry for Senate bill:

Anti-Phishing Act, S. 472, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/senate-bill/472

Sample reference entry for House of Representatives resolution:

Anti-Phishing Act, H.R. 1099, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/house-bill/1099

The Anti-Phishing Act (2005) proposed up to 5 years prison time for people running Internet scams.

These are the three legal areas you may be most apt to cite in your scholarly work. For more examples and explanation, see APA 7, Chapter 11.

Magazine Article

Clay, R. (2008, June). Science vs. ideology: Psychologists fight back about the misuse of research. Monitor on Psychology , 39 (6). https://www.apa.org/monitor/2008/06/ideology

Note that for citations, include only the year: Clay (2008). For magazine articles retrieved from a common academic research database, leave out the URL. For magazine articles from an online news website that is not an online version of a print magazine, follow the format for a webpage reference list entry.

Newspaper Article (Retrieved Online)

Baker, A. (2014, May 7). Connecticut students show gains in national tests. New York Times . http://www.nytimes.com/2014/05/08/nyregion/national-assessment-of-educational-progress-results-in-Connecticut-and-New-Jersey.html

Include the full date in the format Year, Month Day. Do not include a retrieval date for periodical sources found on websites. Note that for citations, include only the year: Baker (2014). For newspaper articles retrieved from a common academic research database, leave out the URL. For newspaper articles from an online news website that is not an online version of a print newspaper, follow the format for a webpage reference list entry.

Online Video/Webcast

Walden University. (2013).  An overview of learning  [Video]. Walden University Canvas.  https://waldenu.instructure.com  

Use this format for online videos such as Walden videos in classrooms. Most of our classroom videos are produced by Walden University, which will be listed as the author in your reference and citation. Note: Some examples of audiovisual materials in the APA manual show the word “Producer” in parentheses after the producer/author area. In consultation with the editors of the APA manual, we have determined that parenthetical is not necessary for the videos in our courses. The manual itself is unclear on the matter, however, so either approach should be accepted. Note that the speaker in the video does not appear in the reference list entry, but you may want to mention that person in your text. For instance, if you are viewing a video where Tobias Ball is the speaker, you might write the following: Tobias Ball stated that APA guidelines ensure a consistent presentation of information in student papers (Walden University, 2013). For more information on citing the speaker in a video, see our page on Common Citation Errors .

Taylor, R. [taylorphd07]. (2014, February 27). Scales of measurement [Video]. YouTube. https://www.youtube.com/watch?v=PDsMUlexaMY

Walden University Academic Skills Center. (2020, April 15). One-way ANCOVA: Introduction [Video]. YouTube. https://youtu.be/_XnNDQ5CNW8

For videos from streaming sites, use the person or organization who uploaded the video in the author space to ensure retrievability, whether or not that person is the speaker in the video. A username can be provided in square brackets. As a change from APA 6 to APA 7, include the publisher after the title, and do not use "Retrieved from" before the URL. See APA 7, Section 10.12 for more information and examples.

See also reference list entry formats for TED Talks .

Technical and Research Reports

Edwards, C. (2015). Lighting levels for isolated intersections: Leading to safety improvements (Report No. MnDOT 2015-05). Center for Transportation Studies. http://www.cts.umn.edu/Publications/ResearchReports/reportdetail.html?id=2402

Technical and research reports by governmental agencies and other research institutions usually follow a different publication process than scholarly, peer-reviewed journals. However, they present original research and are often useful for research papers. Sometimes, researchers refer to these types of reports as gray literature , and white papers are a type of this literature. See APA 7, Section 10.4 for more information.

Reference list entires for TED Talks follow the usual guidelines for multimedia content found online. There are two common places to find TED talks online, with slightly different reference list entry formats for each.

TED Talk on the TED website

If you find the TED Talk on the TED website, follow the format for an online video on an organizational website:

Owusu-Kesse, K. (2020, June). 5 needs that any COVID-19 response should meet [Video]. TED Conferences. https://www.ted.com/talks/kwame_owusu_kesse_5_needs_that_any_covid_19_response_should_meet

The speaker is the author in the reference list entry if the video is posted on the TED website. For citations, use the speaker's surname.

TED Talk on YouTube

If you find the TED Talk on YouTube or another streaming video website, follow the usual format for streaming video sites:

TED. (2021, February 5). The shadow pandemic of domestic violence during COVID-19 | Kemi DaSilvalbru [Video]. YouTube. https://www.youtube.com/watch?v=PGdID_ICFII

TED is the author in the reference list entry if the video is posted on YouTube since it is the channel on which the video is posted. For citations, use TED as the author.

Walden University Course Catalog

To include the Walden course catalog in your reference list, use this format:

Walden University. (2020). 2019-2020 Walden University catalog . https://catalog.waldenu.edu/index.php

If you cite from a specific portion of the catalog in your paper, indicate the appropriate section and paragraph number in your text:

...which reflects the commitment to social change expressed in Walden University's mission statement (Walden University, 2020, Vision, Mission, and Goals section, para. 2).

And in the reference list:

Walden University. (2020). Vision, mission, and goals. In 2019-2020 Walden University catalog. https://catalog.waldenu.edu/content.php?catoid=172&navoid=59420&hl=vision&returnto=search

Vartan, S. (2018, January 30). Why vacations matter for your health . CNN. https://www.cnn.com/travel/article/why-vacations-matter/index.html

For webpages on the open web, include the author, date, webpage title, organization/site name, and URL. (There is a slight variation for online versions of print newspapers or magazines. For those sources, follow the models in the previous sections of this page.)

American Federation of Teachers. (n.d.). Community schools . http://www.aft.org/issues/schoolreform/commschools/index.cfm

If there is no specified author, then use the organization’s name as the author. In such a case, there is no need to repeat the organization's name after the title.

In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list.

Related Resources

Blogger

Knowledge Check: Common Reference List Examples

Didn't find what you need? Search our website or email us .

Read our website accessibility and accommodation statement .

  • Previous Page: Overview
  • Next Page: Common Military Reference List Examples
  • Office of Student Disability Services

Walden Resources

Departments.

  • Academic Residencies
  • Academic Skills
  • Career Planning and Development
  • Customer Care Team
  • Field Experience
  • Military Services
  • Student Success Advising
  • Writing Skills

Centers and Offices

  • Center for Social Change
  • Office of Academic Support and Instructional Services
  • Office of Degree Acceleration
  • Office of Research and Doctoral Services
  • Office of Student Affairs

Student Resources

  • Doctoral Writing Assessment
  • Form & Style Review
  • Quick Answers
  • ScholarWorks
  • SKIL Courses and Workshops
  • Walden Bookstore
  • Walden Catalog & Student Handbook
  • Student Safety/Title IX
  • Legal & Consumer Information
  • Website Terms and Conditions
  • Cookie Policy
  • Accessibility
  • Accreditation
  • State Authorization
  • Net Price Calculator
  • Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

Elsevier QRcode Wechat

  • Manuscript Preparation

How to write your references quickly and easily

  • 3 minute read
  • 210.2K views

Table of Contents

Every scientific paper builds on previous research – even if it’s in a new field, related studies will have preceded and informed it. In peer-reviewed articles, authors must give credit to this previous research, through citations and references. Not only does this show clearly where the current research came from, but it also helps readers understand the content of the paper better.

There is no optimum number of references for an academic article but depending on the subject you could be dealing with more than 100 different papers, conference reports, video articles, medical guidelines or any number of other resources.

That’s a lot of content to manage. Before submitting your manuscript, this needs to be checked, cross-references in the text and the list, organized and formatted.

The exact content and format of the citations and references in your paper will depend on the journal you aim to publish in, so the first step is to check the journal’s Guide for Authors before you submit.

There are two main points to pay attention to – consistency and accuracy. When you go through your manuscript to edit or proofread it, look closely at the citations within the text. Are they all the same? For example, if the journal prefers the citations to be in the format (name, year), make sure they’re all the same: (Smith, 2016).

Your citations must also be accurate and complete. Do they match your references list? Each citation should be included in the list, so cross-checking is important. It’s also common for journals to prefer that most, if not all, of the articles listed in your references be cited within the text – after all, these should be studies that contributed to the knowledge underpinning your work, not just your bedtime reading. So go through them carefully, noting any missing references or citations and filling the gaps.

Each journal has its own requirements when it comes to the content and format of references, as well as where and how you should include them in your submission, so double-check before you hit send!

In general, a reference will include authors’ names and initials, the title of the article, name of the journal, volume and issue, date, page numbers and DOI. On ScienceDirect, articles are linked to their original source (if also published on ScienceDirect) or to their Scopus record, so including the DOI can help link to the correct article.

A spotless reference list

Luckily, compiling and editing the references in your scientific manuscript can be easy – and it no longer has to be manual. Management tools like Mendeley can keep track of all your references, letting you share them with your collaborators. With the Word plugin, it’s possible to select the right citation style for the journal you’re submitting to and the tool will format your references automatically.

Like with any other part of your manuscript, it’s important to make sure your reference list has been checked and edited. Elsevier Author Services Language Editing can help, with professional manuscript editing that will help make sure your references don’t hold you back from publication.

Professional Manuscript Translation

Why it’s best to ask a professional when it comes to translation

How-to-choose-a-Journal-to-submit-an-article

  • Research Process

How to Choose a Journal to Submit an Article

You may also like.

Limitations of a Research

Can Describing Study Limitations Improve the Quality of Your Paper?

Guide to Crafting Impactful Sentences

A Guide to Crafting Shorter, Impactful Sentences in Academic Writing

Write an Excellent Discussion in Your Manuscript

6 Steps to Write an Excellent Discussion in Your Manuscript

How to Write Clear Civil Engineering Papers

How to Write Clear and Crisp Civil Engineering Papers? Here are 5 Key Tips to Consider

what are references on a research paper

The Clear Path to An Impactful Paper: ②

Essentials of Writing to Communicate Research in Medicine

The Essentials of Writing to Communicate Research in Medicine

There are some recognizable elements and patterns often used for framing engaging sentences in English. Find here the sentence patterns in Academic Writing

Changing Lines: Sentence Patterns in Academic Writing

what are references on a research paper

Path to An Impactful Paper: Common Manuscript Writing Patterns and Structure

Input your search keywords and press Enter.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

In-Text Citations: The Basics

OWL logo

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

Note:  This page reflects the latest version of the APA Publication Manual (i.e., APA 7), which released in October 2019. The equivalent resource for the older APA 6 style  can be found here .

Reference citations in text are covered on pages 261-268 of the Publication Manual. What follows are some general guidelines for referring to the works of others in your essay.

Note:  On pages 117-118, the Publication Manual suggests that authors of research papers should use the past tense or present perfect tense for signal phrases that occur in the literature review and procedure descriptions (for example, Jones (1998)  found  or Jones (1998)  has found ...). Contexts other than traditionally-structured research writing may permit the simple present tense (for example, Jones (1998)  finds ).

APA Citation Basics

When using APA format, follow the author-date method of in-text citation. This means that the author's last name and the year of publication for the source should appear in the text, like, for example, (Jones, 1998). One complete reference for each source should appear in the reference list at the end of the paper.

If you are referring to an idea from another work but  NOT  directly quoting the material, or making reference to an entire book, article or other work, you only have to make reference to the author and year of publication and not the page number in your in-text reference.

On the other hand, if you are directly quoting or borrowing from another work, you should include the page number at the end of the parenthetical citation. Use the abbreviation “p.” (for one page) or “pp.” (for multiple pages) before listing the page number(s). Use an en dash for page ranges. For example, you might write (Jones, 1998, p. 199) or (Jones, 1998, pp. 199–201). This information is reiterated below.

Regardless of how they are referenced, all sources that are cited in the text must appear in the reference list at the end of the paper.

In-text citation capitalization, quotes, and italics/underlining

  • Always capitalize proper nouns, including author names and initials: D. Jones.
  • If you refer to the title of a source within your paper, capitalize all words that are four letters long or greater within the title of a source:  Permanence and Change . Exceptions apply to short words that are verbs, nouns, pronouns, adjectives, and adverbs:  Writing New Media ,  There Is Nothing Left to Lose .

( Note:  in your References list, only the first word of a title will be capitalized:  Writing new media .)

  • When capitalizing titles, capitalize both words in a hyphenated compound word:  Natural-Born Cyborgs .
  • Capitalize the first word after a dash or colon: "Defining Film Rhetoric: The Case of Hitchcock's  Vertigo ."
  • If the title of the work is italicized in your reference list, italicize it and use title case capitalization in the text:  The Closing of the American Mind ;  The Wizard of Oz ;  Friends .
  • If the title of the work is not italicized in your reference list, use double quotation marks and title case capitalization (even though the reference list uses sentence case): "Multimedia Narration: Constructing Possible Worlds;" "The One Where Chandler Can't Cry."

Short quotations

If you are directly quoting from a work, you will need to include the author, year of publication, and page number for the reference (preceded by "p." for a single page and “pp.” for a span of multiple pages, with the page numbers separated by an en dash).

You can introduce the quotation with a signal phrase that includes the author's last name followed by the date of publication in parentheses.

If you do not include the author’s name in the text of the sentence, place the author's last name, the year of publication, and the page number in parentheses after the quotation.

Long quotations

Place direct quotations that are 40 words or longer in a free-standing block of typewritten lines and omit quotation marks. Start the quotation on a new line, indented 1/2 inch from the left margin, i.e., in the same place you would begin a new paragraph. Type the entire quotation on the new margin, and indent the first line of any subsequent paragraph within the quotation 1/2 inch from the new margin. Maintain double-spacing throughout, but do not add an extra blank line before or after it. The parenthetical citation should come after the closing punctuation mark.

Because block quotation formatting is difficult for us to replicate in the OWL's content management system, we have simply provided a screenshot of a generic example below.

This image shows how to format a long quotation in an APA seventh edition paper.

Formatting example for block quotations in APA 7 style.

Quotations from sources without pages

Direct quotations from sources that do not contain pages should not reference a page number. Instead, you may reference another logical identifying element: a paragraph, a chapter number, a section number, a table number, or something else. Older works (like religious texts) can also incorporate special location identifiers like verse numbers. In short: pick a substitute for page numbers that makes sense for your source.

Summary or paraphrase

If you are paraphrasing an idea from another work, you only have to make reference to the author and year of publication in your in-text reference and may omit the page numbers. APA guidelines, however, do encourage including a page range for a summary or paraphrase when it will help the reader find the information in a longer work. 

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

  • Knowledge Base
  • Referencing

A Quick Guide to Harvard Referencing | Citation Examples

Published on 14 February 2020 by Jack Caulfield . Revised on 15 September 2023.

Referencing is an important part of academic writing. It tells your readers what sources you’ve used and how to find them.

Harvard is the most common referencing style used in UK universities. In Harvard style, the author and year are cited in-text, and full details of the source are given in a reference list .

Harvard Reference Generator

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

upload-your-document-ai-proofreader

Table of contents

Harvard in-text citation, creating a harvard reference list, harvard referencing examples, referencing sources with no author or date, frequently asked questions about harvard referencing.

A Harvard in-text citation appears in brackets beside any quotation or paraphrase of a source. It gives the last name of the author(s) and the year of publication, as well as a page number or range locating the passage referenced, if applicable:

Note that ‘p.’ is used for a single page, ‘pp.’ for multiple pages (e.g. ‘pp. 1–5’).

An in-text citation usually appears immediately after the quotation or paraphrase in question. It may also appear at the end of the relevant sentence, as long as it’s clear what it refers to.

When your sentence already mentions the name of the author, it should not be repeated in the citation:

Sources with multiple authors

When you cite a source with up to three authors, cite all authors’ names. For four or more authors, list only the first name, followed by ‘ et al. ’:

Sources with no page numbers

Some sources, such as websites , often don’t have page numbers. If the source is a short text, you can simply leave out the page number. With longer sources, you can use an alternate locator such as a subheading or paragraph number if you need to specify where to find the quote:

Multiple citations at the same point

When you need multiple citations to appear at the same point in your text – for example, when you refer to several sources with one phrase – you can present them in the same set of brackets, separated by semicolons. List them in order of publication date:

Multiple sources with the same author and date

If you cite multiple sources by the same author which were published in the same year, it’s important to distinguish between them in your citations. To do this, insert an ‘a’ after the year in the first one you reference, a ‘b’ in the second, and so on:

Prevent plagiarism, run a free check.

A bibliography or reference list appears at the end of your text. It lists all your sources in alphabetical order by the author’s last name, giving complete information so that the reader can look them up if necessary.

The reference entry starts with the author’s last name followed by initial(s). Only the first word of the title is capitalised (as well as any proper nouns).

Harvard reference list example

Sources with multiple authors in the reference list

As with in-text citations, up to three authors should be listed; when there are four or more, list only the first author followed by ‘ et al. ’:

Reference list entries vary according to source type, since different information is relevant for different sources. Formats and examples for the most commonly used source types are given below.

  • Entire book
  • Book chapter
  • Translated book
  • Edition of a book

Journal articles

  • Print journal
  • Online-only journal with DOI
  • Online-only journal with no DOI
  • General web page
  • Online article or blog
  • Social media post

Sometimes you won’t have all the information you need for a reference. This section covers what to do when a source lacks a publication date or named author.

No publication date

When a source doesn’t have a clear publication date – for example, a constantly updated reference source like Wikipedia or an obscure historical document which can’t be accurately dated – you can replace it with the words ‘no date’:

Note that when you do this with an online source, you should still include an access date, as in the example.

When a source lacks a clearly identified author, there’s often an appropriate corporate source – the organisation responsible for the source – whom you can credit as author instead, as in the Google and Wikipedia examples above.

When that’s not the case, you can just replace it with the title of the source in both the in-text citation and the reference list:

Harvard referencing uses an author–date system. Sources are cited by the author’s last name and the publication year in brackets. Each Harvard in-text citation corresponds to an entry in the alphabetised reference list at the end of the paper.

Vancouver referencing uses a numerical system. Sources are cited by a number in parentheses or superscript. Each number corresponds to a full reference at the end of the paper.

A Harvard in-text citation should appear in brackets every time you quote, paraphrase, or refer to information from a source.

The citation can appear immediately after the quotation or paraphrase, or at the end of the sentence. If you’re quoting, place the citation outside of the quotation marks but before any other punctuation like a comma or full stop.

In Harvard referencing, up to three author names are included in an in-text citation or reference list entry. When there are four or more authors, include only the first, followed by ‘ et al. ’

Though the terms are sometimes used interchangeably, there is a difference in meaning:

  • A reference list only includes sources cited in the text – every entry corresponds to an in-text citation .
  • A bibliography also includes other sources which were consulted during the research but not cited.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Caulfield, J. (2023, September 15). A Quick Guide to Harvard Referencing | Citation Examples. Scribbr. Retrieved 22 February 2024, from https://www.scribbr.co.uk/referencing/harvard-style/

Is this article helpful?

Jack Caulfield

Jack Caulfield

Other students also liked, harvard in-text citation | a complete guide & examples, harvard style bibliography | format & examples, referencing books in harvard style | templates & examples, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

what are references on a research paper

  • Search This Site All UCSD Sites Faculty/Staff Search Term
  • Contact & Directions
  • Climate Statement
  • Cognitive Behavioral Neuroscience
  • Cognitive Psychology
  • Developmental Psychology
  • Social Psychology
  • Adjunct Faculty
  • Non-Senate Instructors
  • Researchers
  • Psychology Grads
  • Affiliated Grads
  • New and Prospective Students
  • Honors Program
  • Experiential Learning
  • Programs & Events
  • Psi Chi / Psychology Club
  • Prospective PhD Students
  • Current PhD Students
  • Area Brown Bags
  • Colloquium Series
  • Anderson Distinguished Lecture Series
  • Speaker Videos
  • Undergraduate Program
  • Academic and Writing Resources

Writing Research Papers

  • What Types of References Are Appropriate?

When writing a research paper, there are many different types of sources that you might consider citing.  Which are appropriate?  Which are less appropriate?  Here we discuss the different types of sources that you may wish to use when working on a research paper.   

Please note that the following represents a general set of recommended guidelines that is not specific to any class and does not represent department policy.  The types of allowable sources may vary by course and instructor.

Highly appropriate: peer-reviewed journal articles

In general, you should primarily cite peer-reviewed journal articles in your research papers.  Peer-reviewed journal articles are research papers that have been accepted for publication after having undergone a rigorous editorial review process.  During that review process, the article was carefully evaluated by at least one journal editor and a group of reviewers (usually scientists that are experts in the field or topic under investigation).  Often the article underwent revisions before it was judged to be satisfactory for publication. 

Most articles submitted to high quality journals are not accepted for publication.  As such, research that is successfully published in a respected peer-reviewed journal is generally regarded as higher quality than research that is not published or is published elsewhere, such as in a book, magazine, or on a website.  However, just because a study was published in a peer-reviewed journal does not mean that it is free from error or that its conclusions are correct.  Accordingly, it is important to critically read and carefully evaluate all sources, including peer-reviewed journal articles.

Tips for finding and using peer-reviewed journal articles:

  • Many databases, such as PsycINFO, can be set to only search for peer-reviewed journal articles. Other search engines, such as Google Scholar, typically include both peer-reviewed and not peer-reviewed articles in search results, and thus should be used with greater caution. 
  • Even though a peer-reviewed journal article is, by definition, a source that has been carefully vetted through an editorial process, it should still be critically evaluated by the reader. 

Potentially appropriate: books, encyclopedias, and other scholarly works

Another potential source that you might use when writing a research paper is a book, encyclopedia, or an official online source (such as demographic data drawn from a government website).  When relying on such sources, it is important to carefully consider its accuracy and trustworthiness.  For example, books vary in quality; most have not undergone any form of review process other than basic copyediting.  In many cases, a book’s content is little more than the author’s informed or uninformed opinion. 

However, there are books that have been edited prior to publication, as is the case with many reputable encyclopedias; also, many books from academic publishers are comprised of multiple chapters, each written by one or more researchers, with the entire volume carefully reviewed by one or more editors.  In those cases, the book has undergone a form of peer review, albeit often not as rigorous as that for a peer-reviewed journal article.

Tips for using books, encyclopedias, and other scholarly works:

  • When using books, encyclopedias, and other scholarly works (that is, works written or produced by researchers, official agencies, or corporations), it is important to very carefully evaluate the quality of that source.
  • If the source is an edited volume (in which case in the editor(s) will be listed on the cover), is published by a reputable source (such as Academic Press, MIT Press, and others), or is written by a major expert in the field (such as a researcher with a track record of peer-reviewed journal articles on the subject), then it is more likely to be trustworthy.
  • For online encyclopedias such as Wikipedia, an instructor may or may not consider that an acceptable source (by default, don’t assume that a non-peer reviewed source will be considered acceptable). It is best to ask the instructor for clarification. 1

Usually inappropriate: magazines, blogs, and websites  

Most research papers can be written using only peer-reviewed journal articles as sources.  However, for many topics it is possible to find a plethora of sources that have not been peer-reviewed but also discuss the topic.  These may include articles in popular magazines or postings in blogs, forums, and other websites.  In general, although these sources may be well-written and easy to understand, their scientific value is often not as high as that of peer-reviewed articles.  Exceptions include some magazine and newspaper articles that might be cited in a research paper to make a point about public awareness of a given topic, to illustrate beliefs and attitudes about a given topic among journalists, or to refer to a news event that is relevant to a given topic. 

Tips for using magazines, blogs, and websites:

  • Avoid such references if possible. You should primarily focus on peer-reviewed journal articles as sources for your research paper.  High quality research papers typically do not rely on non-academic and not peer-reviewed sources.
  • Refer to non-academic, not peer-reviewed sources sparingly, and if you do, be sure to carefully evaluate the accuracy and scientific merit of the source.

Downloadable Resources

  • How to Write APA Style Research Papers (a comprehensive guide) [ PDF ]
  • Tips for Writing APA Style Research Papers (a brief summary) [ PDF ]

Further Resources

How-To Videos     

  • Writing Research Paper Videos

Databases and Search Engines (may require connection to UCSD network)

  • Google Scholar
  • PubMed (NIH/NLM)
  • Web of Science  

UCSD Resources on Finding and Evaluating Sources

  • UCSD Library Databases A-Z
  • UCSD Library Psychology Research Guide: Start Page
  • UCSD Library Psychology Research Guide : Finding Articles
  • UCSD Library Psychology Research Guide : Evaluating Sources

External Resources

  • Critically Reading Journal Articles from PSU/ Colby College
  • How to Seriously Read a Journal Article from Science Magazine
  • How to Read Journal Articles from Harvard University
  • How to Read a Scientific Paper Infographic from Elsevier Publishing
  • Tips for searching PsycINFO from UC Berkeley Library
  • Tips for using PsycINFO effectively from the APA Student Science Council

1 Wikipedia articles vary in quality; the site has a peer review system and the very best articles ( Featured Articles ), which go through a multi-stage review process, rival those in traditional encyclopedias and are considered the highest quality articles on the site.

Prepared by s. c. pan for ucsd psychology, graphic adapted from  t-x-generic-apply.svg , a public domain creation by the tango desktop project..

Back to top

  • Research Paper Structure
  • Formatting Research Papers
  • Using Databases and Finding References
  • Evaluating References and Taking Notes
  • Citing References
  • Writing a Literature Review
  • Writing Process and Revising
  • Improving Scientific Writing
  • Academic Integrity and Avoiding Plagiarism
  • Writing Research Papers Videos

University of Leeds logo

  • Study and research support
  • Referencing

Referencing explained

Why and when to reference.

Referencing is an important part of academic work. It puts your work in context, demonstrates the breadth and depth of your research, and acknowledges other people’s work. You should reference whenever you use someone else’s idea.

View video using Microsoft Stream (link opens in a new window, available for University members only)

These webpages explain what referencing is, why it is important and give an overview of the main elements of how to reference. Our Referencing made simple tutorial opens in a new window and covers how to identify your source and create a reference with interactive examples.

Why reference?

Referencing correctly:

  • helps you to avoid plagiarism by making it clear which ideas are your own and which are someone else’s
  • shows your understanding of the topic
  • gives supporting evidence for your ideas, arguments and opinions
  • allows others to identify the sources you have used.

When to reference

Whenever you use an idea from someone else's work, for example from a journal article, textbook or website, you should cite the original author to make it clear where that idea came from. This is the case regardless of whether you have paraphrased, summarised or directly quoted their work. This is a key part of good practice in academic writing.

Read more on:

  • academic integrity
  • quoting, summarising, paraphrasing, and synthesising
  • citing direct quotations in Leeds Harvard or citing direct quotations in Leeds Numeric styles.

University and school policies

The University referencing policy (PDF) sets out the referencing requirements that all taught students and tutors are expected to follow.

Each school in the University requires students to use a specific style of referencing. Check the referencing style used in your school before you begin.

All your citations and references should match the style you are using exactly, including any punctuation, capitalisation, italics and bold, and you should use the same referencing style throughout your assignment.

Book cover

Research Methodology and Scientific Writing pp 361–400 Cite as

References: How to Cite and List Correctly

  • C. George Thomas 2  
  • First Online: 25 February 2021

4081 Accesses

When we write an essay, research paper, thesis, or book, it is normal to include information from the work of others or support our arguments by reference to other published works. All such academic documents draw heavily on the ideas and findings of previous and current researchers available through various sources such as books, journals, theses, newspapers, magazines, government reports, or Internet sources. In all these cases, proper referencing is essential in order to ensure easy retrieval of information. Referencing is the name given to the method of showing and acknowledging the sources from which the author has obtained ideas or information.

Everything deep is also simple and can be reproduced simply as long as its reference to the whole truth is maintained. But what matters is not what is witty but what is true. Albert Schweitzer (1875–1965)

This is a preview of subscription content, log in via an institution .

Buying options

  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
  • Available as EPUB and PDF
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
  • Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Bibliography

AMA [American Medical Association] 2007. AMA Manual of Style: A Guide for Authors and Editors (10 th Ed.). Oxford University Press, New York, 1024p.

Google Scholar  

APA [American Psychological Association] 2013. Publication Manual of the American Psychological Association (6 th Ed.). American Psychological Association, Washington DC, 272p.

Barrass, R. 2002. Scientists Must Write: A Guide to Better Writing for Scientists, Engineers and Students (2 nd Ed.). Routledge, London, 204p.

BSI [British Standards Institution] 1990. Recommendations for Citing and Referencing Published Material (2 nd Ed.). British Standards Institution (BS 5605:1990), 8p.

Cargill, M. and O’Connor, P. 2009. Writing Scientific Research Articles: Strategy and Steps. Wiley-Blackwell, 173p.

COA [Commonwealth of Australia] 2002. Style Manual for Authors, Editors and Printers (6 th Ed.). John Wiley & Sons, Brisbane, Australia, 550 p.

Cousens, R. 1987. Theory and reality of weed control thresholds. Plant Prot. Q. 2: 13–20

Craswell, G. 2004. Writing for Academic Success: A Postgraduate Guide. Sage Publications, London, 271p.

CSE [Council of Science Editors] 2014. Scientific Style and Format: The CSE Manual for Authors, Editors, and Publishers (8 th Ed.): The University of Chicago Press, 722p.

Davis, M. 2005. Scientific Papers and Presentations. Academic Press, Massachusetts, 356p.

Day, R.A. and Gastel, B. 2006. How to Write and Publish a Scientific Paper (6 th Ed.). Cambridge University Press, Cambridge, 320p.

Gustavii, B. 2008. How to Write and Illustrate a Scientific Paper (2 nd Ed.). Cambridge University press, Cambridge, 168p.

Hakansson, S. 1983. Competition and Production in Short-lived Crop Weed Stands: Density Effects. Report 127, Swedish University of Agricultural Sciences, Uppasala, 85p.

ISSN [International Standards Serial Number].2017. The List of title word abbreviations [on line]. Available: http://www.issn.org/services/onlineservices/access-to-the-ltwa/ [Accessed 06 May 2018].

Leggett, G., Mead, C.D., Kramer, M.G., and Beal, R.S. 1985. Handbook for Writers. Prentice- Hall, New Jersey, 558p.

Lester, J.D. and Lester, J.D. Jr. 2009. Writing Research Papers: A Complete Guide (13 th Ed.), Longman, 416p.

MLA [Modern Language Association of America] 2016. MLA Handbook for Writers of Research Papers (8 th Ed.) Modern Language Association of America, 146p

Neville, C. 2010. The Complete Guide to Referencing and Avoiding Plagiarism (2 nd Ed.). McGraw-Hill Education, 288 p.

Renu, S. and Thomas, C.G. 2000. Stale seedbed technique for the management of Sacciolepis interrupta in semi-dry rice. Indian J. Weed Sci. 32(3&4):146-149

Rubens, P. 2004. Science and Technical writing: A Manual of Style (2 nd Ed.). Routledge, New York, 427p.

Sheng, T.C. 1989. Soil Conservation for Small Farmers in the Humid Tropics. FAO Soils Bulletin No. 60. Food and Agriculture Organization of the United Nations, Rome, 104p

Thomas, C. G. 2008. Forage Crop Production in the Tropics. Kalyani Publishers, Ludhiana, 333p.

Thomas, C. G., Abraham, C. T., and Sreedevi, P. 1997. Weed flora and their relative dominance in semi-dry rice culture. J. Trop. Agric. 35:51–53.

Thomas, C. G. 2010. Land Husbandry and Watershed Management. Kalyani Publishers, Ludhiana, 716p.

Turabian, K.L. 2007. A Manual for Writers of Term Papers, Theses, and Dissertations (9 th Ed.). The University of Chicago Press, 461p.

UChicago [University of Chicago] 2017. Chicago Manual of Style [CMOS].

University of Chicago Press, 1144p.

AMA style- http://www.amamanualofstyle.com/oso/public/index.html .

APA style- https://www.apastyle.org/manual

BibSonomy- https://www.bibsonomy.org/

Bibus - http://www.sourceforage.net/projects/bibus-biblio

Bookends-www. sonnysoftware.com/

CAS Source index search tool- http://cassi.cas.org/search.jsp

Citavi- https://citavi.com/

CiteULike- http://www.citeulike.org

EndNote- https://www.endnote.com/

FAO [Food and Agriculture Organization of the United Nations]. 2018. The State of Food Security and Nutrition in the World 2018. Building Climate Resilience for Food Security and Nutrition. FAO, Rome, 181p. Available: http://www.fao.org/3/I9553EN/i9553en.pdf [Accessed 10 Oct. 2018].

Genamics JournalSeek- http://journalseek.net/

JabRef - http://jabref.sourceforge.net/

Journal database, NCBI- http://www.ncbi.nlm.nih.gov/nlmcatalog/journals ISSN, LTWA- http://www.issn.org/2-22661-LTWA-online.php .

Mendeley- https://www.mendeley.com

Papers- http://www.readcube.com/papers/

Qiqqa - http://www.qiqqa.com/

Refbase- http://www.refbase.net

RefWorks- https://www.proquest.com/

Scientific Style and Format (CSE) - https://www.scientificstyleandformat.org/Home.html

SciRef- http://sci-progs.com/

Web of Science journal abbreviations- https://images.webofknowledge.com/ images/help/WOS/A_abrvjt.html

Wikindx - https://sourceforge.net/ projects/wikindx/

Zotero- https://www.zotero.org/

Download references

Author information

Authors and affiliations.

Kerala Agricultural University, Thrissur, Kerala, India

C. George Thomas

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to C. George Thomas .

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s)

About this chapter

Cite this chapter.

Thomas, C.G. (2021). References: How to Cite and List Correctly. In: Research Methodology and Scientific Writing . Springer, Cham. https://doi.org/10.1007/978-3-030-64865-7_15

Download citation

DOI : https://doi.org/10.1007/978-3-030-64865-7_15

Published : 25 February 2021

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-64864-0

Online ISBN : 978-3-030-64865-7

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publish with us

Policies and ethics

  • Find a journal
  • Track your research
  • PRO Courses Guides New Tech Help Pro Expert Videos About wikiHow Pro Upgrade Sign In
  • EDIT Edit this Article
  • EXPLORE Tech Help Pro About Us Random Article Quizzes Request a New Article Community Dashboard This Or That Game Popular Categories Arts and Entertainment Artwork Books Movies Computers and Electronics Computers Phone Skills Technology Hacks Health Men's Health Mental Health Women's Health Relationships Dating Love Relationship Issues Hobbies and Crafts Crafts Drawing Games Education & Communication Communication Skills Personal Development Studying Personal Care and Style Fashion Hair Care Personal Hygiene Youth Personal Care School Stuff Dating All Categories Arts and Entertainment Finance and Business Home and Garden Relationship Quizzes Cars & Other Vehicles Food and Entertaining Personal Care and Style Sports and Fitness Computers and Electronics Health Pets and Animals Travel Education & Communication Hobbies and Crafts Philosophy and Religion Work World Family Life Holidays and Traditions Relationships Youth
  • Browse Articles
  • Learn Something New
  • Quizzes Hot
  • This Or That Game New
  • Train Your Brain
  • Explore More
  • Support wikiHow
  • About wikiHow
  • Log in / Sign up
  • Education and Communications
  • College University and Postgraduate
  • Academic Writing
  • Research Papers

How to Cite a Research Paper

Last Updated: February 19, 2024 Fact Checked

This article was reviewed by Gerald Posner and by wikiHow staff writer, Jennifer Mueller, JD . Gerald Posner is an Author & Journalist based in Miami, Florida. With over 35 years of experience, he specializes in investigative journalism, nonfiction books, and editorials. He holds a law degree from UC College of the Law, San Francisco, and a BA in Political Science from the University of California-Berkeley. He’s the author of thirteen books, including several New York Times bestsellers, the winner of the Florida Book Award for General Nonfiction, and has been a finalist for the Pulitzer Prize in History. He was also shortlisted for the Best Business Book of 2020 by the Society for Advancing Business Editing and Writing. There are 9 references cited in this article, which can be found at the bottom of the page. This article has been fact-checked, ensuring the accuracy of any cited facts and confirming the authority of its sources. This article has been viewed 407,765 times.

When writing a paper for a research project, you may need to cite a research paper you used as a reference. The basic information included in your citation will be the same across all styles. However, the format in which that information is presented is somewhat different depending on whether you're using American Psychological Association (APA), Modern Language Association (MLA), Chicago, or American Medical Association (AMA) style.

Referencing a Research Paper

  • In APA style, cite the paper: Last Name, First Initial. (Year). Title. Publisher.
  • In Chicago style, cite the paper: Last Name, First Name. “Title.” Publisher, Year.
  • In MLA style, cite the paper: Last Name, First Name. “Title.” Publisher. Year.

Citation Help

what are references on a research paper

  • For example: "Kringle, K., & Frost, J."

Step 2 Provide the year the paper was published.

  • For example: "Kringle, K., & Frost, J. (2012)."
  • If the date, or any other information, are not available, use the guide at https://blog.apastyle.org/apastyle/2012/05/missing-pieces.html .

Step 3 List the title of the research paper.

  • For example: "Kringle, K., & Frost, J. (2012). Red noses, warm hearts: The glowing phenomenon among North Pole reindeer."
  • If you found the research paper in a database maintained by a university, corporation, or other organization, include any index number assigned to the paper in parentheses after the title. For example: "Kringle, K., & Frost, J. (2012). Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. (Report No. 1234)."

Step 4 Include information on where you found the paper.

  • For example: "Kringle, K., & Frost, J. (2012). Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. (Report No. 1234). Retrieved from Alaska University Library Archives, December 24, 2017."

Step 5 Use a parenthetical citation in the body of your paper.

  • For example: "(Kringle & Frost, 2012)."
  • If there was no date on the research paper, use the abbreviation n.d. : "(Kringle & Frost, n.d.)."

Step 1 Start with the authors' names.

  • For example: "Kringle, Kris, and Jack Frost."

Step 2 List the title of the research paper.

  • For example: "Kringle, Kris, and Jack Frost. "Red Noses, Warm Hearts: The Glowing Phenomenon among North Pole Reindeer." Master's thesis."

Step 3 Provide the place and year of publication.

  • For example: "Kringle, Kris, and Jack Frost. "Red Noses, Warm Hearts: The Glowing Phenomenon among North Pole Reindeer." Master's thesis, Alaska University, 2012."

Step 4 Include any additional information necessary to locate the paper.

  • For example: "Kringle, Kris, and Jack Frost. "Red Noses, Warm Hearts: The Glowing Phenomenon among North Pole Reindeer." Master's thesis, Alaska University, 2012. Accessed at https://www.northpolemedical.com/raising_rudolf."

Step 5 Follow your instructor's guidance regarding in-text citations.

  • Footnotes are essentially the same as the full citation, although the first and last names of the authors aren't inverted.
  • For parenthetical citations, Chicago uses the Author-Date format. For example: "(Kringle and Frost 2012)."

Step 1 Start with the authors of the paper.

  • For example: "Kringle, Kris, and Frost, Jack."

Step 2 Provide the title of the research paper.

  • For example: "Kringle, Kris, and Frost, Jack. "Red Noses, Warm Hearts: The Glowing Phenomenon Among North Pole Reindeer.""

Step 3 Identify the paper's location.

  • For example, suppose you found the paper in a collection of paper housed in university archives. Your citation might be: "Kringle, Kris, and Frost, Jack. "Red Noses, Warm Hearts: The Glowing Phenomenon Among North Pole Reindeer." Master's Theses 2000-2010. University of Alaska Library Archives. Accessed December 24, 2017."

Step 4 Use parenthetical references in the body of your work.

  • For example: "(Kringle & Frost, p. 33)."

Step 1 Start with the author's last name and first initial.

  • For example: "Kringle K, Frost J."

Step 2 Provide the title in sentence case.

  • For example: "Kringle K, Frost J. Red noses, warm hearts: The glowing phenomenon among North Pole reindeer."

Step 3 Include journal information if the paper was published.

  • For example: "Kringle K, Frost J. Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. Nat Med. 2012; 18(9): 1429-1433."

Step 4 Provide location information if the paper hasn't been published.

  • For example, if you're citing a paper presented at a conference, you'd write: "Kringle K, Frost J. Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. Oral presentation at Arctic Health Association Annual Summit; December, 2017; Nome, Alaska."
  • To cite a paper you read online, you'd write: "Kringle K, Frost J. Red noses, warm hearts: The glowing phenomenon among North Pole reindeer. https://www.northpolemedical.com/raising_rudolf"

Step 5 Use superscript numbers in the body of your paper.

  • For example: "According to Kringle and Frost, these red noses indicate a subspecies of reindeer native to Alaska and Canada that have migrated to the North Pole and mingled with North Pole reindeer. 1 "

Community Q&A

SnowyDay

  • If you used a manual as a source in your research paper, you'll need to learn how to cite the manual also. Thanks Helpful 0 Not Helpful 0
  • If you use any figures in your research paper, you'll also need to know the proper way to cite them in MLA, APA, AMA, or Chicago. Thanks Helpful 0 Not Helpful 0

what are references on a research paper

You Might Also Like

Cite the WHO in APA

  • ↑ https://askus.library.wwu.edu/faq/116659
  • ↑ https://guides.libraries.psu.edu/apaquickguide/intext
  • ↑ https://owl.purdue.edu/owl/research_and_citation/chicago_manual_17th_edition/cmos_formatting_and_style_guide/general_format.html
  • ↑ https://libanswers.snhu.edu/faq/48009
  • ↑ https://www.chicagomanualofstyle.org/tools_citationguide/citation-guide-2.html
  • ↑ https://owl.purdue.edu/owl/research_and_citation/mla_style/mla_formatting_and_style_guide/mla_in_text_citations_the_basics.html
  • ↑ https://morningside.libguides.com/MLA8/location
  • ↑ https://owl.purdue.edu/owl/research_and_citation/ama_style/index.html
  • ↑ https://research.library.oakland.edu/sp/subjects/tutorial.php?faq_id=187

About This Article

Gerald Posner

To cite a paper APA style, start with the author's last name and first initial, and the year of publication. Then, list the title of the paper, where you found it, and the date that you accessed it. In a paper, use a parenthetical reference with the last name of the author and the publication year. For an MLA citation, list the author's last name and then first name and the title of the paper in quotations. Include where you accessed the paper and the date you retrieved it. In your paper, use a parenthetical reference with the author's last name and the page number. Keep reading for tips on Chicago and AMA citations and exceptions to the citation rules! Did this summary help you? Yes No

  • Send fan mail to authors

Did this article help you?

what are references on a research paper

Featured Articles

Make Blueberry Pancakes

Trending Articles

How to Take the Perfect Thirst Trap

Watch Articles

Wrap a Round Gift

  • Terms of Use
  • Privacy Policy
  • Do Not Sell or Share My Info
  • Not Selling Info

wikiHow Tech Help Pro:

Level up your tech skills and stay ahead of the curve

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Indian J Orthop
  • v.53(3); May-Jun 2019

Logo of ijortho

Formatting References for Scientific Manuscripts

Srinivas b s kambhampati.

Sri Dhaatri Orthopaedic, Maternity and Gynaecology Center, Vijayawada, Andhra Pradesh, India

Lalit Maini

1 Department of Orthopaedics, Maulana Azad Medical College, New Delhi, India

While references are an essential and integral part of a scientific manuscript, format and style of references are as varied as the number of journals currently present. International Committee of Medical Journal Editors in their latest recommendations for publication, 1 advice authors to quote original references whenever possible. We would recommend the readers to go through these guidelines as they are given in sufficient detail to submit a good set of references including styling. Other resources for citing references include the PubMed section 2 which gives samples of formatting of different reference types and the eBook: Citing Medicine, 3 published by the U. S. National Library of Medicine, which gives assistance and rules to authors, editors, publishers and librarians for formatting of references for different reference types.

References are formatted in two basic styles – the Vancouver style which is numeric (more commonly used in medical journals) and Harvard which uses author-date style (more commonly used in natural and Social sciences journals). 4 Parts and order of the parts cited differ on what the author is citing (reference type) and the journal that is being submitted to. The most common types of references include journal article, book, book section or chapter, dissertation, monograph, and webpage. As an example, for a journal article, the parts of a reference in the sequence include authors, article title, journal title, date of publication, volume, issue, and location/pagination. Each journal has its own modification of the format for each part and the punctuation marks, or their lack of, between the parts. Formatting style in each part of a reference could involve placement of selected punctuation marks, bold and italics enhancements, alphabetical order or sequential ordering of references and style of citing in the text, making the combination of variations that create a unique reference style as large in number as the number of journals currently published. It is not clear why such a system has evolved, but it requires considerable attention to detail to get the formatting correct and is time-consuming for the author. The tradition of the journal has been thought as one of the reasons. 4 In manuscripts submitted for the Indian Journal of Orthopaedics (IJO), the reference section carries the most formatting errors committed by authors.

The advantage of having a constant style within a journal is two-fold, apart from an esthetic appearance of references across all articles published by the journal. Ease of reading the references at the end of each article and ease of finding reference part by the reader if he/she is used to the format and plans to look up the reference. As a student/professional in the medical field, one would require attention to finer details of his/her research work as well as in their clinical practice and hence exercising attention to the references would help improve those skills. Such a wide variation in the styles of references has also benefitted some software companies who deal with reference managers (RMs). Some RMs are free for use, and the authors are advised to use different RMs to see which one suits their needs best. While some RMs are cloud based, others are computer based and do not require an internet connection while some others are cloud and computer based. The variation in the style of references across journals appears unlikely to be standardized to a single universal format in the near future.

The Citation Style Language (CSL) is an XML-based computer language developed to standardize formatting of citations and references in manuscripts submitting to journals. They are text application editable files which are imported into RMs. An increasing number of RMs use CSL to help users format their list of references according to individual journal guidelines. However, not all journals are supported by CSL files.

There are two main repositories for access to CSL files – One by GitHub 5 and the other by Zotero 6 developed by Corporation for Digital Scholarship and Roy Rosenzweig Center for History and New Media. These contain more than 8500 styles of references. Authors using Zotero, Mendeley, RefWorks, Papers, CrossRef, Bibliography, and some 42 other RMs can use these CSL files to manage references within them. IJO did not have a CSL file until now in both the repositories. In this scenario, there are a few options for the authors preparing references for a manuscript. First, to type and style references manually which would take a long time and prone to human errors. Second, a CSL file similar to IJO may be selected from the repository and used and later manually edited, if any needed. However, this involves searching for an exact match of style for a journal registered in the repository of CSL files. Third, there are RMs with inbuilt options to format references while citing in the text. This option is independent of CSL. The disadvantage here is that the author is bound by the list of reference styles already loaded within the software. They may not be able to add new formats. Fourth, some RMs allow authors to prepare a style, but this would take some time to prepare if the style is not already inbuilt. As a fifth option, a CSL file that is close to IJO may be chosen, and the code of this file tweaked with minor editing to convert it for use with IJO. To do that, the author must be familiar with programming or editing of HTML/XML files since HTML is a language that is closer to XML.

Zotero's repository 7 website has a user-friendly interface in which such searches are easier to perform. It has 9357 styles stored in the repository at the time of writing this article on March 17, 2019. There are 1924 unique styles through which one can search if their required journal is listed. Zotero draws CSL files from GitHub into their repository. Hence, if a file is created in GitHub, it is drawn into Zotero by default. CSL Project 8 is a website sponsored by four well known RMs. These are Zotero, Mendeley, 9 Papers, 10 and RefWorks. 11 This website gives detailed specifications and documentation of CSL language if one is interested in coding these files. If one is proficient with XML, they can create a style and submit it to the GitHub website for others to benefit. Editing is easier if one uses the Zotero RM as it has an inbuilt option to edit style. It can be done even in other managers or with the use of a standard text editing application in Windows or Mac operating systems. Once a new CSL file is developed, in order to publish it, it has to be validated by CSL validator website 12 and submitted at the GitHub site for accepting into the repository. Even finer details like number of author names before et al. while formatting reference, punctuation marks and their placement, style of each part of the reference and each style of the reference, etc., can be edited accurately.

Once developed, the output of references and citations is remarkably consistent, and too much time need not be directed to editing the punctuation marks and styling of the references and citations while preparing the manuscript. The only hurdle after this would be to get full details of the references reliably and accurately into the RM database while importing the references. The author needs to check that the references were properly imported into the database. If verified, they may be used any number of times with precision. With appropriate selection, the citing as well as the list of references can be formatted according to the journal that is being considered, for submission. Those who are already using RMs may be well aware of the advantages and the time such CSL files can save while preparing a manuscript.

We are happy to inform that a CSL file for IJO has now been created in the GitHub repository 1 and Zotero Styles repository 3 and it can be used by authors using the RMs listed in the CSL website and benefit from its use. The direct link of the file in the repository is given 13 [ Figure 1 ]. Basic users of RMs may download it through their RMs by selecting Indian Journal of Orthopaedics option. Advanced users who know where to place this file may access using the weblink given. Examples of reference style and citation for IJO are given in Figure 2 .

An external file that holds a picture, illustration, etc.
Object name is IJOrtho-53-381-g001.jpg

Screenshot of browser shows the web address and search words used to retrieve Citation Style Language file for Indian Journal of Orthopaedics

An external file that holds a picture, illustration, etc.
Object name is IJOrtho-53-381-g002.jpg

Examples of format of references and their citation in text for the Indian Journal of Orthopaedics

  • Privacy Policy
  • SignUp/Login

Research Method

Home » How to Cite Research Paper – All Formats and Examples

How to Cite Research Paper – All Formats and Examples

Table of Contents

Research Paper Citation

Research Paper Citation

Research paper citation refers to the act of acknowledging and referencing a previously published work in a scholarly or academic paper . When citing sources, researchers provide information that allows readers to locate the original source, validate the claims or arguments made in the paper, and give credit to the original author(s) for their work.

The citation may include the author’s name, title of the publication, year of publication, publisher, and other relevant details that allow readers to trace the source of the information. Proper citation is a crucial component of academic writing, as it helps to ensure accuracy, credibility, and transparency in research.

How to Cite Research Paper

There are several formats that are used to cite a research paper. Follow the guide for the Citation of a Research Paper:

Last Name, First Name. Title of Book. Publisher, Year of Publication.

Example : Smith, John. The History of the World. Penguin Press, 2010.

Journal Article

Last Name, First Name. “Title of Article.” Title of Journal, vol. Volume Number, no. Issue Number, Year of Publication, pp. Page Numbers.

Example : Johnson, Emma. “The Effects of Climate Change on Agriculture.” Environmental Science Journal, vol. 10, no. 2, 2019, pp. 45-59.

Research Paper

Last Name, First Name. “Title of Paper.” Conference Name, Location, Date of Conference.

Example : Garcia, Maria. “The Importance of Early Childhood Education.” International Conference on Education, Paris, 5-7 June 2018.

Author’s Last Name, First Name. “Title of Webpage.” Website Title, Publisher, Date of Publication, URL.

Example : Smith, John. “The Benefits of Exercise.” Healthline, Healthline Media, 1 March 2022, https://www.healthline.com/health/benefits-of-exercise.

News Article

Last Name, First Name. “Title of Article.” Name of Newspaper, Date of Publication, URL.

Example : Robinson, Sarah. “Biden Announces New Climate Change Policies.” The New York Times, 22 Jan. 2021, https://www.nytimes.com/2021/01/22/climate/biden-climate-change-policies.html.

Author, A. A. (Year of publication). Title of book. Publisher.

Example: Smith, J. (2010). The History of the World. Penguin Press.

Author, A. A., Author, B. B., & Author, C. C. (Year of publication). Title of article. Title of Journal, volume number(issue number), page range.

Example: Johnson, E., Smith, K., & Lee, M. (2019). The Effects of Climate Change on Agriculture. Environmental Science Journal, 10(2), 45-59.

Author, A. A. (Year of publication). Title of paper. In Editor First Initial. Last Name (Ed.), Title of Conference Proceedings (page numbers). Publisher.

Example: Garcia, M. (2018). The Importance of Early Childhood Education. In J. Smith (Ed.), Proceedings from the International Conference on Education (pp. 60-75). Springer.

Author, A. A. (Year, Month Day of publication). Title of webpage. Website name. URL

Example: Smith, J. (2022, March 1). The Benefits of Exercise. Healthline. https://www.healthline.com/health/benefits-of-exercise

Author, A. A. (Year, Month Day of publication). Title of article. Newspaper name. URL.

Example: Robinson, S. (2021, January 22). Biden Announces New Climate Change Policies. The New York Times. https://www.nytimes.com/2021/01/22/climate/biden-climate-change-policies.html

Chicago/Turabian style

Please note that there are two main variations of the Chicago style: the author-date system and the notes and bibliography system. I will provide examples for both systems below.

Author-Date system:

  • In-text citation: (Author Last Name Year, Page Number)
  • Reference list: Author Last Name, First Name. Year. Title of Book. Place of publication: Publisher.
  • In-text citation: (Smith 2005, 28)
  • Reference list: Smith, John. 2005. The History of America. New York: Penguin Press.

Notes and Bibliography system:

  • Footnote/Endnote citation: Author First Name Last Name, Title of Book (Place of publication: Publisher, Year), Page Number.
  • Bibliography citation: Author Last Name, First Name. Title of Book. Place of publication: Publisher, Year.
  • Footnote/Endnote citation: John Smith, The History of America (New York: Penguin Press, 2005), 28.
  • Bibliography citation: Smith, John. The History of America. New York: Penguin Press, 2005.

JOURNAL ARTICLES:

  • Reference list: Author Last Name, First Name. Year. “Article Title.” Journal Title Volume Number (Issue Number): Page Range.
  • In-text citation: (Johnson 2010, 45)
  • Reference list: Johnson, Mary. 2010. “The Impact of Social Media on Society.” Journal of Communication 60(2): 39-56.
  • Footnote/Endnote citation: Author First Name Last Name, “Article Title,” Journal Title Volume Number, Issue Number (Year): Page Range.
  • Bibliography citation: Author Last Name, First Name. “Article Title.” Journal Title Volume Number, Issue Number (Year): Page Range.
  • Footnote/Endnote citation: Mary Johnson, “The Impact of Social Media on Society,” Journal of Communication 60, no. 2 (2010): 39-56.
  • Bibliography citation: Johnson, Mary. “The Impact of Social Media on Society.” Journal of Communication 60, no. 2 (2010): 39-56.

RESEARCH PAPERS:

  • Reference list: Author Last Name, First Name. Year. “Title of Paper.” Conference Proceedings Title, Location, Date. Publisher, Page Range.
  • In-text citation: (Jones 2015, 12)
  • Reference list: Jones, David. 2015. “The Effects of Climate Change on Agriculture.” Proceedings of the International Conference on Climate Change, Paris, France, June 1-3, 2015. Springer, 10-20.
  • Footnote/Endnote citation: Author First Name Last Name, “Title of Paper,” Conference Proceedings Title, Location, Date (Place of publication: Publisher, Year), Page Range.
  • Bibliography citation: Author Last Name, First Name. “Title of Paper.” Conference Proceedings Title, Location, Date. Place of publication: Publisher, Year.
  • Footnote/Endnote citation: David Jones, “The Effects of Climate Change on Agriculture,” Proceedings of the International Conference on Climate Change, Paris, France, June 1-3, 2015 (New York: Springer, 10-20).
  • Bibliography citation: Jones, David. “The Effects of Climate Change on Agriculture.” Proceedings of the International Conference on Climate Change, Paris, France, June 1-3, 2015. New York: Springer, 10-20.
  • In-text citation: (Author Last Name Year)
  • Reference list: Author Last Name, First Name. Year. “Title of Webpage.” Website Name. URL.
  • In-text citation: (Smith 2018)
  • Reference list: Smith, John. 2018. “The Importance of Recycling.” Environmental News Network. https://www.enn.com/articles/54374-the-importance-of-recycling.
  • Footnote/Endnote citation: Author First Name Last Name, “Title of Webpage,” Website Name, URL (accessed Date).
  • Bibliography citation: Author Last Name, First Name. “Title of Webpage.” Website Name. URL (accessed Date).
  • Footnote/Endnote citation: John Smith, “The Importance of Recycling,” Environmental News Network, https://www.enn.com/articles/54374-the-importance-of-recycling (accessed April 8, 2023).
  • Bibliography citation: Smith, John. “The Importance of Recycling.” Environmental News Network. https://www.enn.com/articles/54374-the-importance-of-recycling (accessed April 8, 2023).

NEWS ARTICLES:

  • Reference list: Author Last Name, First Name. Year. “Title of Article.” Name of Newspaper, Month Day.
  • In-text citation: (Johnson 2022)
  • Reference list: Johnson, Mary. 2022. “New Study Finds Link Between Coffee and Longevity.” The New York Times, January 15.
  • Footnote/Endnote citation: Author First Name Last Name, “Title of Article,” Name of Newspaper (City), Month Day, Year.
  • Bibliography citation: Author Last Name, First Name. “Title of Article.” Name of Newspaper (City), Month Day, Year.
  • Footnote/Endnote citation: Mary Johnson, “New Study Finds Link Between Coffee and Longevity,” The New York Times (New York), January 15, 2022.
  • Bibliography citation: Johnson, Mary. “New Study Finds Link Between Coffee and Longevity.” The New York Times (New York), January 15, 2022.

Harvard referencing style

Format: Author’s Last name, First initial. (Year of publication). Title of book. Publisher.

Example: Smith, J. (2008). The Art of War. Random House.

Journal article:

Format: Author’s Last name, First initial. (Year of publication). Title of article. Title of journal, volume number(issue number), page range.

Example: Brown, M. (2012). The impact of social media on business communication. Harvard Business Review, 90(12), 85-92.

Research paper:

Format: Author’s Last name, First initial. (Year of publication). Title of paper. In Editor’s First initial. Last name (Ed.), Title of book (page range). Publisher.

Example: Johnson, R. (2015). The effects of climate change on agriculture. In S. Lee (Ed.), Climate Change and Sustainable Development (pp. 45-62). Springer.

Format: Author’s Last name, First initial. (Year, Month Day of publication). Title of page. Website name. URL.

Example: Smith, J. (2017, May 23). The history of the internet. Encyclopedia Britannica. https://www.britannica.com/topic/history-of-the-internet

News article:

Format: Author’s Last name, First initial. (Year, Month Day of publication). Title of article. Title of newspaper, page number (if applicable).

Example: Thompson, E. (2022, January 5). New study finds coffee may lower risk of dementia. The New York Times, A1.

IEEE Format

Author(s). (Year of Publication). Title of Book. Publisher.

Smith, J. K. (2015). The Power of Habit: Why We Do What We Do in Life and Business. Random House.

Journal Article:

Author(s). (Year of Publication). Title of Article. Title of Journal, Volume Number (Issue Number), page numbers.

Johnson, T. J., & Kaye, B. K. (2016). Interactivity and the Future of Journalism. Journalism Studies, 17(2), 228-246.

Author(s). (Year of Publication). Title of Paper. Paper presented at Conference Name, Location.

Jones, L. K., & Brown, M. A. (2018). The Role of Social Media in Political Campaigns. Paper presented at the 2018 International Conference on Social Media and Society, Copenhagen, Denmark.

  • Website: Author(s) or Organization Name. (Year of Publication or Last Update). Title of Webpage. Website Name. URL.

Example: National Aeronautics and Space Administration. (2019, August 29). NASA’s Mission to Mars. NASA. https://www.nasa.gov/topics/journeytomars/index.html

  • News Article: Author(s). (Year of Publication). Title of Article. Name of News Source. URL.

Example: Johnson, M. (2022, February 16). Climate Change: Is it Too Late to Save the Planet? CNN. https://www.cnn.com/2022/02/16/world/climate-change-planet-scn/index.html

Vancouver Style

In-text citation: Use superscript numbers to cite sources in the text, e.g., “The study conducted by Smith and Johnson^1 found that…”.

Reference list citation: Format: Author(s). Title of book. Edition if any. Place of publication: Publisher; Year of publication.

Example: Smith J, Johnson L. Introduction to Molecular Biology. 2nd ed. New York: Wiley-Blackwell; 2015.

In-text citation: Use superscript numbers to cite sources in the text, e.g., “Several studies have reported that^1,2,3…”.

Reference list citation: Format: Author(s). Title of article. Abbreviated name of journal. Year of publication; Volume number (Issue number): Page range.

Example: Jones S, Patel K, Smith J. The effects of exercise on cardiovascular health. J Cardiol. 2018; 25(2): 78-84.

In-text citation: Use superscript numbers to cite sources in the text, e.g., “Previous research has shown that^1,2,3…”.

Reference list citation: Format: Author(s). Title of paper. In: Editor(s). Title of the conference proceedings. Place of publication: Publisher; Year of publication. Page range.

Example: Johnson L, Smith J. The role of stem cells in tissue regeneration. In: Patel S, ed. Proceedings of the 5th International Conference on Regenerative Medicine. London: Academic Press; 2016. p. 68-73.

In-text citation: Use superscript numbers to cite sources in the text, e.g., “According to the World Health Organization^1…”.

Reference list citation: Format: Author(s). Title of webpage. Name of website. URL [Accessed Date].

Example: World Health Organization. Coronavirus disease (COVID-19) advice for the public. World Health Organization. https://www.who.int/emergencies/disease/novel-coronavirus-2019/advice-for-public [Accessed 3 March 2023].

In-text citation: Use superscript numbers to cite sources in the text, e.g., “According to the New York Times^1…”.

Reference list citation: Format: Author(s). Title of article. Name of newspaper. Year Month Day; Section (if any): Page number.

Example: Jones S. Study shows that sleep is essential for good health. The New York Times. 2022 Jan 12; Health: A8.

Author(s). Title of Book. Edition Number (if it is not the first edition). Publisher: Place of publication, Year of publication.

Example: Smith, J. Chemistry of Natural Products. 3rd ed.; CRC Press: Boca Raton, FL, 2015.

Journal articles:

Author(s). Article Title. Journal Name Year, Volume, Inclusive Pagination.

Example: Garcia, A. M.; Jones, B. A.; Smith, J. R. Selective Synthesis of Alkenes from Alkynes via Catalytic Hydrogenation. J. Am. Chem. Soc. 2019, 141, 10754-10759.

Research papers:

Author(s). Title of Paper. Journal Name Year, Volume, Inclusive Pagination.

Example: Brown, H. D.; Jackson, C. D.; Patel, S. D. A New Approach to Photovoltaic Solar Cells. J. Mater. Chem. 2018, 26, 134-142.

Author(s) (if available). Title of Webpage. Name of Website. URL (accessed Month Day, Year).

Example: National Institutes of Health. Heart Disease and Stroke. National Heart, Lung, and Blood Institute. https://www.nhlbi.nih.gov/health-topics/heart-disease-and-stroke (accessed April 7, 2023).

News articles:

Author(s). Title of Article. Name of News Publication. Date of Publication. URL (accessed Month Day, Year).

Example: Friedman, T. L. The World is Flat. New York Times. April 7, 2023. https://www.nytimes.com/2023/04/07/opinion/world-flat-globalization.html (accessed April 7, 2023).

In AMA Style Format, the citation for a book should include the following information, in this order:

  • Title of book (in italics)
  • Edition (if applicable)
  • Place of publication
  • Year of publication

Lodish H, Berk A, Zipursky SL, et al. Molecular Cell Biology. 4th ed. New York, NY: W. H. Freeman; 2000.

In AMA Style Format, the citation for a journal article should include the following information, in this order:

  • Title of article
  • Abbreviated title of journal (in italics)
  • Year of publication; volume number(issue number):page numbers.

Chen H, Huang Y, Li Y, et al. Effects of mindfulness-based stress reduction on depression in adolescents and young adults: a systematic review and meta-analysis. JAMA Netw Open. 2020;3(6):e207081. doi:10.1001/jamanetworkopen.2020.7081

In AMA Style Format, the citation for a research paper should include the following information, in this order:

  • Title of paper
  • Name of journal or conference proceeding (in italics)
  • Volume number(issue number):page numbers.

Bredenoord AL, Kroes HY, Cuppen E, Parker M, van Delden JJ. Disclosure of individual genetic data to research participants: the debate reconsidered. Trends Genet. 2011;27(2):41-47. doi:10.1016/j.tig.2010.11.004

In AMA Style Format, the citation for a website should include the following information, in this order:

  • Title of web page or article
  • Name of website (in italics)
  • Date of publication or last update (if available)
  • URL (website address)
  • Date of access (month day, year)

Centers for Disease Control and Prevention. How to protect yourself and others. CDC. Published February 11, 2022. Accessed February 14, 2022. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html

In AMA Style Format, the citation for a news article should include the following information, in this order:

  • Name of newspaper or news website (in italics)
  • Date of publication

Gorman J. Scientists use stem cells from frogs to build first living robots. The New York Times. January 13, 2020. Accessed January 14, 2020. https://www.nytimes.com/2020/01/13/science/living-robots-xenobots.html

Bluebook Format

One author: Daniel J. Solove, The Future of Reputation: Gossip, Rumor, and Privacy on the Internet (Yale University Press 2007).

Two or more authors: Martha Nussbaum and Saul Levmore, eds., The Offensive Internet: Speech, Privacy, and Reputation (Harvard University Press 2010).

Journal article

One author: Daniel J. Solove, “A Taxonomy of Privacy,” University of Pennsylvania Law Review 154, no. 3 (January 2006): 477-560.

Two or more authors: Ethan Katsh and Andrea Schneider, “The Emergence of Online Dispute Resolution,” Journal of Dispute Resolution 2003, no. 1 (2003): 7-19.

One author: Daniel J. Solove, “A Taxonomy of Privacy,” GWU Law School Public Law Research Paper No. 113, 2005.

Two or more authors: Ethan Katsh and Andrea Schneider, “The Emergence of Online Dispute Resolution,” Cyberlaw Research Paper Series Paper No. 00-5, 2000.

WebsiteElectronic Frontier Foundation, “Surveillance Self-Defense,” accessed April 8, 2023, https://ssd.eff.org/.

News article

One author: Mark Sherman, “Court Deals Major Blow to Net Neutrality Rules,” ABC News, January 14, 2014, https://abcnews.go.com/Politics/wireStory/court-deals-major-blow-net-neutrality-rules-21586820.

Two or more authors: Siobhan Hughes and Brent Kendall, “AT&T Wins Approval to Buy Time Warner,” Wall Street Journal, June 12, 2018, https://www.wsj.com/articles/at-t-wins-approval-to-buy-time-warner-1528847249.

In-Text Citation: (Author’s last name Year of Publication: Page Number)

Example: (Smith 2010: 35)

Reference List Citation: Author’s last name First Initial. Title of Book. Edition. Place of publication: Publisher; Year of publication.

Example: Smith J. Biology: A Textbook. 2nd ed. New York: Oxford University Press; 2010.

Example: (Johnson 2014: 27)

Reference List Citation: Author’s last name First Initial. Title of Article. Abbreviated Title of Journal. Year of publication;Volume(Issue):Page Numbers.

Example: Johnson S. The role of dopamine in addiction. J Neurosci. 2014;34(8): 2262-2272.

Example: (Brown 2018: 10)

Reference List Citation: Author’s last name First Initial. Title of Paper. Paper presented at: Name of Conference; Date of Conference; Place of Conference.

Example: Brown R. The impact of social media on mental health. Paper presented at: Annual Meeting of the American Psychological Association; August 2018; San Francisco, CA.

Example: (World Health Organization 2020: para. 2)

Reference List Citation: Author’s last name First Initial. Title of Webpage. Name of Website. URL. Published date. Accessed date.

Example: World Health Organization. Coronavirus disease (COVID-19) pandemic. WHO website. https://www.who.int/emergencies/disease-coronavirus-2019. Updated August 17, 2020. Accessed September 5, 2021.

Example: (Smith 2019: para. 5)

Reference List Citation: Author’s last name First Initial. Title of Article. Title of Newspaper or Magazine. Year of publication; Month Day:Page Numbers.

Example: Smith K. New study finds link between exercise and mental health. The New York Times. 2019;May 20: A6.

Purpose of Research Paper Citation

The purpose of citing sources in a research paper is to give credit to the original authors and acknowledge their contribution to your work. By citing sources, you are also demonstrating the validity and reliability of your research by showing that you have consulted credible and authoritative sources. Citations help readers to locate the original sources that you have referenced and to verify the accuracy and credibility of your research. Additionally, citing sources is important for avoiding plagiarism, which is the act of presenting someone else’s work as your own. Proper citation also shows that you have conducted a thorough literature review and have used the existing research to inform your own work. Overall, citing sources is an essential aspect of academic writing and is necessary for building credibility, demonstrating research skills, and avoiding plagiarism.

Advantages of Research Paper Citation

There are several advantages of research paper citation, including:

  • Giving credit: By citing the works of other researchers in your field, you are acknowledging their contribution and giving credit where it is due.
  • Strengthening your argument: Citing relevant and reliable sources in your research paper can strengthen your argument and increase its credibility. It shows that you have done your due diligence and considered various perspectives before drawing your conclusions.
  • Demonstrating familiarity with the literature : By citing various sources, you are demonstrating your familiarity with the existing literature in your field. This is important as it shows that you are well-informed about the topic and have done a thorough review of the available research.
  • Providing a roadmap for further research: By citing relevant sources, you are providing a roadmap for further research on the topic. This can be helpful for future researchers who are interested in exploring the same or related issues.
  • Building your own reputation: By citing the works of established researchers in your field, you can build your own reputation as a knowledgeable and informed scholar. This can be particularly helpful if you are early in your career and looking to establish yourself as an expert in your field.

About the author

' src=

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Research Paper Conclusion

Research Paper Conclusion – Writing Guide and...

Appendices

Appendices – Writing Guide, Types and Examples

Delimitations

Delimitations in Research – Types, Examples and...

Scope of the Research

Scope of the Research – Writing Guide and...

Research Contribution

Research Contribution – Thesis Guide

Research Paper Title

Research Paper Title – Writing Guide and Example

Roles of references in research papers – a broader assessment

role of references or citation in research paper

In writing research papers and their evaluation, references or citations play a central role. Knowing the role(s) in detail will help you to cite accurately and responsibly.

Know the definition of referencing here: What is referencing in academic writing? – a different perspective.

Commonly understood roles of references

According to several top search results, when “role of references in research” is searched in Google, referencing have the following important roles:

  • Referencing allows you to acknowledge or give credit to the writers and researchers from whom you have borrowed words and ideas, thereby avoiding plagiarism.
  • Referencing is a way to provide evidence to support the assertions and claims in your own assignments.
  • References allow readers to trace the sources of information you have used.

Subtle but critical roles of referencing

However, these commonly understood roles don’t clearly reflect the deeper impact (as listed below) references make on the research paper itself and science overall.

References (summarised from here ):

  • demonstrate the foundation of the study.
  • support the novelty and value of the study.
  • link one study to others creating a web of knowledge that carries meaning.
  • allows researchers to identify work as relevant in general and relevant to them.
  • create values that are internal to science (e.g., relevance, credit).
  • create values that are external to science (e.g., provide avenues to determine accountability and researchers or funding performance).

A wrong perception of referencing

According to many search results, the most common role of referencing is to acknowledge or give credit to other researchers. This creates a wrong perception among authors that they need to cite references neutrally without constructive scientific evaluation (praise or criticism). No wonder an editorial published in Nature genetics reported that neutral, flavourless or unexamined citations frequently occur in research articles and supporting or contradictory publications are rarely cited. This is an increasing problem for the integrity of scientific communication.

This situation can be improved by acknowledging that referencing is not a neutral act but a political act (without bad politics). In fact, acknowledging the political nature of referencing is a vital rule for responsible referencing. For more on responsible referencing, see here .

But why referencing is a political act?

By assessing many research articles, you select the articles you want to cite and discard others. To pose the research questions and support the claims and novelty of your research work, rather than neutral representations of the references, you accurately represent, inflate, or deflate the contributions of other researchers’ works. See the following examples.

Example 1: Using references to establish the research question.

Many of us have encountered instances where the support of an assertion by the cited reference proves to be ambiguous, non-existent, or even contradictory (often we only notice this when our own work has been mis-cited!). A related practice is the citing of “empty” references (Harzing 2002), also known as “lazy author syndrome” (Gavras 2002), where the citation actually attributes a finding or an opinion to a secondary source such as a review paper, editorial, etc. But how pervasive is citation malpractice and how can it be controlled? SOURCE: Tood, Yeo, Li, and Ladle, Oikos 116, 1599 (2007)

Example 2: Using references to support claim and novelty of the research paper.

We found that the original assertion was “clearly supported” by the citation in 76.1% of the cases; the support was “ambiguous” in 11.1% of the cases; and the citation did “not support” the original statement in 7.2% of the cases. The remaining 5.6% of the cases were classified as “empty”. How do these mis-citation rates compare with other disciplines? A number of biomedical studies have used an approach similar to our own, although they applied the analogous categories “major error” and “minor error” rather than “no support” and “ambiguous”. Combined error rates found by Fenton et al. 2000 (17%) and Lukic et al. 2004 (19%) are comparable to our result of 18.3% for “no support” plus “ambiguous”, though other results for medical journals range from 12.3% (Gosling et al. 2004) to 35.2% (Goldberg et al. 1993). To our knowledge, empty citation data are absent for all the sciences. SOURCE: Tood, Yeo, Li, and Ladle, Oikos 116, 1599 (2007)

Appreciation of the broader roles of referencing and its political nature will enable researchers and scientists to produce research works by making citations accurately.

Does your reference manager support you in demonstrating how accurately you represented other researchers’ contributions in your research work? For the first time, with nXr.iCite you can search quotes/images; make citations based on quotes/images and share all the cited materials with your mentors/reviewers for peer-review. This enables them to validate your research paper with the content of the references side-by-side without leaving the paper. https://nxref.com
  • Systematic review
  • Open access
  • Published: 19 February 2024

‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice

  • Annette Boaz   ORCID: orcid.org/0000-0003-0557-1294 1 ,
  • Juan Baeza 2 ,
  • Alec Fraser   ORCID: orcid.org/0000-0003-1121-1551 2 &
  • Erik Persson 3  

Implementation Science volume  19 , Article number:  15 ( 2024 ) Cite this article

1758 Accesses

68 Altmetric

Metrics details

The gap between research findings and clinical practice is well documented and a range of strategies have been developed to support the implementation of research into clinical practice. The objective of this study was to update and extend two previous reviews of systematic reviews of strategies designed to implement research evidence into clinical practice.

We developed a comprehensive systematic literature search strategy based on the terms used in the previous reviews to identify studies that looked explicitly at interventions designed to turn research evidence into practice. The search was performed in June 2022 in four electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched from January 2010 up to June 2022 and applied no language restrictions. Two independent reviewers appraised the quality of included studies using a quality assessment checklist. To reduce the risk of bias, papers were excluded following discussion between all members of the team. Data were synthesised using descriptive and narrative techniques to identify themes and patterns linked to intervention strategies, targeted behaviours, study settings and study outcomes.

We identified 32 reviews conducted between 2010 and 2022. The reviews are mainly of multi-faceted interventions ( n  = 20) although there are reviews focusing on single strategies (ICT, educational, reminders, local opinion leaders, audit and feedback, social media and toolkits). The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Furthermore, a lot of nuance lies behind these headline findings, and this is increasingly commented upon in the reviews themselves.

Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been identified. We need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of research perspectives (including social science) in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed.

Peer Review reports

Contribution to the literature

Considerable time and money is invested in implementing and evaluating strategies to increase the implementation of research into clinical practice.

The growing body of evidence is not providing the anticipated clear lessons to support improved implementation.

Instead what is needed is better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice.

This would involve a more central role in implementation science for a wider range of perspectives, especially from the social, economic, political and behavioural sciences and for greater use of different types of synthesis, such as realist synthesis.

Introduction

The gap between research findings and clinical practice is well documented and a range of interventions has been developed to increase the implementation of research into clinical practice [ 1 , 2 ]. In recent years researchers have worked to improve the consistency in the ways in which these interventions (often called strategies) are described to support their evaluation. One notable development has been the emergence of Implementation Science as a field focusing explicitly on “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice” ([ 3 ] p. 1). The work of implementation science focuses on closing, or at least narrowing, the gap between research and practice. One contribution has been to map existing interventions, identifying 73 discreet strategies to support research implementation [ 4 ] which have been grouped into 9 clusters [ 5 ]. The authors note that they have not considered the evidence of effectiveness of the individual strategies and that a next step is to understand better which strategies perform best in which combinations and for what purposes [ 4 ]. Other authors have noted that there is also scope to learn more from other related fields of study such as policy implementation [ 6 ] and to draw on methods designed to support the evaluation of complex interventions [ 7 ].

The increase in activity designed to support the implementation of research into practice and improvements in reporting provided the impetus for an update of a review of systematic reviews of the effectiveness of interventions designed to support the use of research in clinical practice [ 8 ] which was itself an update of the review conducted by Grimshaw and colleagues in 2001. The 2001 review [ 9 ] identified 41 reviews considering a range of strategies including educational interventions, audit and feedback, computerised decision support to financial incentives and combined interventions. The authors concluded that all the interventions had the potential to promote the uptake of evidence in practice, although no one intervention seemed to be more effective than the others in all settings. They concluded that combined interventions were more likely to be effective than single interventions. The 2011 review identified a further 13 systematic reviews containing 313 discrete primary studies. Consistent with the previous review, four main strategy types were identified: audit and feedback; computerised decision support; opinion leaders; and multi-faceted interventions (MFIs). Nine of the reviews reported on MFIs. The review highlighted the small effects of single interventions such as audit and feedback, computerised decision support and opinion leaders. MFIs claimed an improvement in effectiveness over single interventions, although effect sizes remained small to moderate and this improvement in effectiveness relating to MFIs has been questioned in a subsequent review [ 10 ]. In updating the review, we anticipated a larger pool of reviews and an opportunity to consolidate learning from more recent systematic reviews of interventions.

This review updates and extends our previous review of systematic reviews of interventions designed to implement research evidence into clinical practice. To identify potentially relevant peer-reviewed research papers, we developed a comprehensive systematic literature search strategy based on the terms used in the Grimshaw et al. [ 9 ] and Boaz, Baeza and Fraser [ 8 ] overview articles. To ensure optimal retrieval, our search strategy was refined with support from an expert university librarian, considering the ongoing improvements in the development of search filters for systematic reviews since our first review [ 11 ]. We also wanted to include technology-related terms (e.g. apps, algorithms, machine learning, artificial intelligence) to find studies that explored interventions based on the use of technological innovations as mechanistic tools for increasing the use of evidence into practice (see Additional file 1 : Appendix A for full search strategy).

The search was performed in June 2022 in the following electronic databases: Medline, Embase, Cochrane and Epistemonikos. We searched for articles published since the 2011 review. We searched from January 2010 up to June 2022 and applied no language restrictions. Reference lists of relevant papers were also examined.

We uploaded the results using EPPI-Reviewer, a web-based tool that facilitated semi-automation of the screening process and removal of duplicate studies. We made particular use of a priority screening function to reduce screening workload and avoid ‘data deluge’ [ 12 ]. Through machine learning, one reviewer screened a smaller number of records ( n  = 1200) to train the software to predict whether a given record was more likely to be relevant or irrelevant, thus pulling the relevant studies towards the beginning of the screening process. This automation did not replace manual work but helped the reviewer to identify eligible studies more quickly. During the selection process, we included studies that looked explicitly at interventions designed to turn research evidence into practice. Studies were included if they met the following pre-determined inclusion criteria:

The study was a systematic review

Search terms were included

Focused on the implementation of research evidence into practice

The methodological quality of the included studies was assessed as part of the review

Study populations included healthcare providers and patients. The EPOC taxonomy [ 13 ] was used to categorise the strategies. The EPOC taxonomy has four domains: delivery arrangements, financial arrangements, governance arrangements and implementation strategies. The implementation strategies domain includes 20 strategies targeted at healthcare workers. Numerous EPOC strategies were assessed in the review including educational strategies, local opinion leaders, reminders, ICT-focused approaches and audit and feedback. Some strategies that did not fit easily within the EPOC categories were also included. These were social media strategies and toolkits, and multi-faceted interventions (MFIs) (see Table  2 ). Some systematic reviews included comparisons of different interventions while other reviews compared one type of intervention against a control group. Outcomes related to improvements in health care processes or patient well-being. Numerous individual study types (RCT, CCT, BA, ITS) were included within the systematic reviews.

We excluded papers that:

Focused on changing patient rather than provider behaviour

Had no demonstrable outcomes

Made unclear or no reference to research evidence

The last of these criteria was sometimes difficult to judge, and there was considerable discussion amongst the research team as to whether the link between research evidence and practice was sufficiently explicit in the interventions analysed. As we discussed in the previous review [ 8 ] in the field of healthcare, the principle of evidence-based practice is widely acknowledged and tools to change behaviour such as guidelines are often seen to be an implicit codification of evidence, despite the fact that this is not always the case.

Reviewers employed a two-stage process to select papers for inclusion. First, all titles and abstracts were screened by one reviewer to determine whether the study met the inclusion criteria. Two papers [ 14 , 15 ] were identified that fell just before the 2010 cut-off. As they were not identified in the searches for the first review [ 8 ] they were included and progressed to assessment. Each paper was rated as include, exclude or maybe. The full texts of 111 relevant papers were assessed independently by at least two authors. To reduce the risk of bias, papers were excluded following discussion between all members of the team. 32 papers met the inclusion criteria and proceeded to data extraction. The study selection procedure is documented in a PRISMA literature flow diagram (see Fig.  1 ). We were able to include French, Spanish and Portuguese papers in the selection reflecting the language skills in the study team, but none of the papers identified met the inclusion criteria. Other non- English language papers were excluded.

figure 1

PRISMA flow diagram. Source: authors

One reviewer extracted data on strategy type, number of included studies, local, target population, effectiveness and scope of impact from the included studies. Two reviewers then independently read each paper and noted key findings and broad themes of interest which were then discussed amongst the wider authorial team. Two independent reviewers appraised the quality of included studies using a Quality Assessment Checklist based on Oxman and Guyatt [ 16 ] and Francke et al. [ 17 ]. Each study was rated a quality score ranging from 1 (extensive flaws) to 7 (minimal flaws) (see Additional file 2 : Appendix B). All disagreements were resolved through discussion. Studies were not excluded in this updated overview based on methodological quality as we aimed to reflect the full extent of current research into this topic.

The extracted data were synthesised using descriptive and narrative techniques to identify themes and patterns in the data linked to intervention strategies, targeted behaviours, study settings and study outcomes.

Thirty-two studies were included in the systematic review. Table 1. provides a detailed overview of the included systematic reviews comprising reference, strategy type, quality score, number of included studies, local, target population, effectiveness and scope of impact (see Table  1. at the end of the manuscript). Overall, the quality of the studies was high. Twenty-three studies scored 7, six studies scored 6, one study scored 5, one study scored 4 and one study scored 3. The primary focus of the review was on reviews of effectiveness studies, but a small number of reviews did include data from a wider range of methods including qualitative studies which added to the analysis in the papers [ 18 , 19 , 20 , 21 ]. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. In this section, we discuss the different EPOC-defined implementation strategies in turn. Interestingly, we found only two ‘new’ approaches in this review that did not fit into the existing EPOC approaches. These are a review focused on the use of social media and a review considering toolkits. In addition to single interventions, we also discuss multi-faceted interventions. These were the most common intervention approach overall. A summary is provided in Table  2 .

Educational strategies

The overview identified three systematic reviews focusing on educational strategies. Grudniewicz et al. [ 22 ] explored the effectiveness of printed educational materials on primary care physician knowledge, behaviour and patient outcomes and concluded they were not effective in any of these aspects. Koota, Kääriäinen and Melender [ 23 ] focused on educational interventions promoting evidence-based practice among emergency room/accident and emergency nurses and found that interventions involving face-to-face contact led to significant or highly significant effects on patient benefits and emergency nurses’ knowledge, skills and behaviour. Interventions using written self-directed learning materials also led to significant improvements in nurses’ knowledge of evidence-based practice. Although the quality of the studies was high, the review primarily included small studies with low response rates, and many of them relied on self-assessed outcomes; consequently, the strength of the evidence for these outcomes is modest. Wu et al. [ 20 ] questioned if educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes. Although based on evaluation projects and qualitative data, their results also suggest that positive changes on patient outcomes can be made following the implementation of specific evidence-based approaches (or projects). The differing positive outcomes for educational strategies aimed at nurses might indicate that the target audience is important.

Local opinion leaders

Flodgren et al. [ 24 ] was the only systemic review focusing solely on opinion leaders. The review found that local opinion leaders alone, or in combination with other interventions, can be effective in promoting evidence‐based practice, but this varies both within and between studies and the effect on patient outcomes is uncertain. The review found that, overall, any intervention involving opinion leaders probably improves healthcare professionals’ compliance with evidence-based practice but varies within and across studies. However, how opinion leaders had an impact could not be determined because of insufficient details were provided, illustrating that reporting specific details in published studies is important if diffusion of effective methods of increasing evidence-based practice is to be spread across a system. The usefulness of this review is questionable because it cannot provide evidence of what is an effective opinion leader, whether teams of opinion leaders or a single opinion leader are most effective, or the most effective methods used by opinion leaders.

Pantoja et al. [ 26 ] was the only systemic review focusing solely on manually generated reminders delivered on paper included in the overview. The review explored how these affected professional practice and patient outcomes. The review concluded that manually generated reminders delivered on paper as a single intervention probably led to small to moderate increases in adherence to clinical recommendations, and they could be used as a single quality improvement intervention. However, the authors indicated that this intervention would make little or no difference to patient outcomes. The authors state that such a low-tech intervention may be useful in low- and middle-income countries where paper records are more likely to be the norm.

ICT-focused approaches

The three ICT-focused reviews [ 14 , 27 , 28 ] showed mixed results. Jamal, McKenzie and Clark [ 14 ] explored the impact of health information technology on the quality of medical and health care. They examined the impact of electronic health record, computerised provider order-entry, or decision support system. This showed a positive improvement in adherence to evidence-based guidelines but not to patient outcomes. The number of studies included in the review was low and so a conclusive recommendation could not be reached based on this review. Similarly, Brown et al. [ 28 ] found that technology-enabled knowledge translation interventions may improve knowledge of health professionals, but all eight studies raised concerns of bias. The De Angelis et al. [ 27 ] review was more promising, reporting that ICT can be a good way of disseminating clinical practice guidelines but conclude that it is unclear which type of ICT method is the most effective.

Audit and feedback

Sykes, McAnuff and Kolehmainen [ 29 ] examined whether audit and feedback were effective in dementia care and concluded that it remains unclear which ingredients of audit and feedback are successful as the reviewed papers illustrated large variations in the effectiveness of interventions using audit and feedback.

Non-EPOC listed strategies: social media, toolkits

There were two new (non-EPOC listed) intervention types identified in this review compared to the 2011 review — fewer than anticipated. We categorised a third — ‘care bundles’ [ 36 ] as a multi-faceted intervention due to its description in practice and a fourth — ‘Technology Enhanced Knowledge Transfer’ [ 28 ] was classified as an ICT-focused approach. The first new strategy was identified in Bhatt et al.’s [ 30 ] systematic review of the use of social media for the dissemination of clinical practice guidelines. They reported that the use of social media resulted in a significant improvement in knowledge and compliance with evidence-based guidelines compared with more traditional methods. They noted that a wide selection of different healthcare professionals and patients engaged with this type of social media and its global reach may be significant for low- and middle-income countries. This review was also noteworthy for developing a simple stepwise method for using social media for the dissemination of clinical practice guidelines. However, it is debatable whether social media can be classified as an intervention or just a different way of delivering an intervention. For example, the review discussed involving opinion leaders and patient advocates through social media. However, this was a small review that included only five studies, so further research in this new area is needed. Yamada et al. [ 31 ] draw on 39 studies to explore the application of toolkits, 18 of which had toolkits embedded within larger KT interventions, and 21 of which evaluated toolkits as standalone interventions. The individual component strategies of the toolkits were highly variable though the authors suggest that they align most closely with educational strategies. The authors conclude that toolkits as either standalone strategies or as part of MFIs hold some promise for facilitating evidence use in practice but caution that the quality of many of the primary studies included is considered weak limiting these findings.

Multi-faceted interventions

The majority of the systematic reviews ( n  = 20) reported on more than one intervention type. Some of these systematic reviews focus exclusively on multi-faceted interventions, whilst others compare different single or combined interventions aimed at achieving similar outcomes in particular settings. While these two approaches are often described in a similar way, they are actually quite distinct from each other as the former report how multiple strategies may be strategically combined in pursuance of an agreed goal, whilst the latter report how different strategies may be incidentally used in sometimes contrasting settings in the pursuance of similar goals. Ariyo et al. [ 35 ] helpfully summarise five key elements often found in effective MFI strategies in LMICs — but which may also be transferrable to HICs. First, effective MFIs encourage a multi-disciplinary approach acknowledging the roles played by different professional groups to collectively incorporate evidence-informed practice. Second, they utilise leadership drawing on a wide set of clinical and non-clinical actors including managers and even government officials. Third, multiple types of educational practices are utilised — including input from patients as stakeholders in some cases. Fourth, protocols, checklists and bundles are used — most effectively when local ownership is encouraged. Finally, most MFIs included an emphasis on monitoring and evaluation [ 35 ]. In contrast, other studies offer little information about the nature of the different MFI components of included studies which makes it difficult to extrapolate much learning from them in relation to why or how MFIs might affect practice (e.g. [ 28 , 38 ]). Ultimately, context matters, which some review authors argue makes it difficult to say with real certainty whether single or MFI strategies are superior (e.g. [ 21 , 27 ]). Taking all the systematic reviews together we may conclude that MFIs appear to be more likely to generate positive results than single interventions (e.g. [ 34 , 45 ]) though other reviews should make us cautious (e.g. [ 32 , 43 ]).

While multi-faceted interventions still seem to be more effective than single-strategy interventions, there were important distinctions between how the results of reviews of MFIs are interpreted in this review as compared to the previous reviews [ 8 , 9 ], reflecting greater nuance and debate in the literature. This was particularly noticeable where the effectiveness of MFIs was compared to single strategies, reflecting developments widely discussed in previous studies [ 10 ]. We found that most systematic reviews are bounded by their clinical, professional, spatial, system, or setting criteria and often seek to draw out implications for the implementation of evidence in their areas of specific interest (such as nursing or acute care). Frequently this means combining all relevant studies to explore the respective foci of each systematic review. Therefore, most reviews we categorised as MFIs actually include highly variable numbers and combinations of intervention strategies and highly heterogeneous original study designs. This makes statistical analyses of the type used by Squires et al. [ 10 ] on the three reviews in their paper not possible. Further, it also makes extrapolating findings and commenting on broad themes complex and difficult. This may suggest that future research should shift its focus from merely examining ‘what works’ to ‘what works where and what works for whom’ — perhaps pointing to the value of realist approaches to these complex review topics [ 48 , 49 ] and other more theory-informed approaches [ 50 ].

Some reviews have a relatively small number of studies (i.e. fewer than 10) and the authors are often understandably reluctant to engage with wider debates about the implications of their findings. Other larger studies do engage in deeper discussions about internal comparisons of findings across included studies and also contextualise these in wider debates. Some of the most informative studies (e.g. [ 35 , 40 ]) move beyond EPOC categories and contextualise MFIs within wider systems thinking and implementation theory. This distinction between MFIs and single interventions can actually be very useful as it offers lessons about the contexts in which individual interventions might have bounded effectiveness (i.e. educational interventions for individual change). Taken as a whole, this may also then help in terms of how and when to conjoin single interventions into effective MFIs.

In the two previous reviews, a consistent finding was that MFIs were more effective than single interventions [ 8 , 9 ]. However, like Squires et al. [ 10 ] this overview is more equivocal on this important issue. There are four points which may help account for the differences in findings in this regard. Firstly, the diversity of the systematic reviews in terms of clinical topic or setting is an important factor. Secondly, there is heterogeneity of the studies within the included systematic reviews themselves. Thirdly, there is a lack of consistency with regards to the definition and strategies included within of MFIs. Finally, there are epistemological differences across the papers and the reviews. This means that the results that are presented depend on the methods used to measure, report, and synthesise them. For instance, some reviews highlight that education strategies can be useful to improve provider understanding — but without wider organisational or system-level change, they may struggle to deliver sustained transformation [ 19 , 44 ].

It is also worth highlighting the importance of the theory of change underlying the different interventions. Where authors of the systematic reviews draw on theory, there is space to discuss/explain findings. We note a distinction between theoretical and atheoretical systematic review discussion sections. Atheoretical reviews tend to present acontextual findings (for instance, one study found very positive results for one intervention, and this gets highlighted in the abstract) whilst theoretically informed reviews attempt to contextualise and explain patterns within the included studies. Theory-informed systematic reviews seem more likely to offer more profound and useful insights (see [ 19 , 35 , 40 , 43 , 45 ]). We find that the most insightful systematic reviews of MFIs engage in theoretical generalisation — they attempt to go beyond the data of individual studies and discuss the wider implications of the findings of the studies within their reviews drawing on implementation theory. At the same time, they highlight the active role of context and the wider relational and system-wide issues linked to implementation. It is these types of investigations that can help providers further develop evidence-based practice.

This overview has identified a small, but insightful set of papers that interrogate and help theorise why, how, for whom, and in which circumstances it might be the case that MFIs are superior (see [ 19 , 35 , 40 ] once more). At the level of this overview — and in most of the systematic reviews included — it appears to be the case that MFIs struggle with the question of attribution. In addition, there are other important elements that are often unmeasured, or unreported (e.g. costs of the intervention — see [ 40 ]). Finally, the stronger systematic reviews [ 19 , 35 , 40 , 43 , 45 ] engage with systems issues, human agency and context [ 18 ] in a way that was not evident in the systematic reviews identified in the previous reviews [ 8 , 9 ]. The earlier reviews lacked any theory of change that might explain why MFIs might be more effective than single ones — whereas now some systematic reviews do this, which enables them to conclude that sometimes single interventions can still be more effective.

As Nilsen et al. ([ 6 ] p. 7) note ‘Study findings concerning the effectiveness of various approaches are continuously synthesized and assembled in systematic reviews’. We may have gone as far as we can in understanding the implementation of evidence through systematic reviews of single and multi-faceted interventions and the next step would be to conduct more research exploring the complex and situated nature of evidence used in clinical practice and by particular professional groups. This would further build on the nuanced discussion and conclusion sections in a subset of the papers we reviewed. This might also support the field to move away from isolating individual implementation strategies [ 6 ] to explore the complex processes involving a range of actors with differing capacities [ 51 ] working in diverse organisational cultures. Taxonomies of implementation strategies do not fully account for the complex process of implementation, which involves a range of different actors with different capacities and skills across multiple system levels. There is plenty of work to build on, particularly in the social sciences, which currently sits at the margins of debates about evidence implementation (see for example, Normalisation Process Theory [ 52 ]).

There are several changes that we have identified in this overview of systematic reviews in comparison to the review we published in 2011 [ 8 ]. A consistent and welcome finding is that the overall quality of the systematic reviews themselves appears to have improved between the two reviews, although this is not reflected upon in the papers. This is exhibited through better, clearer reporting mechanisms in relation to the mechanics of the reviews, alongside a greater attention to, and deeper description of, how potential biases in included papers are discussed. Additionally, there is an increased, but still limited, inclusion of original studies conducted in low- and middle-income countries as opposed to just high-income countries. Importantly, we found that many of these systematic reviews are attuned to, and comment upon the contextual distinctions of pursuing evidence-informed interventions in health care settings in different economic settings. Furthermore, systematic reviews included in this updated article cover a wider set of clinical specialities (both within and beyond hospital settings) and have a focus on a wider set of healthcare professions — discussing both similarities, differences and inter-professional challenges faced therein, compared to the earlier reviews. These wider ranges of studies highlight that a particular intervention or group of interventions may work well for one professional group but be ineffective for another. This diversity of study settings allows us to consider the important role context (in its many forms) plays on implementing evidence into practice. Examining the complex and varied context of health care will help us address what Nilsen et al. ([ 6 ] p. 1) described as, ‘society’s health problems [that] require research-based knowledge acted on by healthcare practitioners together with implementation of political measures from governmental agencies’. This will help us shift implementation science to move, ‘beyond a success or failure perspective towards improved analysis of variables that could explain the impact of the implementation process’ ([ 6 ] p. 2).

This review brings together 32 papers considering individual and multi-faceted interventions designed to support the use of evidence in clinical practice. The majority of reviews report strategies achieving small impacts (normally on processes of care). There is much less evidence that these strategies have shifted patient outcomes. Combined with the two previous reviews, 86 systematic reviews of strategies to increase the implementation of research into clinical practice have been conducted. As a whole, this substantial body of knowledge struggles to tell us more about the use of individual and MFIs than: ‘it depends’. To really move forwards in addressing the gap between research evidence and practice, we may need to shift the emphasis away from isolating individual and multi-faceted interventions to better understanding and building more situated, relational and organisational capability to support the use of research in clinical practice. This will involve drawing on a wider range of perspectives, especially from the social, economic, political and behavioural sciences in primary studies and diversifying the types of synthesis undertaken to include approaches such as realist synthesis which facilitate exploration of the context in which strategies are employed. Harvey et al. [ 53 ] suggest that when context is likely to be critical to implementation success there are a range of primary research approaches (participatory research, realist evaluation, developmental evaluation, ethnography, quality/ rapid cycle improvement) that are likely to be appropriate and insightful. While these approaches often form part of implementation studies in the form of process evaluations, they are usually relatively small scale in relation to implementation research as a whole. As a result, the findings often do not make it into the subsequent systematic reviews. This review provides further evidence that we need to bring qualitative approaches in from the periphery to play a central role in many implementation studies and subsequent evidence syntheses. It would be helpful for systematic reviews, at the very least, to include more detail about the interventions and their implementation in terms of how and why they worked.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Before and after study

Controlled clinical trial

Effective Practice and Organisation of Care

High-income countries

Information and Communications Technology

Interrupted time series

Knowledge translation

Low- and middle-income countries

Randomised controlled trial

Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30. https://doi.org/10.1016/S0140-6736(03)14546-1 .

Article   PubMed   Google Scholar  

Green LA, Seifert CM. Translation of research into practice: why we can’t “just do it.” J Am Board Fam Pract. 2005;18:541–5. https://doi.org/10.3122/jabfm.18.6.541 .

Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1:1–3. https://doi.org/10.1186/1748-5908-1-1 .

Article   PubMed Central   Google Scholar  

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:2–14. https://doi.org/10.1186/s13012-015-0209-1 .

Article   Google Scholar  

Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:1–8. https://doi.org/10.1186/s13012-015-0295-0 .

Nilsen P, Ståhl C, Roback K, et al. Never the twain shall meet? - a comparison of implementation science and policy implementation research. Implementation Sci. 2013;8:2–12. https://doi.org/10.1186/1748-5908-8-63 .

Rycroft-Malone J, Seers K, Eldh AC, et al. A realist process evaluation within the Facilitating Implementation of Research Evidence (FIRE) cluster randomised controlled international trial: an exemplar. Implementation Sci. 2018;13:1–15. https://doi.org/10.1186/s13012-018-0811-0 .

Boaz A, Baeza J, Fraser A, European Implementation Score Collaborative Group (EIS). Effective implementation of research into practice: an overview of systematic reviews of the health literature. BMC Res Notes. 2011;4:212. https://doi.org/10.1186/1756-0500-4-212 .

Article   PubMed   PubMed Central   Google Scholar  

Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, et al. Changing provider behavior – an overview of systematic reviews of interventions. Med Care. 2001;39 8Suppl 2:II2–45.

Google Scholar  

Squires JE, Sullivan K, Eccles MP, et al. Are multifaceted interventions more effective than single-component interventions in changing health-care professionals’ behaviours? An overview of systematic reviews. Implement Sci. 2014;9:1–22. https://doi.org/10.1186/s13012-014-0152-6 .

Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R. Development of an efficient search filter to retrieve systematic reviews from PubMed. J Med Libr Assoc. 2021;109:561–74. https://doi.org/10.5195/jmla.2021.1223 .

Thomas JM. Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation? OA Evid Based Med. 2013;1:1–6.

Effective Practice and Organisation of Care (EPOC). The EPOC taxonomy of health systems interventions. EPOC Resources for review authors. Oslo: Norwegian Knowledge Centre for the Health Services; 2016. epoc.cochrane.org/epoc-taxonomy . Accessed 9 Oct 2023.

Jamal A, McKenzie K, Clark M. The impact of health information technology on the quality of medical and health care: a systematic review. Health Inf Manag. 2009;38:26–37. https://doi.org/10.1177/183335830903800305 .

Menon A, Korner-Bitensky N, Kastner M, et al. Strategies for rehabilitation professionals to move evidence-based knowledge into practice: a systematic review. J Rehabil Med. 2009;41:1024–32. https://doi.org/10.2340/16501977-0451 .

Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44:1271–8. https://doi.org/10.1016/0895-4356(91)90160-b .

Article   CAS   PubMed   Google Scholar  

Francke AL, Smit MC, de Veer AJ, et al. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;8:1–11. https://doi.org/10.1186/1472-6947-8-38 .

Jones CA, Roop SC, Pohar SL, et al. Translating knowledge in rehabilitation: systematic review. Phys Ther. 2015;95:663–77. https://doi.org/10.2522/ptj.20130512 .

Scott D, Albrecht L, O’Leary K, Ball GDC, et al. Systematic review of knowledge translation strategies in the allied health professions. Implement Sci. 2012;7:1–17. https://doi.org/10.1186/1748-5908-7-70 .

Wu Y, Brettle A, Zhou C, Ou J, et al. Do educational interventions aimed at nurses to support the implementation of evidence-based practice improve patient outcomes? A systematic review. Nurse Educ Today. 2018;70:109–14. https://doi.org/10.1016/j.nedt.2018.08.026 .

Yost J, Ganann R, Thompson D, Aloweni F, et al. The effectiveness of knowledge translation interventions for promoting evidence-informed decision-making among nurses in tertiary care: a systematic review and meta-analysis. Implement Sci. 2015;10:1–15. https://doi.org/10.1186/s13012-015-0286-1 .

Grudniewicz A, Kealy R, Rodseth RN, Hamid J, et al. What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses. Implement Sci. 2015;10:2–12. https://doi.org/10.1186/s13012-015-0347-5 .

Koota E, Kääriäinen M, Melender HL. Educational interventions promoting evidence-based practice among emergency nurses: a systematic review. Int Emerg Nurs. 2018;41:51–8. https://doi.org/10.1016/j.ienj.2018.06.004 .

Flodgren G, O’Brien MA, Parmelli E, et al. Local opinion leaders: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD000125.pub5 .

Arditi C, Rège-Walther M, Durieux P, et al. Computer-generated reminders delivered on paper to healthcare professionals: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2017. https://doi.org/10.1002/14651858.CD001175.pub4 .

Pantoja T, Grimshaw JM, Colomer N, et al. Manually-generated reminders delivered on paper: effects on professional practice and patient outcomes. Cochrane Database Syst Rev. 2019. https://doi.org/10.1002/14651858.CD001174.pub4 .

De Angelis G, Davies B, King J, McEwan J, et al. Information and communication technologies for the dissemination of clinical practice guidelines to health professionals: a systematic review. JMIR Med Educ. 2016;2:e16. https://doi.org/10.2196/mededu.6288 .

Brown A, Barnes C, Byaruhanga J, McLaughlin M, et al. Effectiveness of technology-enabled knowledge translation strategies in improving the use of research in public health: systematic review. J Med Internet Res. 2020;22:e17274. https://doi.org/10.2196/17274 .

Sykes MJ, McAnuff J, Kolehmainen N. When is audit and feedback effective in dementia care? A systematic review. Int J Nurs Stud. 2018;79:27–35. https://doi.org/10.1016/j.ijnurstu.2017.10.013 .

Bhatt NR, Czarniecki SW, Borgmann H, et al. A systematic review of the use of social media for dissemination of clinical practice guidelines. Eur Urol Focus. 2021;7:1195–204. https://doi.org/10.1016/j.euf.2020.10.008 .

Yamada J, Shorkey A, Barwick M, Widger K, et al. The effectiveness of toolkits as knowledge translation strategies for integrating evidence into clinical care: a systematic review. BMJ Open. 2015;5:e006808. https://doi.org/10.1136/bmjopen-2014-006808 .

Afari-Asiedu S, Abdulai MA, Tostmann A, et al. Interventions to improve dispensing of antibiotics at the community level in low and middle income countries: a systematic review. J Glob Antimicrob Resist. 2022;29:259–74. https://doi.org/10.1016/j.jgar.2022.03.009 .

Boonacker CW, Hoes AW, Dikhoff MJ, Schilder AG, et al. Interventions in health care professionals to improve treatment in children with upper respiratory tract infections. Int J Pediatr Otorhinolaryngol. 2010;74:1113–21. https://doi.org/10.1016/j.ijporl.2010.07.008 .

Al Zoubi FM, Menon A, Mayo NE, et al. The effectiveness of interventions designed to increase the uptake of clinical practice guidelines and best practices among musculoskeletal professionals: a systematic review. BMC Health Serv Res. 2018;18:2–11. https://doi.org/10.1186/s12913-018-3253-0 .

Ariyo P, Zayed B, Riese V, Anton B, et al. Implementation strategies to reduce surgical site infections: a systematic review. Infect Control Hosp Epidemiol. 2019;3:287–300. https://doi.org/10.1017/ice.2018.355 .

Borgert MJ, Goossens A, Dongelmans DA. What are effective strategies for the implementation of care bundles on ICUs: a systematic review. Implement Sci. 2015;10:1–11. https://doi.org/10.1186/s13012-015-0306-1 .

Cahill LS, Carey LM, Lannin NA, et al. Implementation interventions to promote the uptake of evidence-based practices in stroke rehabilitation. Cochrane Database Syst Rev. 2020. https://doi.org/10.1002/14651858.CD012575.pub2 .

Pedersen ER, Rubenstein L, Kandrack R, Danz M, et al. Elusive search for effective provider interventions: a systematic review of provider interventions to increase adherence to evidence-based treatment for depression. Implement Sci. 2018;13:1–30. https://doi.org/10.1186/s13012-018-0788-8 .

Jenkins HJ, Hancock MJ, French SD, Maher CG, et al. Effectiveness of interventions designed to reduce the use of imaging for low-back pain: a systematic review. CMAJ. 2015;187:401–8. https://doi.org/10.1503/cmaj.141183 .

Bennett S, Laver K, MacAndrew M, Beattie E, et al. Implementation of evidence-based, non-pharmacological interventions addressing behavior and psychological symptoms of dementia: a systematic review focused on implementation strategies. Int Psychogeriatr. 2021;33:947–75. https://doi.org/10.1017/S1041610220001702 .

Noonan VK, Wolfe DL, Thorogood NP, et al. Knowledge translation and implementation in spinal cord injury: a systematic review. Spinal Cord. 2014;52:578–87. https://doi.org/10.1038/sc.2014.62 .

Albrecht L, Archibald M, Snelgrove-Clarke E, et al. Systematic review of knowledge translation strategies to promote research uptake in child health settings. J Pediatr Nurs. 2016;31:235–54. https://doi.org/10.1016/j.pedn.2015.12.002 .

Campbell A, Louie-Poon S, Slater L, et al. Knowledge translation strategies used by healthcare professionals in child health settings: an updated systematic review. J Pediatr Nurs. 2019;47:114–20. https://doi.org/10.1016/j.pedn.2019.04.026 .

Bird ML, Miller T, Connell LA, et al. Moving stroke rehabilitation evidence into practice: a systematic review of randomized controlled trials. Clin Rehabil. 2019;33:1586–95. https://doi.org/10.1177/0269215519847253 .

Goorts K, Dizon J, Milanese S. The effectiveness of implementation strategies for promoting evidence informed interventions in allied healthcare: a systematic review. BMC Health Serv Res. 2021;21:1–11. https://doi.org/10.1186/s12913-021-06190-0 .

Zadro JR, O’Keeffe M, Allison JL, Lembke KA, et al. Effectiveness of implementation strategies to improve adherence of physical therapist treatment choices to clinical practice guidelines for musculoskeletal conditions: systematic review. Phys Ther. 2020;100:1516–41. https://doi.org/10.1093/ptj/pzaa101 .

Van der Veer SN, Jager KJ, Nache AM, et al. Translating knowledge on best practice into improving quality of RRT care: a systematic review of implementation strategies. Kidney Int. 2011;80:1021–34. https://doi.org/10.1038/ki.2011.222 .

Pawson R, Greenhalgh T, Harvey G, et al. Realist review–a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10Suppl 1:21–34. https://doi.org/10.1258/1355819054308530 .

Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research. Implementation Sci. 2012;7:1–10. https://doi.org/10.1186/1748-5908-7-33 .

Johnson MJ, May CR. Promoting professional behaviour change in healthcare: what interventions work, and why? A theory-led overview of systematic reviews. BMJ Open. 2015;5:e008592. https://doi.org/10.1136/bmjopen-2015-008592 .

Metz A, Jensen T, Farley A, Boaz A, et al. Is implementation research out of step with implementation practice? Pathways to effective implementation support over the last decade. Implement Res Pract. 2022;3:1–11. https://doi.org/10.1177/26334895221105585 .

May CR, Finch TL, Cornford J, Exley C, et al. Integrating telecare for chronic disease management in the community: What needs to be done? BMC Health Serv Res. 2011;11:1–11. https://doi.org/10.1186/1472-6963-11-131 .

Harvey G, Rycroft-Malone J, Seers K, Wilson P, et al. Connecting the science and practice of implementation – applying the lens of context to inform study design in implementation research. Front Health Serv. 2023;3:1–15. https://doi.org/10.3389/frhs.2023.1162762 .

Download references

Acknowledgements

The authors would like to thank Professor Kathryn Oliver for her support in the planning the review, Professor Steve Hanney for reading and commenting on the final manuscript and the staff at LSHTM library for their support in planning and conducting the literature search.

This study was supported by LSHTM’s Research England QR strategic priorities funding allocation and the National Institute for Health and Care Research (NIHR) Applied Research Collaboration South London (NIHR ARC South London) at King’s College Hospital NHS Foundation Trust. Grant number NIHR200152. The views expressed are those of the author(s) and not necessarily those of the NIHR, the Department of Health and Social Care or Research England.

Author information

Authors and affiliations.

Health and Social Care Workforce Research Unit, The Policy Institute, King’s College London, Virginia Woolf Building, 22 Kingsway, London, WC2B 6LE, UK

Annette Boaz

King’s Business School, King’s College London, 30 Aldwych, London, WC2B 4BG, UK

Juan Baeza & Alec Fraser

Federal University of Santa Catarina (UFSC), Campus Universitário Reitor João Davi Ferreira Lima, Florianópolis, SC, 88.040-900, Brazil

Erik Persson

You can also search for this author in PubMed   Google Scholar

Contributions

AB led the conceptual development and structure of the manuscript. EP conducted the searches and data extraction. All authors contributed to screening and quality appraisal. EP and AF wrote the first draft of the methods section. AB, JB and AF performed result synthesis and contributed to the analyses. AB wrote the first draft of the manuscript and incorporated feedback and revisions from all other authors. All authors revised and approved the final manuscript.

Corresponding author

Correspondence to Annette Boaz .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: appendix a., additional file 2: appendix b., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Boaz, A., Baeza, J., Fraser, A. et al. ‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice. Implementation Sci 19 , 15 (2024). https://doi.org/10.1186/s13012-024-01337-z

Download citation

Received : 01 November 2023

Accepted : 05 January 2024

Published : 19 February 2024

DOI : https://doi.org/10.1186/s13012-024-01337-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Implementation
  • Interventions
  • Clinical practice
  • Research evidence
  • Multi-faceted

Implementation Science

ISSN: 1748-5908

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

what are references on a research paper

Help | Advanced Search

Computer Science > Artificial Intelligence

Title: an interactive agent foundation model.

Abstract: The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide range of applications. We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and tasks. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction, enabling a versatile and adaptable AI framework. We demonstrate the performance of our framework across three separate domains -- Robotics, Gaming AI, and Healthcare. Our model demonstrates its ability to generate meaningful and contextually relevant outputs in each area. The strength of our approach lies in its generality, leveraging a variety of data sources such as robotics sequences, gameplay data, large-scale video datasets, and textual information for effective multimodal and multi-task learning. Our approach provides a promising avenue for developing generalist, action-taking, multimodal systems.

Submission history

Access paper:.

  • Download PDF
  • Other Formats

license icon

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

BibSonomy logo

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

  • Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • My Account Login
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • Open access
  • Published: 19 February 2024

Selection, optimization and validation of ten chronic disease polygenic risk scores for clinical implementation in diverse US populations

  • Niall J. Lennon   ORCID: orcid.org/0000-0002-2874-7371 1   na1 ,
  • Leah C. Kottyan 2   na1 ,
  • Christopher Kachulis   ORCID: orcid.org/0000-0003-2095-7419 1 ,
  • Noura S. Abul-Husn   ORCID: orcid.org/0000-0002-5179-1944 3 ,
  • Josh Arias   ORCID: orcid.org/0000-0001-6545-6656 4 ,
  • Gillian Belbin 3 ,
  • Jennifer E. Below   ORCID: orcid.org/0000-0002-1346-1872 5 ,
  • Sonja I. Berndt 4 ,
  • Wendy K. Chung   ORCID: orcid.org/0000-0003-3438-5685 6 ,
  • James J. Cimino   ORCID: orcid.org/0000-0003-4101-1622 7 ,
  • Ellen Wright Clayton   ORCID: orcid.org/0000-0002-0308-4110 5 ,
  • John J. Connolly 8 ,
  • David R. Crosslin 9 , 10 ,
  • Ozan Dikilitas   ORCID: orcid.org/0000-0002-9906-8608 11 ,
  • Digna R. Velez Edwards 5 ,
  • QiPing Feng   ORCID: orcid.org/0000-0002-6213-793X 5 ,
  • Marissa Fisher 1 ,
  • Robert R. Freimuth 11 ,
  • Tian Ge 12 ,
  • The GIANT Consortium ,
  • The All of Us Research Program ,
  • Joseph T. Glessner   ORCID: orcid.org/0000-0001-5131-2811 8 ,
  • Adam S. Gordon   ORCID: orcid.org/0000-0002-2058-7289 13 ,
  • Candace Patterson 1 ,
  • Hakon Hakonarson   ORCID: orcid.org/0000-0003-2814-7461 8 ,
  • Maegan Harden   ORCID: orcid.org/0000-0002-3607-6416 1 ,
  • Margaret Harr 8 ,
  • Joel N. Hirschhorn 1 , 14 ,
  • Clive Hoggart 3 ,
  • Li Hsu   ORCID: orcid.org/0000-0001-8168-4712 15 ,
  • Marguerite R. Irvin 7 ,
  • Gail P. Jarvik 10 ,
  • Elizabeth W. Karlson 12 ,
  • Atlas Khan   ORCID: orcid.org/0000-0002-6651-2725 6 ,
  • Amit Khera 1 ,
  • Krzysztof Kiryluk   ORCID: orcid.org/0000-0002-5047-6715 6 ,
  • Iftikhar Kullo   ORCID: orcid.org/0000-0002-6524-3471 11 ,
  • Katie Larkin 1 ,
  • Nita Limdi 7 ,
  • Jodell E. Linder   ORCID: orcid.org/0000-0002-0081-4712 5 ,
  • Ruth J. F. Loos 16 , 17 ,
  • Yuan Luo   ORCID: orcid.org/0000-0003-0195-7456 13 ,
  • Edyta Malolepsza 1 ,
  • Teri A. Manolio   ORCID: orcid.org/0000-0001-5844-4382 4 ,
  • Lisa J. Martin   ORCID: orcid.org/0000-0001-8702-9946 2 ,
  • Li McCarthy 1 ,
  • Elizabeth M. McNally 13 ,
  • James B. Meigs 12 ,
  • Tesfaye B. Mersha   ORCID: orcid.org/0000-0002-9189-8447 2 ,
  • Jonathan D. Mosley   ORCID: orcid.org/0000-0001-6421-2887 5 ,
  • Anjene Musick   ORCID: orcid.org/0000-0001-7770-299X 18 ,
  • Bahram Namjou   ORCID: orcid.org/0000-0003-4452-7878 2 ,
  • Nihal Pai 1 ,
  • Lorenzo L. Pesce 13 ,
  • Ulrike Peters 15 ,
  • Josh F. Peterson 5 ,
  • Cynthia A. Prows 2 ,
  • Megan J. Puckelwartz 13 ,
  • Heidi L. Rehm   ORCID: orcid.org/0000-0002-6025-0015 1 ,
  • Dan M. Roden   ORCID: orcid.org/0000-0002-6302-0389 5 ,
  • Elisabeth A. Rosenthal   ORCID: orcid.org/0000-0001-6042-4487 10 ,
  • Robb Rowley 4 ,
  • Konrad Teodor Sawicki 13 ,
  • Daniel J. Schaid 11 ,
  • Roelof A. J. Smit 3 ,
  • Johanna L. Smith   ORCID: orcid.org/0000-0002-5861-0413 11 ,
  • Jordan W. Smoller   ORCID: orcid.org/0000-0002-0381-6334 12 ,
  • Minta Thomas 15 ,
  • Hemant Tiwari 7 ,
  • Diana M. Toledo 1 ,
  • Nataraja Sarma Vaitinadin 5 ,
  • David Veenstra 10 ,
  • Theresa L. Walunas   ORCID: orcid.org/0000-0002-7653-3650 13 ,
  • Zhe Wang   ORCID: orcid.org/0000-0002-8046-4969 3 ,
  • Wei-Qi Wei   ORCID: orcid.org/0000-0003-4985-056X 5 ,
  • Chunhua Weng 6 ,
  • Georgia L. Wiesner 5 ,
  • Xianyong Yin   ORCID: orcid.org/0000-0001-6454-2384 19 &
  • Eimear E. Kenny 3  

Nature Medicine volume  30 ,  pages 480–487 ( 2024 ) Cite this article

11k Accesses

373 Altmetric

Metrics details

  • Clinical genetics
  • Risk factors

Polygenic risk scores (PRSs) have improved in predictive performance, but several challenges remain to be addressed before PRSs can be implemented in the clinic, including reduced predictive performance of PRSs in diverse populations, and the interpretation and communication of genetic results to both providers and patients. To address these challenges, the National Human Genome Research Institute-funded Electronic Medical Records and Genomics (eMERGE) Network has developed a framework and pipeline for return of a PRS-based genome-informed risk assessment to 25,000 diverse adults and children as part of a clinical study. From an initial list of 23 conditions, ten were selected for implementation based on PRS performance, medical actionability and potential clinical utility, including cardiometabolic diseases and cancer. Standardized metrics were considered in the selection process, with additional consideration given to strength of evidence in African and Hispanic populations. We then developed a pipeline for clinical PRS implementation (score transfer to a clinical laboratory, validation and verification of score performance), and used genetic ancestry to calibrate PRS mean and variance, utilizing genetically diverse data from 13,475 participants of the All of Us Research Program cohort to train and test model parameters. Finally, we created a framework for regulatory compliance and developed a PRS clinical report for return to providers and for inclusion in an additional genome-informed risk assessment. The initial experience from eMERGE can inform the approach needed to implement PRS-based testing in diverse clinical settings.

Polygenic risk scores (PRSs) aggregate the effects of many genetic risk variants and can be used to predict an individual’s genetic predisposition to a disease or phenotype 1 . PRSs are being calculated and disseminated at a prodigious rate 1 , 2 , but their development and application to clinical care, particularly among ancestrally diverse individuals, present substantial challenges 3 , 4 , 5 . Incorporation of genomic risk information has the potential to improve risk estimation and management 4 , 6 , particularly at younger ages 7 . Clinical use of PRSs may ultimately prevent disease or enable its detection at earlier, more treatable stages 7 , 8 , 9 , 10 . Improved estimation of risk may also enable targeting of preventive or therapeutic interventions to those most likely to benefit from them while avoiding unnecessary testing or overtreatment 10 , 11 . However, clinical use of Eurocentric PRSs in diverse patient samples risks exacerbating existing health disparities 12 , 13 , 14 .

PRSs for individual conditions are typically generated from summary statistics derived from genome-wide association studies (GWASs), which are themselves derived from populations that are heavily overrepresented by individuals of European ancestry 12 . Such scores have been shown to have limited prediction accuracy with increasing genetic distance from European populations 12 , 15 . PRSs can be improved if developed and validated using multiancestry cohorts 16 . Clinical and environmental data combined with monogenic and polygenic risk measurements can improve risk prediction, as demonstrated in ref. 17 and other studies 18 . Approaches for combining genomic and nongenomic information, optimizing models for populations of diverse genetic ancestry and across age groups, and conveying this information to clinicians and patients have yet to be developed and applied in clinical care. Various forms of PRSs are available to consumers through commercial platforms such as 23andMe, Myriad Genetics (riskScore), Allelica, Ambry Genetics, and others, and several noncommercial studies have explored the clinical use of PRSs in direct-to-participant models 19 , 20 , 21 ; however, there is limited information on the clinical implementation considerations of returning PRSs across multiple phenotypes in a primary care setting 20 . Even before assessing the ability of PRSs to improve health outcomes, reduce risk and enhance clinical care, large multicenter prospective pragmatic studies are needed to assess how patients and care providers interact with and respond to PRSs in a primary care setting 22 .

The Electronic Medical Records and Genomics (eMERGE) Network is a multicenter consortium established in 2007 to conduct genomic research in biobanks with electronic medical records 23 , 24 . In 2020, eMERGE embarked on a study of genomic risk assessment and management in 5,000 children and 20,000 adults of diverse ancestry, beginning with efforts to identify and validate published PRSs across multiple race-ethnic groups (and inferred genetic ancestries) in ten common diseases with complex genetic etiologies. The study plans for 25,000 individuals (aged 3–75 years) to be recruited from general healthcare system populations. Six of the ten recruitment sites are committed to recruiting an ‘enhanced diversity cohort’, meaning that their enrollment will target 75% of enrolled individuals belonging to a racial or ethnic minority or medically underserved population, whereas the remainder of clinical sites will target 35% minority participants 22 . Enrollment is not targeted to individuals with specific conditions, although individuals with prevalent conditions can be included. For this prospective, pragmatic study, the primary outcome being measured is the number of new healthcare actions after return of the genome-informed risk assessment. This paper describes (1) identification, selection and optimization of the PRSs that are included in the study; (2) calibration of ancestry for PRS estimation using a modified method developed for eMERGE; (3) development and launch of clinical reporting tools; and (4) an overview of the first 2,500 samples processed as part of the study.

PRS auditing and evaluation

To select the PRSs for clinical implementation, the Network conducted a multistage process to evaluate proposed scores (Fig. 1 ). An initial set of 23 conditions was selected based on considerations including relevance to population health (condition prevalence and heritability), strength of evidence for PRS performance, clinical expertise in the eMERGE Network, and data availability that would facilitate validation of the PRS in diverse populations. These conditions were abdominal aortic aneurysm, age-related macular degeneration, asthma, atopic dermatitis, atrial fibrillation, bone mineral density, breast cancer, Crohn’s disease, chronic kidney disease, colorectal cancer, coronary heart disease, depression, hypercholesterolemia, hypertension, ischemic stroke, lupus, nonalcoholic fatty liver disease, obesity, primary open angle glaucoma, prostate cancer, rheumatoid arthritis, type 1 diabetes and type 2 diabetes.

figure 1

a ,Timeline and process for selection, evaluation, optimization, transfer, validation and implementation of the clinical PRS test pipeline. Dashed lines represent pivotal moments in the progression of the project with duration between these events indicated in months (mo) above the blue arrow. Numbers in white represent the number of conditions being examined at each stage and their fates. List of ten conditions on the right-hand side indicates the conditions that were implemented in the clinical pipeline for this study. b , Overview of the eMERGE PRS process. Participant DNA is genotyped using the Illumina Global Diversity Array, which assesses 1.8 million sites. Genotyping data are phased and imputed with a reference panel derived from the 1,000 Genomes Project. For each participant, raw PRSs are calculated for each condition ( PRS raw ). Each participant’s genetic ancestry is algorithmically determined in the projection step. For each condition, an ancestry calibration model is applied to each participant’s z- scores based on model parameters derived from the All of Us Research Program (Calibration) and an adjusted z -score is calculated ( PRS adjusted ). Participants whose adjusted scores cross the predefined threshold for high PRS are identified and a pdf report is generated. The report is electronically signed after data review by a clinical laboratory director and delivered to the study portal for return to the clinical sites.

Network sites completed a comprehensive literature review on 23 proposed conditions and the corresponding PRSs. A summary of the features of the PRS for each of the final conditions chosen is shown in Supplementary Table 1 . The collated information included analytic viability—a description of covariates, the age, and ancestry effects of the original PRS model; feasibility—access to sufficiently diverse validation datasets (genetic ancestry and age) as well as condition prevalence and relevance to preventative care; potential clinical actionability—existing screening or treatment strategies, and magnitude (odds ratio) of risk in the high-risk group; and translatability—expected public health impact across diverse populations. Candidate PRSs were restricted to those that were either previously validated and published (journal or preprint) or for which there was sufficient access to information to develop and/or optimize new PRSs, which could then be validated.

In auditing and evaluating evidence of PRS performance, the eMERGE steering committee considered PRSs for conditions that could be implemented in pediatric and/or adult populations, and for diseases with a range of age of onset (0 to >65 years of age). We considered published single nucleotide polymorphism (SNP)-based heritability estimates available for ten of the 23 conditions, ranging from 3% to 58%. The majority of PRSs under consideration aimed to identify individuals at high risk for disease; however, PRSs to predict disease severity and drug response were also considered. Two of the conditions, breast cancer and prostate cancer, were only considered for implementation in individuals whose biological sex was female or male, respectively. As the eMERGE Network plans to enroll >50% participants from underrepresented groups (including racial and ethnic minority groups; people with lower socioeconomic status; underserved rural communities; sexual and gender minority groups) 25 , emphasis was placed on PRSs that were already available for, or could be developed and validated in, diverse population groups.

To define population groups, study-level population descriptors were first extracted from published literature, preprints or information shared directly by collaborators on data used to develop and/or optimize and/or validate PRSs. Methods for defining population groups across studies ranged from self-reporting, extraction from health system data and/or analysis of genetic ancestry. We designated four population groups: European ancestry (that is, study population descriptors included European, European-American or other European descent diaspora groups), African (African, African American or other African descent diaspora groups), Hispanic (that is, Hispanic, Latina/o/x or those who have origins in countries in the Caribbean and Latin America) and Asian (that is, South Asian, East Asian, South-East Asian, Asian-American or other diaspora Asian groups).

Thirteen conditions were considered and not selected for clinical implementation (Fig. 1 ). Of the six conditions dropped from consideration in August 2020, low disease prevalence across ancestral groups (age-related macular degeneration), availability of diverse genetic datasets for validation (primary open angle glaucoma, rheumatoid arthritis and Crohn’s disease) and the lack of a validated algorithm to identify patients and controls based upon electronic health record (EHR) (bone mineral density) were the driving factors. In March 2021, five additional conditions were dropped from consideration for clinical implementation based upon the progress of the development and validation of a multiancestral PRSs (depression, ischemic stroke), the low predictive value of candidate PRSs (hypertension, nonalcoholic fatty liver disease) and ethical considerations around returning results to a condition with low population prevalence (lupus).

Conditions not prioritized for implementation continued on a ‘developmental’ pathway for further refinement. Each of the 12 conditions that were selected to move forward from the March 2021 review was assigned a ‘lead’ and ‘co-lead’ site, which worked together to develop, validate and transfer the score to the clinical laboratory for instantiation and Clinical Laboratory Improvement Amendments (CLIA) validation. Assignment of leads was based on site preference, expertise and distribution of workload.

Selection, optimization and validation

A systematic framework was developed to evaluate the performance for the remaining 12 PRSs, in accordance with best practices outlined in ref. 26 . An in-depth evaluation matrix of the 12 chosen conditions can be found in Supplementary Table 2 . The Network carefully considered a variety of strategies to optimize PRS generalizability and portability. The Network prioritized validation across four ancestries with an emphasis on African and Hispanic ancestry due to their underrepresentation in genetic research and projected representation within the study cohort. We determined that a PRS was validated if the odds ratios were statistically significant in a minimum of two and up to four ancestral populations: African/African American, Asian, European ancestry, and Hispanic/Latino. The PRS Working Group members conducted an extensive scoping exercise to identify suitable datasets of multiple ancestries for disease-specific PRS validation. These included datasets from early phases of eMERGE (2007–2019) as well as external datasets such as the UK Biobank and Million Veteran Program. These larger population-level databases had the advantage of large sample sizes and less case–control ascertainment bias (though other sources of bias can still be an issue; ‘Discussion’). A standardized set of questions was addressed by the disease leads that included the source of discovery and validation datasets, the availability of multiancestry validation datasets, the availability of cross-ancestry PRSs (that is, PRS models that were developed and validated in more than one genetic ancestry), proposed percentile thresholds for identifying high-risk status, model discrimination (AUC) and effect sizes (odds ratios) associated with high-risk versus not high-risk status (Supplementary Table 2 ). For seven out of the 12 candidate scores, no further optimization of the original model was performed. For five scores, an additional optimization effort was undertaken to further refine the score performance in multiple ancestries. Details of the optimization can be found in Supplementary Table 3 . A specific score optimization was applied for chronic kidney disease. This optimization consisted of adding the effect of APOL1 risk genotypes to a polygenic component, which has been found to improve risk predictions in African ancestry cohorts 27 .

For the final selection of PRSs to be included in the prospective clinical study, the steering committee considered the score performance summaries (presented by condition leads) in addition to the actionable and measurable recommendations relevant for return, for each condition, in the prospective cohort. Abdominal aortic aneurysm was removed from the clinical pathway in June 2021 based on inability to pull a critical risk factor from the EHR (smoking) and a relatively low disease prevalence in Asian and Hispanic populations. Colorectal cancer was removed in June 2021 because the development and validation of the PRS was not complete for all the ancestral groups (Fig. 1 ). For the ten remaining phenotypes, the prospective pragmatic study required a small number of measurable primary clinical recommendations per phenotype so that the utility of the PRS to change physician and patient behavior can be measured. These recommendations can be found in Supplementary Tables 2 and 4 of ref. 22 .

Population-based z -score calibration

In this study, the focus is on integration and implementation of validated PRSs in clinical practice rather than novel PRS development. Ultimately, the Network opted to balance generalizability and feasibility by validating and returning cross-ancestry PRSs. However, even with cross-ancestry scores, differences remain in the distribution of z -scores for the PRSs across genetic ancestries that can result in inconsistent categorization of individuals into ‘high’ or ‘not high’ polygenic risk categories for a given condition 28 . To that end, the Network chose to develop methods to genetically infer each participant’s ancestry and calibrate the distribution of resulting z -scores through a population-based calibration model 28 , 29 (see below). An alternative would have been to apply existing PRSs in available samples of different ancestries and derive ancestry-specific effect estimates. However, returning ancestry-specific risk estimates is challenging in real-world implementations as it would require self-reporting of ancestry by patients (who may not be able to provide this with accuracy) and developing multiple ancestry-specific reports for each health condition. In addition, such PRSs would be problematic to return to patients of mixed ancestry.

PRSs often have different means and standard deviations for individuals from different genetic ancestries. While some of these differences could be due to true biological differences in risk, they also result from allele frequency and linkage disequilibrium structure differences between populations 30 . This problem is more acute when a PRS is calculated for an individual whose ancestry does not match the ancestries used to develop the PRS. A clinically implemented PRS test to return disease risk estimates, therefore, must be adjusted to account for these differences due to ancestral background. A calibration method based on principal component analysis (PCA), which was initially described in ref. 28 , was modified to model both the variance and means of scores as ancestry dependent, as compared to the previous method ( Methods ), which modeled only the means as dependent on ancestry. This modification was found to be necessary because some conditions were found to exhibit highly ancestry-dependent variance, which would have led to many more or fewer participants of certain ancestries receiving a ‘high PRS’ determination than was intended. One option considered to create and train the calibration model was to enroll and process a representative number of participants then pause on the return of results while the model was trained and the older data reprocessed. This stop–start approach was deemed suboptimal. Instead, the model was fit, with permission, to a portion of the All of Us (AoU) Research Program ( https://www.researchallofus.org/ ) cohort genotyping data, which allowed for continuous return of results to eMERGE participants once the study began. Of note, the All of Us Research Program cohorts selected for both training and testing the calibration model exhibited high degrees of genetic admixture, which would be expected to accurately reflect the study enrollment population. Importantly, because no ancestry group is homogenous, when individuals are compared directly to other individuals in their assigned population group, a dependence between admixture fraction and PRS can result. This dependence is removed by the described PCA calibration method, and the resulting calibrated PRSs are independent of admixture fraction. More details about the ancestry calibration can be found in Methods .

Transfer and implementation

Once the final ten conditions had been selected, condition leads worked with computational scientists at the clinical laboratory (Clinical Research Sequencing Platform, LLC at the Broad Institute) to transfer the PRS models and create the sample and data-processing workflow (Fig. 2 ). Condition-specific models were run with outputs from the lab’s genotyping (Illumina Global Diversity Array (GDA)), phasing (Eagle2 (ref. 31 ) https://github.com/poruloh/Eagle ) and imputation (Minimac4 (ref. 32 ) https://genome.sph.umich.edu/wiki/Minimac4 ) pipelines to assess genomic site representation (see Methods for more information on the architecture and components of the pipeline). Several rounds of iteration between the clinical laboratory and condition leads followed in which any issues with the pipeline were resolved and the effect of genomic site missingness was assessed (Table 1 ). The final version of the implemented models was returned to the condition leads to recalculate effect sizes in the validation cohorts.

figure 2

‘High-PRS threshold’ represents the percentile that is deemed to be the cutoff for a specific condition above which a high-PRS result is reported for that condition. Odds ratios are reported as the mean odds ratios (square dot) associated with having a score above the specified threshold, compared to having a score below the specified threshold, along with 95% confidence intervals (CIs), shown in the whiskers. The number of case and control samples used to derive these odds ratios and CIs for each condition can be found in Supplementary Table 2 . Note that the odds ratio for obesity is not reported here, as it will be published by the Genetic Investigation of ANthropometric Traits consortium (Smit et al., manuscript in preparation). ‘Number of SNPs’ represents the range of numbers or sites included in each score. ‘Age ranges for return’ indicates the participant ages at which a PRS is calculated for a given condition. AFIB, atrial fibrillation; BC, breast cancer; CKD, chronic kidney disease; CHD, coronary heart disease; HC, hypercholesterolemia; PC, prostate cancer; T1D, type 1 diabetes; T2D, type 2 diabetes.

Finally, as part of the implementation of the PRS pipelines as a clinical test in a CLIA laboratory, a validation study was performed (see Methods for a detailed description; Table 1 summarizes some of the results). Briefly, this study leveraged 70 reference cell lines from diverse ancestry groups (Coriell) where 30X whole genome sequencing data were generated to form a variant truth set from which the technical accuracy and reproducibility of imputation and PRS calling was assessed. A second sample set of 20 matched donor blood and saliva specimens was procured to assess the performance of the pipeline with different input materials. A set of three samples, each with six replicates, was run end-to-end through the wet lab and analytical pipelines as an assessment of reproducibility. As a verification of the clinical validity of the scores, cohorts of cases for eight of the ten conditions were created using the eMERGE phase III imputed dataset (available on https://anvil.terra.bio/#workspaces/anvil-datastorage/AnVIL_eMERGE_GWAS/data (registration required)). PRS performance measures were calculated to confirm associations between scores and conditions. Due to limitations in the eMERGE phase III imputation (no chromosome X, different imputation pipeline), the odds ratios from this analysis were not included in the final reports; rather, the odds ratios calculated in the condition-specific validation cohorts (using the final clinical lab pipeline) were used (Fig. 2 and Table 1 ). A validation report was created for each condition. This report was reviewed and approved by the Laboratory Director in compliance with CLIA regulations for the development of a laboratory-developed test. Personnel were trained on laboratory and analytical procedures, and standard operating procedures were implemented. Data review metrics were established, sample pass/fail criteria were defined, and order and report data-transfer pipelines were built as described in ref. 22 .

Creation of pipeline for report creation, review, sign-out and release

A software pipeline was built to facilitate the data review and clinical report generation. Reports were created both as documents (in pdf format) and structured data (in JSON format; a sample report is included in the Supplementary Information ). Automated rules for case triage were built into the PRS calculation and reporting pipeline to account for differences in return based on age and sex at birth for certain conditions. For instance, the PRS for breast cancer is only calculated for participants who report sex at birth as female; similarly, prostate cancer scores are only generated for participants who report sex at birth as male. Age-related restrictions were similarly coded into the pipeline to account for study policies on return. Data review by an appropriately qualified, trained individual is required for high complexity clinical testing. In the PRS clinical pipeline, this review takes the form of a set of metrics that are exposed by the pipeline to the reviewer. These include a z -score range for each condition (passing samples will have a score −5 <  z  < +5), a PCA plot per batch against a reference sample set (visual representation of outlier samples), monitoring the z -score range for each control per condition (one control on each plate; NA12878) and flagging any samples with multiple ‘high risk’ results for further review.

Each participant’s sample is also run on an orthogonal fingerprinting assay (Fluidigm biomark) that creates a genotype-based fingerprint for that DNA aliquot. Infinium genotyping data are compared to this fingerprint as a primary check of sample chain-of-custody fidelity and to preclude sample or plate swaps during lab processing. Reviewed and approved data for a participant are processed into a clinical report. The text and format of this report were created during an iterative review process by consortium work groups. For this pragmatic clinical implementation study, two results are returned to participants: ‘high risk’ or ‘not high risk’ based on the PRS 22 . In the clinical report, a qualitative framework has been developed to indicate for which condition(s) a participant has been determined to have a high PRS (if any). Quantitative values ( z- scores) are not included for any condition in the main results panel. For breast cancer and CHD, the z -score is presented in another section of the report for inclusion in integrated score models for those conditions. For breast cancer specifically, the provided z -score is used with the BOADICEA 33 model to generate an integrated risk that is included in the genome-informed risk assessment (GIRA), as described in ref. 22 .

Overview of the first 2,500 clinical samples processed

Between the launch in July 2022 and May 2023, 2,500 participants were processed through the clinical PRS pipeline (representing ∼ 10% of the proposed cohort). Of the first 2,500 participants processed, 64.5% (1,612) indicated sex at birth as female, while 35.5% (886) indicated male. Median age at sample collection was 51 years (range: 3 years to 75 years). Participants self-reported race/ancestry, with 32.8% (820) identifying as ‘White (for example, English, European, French, German, Irish, Italian, Polish, etc.)’; 32.8% (820) identified as ‘Black, African American or African (for example, African American, Ethiopian, Haitian, Jamaican, Nigerian, Somali, etc.)’; 25.4% (636) identified as ‘Hispanic, Latino or Spanish (for example, Colombian, Cuban, Dominican, Mexican or Mexican American, Puerto Rican, Salvadoran, etc.)’; 5% (124) identified as ‘Asian (for example, Asian, Indian, Chinese, Filipino, Japanese, Korean, Vietnamese, etc.)’; 1.5% (38) identified as American Indian or Alaska Native (for example, Aztec, Blackfeet Tribe, Mayan, Navajo Nation, Native Village of Barrow (Utqiagvik) Inupiat Traditional Government, Nome Eskimo Community, etc.); 0.9% (22) identified as Middle Eastern or North African (for example, Algerian, Egyptian, Iranian, Lebanese, Moroccan, Syrian, etc.); 0.8% (21) selected ‘None of these fully describe [me_or_my_child]’; 0.7% (17) selected ‘Prefer not to answer’; 0.1% (2) participants had incomplete data. A summary of the performance of the first 2,500 samples and resulting high-PRS metrics are shown in Fig. 3 . In the first 2,500 participants, we identified 515 participants (20.6%) with a high PRS for one of the ten conditions, 64 participants (2.6%) had a high PRS for two conditions, and two participants (0.08%) had a high PRS for three conditions. The remaining 1,919 participants had no high PRS found. High-PRS participants spanned the spectrum of genetic ancestry when projected onto principal component space (Fig. 3 ). Observed numbers of high-PRS assessments were largely consistent with the corresponding expected numbers. The P values in Fig. 3c are two-sided P values, which are calculated taking into account both the finite size of the eMERGE cohort and the finite size of the training data used to estimate the ancestry adjustment parameters. The P values are further adjusted for multiple hypothesis testing using the Holm–Šidák procedure 34 .

figure 3

a , PCA of ancestry indicating participants with a result of ‘high PRS’ for any condition (red dots) compared to participants who did not have a high PRS identified (gray dots). b , Summary of number of high-risk conditions found per participant. c , Observed numbers of high PRS called per condition compared to the expected numbers of high PRS per condition. P values are two-sided P values calculated by simulation to account for the uncertainty in the All of Us (AoU) derived ancestry calibration parameters due to the finite size of the AoU training cohort, and further adjusted for multiple hypothesis testing using the Holm–Šidák procedure. Note not all participants get scored for every condition based on age and sex at birth filters.

While the predictive performance of PRSs has improved substantially in recent years, challenges remain in ensuring that PRSs are applicable and effective in diverse populations. In particular, the vast majority of GWASs have focused on individuals of European ancestry, and the predictive accuracy of PRSs declines with increasing genetic distance from the discovery population 5 , 30 , 35 . This risks exacerbating existing health disparities, as clinical use of Eurocentric PRSs in diverse patient samples may not accurately reflect disease risk in non-European populations. To address these challenges, the eMERGE Network has conducted a multistage process to evaluate and optimize PRS selection, development and validation. The Network has prioritized conditions with high prevalence and heritability, existing literature, clinical actionability and the potential for health disparities, and has developed strategies to optimize PRS generalizability and portability across diverse populations. In particular, the Network has emphasized performance across four major ancestry groups (African, Asian, European, Hispanic, as reflected by self-identified race/ethnicity) and has developed a pipeline for clinical PRS implementation, a framework for regulatory compliance and a PRS clinical report.

The potential impact of PRS-based risk assessment in clinical practice is substantial. By enabling targeted interventions and preventative measures, PRS-based risk assessment has the potential to reduce the burden of a range of conditions 22 . Moreover, the development of PRS-based risk assessment in diverse populations has the potential to reduce health disparities by ensuring that clinical use of PRSs accurately reflects disease risk in diverse populations.

However, challenges remain in the successful implementation of PRS-based risk assessment in clinical practice. Participation bias in training or validation datasets that do not accurately represent the broader populations, for example the United Kingdom BioBank, can lead to skewed results and reduced generalizability in PRS test development 36 . Other challenges include concerns about genetic determinism, the potential for stigmatization and the need for robust regulatory frameworks to ensure that PRS-based risk assessment is deployed safely and effectively. Furthermore, to have more clinical utility, an individual’s PRS-based risk would be calculated as age-based absolute risk. Challenges also remain in healthcare provider and patient understanding and interpretation of PRS results and how to effectively communicate these results. Additionally, one of the biggest challenges is the implementation of effective disease prevention strategies after the return of the results. Return of the results will not result in a benefit without effective disease prevention or early detection strategies. The eMERGE Network’s work provides a blueprint for addressing these challenges, but ongoing research and evaluation will be necessary to ensure that PRS-based risk assessment is implemented in a responsible and effective manner. While this study will not answer all of the unanswered challenges and questions, the results from the 25,000 subjects from the eMERGE study will provide additional data to existing risk stratification to model harms and benefits over patient lifetimes.

Future groups developing, transferring and implementing PRSs into a clinical setting could build upon the eMERGE experience. Slightly less than half of the phenotypes originally considered for PRS development were able to be continued through clinical implementation based on varying considerations, suggesting that a moderately high number of phenotypes with measurable genetic contributions will be appropriate for PRS-based clinical tools. Thresholds for returning ‘high risk’ PRS were identified by each phenotype working group based in part upon the statistical significance between the ‘high-risk’ and ‘not high-risk’ groups. Future studies might consider standardizing the analyses and methods used to define these thresholds. Additionally, to have more clinical utility, an individual’s PRS-based risk would be calculated as an age-based absolute risk. While data for these risk assessments are available for some phenotypes (for example, cardiovascular and cancer), age of onset data are lacking for many clinically important phenotypes. Finally, the standards, guidance and the development of best practices for the integration of PRSs into clinical processes are yet to be developed. Future studies can learn from eMERGE and other groups' experiences will be a foundation for ongoing opportunities for the integration of polygenic risk predictions in clinical care settings.

In conclusion, the eMERGE Network’s work in PRS development represents an important step forward in the implementation of PRS-based risk assessment (in combination with other risk estimates from monogenic testing and family history) in clinical practice.

Consent and ethical approval

The study was conducted in accordance with the Declaration of Helsinki, and the central institutional regulatory board protocol was approved by the Ethics Committee of Vanderbilt University. All participants for eMERGE are consented, using a global primary consent and a site-specific consent. Minors acknowledge study participation by signing an assent (if local policy dictates) and the child’s parent/guardian signs a parental permission form. The Vanderbilt University Medical Center Co-ordinating Center is the institutional review board of record (no. 211043) for the Network’s single institutional review board, approved in July 2021.

For the All of Us Research Program, informed consent for all participants is conducted in person or through an eConsent platform that includes primary consent, Health Insurance Portability and Accountability Act authorization for research EHRs and consent for return of genomic results. The protocol was reviewed by the Institutional Review Board (IRB) of the All of Us Research Program. The All of Us Institutional Review Board follows the regulations and guidance of the National Institutes of Health Office for Human Research Protections for all studies, ensuring that the rights and welfare of research participants are overseen and protected uniformly.

Clinical trials registration

The eMERGE genomic risk assessment study is a registered, prospective, interventional clinical trial registered with clinicaltrials.gov (Identifier: NCT05277116 ). The purpose of the study is to determine if providing a GIRA will impact clinical actions taken by providers and patients to manage disease risk and the propensity of participants to develop a disease reported in the GIRA. For this prospective, pragmatic study, the primary outcome being measured is the number of new healthcare actions after return of the genome-informed risk assessment. Number of new healthcare actions will be measured by electronic health record data and participant-reported outcomes through a REDCap survey. Prespecified actions will include a condition-specific composite of new encounters, clinical orders or specialty referrals for clinical evaluation associated with the condition(s), placed by a provider within six months of result disclosure.

Secondary outcomes are the number of newly diagnosed conditions after return of the genome-informed risk assessment and the number of risk-reducing interventions after return of the genome-informed risk assessment (time frame: six months and 12 months post return of results to participant).

Population group definition

In the score auditing and evaluation phase, condition leads cataloged population groups used in the development or validation of given scores from available publications, preprints or information shared directly from collaborators. Across the initial list of evaluated scores, methods for defining population groups included self-reporting, extraction from health system data and/or analysis of genetic ancestry. In the optimization phase, populations were defined using either computational analysis alone or both computational analysis and self-reported ancestry, as indicated in Supplementary Table 3 . For creation of the training model for PRS ancestry calibration, populations were computationally determined as described in ‘PRS ancestry calibration overview’ below.

Populations with that are underserved and more frequently experience health disparities include racial and ethnic minority groups; people with lower socioeconomic status; underserved rural communities; sexual and gender minority groups; and people with disabilities 25 .

Analytical and technical validation studies

Broad imputation pipeline overview.

An imputation pipeline that takes as an input a variant call format (VCF) file generated from a genotyping microarray and imputes the genotypes at additional sites across the genome was developed. The pipeline architecture and composition was based on the widely used University of Michigan Imputation Server, which uses a software called Eagle ( https://github.com/poruloh/Eagle ) for phasing and Minimac4 ( https://genome.sph.umich.edu/wiki/Minimac4 ) for the imputation. The pipeline uses a curated version of the 1,000 Genomes Project (1KG, www.internationalgenome.org ) as the reference panel. Additional details on the imputation pipeline can be found at https://broadinstitute.github.io/warp/docs/Pipelines/Imputation_Pipeline/README .

Broad curated 1KG reference panel

During the validation process, we determined that some sites in the 1KG reference panel were incorrectly genotyped compared to the sites in matching whole genome sequencing data. To increase accuracy of the imputation and PRS scoring, we curated the original panel by removing sites that were likely incorrectly genotyped based on comparing allele frequencies to those reported in gnomAD v.2 ( https://gnomad.broadinstitute.org/ ). Documentation of this curation can be found at https://broadinstitute.github.io/warp/docs/Pipelines/Imputation_Pipeline/references_overview and a publicly available version of the panel at the following Google Cloud location (accessible via the gsutil utility): gs://broad-gotc-test-storage/imputation/1000G_reference_panel/.

Selection of a reference panel for imputation as an input to a PRS is an important consideration. Some reference panels (for example, Trans-Omics for Precision Medicine (TOPMed)) have more samples than the default used in our pipeline (that is, 1KG). This leads to more variants being imputed. The question is whether this would materially change the PRSs calculated from samples imputed with the TOPMed panel. Access to this panel computationally is restricted (and local download prohibited) so it was deemed infeasible to implement in our clinical production environment. The performance of a non-eMERGE PRS (for CHD; ref. 28 ) using the two different reference panels was determined for 20 GDA saliva specimens and for 42 AoU array v.1 specimens. The cohort was imputed both by the Broad imputation pipeline with curated 1KG as the reference panel as well as on the TOPMed imputation server with TOPMed as the reference panel. Imputed arrays were scored by the PRS pipeline.

The PRS percentiles computed with each method are highly concordant for both cohorts. The Pearson correlation coefficient is 0.996 for both cohorts, the P value of the Welch two-sample t- test is equal to 0.93 and 0.85 (indicating no statistical difference between the methods) for GDA and AoU v.1 cohorts, respectively.

Performance verification of the imputation pipeline

Imputation accuracy was determined for 42 specimens that were processed through a genotyping microarray (AoU v.1 array—the precursor to the commercial Global Diversity Array) and imputed with curated 1KG as the reference panel where corresponding deep-coverage (>30X) PCR-free whole genome sequencing data were used as a truth call set to calculate sensitivity and specificity. The arrays were also imputed on the Michigan Imputation Server with 1KG as the reference panel.

Within the cohort, four different ancestries were represented: non-Finnish Europeans, East Asians, South Asian (SAS), African (AFR). Broad imputation pipeline sensitivity for SNPs is >97% and insertions/deletions (INDELs) >95% for all ancestries. Similarly, specificity for SNPs from the Broad imputation pipeline is above 99% and the specificity for INDELs is >98%. See Extended Data Table 1 for a table of results. Results were highly concordant with those returned by the remote server at Michigan.

Performance evaluation of different input material types

To assess the performance of specimens derived from both saliva and whole blood, a set of 20 matched blood and saliva pairs were run through the GDA genotyping process and the resulting VCFs were imputed using the Broad pipeline to be compared against results for matched blood-derived whole genome data. The Pearson correlation between sensitivity and specificity of blood- and saliva-derived samples are equal to 100% and 100%, respectively. For the same pairs, the Welch two-sample t- test statistic is 0.997 and 0.987, respectively. There is no significant difference between the different input sample types.

Imputation repeatability and reproducibility

Imputation pipeline repeatability was assessed by repeating imputation of a cohort of 1,000 Global Screening Array arrays ten times over the course of two weeks and was found to be 100% concordant. Imputation pipeline precision (reproducibility) was also tested on technical replicates. Three individual samples derived from saliva were each genotyped six times, followed by an imputation in a cohort of all saliva-derived samples. In each set of technical replicates, all pairs and variants in each pair were compared (making a total of 45 pairs for which genotypes were compared). Reproducibility is measured using Jaccard scores. ‘Reproducibility over variants’ was calculated only over sites where at least one of the two replicates in a pair calls a non hom-ref genotype and was found to be 99.91% (95% CI 99.89–99.93) for SNPs and 99.87% (95% CI 99.85–99.90) for INDELs. ‘Reproducibility over all sites’ was calculated over all genotyped sites, including sites genotyped as hom-ref in both replicates and was found to be 100% (95% CI 100–100) for both SNPs and INDELs.

Imputation performance as a function of variant frequency

Because we expect accuracy to be impacted by the frequency of a variant in the population (rare variants are less likely to be in the reference panel and therefore less accurately imputed), we further subdivided the performance assessment by allele frequencies on two cohorts: 42 AoU v.1 arrays and 20 blood–saliva pairs of GDA arrays. Accuracy of imputation of variants as a function of population allele frequency performed as expected, with rare variants in the population not being as accurately represented. Imputation is more accurate for variants that are more frequently observed in the population (≥0.1 allele frequency (AF)).

Impact of genotyping array call rate on imputation performance

The impact of call rate on the imputation was assessed by generating a downsampled series of 42 arrays, each with call rates of 90%, 95%, 97% and 98%. Pearson correlation values for SNPs and INDELs were calculated across bins of allele frequencies, assessed against gnomAD common variants (AF > 0.1), for the cohorts with downsampled call rates. Call rates below 95% were found to produce suboptimal results. At this rate the mean R 2 dosage score for sites with AF ≥ 0.1 was found to be 0.98% (95% CI 0.98–0.98) for both SNPs and INDELs compared to 0.99% for call rates of 97% and 98%.

Impact of imputation batch size on performance

Batch size effect of the imputation pipeline was assessed by imputing and analyzing arrays in a cohort of size 1,000 (randomly chosen), ten cohorts of size 100 (nonoverlapping subsets of the 1,000 cohort) and ten cohorts of size ten (nonoverlapping subsets of one of the 100 cohorts). Pearson correlations of dosage scores were calculated across bins for allele frequencies (assessed against gnomAD) for smaller cohorts versus larger cohorts. The data show that imputation is highly correlated across batch sizes with batches down to as few as ten samples, producing acceptable performance. The mean R 2 correlation of dosage scores for sites with allele frequency greater or equal to 0.1 is above 0.97 in all cases both for SNPs and INDELs and increases to 0.98 for the larger studied cohorts. Increasing batch sizes produces very slight improvements in imputation but these are not significant and the choice of imputation batch size (above or equal to ten samples) can be made on practical and operational grounds.

Broad PRS pipeline overview

The PRS pipeline begins by calculating a raw score using plink2 ( https://www.cog-genomics.org/plink/2.0/ ). For each condition, effect alleles and weights are defined for a set of genomic sites stored in a weights file. At each site, the effect allele dosage observed in the imputed VCF is multiplied by the effect weight in the weights file. The raw score is the sum of these products over all the specified sites.

Validation of technical and analytical performance of the PRS pipeline

For each of the ten conditions chosen by the consortium for clinical return, a validation study was performed to assess the technical and analytical performance as well as to verify the association between score and disease risk. See Extended Data Table 2 for a summary of the validation measures.

PRS pipeline accuracy

Accuracy of the pipeline was determined by calculating the Pearson correlation between PRSs calculated from 70 specimens imputed from GDA array data and PRSs of corresponding deep-coverage PCR-free whole genome sequencing data (used as a truth call set).

Input material performance

Accuracy of PRS scoring when different sample types (blood or saliva) are used as inputs was determined by comparing the PRSs from matched blood and saliva pairs collected from 20 individuals.

PRS pipeline repeatability

PRS pipeline repeatability was assessed by running the pipeline on the same dataset of 70 imputed GDA arrays ten times over the course of two weeks (without call caching). Scores generated from the different processing runs were compared to determine if there are any differences observed for a given PRS when the pipeline is run at different times.

PRS pipeline reproducibility

PRS pipeline precision (reproducibility) was assessed using three samples each run six times end-to-end and then compared in a pairwise manner. The z -score standard deviation is used as a measure of variability.

PRS site representation

The SNP weight sites that are not called during genotyping or imputation were determined. These are sites not present in the intersection of an imputed GDA array and the reference panel. Ideally, all sites required for PRS calculation are present either as genotyped or imputed sites; however, in practice, a small number of sites are not present due to differences in the data used to create the score and the specific array and imputation reference panel used in this study.

Performance verification using eMERGE I–III cohort

A cohort of samples with known phenotypic information was used to verify the relationship between PRS as determined by our pipeline and disease risk. For conditions where cases and controls could be identified in the eMERGE I–III cohort, we determined performance using metrics outlined in the ClinGen working group recommendations 26 . Specifically, we determined the PRS distributions for cases and controls, we examined the impact of ancestry adjustment on the distributions and we examined the relationship between observed and predicted risk. An example of this analysis (for T2D is shown below).

The T2D weight file used for PRSs in this validation report comes from a GWAS by Ge et al. 29 where they reported that individuals in the top 2% of the PRSs in the population have an increased risk of developing T2D.

The T2D cohort in the eMERGE I–III dataset consisted of 19,145 cases and 68,823 control samples. The mean adjusted PRS for case samples was 0.435, while the mean for control samples was −0.042. Individuals with higher adjusted PRS scores tend to be more likely to develop disease (see Extended Data Fig. 1 for a histogram of T2D PRSs in cases and controls).

There are some limitations to this analysis: (1) the eMERGE I–III dataset being used for this analysis was generated from different array platforms and was imputed with a different pipeline including a different version of the 1KG reference panel than the one currently implemented; (2) the eMERGE I–III imputed dataset does not include variants from chromosomes X or Y. For these reasons, the PRS disease association analysis represents a verification of the clinical validation performed by eMERGE condition leads rather than the quantitative measure of the impact of the score on risk. The clinical associations (odds ratios) that are reported on the clinical report for each condition were independently determined by eMERGE disease-specific expert teams.

Validation of pipeline and ancestry adjustment in original case–control cohorts

The final pipeline was made available to computational scientists at each of the eMERGE disease-specific expert teams who had access to appropriate case–control cohorts. These groups confirmed the performance of the final pipeline on their cohorts. The odds ratios for each condition that are reported on the clinical reports come from these cohorts rather than the eMERGE cohort for the reasons described above.

PRS ancestry calibration overview

Pca method description.

For a PRS, which is a sum of SNP effects (linear weights), the central limit theorem states that the distribution of scores in a homogenous population will tend towards a normal distribution as the number of SNPs becomes large. When two different homogenous populations are randomly mixed, the additive property of the PRS leads the resulting distribution to be similarly normally distributed, with mean and variance depending on the mean and variance of the original homogenous populations 37 , 38 . We can therefore model the distribution of the PRS as being normally distributed, with mean and variance being functions of genetic ancestry. Practically, we implement this as

with genetic ancestry being represented by projection into principal component (PC) space 39 . The α and β parameters are found by jointly fitting them to a cohort of training data. This fit is performed by minimizing the negative log likelihood:

where i runs over the individuals in the training cohort, prs i is the i th individual’s raw PRS, and μ i and σ i are calculated using equations ( 2 ) and ( 3 ) above by projecting the i th individual into PC space. Note that, due to the simplicity of the model, overfitting is unlikely to be a problem, and so no regularization or other overfitting avoidance technique is implemented. An individual’s PRS z -score can then be calculated as

where μ and σ have again been calculated based on the specific individual’s projection into PC space. In this way, once the model has been trained, the z -score calculation is fully defined by the fitted model parameters, and z -scores can be calculated without needing additional access to the original training cohort.

Generating trained models from All of Us data

Generating the trained models consisted of three steps: (1) selecting the training cohort; (2) imputation of the training cohort; and (3) training the models on the training cohort. A test cohort was also generated to test the performance of the training.

Ancestry-balanced training and test cohorts were generated by subsampling from an initial cohort of around 100,000 All of Us samples. For the purposes of balancing the cohort, each sample was assigned to one of the five 1KG super populations. Principal component analysis was first performed on a random subset of 20,000 samples. The 1KG samples were projected onto these principal components, and a support vector machine was trained on 1KG to predict ancestry. The support vector machine was then used to assign 108,000 AoU samples to one of the five 1KG super populations. A balanced training cohort was selected based on these predicted ancestries, and principal components were recalculated using this balanced training cohort. A similarly balanced test cohort was selected based on ancestries estimated from projection on the training set PCs. The resulting breakdown of the cohorts by estimated ancestry is shown in Extended Data Table 3 .

Both the training and testing cohorts include a number of individuals with highly admixed ancestry. Admixture was quantified using the tool Admixture 40 with five ancestral populations. The resulting admixtures of each cohort are shown in Extended Data Fig. 2 , and the most common admixed ancestries in each cohort are summarized in Extended Data Table 4 .

Each cohort was imputed using the imputation pipeline described above, with 1KG as the reference panel. By keeping the imputation pipeline identical to the pipeline used for the eMERGE dataset, and because the AoU dataset uses the same GDA array as the eMERGE dataset, any potential biases resulting from differing data production and processing methods were removed. The training cohort was scored for each of the ten conditions, and model parameters were fit by minimizing the negative log likelihood as described. The test cohort was then used to evaluate the generalizability of these model parameters.

Performance on test cohort

Extended Data Fig. 3 illustrates the distribution of calibrated z -scores in the test cohort using the parameters fit in the training cohort. As can be seen, all ancestries show the intended standard normal distribution of calibrated scores.

One of the main improvements of this method over previous methods is the inclusion of an ancestry-dependent variance in addition to the ancestry-dependent mean. The importance of this is shown for the hypercholesterolemia PRS in Extended Data Fig. 4 . The variance of this score differs significantly across ancestries, so that a method that only fits the mean of the distribution as ancestry dependent can result in z -score distributions that have been attenuated towards zero or expanded away from zero for some ancestries. By also treating variance as ancestry dependent, this method results in z -score distributions that are more standardized across ancestries.

In addition to improving calibration across ancestries, this method can improve calibration within ancestries, particularly for highly admixed individuals. An example of this can be seen in Extended Data Fig. 5 . Because no ancestry group is homogenous, when individuals are compared directly to other individuals in their assigned population group, a dependence between admixture fraction and PRS can result. This dependence is removed by the described PCA calibration method, and the resulting calibrated PRSs are independent of admixture fraction.

Reporting summary

Further information on the research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

Underlying data used to verify the performance of the PRS pipeline are available in dbGaP https://www.ncbi.nlm.nih.gov/projects/gap/cgi-bin/study.cgi?study_id=phs001584.v1.p1 . De-identified data relating to trial participants will be available through dbGaP ( https://www.ncbi.nlm.nih.gov/gap/ ) access and the AnVIL platform ( https://anvil.terra.bio/ ) as an interim analysis in 2024 and final dataset at the end of the study, expected in 2026. Information (sites and weights) on the implemented scores can be found at https://github.com/broadinstitute/eMERGE-implemented-PRS-models-Lennon-et-al and also on the UCSC browser https://genome.ucsc.edu/s/Max/emerge . Additionally, PGS Catalog IDs for most of the implemented scores are indicated in Supplementary Table 3 .

Code availability

Codes used in this work to create and operate the imputation and PRS pipelines are hosted at https://github.com/broadinstitute/palantir-workflows/tree/main/ImputationPipeline . Code for the PRS ancestry calibration can also be found in the AoU demonstration workspace https://workbench.researchallofus.org/workspaces/aou-rw-bef5bf62/demopolygenicriskscoregeneticancestrycalibration/data (open access but researcher registration required).

Lambert, S. A. et al. The Polygenic Score Catalog as an open database for reproducibility and systematic evaluation. Nat. Genet. 53 , 420–425 (2021).

Article   CAS   PubMed   Google Scholar  

Lewis, C. M. & Vassos, E. Polygenic risk scores: from research tools to clinical instruments. Genome Med. 12 , 44 (2020).

Article   PubMed   PubMed Central   Google Scholar  

Polygenic Risk Score Task Force of the International Common Disease Alliance. Responsible use of polygenic risk scores in the clinic: potential benefits, risks and gaps. Nat. Med. 27 , 1876–1884 (2021).

Article   CAS   Google Scholar  

Torkamani, A., Wineinger, N. E. & Topol, E. J. The personal and clinical utility of polygenic risk scores. Nat. Rev. Genet. 19 , 581–590 (2018).

Duncan, L. et al. Analysis of polygenic risk score usage and performance in diverse human populations. Nat. Commun. 10 , 3328 (2019).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Hurson, A. N. et al. Prospective evaluation of a breast-cancer risk model integrating classical risk factors and polygenic risk in 15 cohorts from six countries. Int. J. Epidemiol. 50 , 1897–1911 (2022).

Article   PubMed   Google Scholar  

Inouye, M. et al. Genomic risk prediction of coronary artery disease in 480,000 adults: implications for primary prevention. J. Am. Coll. Cardiol. 72 , 1883–1893 (2018).

Guo, F. et al. Polygenic risk score for defining personalized surveillance intervals after adenoma detection and removal at colonoscopy. Clin. Gastroenterol. Hepatol. 21 , 210–219.e11 (2023).

Fantus, R. J. & Helfand, B. T. Germline genetics of prostate cancer: time to incorporate genetics into early detection tools. Clin. Chem. 65 , 74–79 (2019).

Pharoah, P. D. P., Antoniou, A. C., Easton, D. F. & Ponder, B. A. J. Polygenes, risk prediction, and targeted prevention of breast cancer. N. Engl. J. Med. 358 , 2796–2803 (2008).

Willoughby, A., Andreassen, P. R. & Toland, A. E. Genetic testing to guide risk-stratified screens for breast cancer. J. Pers. Med 9 , 15 (2019).

Martin, A. R. et al. Clinical use of current polygenic risk scores may exacerbate health disparities. Nat. Genet. 51 , 584–591 (2019).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mars, N. et al. Systematic comparison of family history and polygenic risk across 24 common diseases. Am. J. Hum. Genet. 109 , 2152–2162 (2022).

Wang, Z. et al. The value of rare genetic variation in the prediction of common obesity in European ancestry populations. Front. Endocrinol. 13 , 863893 (2022).

Ruan, Y. et al. Improving polygenic prediction in ancestrally diverse populations. Nat. Genet. 54 , 573–580 (2022).

Márquez-Luna, C., Loh, P.-R., South Asian Type 2 Diabetes (SAT2D) Consortium & Price, A. L. Multiethnic polygenic risk scores improve risk prediction in diverse populations. Genet. Epidemiol. 41 , 811–823 (2017).

Hujoel, M. L. A., Loh, P.-R., Neale, B. M. & Price, A. L. Incorporating family history of disease improves polygenic risk scores in diverse populations. Cell Genom. 2 , 100152 (2022).

Elliott, J. et al. Predictive accuracy of a polygenic risk score-enhanced prediction model vs a clinical risk score for coronary artery disease. JAMA 323 , 636–645 (2020).

Folkersen, L. et al. Impute.me: an open-source, non-profit tool for using data from direct-to-consumer genetic testing to calculate and interpret polygenic risk scores. Front. Genet. 11 , 515901 (2020).

Article   Google Scholar  

Hao, L. et al. Development of a clinical polygenic risk score assay and reporting workflow. Nat. Med. 28 , 1006–1013 (2022).

Vassy, J. L. et al. Cardiovascular disease risk assessment using traditional risk factors and polygenic risk scores in the million veteran program. JAMA Cardiol. 8 , 564–574 (2023).

Linder, J. E. et al. Returning integrated genomic risk and clinical recommendations: the eMERGE study. Genet. Med. 25 , 100006 (2023).

McCarty, C. A. et al. The eMERGE Network: a consortium of biorepositories linked to electronic medical records data for conducting genomic studies. BMC Med. Genomics 4 , 13 (2011).

Gottesman, O. et al. The Electronic Medical Records and Genomics (eMERGE) Network: past, present, and future. Genet. Med. 15 , 761–771 (2013).

NIMHD. Overview. https://www.nimhd.nih.gov/about/overview/ . Accessed 11 Dec 2023.

Wand, H. et al. Improving reporting standards for polygenic scores in risk prediction studies. Nature 591 , 211–219 (2021).

Khan, A. et al. Genome-wide polygenic score to predict chronic kidney disease across ancestries. Nat. Med. 28 , 1412–1420 (2022).

Khera, A. V. et al. Genome-wide polygenic scores for common diseases identify individuals with risk equivalent to monogenic mutations. Nat. Genet. 50 , 1219–1224 (2018).

Ge, T. et al. Development and validation of a trans-ancestry polygenic risk score for type 2 diabetes in diverse populations. Genome Med. 14 , 70 (2022).

Martin, A. R. et al. Human demographic history impacts genetic risk prediction across diverse populations. Am. J. Hum. Genet. 100 , 635–649 (2017).

Loh, P.-R. et al. Reference-based phasing using the Haplotype Reference Consortium panel. Nat. Genet. 48 , 1443–1448 (2016).

Howie, B., Fuchsberger, C., Stephens, M., Marchini, J. & Abecasis, G. R. Fast and accurate genotype imputation in genome-wide association studies through pre-phasing. Nat. Genet. 44 , 955–959 (2012).

Lee, A. et al. BOADICEA: a comprehensive breast cancer risk prediction model incorporating genetic and nongenetic risk factors. Genet. Med. 21 , 1708–1718 (2019).

Holm, S. A simple sequentially rejective multiple test procedure. Scand. J. Stat. 6 , 65–70 (1979).

Ding, Y. et al. Large uncertainty in individual polygenic risk score estimation impacts PRS-based risk stratification. Nat. Genet. 54 , 30–39 (2022).

Schoeler, T. et al. Participation bias in the UK Biobank distorts genetic associations and downstream analyses. Nat. Hum. Behav. 7 , 1216–1227 (2023).

Privé, F. et al. Portability of 245 polygenic scores when derived from the UK Biobank and applied to 9 ancestry groups from the same cohort. Am. J. Hum. Genet. 109 , 12–23 (2022).

Ding, Y. et al. Polygenic scoring accuracy varies across the genetic ancestry continuum. Nature 618 , 774–781 (2023).

Scutari, M., Mackay, I. & Balding, D. Using genetic distance to infer the accuracy of genomic prediction. PLoS Genet. 12 , e1006288 (2016).

Alexander, D. H., Novembre, J. & Lange, K. Fast model-based estimation of ancestry in unrelated individuals. Genome Res. 19 , 1655–1664 (2009).

Download references

Acknowledgements

We thank the past and future participants of the eMERGE Network projects. We thank M. O’Reilly for help with figure creation. We thank our All of Us Research Program colleagues, A. Ramirez, S. Lim, B. Mapes, A. Green and A. Musick, for providing their support and input throughout the ancestry calibration demonstration project lifecycle. We also thank the All of Us Science Committee and All of Us Steering Committee for their efforts evaluating and finalizing the approved demonstration projects. The All of Us Research Program would not be possible without the partnership of contributions made by its participants. To learn more about the All of Us Research Program’s research data repository, please visit https://www.researchallofus.org/ . This phase of the eMERGE Network was initiated and funded by the National Human Genome Research Institute through the following grants: U01HG011172 (Cincinnati Children’s Hospital Medical Center); U01HG011175 (Children’s Hospital of Philadelphia); U01HG008680 (Columbia University); U01HG011176 (Icahn School of Medicine at Mount Sinai); U01HG008685 (Mass General Brigham); U01HG006379 (Mayo Clinic); U01HG011169 (Northwestern University); U01HG011167 (University of Alabama at Birmingham); U01HG008657 (University of Washington); U01HG011181 (Vanderbilt University Medical Center); U01HG011166 (Vanderbilt University Medical Center serving as the Coordinating Center). The All of Us Research Program is supported by the National Institutes of Health, Office of the Director: Regional Medical Centers: 1 OT2 OD026549; 1 OT2 OD026554; 1 OT2 OD026557; 1 OT2 OD026556; 1 OT2 OD026550; 1 OT2 OD 026552; 1 OT2 OD026548; 1 OT2 OD026551; 1 OT2 OD026555; IAA#: AOD 16037; Federally Qualified Health Centers: 75N98019F01202; Data and Research Center: 1 OT2 OD35404; Biobank: 1 U24 OD023121; The Participant Center: U24 OD023176; Participant Technology Systems Center: 1 OT2 OD030043; Community Partners: 1 OT2 OD025277; 3 OT2 OD025315; 1 OT2 OD025337; 1 OT2 OD025276.

Author information

These authors contributed equally: Niall J. Lennon, Leah C. Kottyan.

Full lists of members and their affiliations appear in the Supplementary Information.

Authors and Affiliations

Broad Institute of MIT and Harvard, Cambridge, MA, USA

Niall J. Lennon, Christopher Kachulis, Marissa Fisher, Joel Hirschhorn, Candace Patterson, Maegan Harden, Joel N. Hirschhorn, Amit Khera, Katie Larkin, Edyta Malolepsza, Li McCarthy, Nihal Pai, Heidi L. Rehm & Diana M. Toledo

Cincinnati Children’s Hospital Medical Center, University of Cincinnati, Cincinnati, OH, USA

Leah C. Kottyan, Lisa J. Martin, Tesfaye B. Mersha, Bahram Namjou & Cynthia A. Prows

Icahn School of Medicine at Mount Sinai, New York, NY, USA

Noura S. Abul-Husn, Gillian Belbin, Clive Hoggart, Roelof A. J. Smit, Zhe Wang & Eimear E. Kenny

National Human Genome Research Institute, National Institutes of Health, Bethesda, MD, USA

Josh Arias, Sonja I. Berndt, Sonja Berndt, Teri A. Manolio & Robb Rowley

Vanderbilt University Medical Center, Nashville, TN, USA

Jennifer E. Below, Ellen Wright Clayton, Digna R. Velez Edwards, QiPing Feng, Jodell E. Linder, Jonathan D. Mosley, Josh F. Peterson, Dan M. Roden, Nataraja Sarma Vaitinadin, Wei-Qi Wei & Georgia L. Wiesner

Columbia University, New York, NY, USA

Wendy K. Chung, Atlas Khan, Krzysztof Kiryluk & Chunhua Weng

University of Alabama at Birmingham, Birmingham, AL, USA

James J. Cimino, Marguerite R. Irvin, Nita Limdi & Hemant Tiwari

Children’s Hospital of Philadelphia, Philadelphia, PA, USA

John J. Connolly, Joseph T. Glessner, Hakon Hakonarson & Margaret Harr

Tulane University, New Orleans, LA, USA

David R. Crosslin

University of Washington, Seattle, WA, USA

David R. Crosslin, Gail P. Jarvik, Elisabeth A. Rosenthal & David Veenstra

Mayo Clinic, Rochester, MI, USA

Ozan Dikilitas, Robert R. Freimuth, Iftikhar Kullo, Daniel J. Schaid & Johanna L. Smith

Mass General Brigham, Boston, MA, USA

Tian Ge, Elizabeth W. Karlson, James B. Meigs & Jordan W. Smoller

Northwestern University, Evanston, IL, USA

Adam S. Gordon, Yuan Luo, Elizabeth M. McNally, Lorenzo L. Pesce, Megan J. Puckelwartz, Konrad Teodor Sawicki & Theresa L. Walunas

Boston Children’s Hospital, Boston, MA, USA

Joel Hirschhorn & Joel N. Hirschhorn

Fred Hutchinson Cancer Center, Seattle, WA, USA

Li Hsu, Ulrike Peters & Minta Thomas

Novo Nordisk Foundation Center for Basic Metabolic Research, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark

Ruth Loos & Ruth J. F. Loos

The Charles Bronfman Institute for Personalized Medicine, Icahn School of Medicine at Mount Sinai, New York, NY, USA

National Institutes of Health, Bethesda, MD, USA

  • Anjene Musick

Nanjing Medical University, Nanjing, China

Xianyong Yin

You can also search for this author in PubMed   Google Scholar

The GIANT Consortium

  • Sonja Berndt
  • , Joel Hirschhorn
  • , Ruth Loos
  • , Roelof A. J. Smit
  •  & Xianyong Yin

The All of Us Research Program

Contributions.

N.J.L. and L.C.K. contributed equally. N.J.L., L.C.K., D.R.C., O.D., T.G., J.T.G., H.H., L.H., E.W.K., R.L., E.M.M., J.B.M., B.N., R.A.J.S. and E.E.K. were responsible for PRS development. N.J.L., L.C.K., D.R.C., O.D., T.G., J.T.G., A.S.G., H.H., L.H., E.W.K., J.E.L., R.L., Y.L., E.M., L.M., J.B.M., B.N., L.L.P., J.F.P., M.J.P., R.R., K.T.S., R.A.J.S., J.L.S., C.W., W.-Q.W. and E.E.K. conducted PRS evaluation. PRS selection, optimization and validation was done by N.J.L., L.C.K., D.R.C., Q.F., O.D., T.G., J.T.G., A.S.G., H.H., L.H., E.W.K., J.E.L., R.L., Y.L., E.M., L.M., B.N., L.L.P., M.J.P., R.R., K.T.S., R.A.J.S., J.L.S., G.L.W., C.W., W.-Q.W. and E.E.K. Population-based z- score calibration was done by N.J.L., C.K., T.G., B.N. and E.E.K. N.J.L., L.C.K., C.K., J. J. Connolly., D.R.C., T.G., E.M., B.N., M.J.P., R.A.J.S. and E.E.K. were responsible for PRS transfer and implementation. Assessment of the first 2,500 participants was done by N.J.L., C.K., Q.F., B.N. and E.E.K. N.J.L., L.C.K., C.K., J.E.L., T.A.M., J.W.S., J.F.P. and E.E.K. wrote the first draft of the paper. All authors reviewed the paper.

Corresponding author

Correspondence to Niall J. Lennon .

Ethics declarations

Competing interests.

N.S.A.-H. is an employee and equity holder of 23andMe; serves as a scientific advisory board member for Allelica, Inc; received personal fees from Genentech Inc, Allelica Inc, and 23andMe; received research funding from Akcea Therapeutics; and was previously employed by Regeneron Pharmaceuticals. E.E.K. received personal fees from Illumina Inc, 23andMe and Regeneron Pharmaceuticals and serves as a scientific advisory board member for Encompass Bioscience, Foresite Labs and Galateo Bio. J.N.H. has equity in Camp4 Therapeutics and has been a consultant to Amgen, AstraZeneca, Cytokinetics, PepGen, Pfizer and Tenaya Therapeutics and is the founder of Ikaika Therapeutics. J.F.P. is a paid consultant for Natera Inc. A. Khera. is an employee of Verve Therapeutics. N.L. received personal fees from Illumina Inc and is a scientific advisory board member for FYR Diagnostics. J.F.P. is a consultant for Myome. D.V. is a consultant for Illumina and has grant support from GeneDx. T.L.W. has grant funding from Gilead Sciences, Inc. The other authors declare no competing interests.

Peer review

Peer review information.

Nature Medicine thanks Cathryn Lewis, Lili Milani, Bjarni Vilhjálmsson and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Anna Maria Ranzoni, in collaboration with the Nature Medicine team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended data fig. 1 case-control prs histograms..

Histograms of T2D PRS scores for case and control samples in the eMERGE I-III dataset.

Extended Data Fig. 2 Representation of the genetic ancestry admixture composition of both the Test and Training cohorts.

The x-axis represents individuals within the cohorts and the color-coding highlights the proportion of genetic admixture observed.

Extended Data Fig. 3 Calibrated z-scores.

The distributions of calibrated z-scores in the test cohort when the training cohort parameters are applied.

Extended Data Fig. 4 Hypercholesterolemia PRS calibrated z-scores of training cohort.

Note the improvement when an ancestry dependent variance is used over a constant variance method.

Extended Data Fig. 5 PRS z-score as a function of African Admixture Fraction, for individuals of African ancestry.

In the ‘Bucketing’ method, a z-score is calculated by comparing to the mean and variance of all individuals of African ancestry in the cohort. The ‘PCA Calibrated’ method is the method described above. Note the dependence on admixture fraction in the ‘Bucketing’ method, which has been removed in the ‘PCA Calibrated’ method.

Supplementary information

Supplementary information.

Sample clinical report and list of consortia members.

Reporting Summary

Supplementary table 1.

Supplementary Tables 1–3 (tabs in a single worksheet).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Lennon, N.J., Kottyan, L.C., Kachulis, C. et al. Selection, optimization and validation of ten chronic disease polygenic risk scores for clinical implementation in diverse US populations. Nat Med 30 , 480–487 (2024). https://doi.org/10.1038/s41591-024-02796-z

Download citation

Received : 25 May 2023

Accepted : 02 January 2024

Published : 19 February 2024

Issue Date : February 2024

DOI : https://doi.org/10.1038/s41591-024-02796-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

what are references on a research paper

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base
  • Research paper
  • Research Paper Format | APA, MLA, & Chicago Templates

Research Paper Format | APA, MLA, & Chicago Templates

Published on November 19, 2022 by Jack Caulfield . Revised on January 20, 2023.

The formatting of a research paper is different depending on which style guide you’re following. In addition to citations , APA, MLA, and Chicago provide format guidelines for things like font choices, page layout, format of headings and the format of the reference page.

Scribbr offers free Microsoft Word templates for the most common formats. Simply download and get started on your paper.

APA |  MLA | Chicago author-date | Chicago notes & bibliography

  • Generate an automatic table of contents
  • Generate a list of tables and figures
  • Ensure consistent paragraph formatting
  • Insert page numbering

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

upload-your-document-ai-proofreader

Table of contents

Formatting an apa paper, formatting an mla paper, formatting a chicago paper, frequently asked questions about research paper formatting.

The main guidelines for formatting a paper in APA Style are as follows:

  • Use a standard font like 12 pt Times New Roman or 11 pt Arial.
  • Set 1 inch page margins.
  • Apply double line spacing.
  • If submitting for publication, insert a APA running head on every page.
  • Indent every new paragraph ½ inch.

Watch the video below for a quick guide to setting up the format in Google Docs.

The image below shows how to format an APA Style title page for a student paper.

APA title page - student version (7th edition)

Running head

If you are submitting a paper for publication, APA requires you to include a running head on each page. The image below shows you how this should be formatted.

APA running head (7th edition)

For student papers, no running head is required unless you have been instructed to include one.

APA provides guidelines for formatting up to five levels of heading within your paper. Level 1 headings are the most general, level 5 the most specific.

APA headings (7th edition)

Reference page

APA Style citation requires (author-date) APA in-text citations throughout the text and an APA Style reference page at the end. The image below shows how the reference page should be formatted.

APA reference page (7th edition)

Note that the format of reference entries is different depending on the source type. You can easily create your citations and reference list using the free APA Citation Generator.

Generate APA citations for free

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

what are references on a research paper

Try for free

The main guidelines for writing an MLA style paper are as follows:

  • Use an easily readable font like 12 pt Times New Roman.
  • Use title case capitalization for headings .

Check out the video below to see how to set up the format in Google Docs.

On the first page of an MLA paper, a heading appears above your title, featuring some key information:

  • Your full name
  • Your instructor’s or supervisor’s name
  • The course name or number
  • The due date of the assignment

MLA heading

Page header

A header appears at the top of each page in your paper, including your surname and the page number.

MLA page header

Works Cited page

MLA in-text citations appear wherever you refer to a source in your text. The MLA Works Cited page appears at the end of your text, listing all the sources used. It is formatted as shown below.

The format of the MLA Works Cited page

You can easily create your MLA citations and save your Works Cited list with the free MLA Citation Generator.

Generate MLA citations for free

The main guidelines for writing a paper in Chicago style (also known as Turabian style) are:

  • Use a standard font like 12 pt Times New Roman.
  • Use 1 inch margins or larger.
  • Place page numbers in the top right or bottom center.

Format of a Chicago Style paper

Chicago doesn’t require a title page , but if you want to include one, Turabian (based on Chicago) presents some guidelines. Lay out the title page as shown below.

Example of a Chicago Style title page

Bibliography or reference list

Chicago offers two citation styles : author-date citations plus a reference list, or footnote citations plus a bibliography. Choose one style or the other and use it consistently.

The reference list or bibliography appears at the end of the paper. Both styles present this page similarly in terms of formatting, as shown below.

Chicago bibliography

To format a paper in APA Style , follow these guidelines:

  • Use a standard font like 12 pt Times New Roman or 11 pt Arial
  • Set 1 inch page margins
  • Apply double line spacing
  • Include a title page
  • If submitting for publication, insert a running head on every page
  • Indent every new paragraph ½ inch
  • Apply APA heading styles
  • Cite your sources with APA in-text citations
  • List all sources cited on a reference page at the end

The main guidelines for formatting a paper in MLA style are as follows:

  • Use an easily readable font like 12 pt Times New Roman
  • Include a four-line MLA heading on the first page
  • Center the paper’s title
  • Use title case capitalization for headings
  • Cite your sources with MLA in-text citations
  • List all sources cited on a Works Cited page at the end

The main guidelines for formatting a paper in Chicago style are to:

  • Use a standard font like 12 pt Times New Roman
  • Use 1 inch margins or larger
  • Place page numbers in the top right or bottom center
  • Cite your sources with author-date citations or Chicago footnotes
  • Include a bibliography or reference list

To automatically generate accurate Chicago references, you can use Scribbr’s free Chicago reference generator .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Caulfield, J. (2023, January 20). Research Paper Format | APA, MLA, & Chicago Templates. Scribbr. Retrieved February 23, 2024, from https://www.scribbr.com/research-paper/research-paper-format/

Is this article helpful?

Jack Caulfield

Jack Caulfield

Other students also liked, apa format for academic papers and essays, mla format for academic papers and essays, chicago style format for papers | requirements & examples, what is your plagiarism score.

IMAGES

  1. APA Basics: Fundamentals of Formatting Research Papers in APA Style

    what are references on a research paper

  2. Essay Basics: Format a References Page in APA Style

    what are references on a research paper

  3. How to Do a Reference Page for a Research Paper: 11 Steps

    what are references on a research paper

  4. Research Paper Format

    what are references on a research paper

  5. APA Reference Page Examples and Format Guide

    what are references on a research paper

  6. Harvard Style

    what are references on a research paper

VIDEO

  1. Citing Article Cross Referencing within Research Paper by Dr. Yashwant Chapke

  2. How to Read Research Paper Quickly

  3. Secret To Writing A Research Paper

  4. BASICS OF RESEARCH AND ACADEMIC INTEGRITY- DAY

  5. cs101 Word Project 2 Part 2 of 3 2023 09 13 09 25 14

  6. Learn Zotero Citation & Reference Management

COMMENTS

  1. References in Research

    Definition: References in research are a list of sources that a researcher has consulted or cited while conducting their study. They are an essential component of any academic work, including research papers, theses, dissertations, and other scholarly publications. Types of References

  2. Reference List: Basic Rules

    Reference List: Basic Rules. This resourse, revised according to the 7 th edition APA Publication Manual, offers basic guidelines for formatting the reference list at the end of a standard APA research paper. Most sources follow fairly straightforward rules. However, because sources obtained from academic journals carry special weight in research writing, these sources are subject to special ...

  3. How to Cite Sources

    The most commonly used citation styles are APA and MLA. The free Scribbr Citation Generator is the quickest way to cite sources in these styles. Simply enter the URL, DOI, or title, and we'll generate an accurate, correctly formatted citation. Generate accurate citations with Scribbr Instantly correct all language mistakes in your text

  4. Citations, References and Bibliography in Research Papers [Beginner's

    Reference in research papers: A reference is a detailed description of the source of information that you want to give credit to via a citation. The references in research papers are usually in the form of a list at the end of the paper.

  5. Reference List: Common Reference List Examples

    These instructional pages offer examples of reference list entries for different types of sources as well as guidance on the variations for citing online materials using doi numbers and URLs. ... they present original research and are often useful for research papers. Sometimes, researchers refer to these types of reports as gray literature ...

  6. How To Write Your References Quickly And Easily

    In general, a reference will include authors' names and initials, the title of the article, name of the journal, volume and issue, date, page numbers and DOI. On ScienceDirect, articles are linked to their original source (if also published on ScienceDirect) or to their Scopus record, so including the DOI can help link to the correct article.

  7. How to Cite in APA Format (7th edition)

    In-text citations are brief references in the running text that direct readers to the reference entry at the end of the paper. You include them every time you quote or paraphrase someone else's ideas or words to avoid plagiarism.

  8. In-Text Citations: The Basics

    When using APA format, follow the author-date method of in-text citation. This means that the author's last name and the year of publication for the source should appear in the text, like, for example, (Jones, 1998). One complete reference for each source should appear in the reference list at the end of the paper.

  9. Citation Styles Guide

    A citation style is a set of guidelines on how to cite sources in your academic writing. You always need a citation whenever you quote, paraphrase, or summarize a source to avoid plagiarism. How you present these citations depends on the style you follow. Scribbr's citation generator can help!

  10. A Quick Guide to Referencing

    A reference list or bibliography at the end listing full details of all your sources. The most common method of referencing in UK universities is Harvard style, which uses author-date citations in the text. Our free Harvard Reference Generator automatically creates accurate references in this style.

  11. How to write references in research papers| Researcher.Life

    One of the the most common types of references in research papers is other research papers published in journals; other common sources include technical reports, handbooks, presentations at conferences, and books. Increasingly, the sources in research papers are digital and include web pages, databases, blog posts, and even tweets and emails.

  12. A Quick Guide to Harvard Referencing

    When you cite a source with up to three authors, cite all authors' names. For four or more authors, list only the first name, followed by ' et al. ': Number of authors. In-text citation example. 1 author. (Davis, 2019) 2 authors. (Davis and Barrett, 2019) 3 authors.

  13. What Types of References Are Appropriate?

    What Types of References Are Appropriate? When writing a research paper, there are many different types of sources that you might consider citing. Which are appropriate? Which are less appropriate? Here we discuss the different types of sources that you may wish to use when working on a research paper.

  14. How To Cite a Research Paper in 2024: Citation Styles Guide

    As such, this article should serve as your guide on how to reference a research paper in popular formats: APA, MLA, Chicago/Turabian, and the IEEE style. A list of digital tools that can make citation easier and a quick tutorial will also be provided.

  15. Why and when to reference

    Referencing is an important part of academic work. It puts your work in context, demonstrates the breadth and depth of your research, and acknowledges other people's work. You should reference whenever you use someone else's idea. View video using Microsoft Stream (link opens in a new window, available for University members only) These ...

  16. References: How to Cite and List Correctly

    For an original article, thesis, or review, use the term ' References ', as the list of sources comprises of complete bibliographic details of every publication. Some prefer to use ' Works cited ' or ' Sources cited ' to include non-traditional sources such as online sources, video recordings, and personal communications (e-mail, interview, etc.).

  17. 5 Ways to Cite a Research Paper

    Referencing a Research Paper In APA style, cite the paper: Last Name, First Initial. (Year). Title. Publisher. In Chicago style, cite the paper: Last Name, First Name. "Title." Publisher, Year. In MLA style, cite the paper: Last Name, First Name. "Title." Publisher. Year. Citation Help APA Research Paper Citation MLA Research Paper Citation

  18. How to Write a Research Paper

    A research paper is a piece of academic writing that provides analysis, interpretation, and argument based on in-depth independent research. Research papers are similar to academic essays, but they are usually longer and more detailed assignments, designed to assess not only your writing skills but also your skills in scholarly research.

  19. Formatting References for Scientific Manuscripts

    The Citation Style Language (CSL) is an XML-based computer language developed to standardize formatting of citations and references in manuscripts submitting to journals. They are text application editable files which are imported into RMs. An increasing number of RMs use CSL to help users format their list of references according to individual ...

  20. How to Cite Research Paper

    Research paper: In-text citation: Use superscript numbers to cite sources in the text, e.g., "Previous research has shown that^1,2,3…". Reference list citation: Format: Author (s). Title of paper. In: Editor (s). Title of the conference proceedings. Place of publication: Publisher; Year of publication. Page range.

  21. Roles of references in research papers

    References (summarised from here ): demonstrate the foundation of the study. support the novelty and value of the study. link one study to others creating a web of knowledge that carries meaning. allows researchers to identify work as relevant in general and relevant to them. create values that are internal to science (e.g., relevance, credit).

  22. 'It depends': what 86 systematic reviews tell us about what strategies

    To identify potentially relevant peer-reviewed research papers, we developed a comprehensive systematic literature search strategy based on the terms used in the Grimshaw et al. ... Reference lists of relevant papers were also examined. We uploaded the results using EPPI-Reviewer, a web-based tool that facilitated semi-automation of the ...

  23. [2402.05929] An Interactive Agent Foundation Model

    The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide range of applications. We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and tasks. Our training ...

  24. Selection, optimization and validation of ten chronic disease ...

    a,Timeline and process for selection, evaluation, optimization, transfer, validation and implementation of the clinical PRS test pipeline.Dashed lines represent pivotal moments in the progression ...

  25. Setting Up the APA Reference Page

    References are ordered alphabetically by the first author's last name. If the author is unknown, order the reference entry by the first meaningful word of the title (ignoring articles: "the", "a", or "an"). Word processors like Word or Google Docs and citation generators can usually order the reference list automatically.

  26. Solving the puzzle of Long Covid

    From an extensive body of mechanistic research in people affected by Long Covid, there appear to be multiple potential pathogenic pathways, including persistence of the virus or its components in tissue reservoirs; autoimmune or an unchecked, dysregulated immune response; mitochondrial dysfunction; vascular (endothelial) and/or neuronal inflammation; and microbiome dysbiosis ().

  27. Research Paper Format

    Research paper Research Paper Format | APA, MLA, & Chicago Templates Research Paper Format | APA, MLA, & Chicago Templates Published on November 19, 2022 by Jack Caulfield . Revised on January 20, 2023. The formatting of a research paper is different depending on which style guide you're following.