paper research methods

How to write the Methods section of a research paper

Dr. Dhriti Bhattacharyya

The Methods section of a research article is like a roadmap leading to the core of the research, guiding the readers through the actual journey the authors took to reach their destination. In the manuscript, this section contains the essential details for other scientists to replicate the experiments of the study and help the common readers to understand the study better.

In this article, we will share some tips to make the Methods section of your manuscript interesting and informative. While the article uses examples mostly from the biomedical and clinical research studies, authors from other fields too would find the tips useful for preparing their next manuscript.

Break ice between the readers and the Methods section

First, let’s ponder over the issue of the perception of boredom we often associate with the Methods section of an article. It may be the names of the reagents and instruments, separated by some numbers in terms of some concentrations or the technical terminologies that make the reading a heavy-duty task. Listed below are some useful ways of breaking the ice between the Methods section and the readers:

1. Explanation : Usually, each paragraph or subsection of the Methods section talks about a specific experiment. Early in each paragraph, explain the rationale behind your choices of that particular experiment.; for example, why you used a certain compound, a specific strain of mice as the experimental model or the particular concentration of that key reagent.

For clinical research, providing a detailed rationale for selecting the exclusion or inclusion criteria can be a good idea to present early in the Methods section. If you took a conventional or widely used method, you certainly don’t need to appear stating the obvious, but for less conventional approaches sharing your reasoning of the study design instantly makes the readers curious and engaged with your paper.

2. Visual presentation : To help the readers follow the study design or methodology better, visual elements like the schematic diagram, flowchart, and table can be used in this section. They help in breaking the monotony and making the absorption of complex information easy.

The dos and don’ts of writing the Methods section

Secondly, the information in the methods section is closely scrutinized by the journal editors and peer reviewers to assess whether the most appropriate technique was used to reach your research goal. While every detail of your experiment need not be included, the essential and critical steps should be well described to receive a positive peer review.

The essential do’s and don’ts of writing a technically sound Methods section:

1. Adhere to the specific guidelines: Read the author’s instruction section of your target journal carefully and follow the specific instructions. For example, the heading of the section “Materials and Methods” may need to be changed to “Patients and the Method” to follow the guidelines of your target journal or the name of the institutes could be omitted for the journals that do not prefer open-label reporting. Also, you may be expected to follow a particular style guideline like the one published by the American Psychological Association while writing the Methods section.

Biomedical researchers would benefit from using the checklists for different study types to ensure the essential details are included in the Methods. Some of the standardized and widely referred checklists include the ones for randomized clinical trials CONSORT (Consolidated Standards of Reporting Trials), cohort, case-control, cross‐sectional studies STROBE (STrengthening the Reporting of OBservational studies in Epidemiology), diagnostic accuracy STARD (STAndards for the Reporting of Diagnostic accuracy studies), systematic reviews and meta‐analyses PRISMA (Preferred Reporting Items for Systematic reviews and Meta‐Analyses), and Case reports CARE (CAse REport).

2. Structure the section so that it tells the story of your research : All the experiments should be presented in a logical manner that helps the reader retrace the gradual and development and nuances of the study. A useful way of achieving this is to describe the methods in a chronological order of the experiments. For example: for a clinical trial, you may start with the setting and time of the study ( the beginning and termination dates of the study) , followed by the details of the patient recruitment ( Number of subjects/patients etc.) , study design (prospective, retrospective or other), randomization (if any), assigning into groups, intervention, and describing the techniques used to collect, measure, and analyse data.

3. Follow the order of the results: To improve the readability and flow of your manuscript, match the order of specific methods to the order of the results that were achieved using those methods.

4. Use subheadings: Dividing the Methods section in terms of the experiments helps the reader to follow the section better. You may write the specific objective of each experiment as a subheading. Alternatively, if applicable, the name of each experiment can also be used as subheading.

5. Provide all details meticulously: Provide the details that you considered while designing the study or collecting the data because the smallest variations in these steps may affect the results and interpretation of their significance. When employing the outcome measures, the readers would like to know the information regarding validity and reliability. The correct way of reporting the reliability and the validity depends on the specific research design. Usually, information from existing literature is presented to support for the reliability and the validity of a measure.

Carefully describe the materials, equipment (like testing instruments and technical equipment), or stimuli used in the experiment. If your study involved a survey or any psychological assessment, mention the questionnaire, scoring methods, and validation of scales with every possible detail.

Also, be careful about one common manuscript error i.e. not mentioning the sample size estimation (whenever relevant). Although the estimated sample size is computed before the actual study starts, it helps the reader assess the expected change in the outcome variables and the number of subjects needed to detect that change within a certain confidence range. Similarly, mentioning power calculation is a critical point to be mentioned in the Methods section.

6. Mention the ethical approval: If relevant, early in the Methods section mention whether your study was approved by the ethics committee or institutional review board, and whether you have received oral/ written informed consent from the patients or the guardians.

7. Specify the variables : Clearly mention not only the control variables, independent variables, dependent variables but also if there were any extraneous variables that might influence the result of your study. For example, in a tutorial on learning how to write ‘Research Methodology’, one group is provided with a traditional text while the other group is provided with an interactive online tool. However, if some participants already have prior knowledge of ‘how to write the Methods section’, this pre-knowledge will act as an extraneous variable.

8. Statistical analysis: In this section, describe all statistical tests, levels of significance, and software packages used to conduct the statistical analysis. You may also consult the biostatistician of your team to receive help to write this section . Don’t forget to indicate if the recommendations of a knowledgeable and experienced statistician were considered. Finally, it is important to provide the justification of the preferred statistical method used in the study. For example, why the author is using a one-tailed or two-tailed analysis.

1. Do not describe well-known methods in detail: For the sake of brevity, avoid listing the details of the experiments that are widely used or already published in numerous articles in your field of research. Instead, mention and cite the specific experiment and mention that the referred process was followed. However, if you have modified the standard process to meet the specific aim of your study, do describe the modifications and the reasons for those in sufficient detail.

2. Do not provide unnecessary details: Avoid unnecessary details that are not relevant to the result of the experiment. For example, you need not mention trivial details such as the color of the bucket that held the ice. Try to stick only to the details that are relevant and have an impact on your study.

3. Do not discuss the pros and cons of other methods: While it may be tempting to discuss the reasons why you did not use a particular method or how your chosen method is superior to others, save these details for the Discussion section. Utilize the Methods section only to mention the details of the methods you chose.

To summarize all the tips stated above, the Methods section of an ideal manuscript aims to share the scientific knowledge with transparency and also establishes the robustness of the study. I hope that this article helps you to reach the goal of writing a perfect manuscript!

The correct way to report p values

How to choose the research methodology best suited for your study

Planning to Write

Manuscript structure: How to convey your most important ideas through…

How to write the Methods section of a research paper 8 min read

10 Tips on how to write an effective research grant proposal 9 min read

11 Commonly confused elements of a research paper 16 min read

Manuscript structure: How to convey your most important ideas through your paper 6 min read

4 Step approach to writing the Introduction section of a research paper 10 min read

Recent Searches

Review paper
Responding to reviewer comments
Predatory publishers
Scope and delimitations
Open access
Plagiarism in research
Journal selection tips
Editor assigned
Types of articles
"Reject and Resubmit" status
Decision in process
Conflict of interest

When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

PLOS Biology
PLOS Climate
PLOS Complex Systems
PLOS Computational Biology
PLOS Digital Health
PLOS Genetics
PLOS Global Public Health
PLOS Medicine
PLOS Mental Health
PLOS Neglected Tropical Diseases
PLOS Pathogens
PLOS Sustainability and Transformation
PLOS Collections
How to Write Your Methods

Ensure understanding, reproducibility and replicability

What should you include in your methods section, and how much detail is appropriate?

Why Methods Matter

The methods section was once the most likely part of a paper to be unfairly abbreviated, overly summarized, or even relegated to hard-to-find sections of a publisher’s website. While some journals may responsibly include more detailed elements of methods in supplementary sections, the movement for increased reproducibility and rigor in science has reinstated the importance of the methods section. Methods are now viewed as a key element in establishing the credibility of the research being reported, alongside the open availability of data and results.

A clear methods section impacts editorial evaluation and readers’ understanding, and is also the backbone of transparency and replicability.

For example, the Reproducibility Project: Cancer Biology project set out in 2013 to replicate experiments from 50 high profile cancer papers, but revised their target to 18 papers once they understood how much methodological detail was not contained in the original papers.

What to include in your methods section

What you include in your methods sections depends on what field you are in and what experiments you are performing. However, the general principle in place at the majority of journals is summarized well by the guidelines at PLOS ONE : “The Materials and Methods section should provide enough detail to allow suitably skilled investigators to fully replicate your study. ” The emphases here are deliberate: the methods should enable readers to understand your paper, and replicate your study. However, there is no need to go into the level of detail that a lay-person would require—the focus is on the reader who is also trained in your field, with the suitable skills and knowledge to attempt a replication.

A constant principle of rigorous science

A methods section that enables other researchers to understand and replicate your results is a constant principle of rigorous, transparent, and Open Science. Aim to be thorough, even if a particular journal doesn’t require the same level of detail . Reproducibility is all of our responsibility. You cannot create any problems by exceeding a minimum standard of information. If a journal still has word-limits—either for the overall article or specific sections—and requires some methodological details to be in a supplemental section, that is OK as long as the extra details are searchable and findable .

Imagine replicating your own work, years in the future

As part of PLOS’ presentation on Reproducibility and Open Publishing (part of UCSF’s Reproducibility Series ) we recommend planning the level of detail in your methods section by imagining you are writing for your future self, replicating your own work. When you consider that you might be at a different institution, with different account logins, applications, resources, and access levels—you can help yourself imagine the level of specificity that you yourself would require to redo the exact experiment. Consider:

Which details would you need to be reminded of?
Which cell line, or antibody, or software, or reagent did you use, and does it have a Research Resource ID (RRID) that you can cite?
Which version of a questionnaire did you use in your survey?
Exactly which visual stimulus did you show participants, and is it publicly available?
What participants did you decide to exclude?
What process did you adjust, during your work?

Tip: Be sure to capture any changes to your protocols

You yourself would want to know about any adjustments, if you ever replicate the work, so you can surmise that anyone else would want to as well. Even if a necessary adjustment you made was not ideal, transparency is the key to ensuring this is not regarded as an issue in the future. It is far better to transparently convey any non-optimal methods, or methodological constraints, than to conceal them, which could result in reproducibility or ethical issues downstream.

Visual aids for methods help when reading the whole paper

Consider whether a visual representation of your methods could be appropriate or aid understanding your process. A visual reference readers can easily return to, like a flow-diagram, decision-tree, or checklist, can help readers to better understand the complete article, not just the methods section.

Ethical Considerations

In addition to describing what you did, it is just as important to assure readers that you also followed all relevant ethical guidelines when conducting your research. While ethical standards and reporting guidelines are often presented in a separate section of a paper, ensure that your methods and protocols actually follow these guidelines. Read more about ethics .

Existing standards, checklists, guidelines, partners

While the level of detail contained in a methods section should be guided by the universal principles of rigorous science outlined above, various disciplines, fields, and projects have worked hard to design and develop consistent standards, guidelines, and tools to help with reporting all types of experiment. Below, you’ll find some of the key initiatives. Ensure you read the submission guidelines for the specific journal you are submitting to, in order to discover any further journal- or field-specific policies to follow, or initiatives/tools to utilize.

Tip: Keep your paper moving forward by providing the proper paperwork up front

Be sure to check the journal guidelines and provide the necessary documents with your manuscript submission. Collecting the necessary documentation can greatly slow the first round of peer review, or cause delays when you submit your revision.

Randomized Controlled Trials – CONSORT The Consolidated Standards of Reporting Trials (CONSORT) project covers various initiatives intended to prevent the problems of inadequate reporting of randomized controlled trials. The primary initiative is an evidence-based minimum set of recommendations for reporting randomized trials known as the CONSORT Statement .

Systematic Reviews and Meta-Analyses – PRISMA The Preferred Reporting Items for Systematic Reviews and Meta-Analyses ( PRISMA ) is an evidence-based minimum set of items focusing on the reporting of reviews evaluating randomized trials and other types of research.

Research using Animals – ARRIVE The Animal Research: Reporting of In Vivo Experiments ( ARRIVE ) guidelines encourage maximizing the information reported in research using animals thereby minimizing unnecessary studies. (Original study and proposal , and updated guidelines , in PLOS Biology .)

Laboratory Protocols Protocols.io has developed a platform specifically for the sharing and updating of laboratory protocols , which are assigned their own DOI and can be linked from methods sections of papers to enhance reproducibility. Contextualize your protocol and improve discovery with an accompanying Lab Protocol article in PLOS ONE .

Consistent reporting of Materials, Design, and Analysis – the MDAR checklist A cross-publisher group of editors and experts have developed, tested, and rolled out a checklist to help establish and harmonize reporting standards in the Life Sciences . The checklist , which is available for use by authors to compile their methods, and editors/reviewers to check methods, establishes a minimum set of requirements in transparent reporting and is adaptable to any discipline within the Life Sciences, by covering a breadth of potentially relevant methodological items and considerations. If you are in the Life Sciences and writing up your methods section, try working through the MDAR checklist and see whether it helps you include all relevant details into your methods, and whether it reminded you of anything you might have missed otherwise.

Summary Writing tips

The main challenge you may find when writing your methods is keeping it readable AND covering all the details needed for reproducibility and replicability. While this is difficult, do not compromise on rigorous standards for credibility!

Keep in mind future replicability, alongside understanding and readability.
Follow checklists, and field- and journal-specific guidelines.
Consider a commitment to rigorous and transparent science a personal responsibility, and not just adhering to journal guidelines.
Establish whether there are persistent identifiers for any research resources you use that can be specifically cited in your methods section.
Deposit your laboratory protocols in Protocols.io, establishing a permanent link to them. You can update your protocols later if you improve on them, as can future scientists who follow your protocols.
Consider visual aids like flow-diagrams, lists, to help with reading other sections of the paper.
Be specific about all decisions made during the experiments that someone reproducing your work would need to know.

Don’t

Summarize or abbreviate methods without giving full details in a discoverable supplemental section.
Presume you will always be able to remember how you performed the experiments, or have access to private or institutional notebooks and resources.
Attempt to hide constraints or non-optimal decisions you had to make–transparency is the key to ensuring the credibility of your research.
How to Write a Great Title
How to Write an Abstract
How to Report Statistics
How to Write Discussions and Conclusions
How to Edit Your Work

The contents of the Peer Review Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

The contents of the Writing Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

There’s a lot to consider when deciding where to submit your work. Learn how to choose a journal that will help your study reach its audience, while reflecting your values as a researcher…

Locations and Hours
UCLA Library
Research Guides
Research Tips and Tools

Advanced Research Methods

Writing the research paper.

What Is Research?
Library Research
Writing a Research Proposal

Before Writing the Paper

Methods, thesis, and hypothesis, clarity, precision, and academic expression, format your paper, typical problems, a few suggestions, avoid plagiarism.

Presenting the Research Paper
Try to find a subject that really interests you.
While you explore the topic, narrow or broaden your target and focus on something that gives the most promising results.
Don't choose a huge subject if you have to write a 3 page long paper, and broaden your topic sufficiently if you have to submit at least 25 pages.
Consult your class instructor (and your classmates) about the topic.
Find primary and secondary sources in the library.
Read and critically analyse them.
Take notes.
Compile surveys, collect data, gather materials for quantitative analysis (if these are good methods to investigate the topic more deeply).
Come up with new ideas about the topic. Try to formulate your ideas in a few sentences.
Review your notes and other materials and enrich the outline.
Try to estimate how long the individual parts will be.
Do others understand what you want to say?
Do they accept it as new knowledge or relevant and important for a paper?
Do they agree that your thoughts will result in a successful paper?
Qualitative: gives answers on questions (how, why, when, who, what, etc.) by investigating an issue
Quantitative:requires data and the analysis of data as well
the essence, the point of the research paper in one or two sentences.
a statement that can be proved or disproved.
Be specific.
Avoid ambiguity.
Use predominantly the active voice, not the passive.
Deal with one issue in one paragraph.
Be accurate.
Double-check your data, references, citations and statements.

Academic Expression

Don't use familiar style or colloquial/slang expressions.
Write in full sentences.
Check the meaning of the words if you don't know exactly what they mean.
Avoid metaphors.
Almost the rough content of every paragraph.
The order of the various topics in your paper.
On the basis of the outline, start writing a part by planning the content, and then write it down.
Put a visible mark (which you will later delete) where you need to quote a source, and write in the citation when you finish writing that part or a bigger part.
Does the text make sense?
Could you explain what you wanted?
Did you write good sentences?
Is there something missing?
Check the spelling.
Complete the citations, bring them in standard format.

Use the guidelines that your instructor requires (MLA, Chicago, APA, Turabian, etc.).

Adjust margins, spacing, paragraph indentation, place of page numbers, etc.
Standardize the bibliography or footnotes according to the guidelines.

EndNote and EndNote Basic by UCLA Library Last Updated Aug 29, 2023 2028 views this year
Zotero by UCLA Library Last Updated Jan 18, 2023 1305 views this year

(Based on English Composition 2 from Illinois Valley Community College):

Weak organization
Poor support and development of ideas
Weak use of secondary sources
Excessive errors
Stylistic weakness

When collecting materials, selecting research topic, and writing the paper:

Be systematic and organized (e.g. keep your bibliography neat and organized; write your notes in a neat way, so that you can find them later on.
Use your critical thinking ability when you read.
Write down your thoughts (so that you can reconstruct them later).
Stop when you have a really good idea and think about whether you could enlarge it to a whole research paper. If yes, take much longer notes.
When you write down a quotation or summarize somebody else's thoughts in your notes or in the paper, cite the source (i.e. write down the author, title, publication place, year, page number).
If you quote or summarize a thought from the internet, cite the internet source.
Write an outline that is detailed enough to remind you about the content.
Read your paper for yourself or, preferably, somebody else.
When you finish writing, check the spelling;
Use the citation form (MLA, Chicago, or other) that your instructor requires and use it everywhere.

Plagiarism : somebody else's words or ideas presented without citation by an author

Cite your source every time when you quote a part of somebody's work.
Cite your source every time when you summarize a thought from somebody's work.
Cite your source every time when you use a source (quote or summarize) from the Internet.

Consult the Citing Sources research guide for further details.

<< Previous: Writing a Research Proposal
Next: Presenting the Research Paper >>
Last Updated: Sep 18, 2023 4:52 PM
URL: https://guides.library.ucla.edu/research-methods

Research Methods: What are research methods?

What are research methods?
Searching specific databases

What are research methods

Research methods are the strategies, processes or techniques utilized in the collection of data or evidence for analysis in order to uncover new information or create better understanding of a topic.

There are different types of research methods which use different tools for data collection.

Types of research

Qualitative Research
Quantitative Research
Mixed Methods Research

Qualitative Research gathers data about lived experiences, emotions or behaviours, and the meanings individuals attach to them. It assists in enabling researchers to gain a better understanding of complex concepts, social interactions or cultural phenomena. This type of research is useful in the exploration of how or why things have occurred, interpreting events and describing actions.

Quantitative Research gathers numerical data which can be ranked, measured or categorised through statistical analysis. It assists with uncovering patterns or relationships, and for making generalisations. This type of research is useful for finding out how many, how much, how often, or to what extent.

Mixed Methods Research integrates both Q ualitative and Quantitative Research . It provides a holistic approach combining and analysing the statistical data with deeper contextualised insights. Using Mixed Methods also enables Triangulation, or verification, of the data from two or more sources.

Finding Mixed Methods research in the Databases

“mixed model*” OR “mixed design*” OR “multiple method*” OR multimethod* OR triangulat*

Data collection tools

Sage research methods.

SAGE research methods online This link opens in a new window Research methods tool to help researchers gather full-text resources, design research projects, understand a particular method and write up their research. Includes access to collections of video, business cases and eBooks,

Help and Information

Next: Finding qualitative research >>
Last Updated: Mar 14, 2023 12:34 PM
URL: https://libguides.newcastle.edu.au/researchmethods

Organizing Academic Research Papers: 6. The Methodology

Purpose of Guide
Design Flaws to Avoid
Glossary of Research Terms
Narrowing a Topic Idea
Broadening a Topic Idea
Extending the Timeliness of a Topic Idea
Academic Writing Style
Choosing a Title
Making an Outline
Paragraph Development
Executive Summary
Background Information
The Research Problem/Question
Theoretical Framework
Citation Tracking
Content Alert Services
Evaluating Sources
Primary Sources
Secondary Sources
Tertiary Sources
What Is Scholarly vs. Popular?
Qualitative Methods
Quantitative Methods
Using Non-Textual Elements
Limitations of the Study
Common Grammar Mistakes
Avoiding Plagiarism
Footnotes or Endnotes?
Further Readings
Annotated Bibliography
Dealing with Nervousness
Using Visual Aids
Grading Someone Else's Paper
How to Manage Group Projects
Multiple Book Review Essay
Reviewing Collected Essays
About Informed Consent
Writing Field Notes
Writing a Policy Memo
Writing a Research Proposal
Acknowledgements

The methods section of a research paper provides the information by which a study’s validity is judged. The method section answers two main questions: 1) How was the data collected or generated? 2) How was it analyzed? The writing should be direct and precise and written in the past tense.

Importance of a Good Methodology Section

You must explain how you obtained and analyzed your results for the following reasons:

Readers need to know how the data was obtained because the method you choose affects the results and, by extension, how you likely interpreted those results.
Methodology is crucial for any branch of scholarship because an unreliable method produces unreliable results and it misappropriates interpretations of findings .
In most cases, there are a variety of different methods you can choose to investigate a research problem. Your methodology section of your paper should make clear the reasons why you chose a particular method or procedure .
The reader wants to know that the data was collected or generated in a way that is consistent with accepted practice in the field of study. For example, if you are using a questionnaire, readers need to know that it offered your respondents a reasonable range of answers to choose from.
The research method must be appropriate to the objectives of the study . For example, be sure you have a large enough sample size to be able to generalize and make recommendations based upon the findings.
The methodology should discuss the problems that were anticipated and the steps you took to prevent them from occurring . For any problems that did arise, you must describe the ways in which their impact was minimized or why these problems do not affect the findings in any way that impacts your interpretation of the data.
Often in social science research, it is useful for other researchers to adapt or replicate your methodology. Therefore, it is important to always provide sufficient information to allow others to use or replicate the study . This information is particularly important when a new method had been developed or an innovative use of an existing method has been utilized.

Bem, Daryl J. Writing the Empirical Journal Article . Psychology Writing Center. University of Washington; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008.

Structure and Writing Style

I. Groups of Research Methods

There are two main groups of research methods in the social sciences:

The empirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences. This type of research focuses on objective knowledge, research questions that can be answered yes or no, and operational definitions of variables to be measured. The empirical-analytical group employs deductive reasoning that uses existing theory as a foundation for hypotheses that need to be tested. This approach is focused on explanation .
The interpretative group is focused on understanding phenomenon in a comprehensive, holistic way . This research method allows you to recognize your connection to the subject under study. Because the interpretative group focuses more on subjective knowledge, it requires careful interpretation of variables.

II. Content

An effectively written methodology section should:

Introduce the overall methodological approach for investigating your research problem . Is your study qualitative or quantitative or a combination of both (mixed method)? Are you going to take a special approach, such as action research, or a more neutral stance?
Indicate how the approach fits the overall research design . Your methods should have a clear connection with your research problem. In other words, make sure that your methods will actually address the problem. One of the most common deficiencies found in research papers is that the proposed methodology is unsuited to achieving the stated objective of your paper.
Describe the specific methods of data collection you are going to use , such as, surveys, interviews, questionnaires, observation, archival research. If you are analyzing existing data, such as a data set or archival documents, describe how it was originally created or gathered and by whom.
Explain how you intend to analyze your results . Will you use statistical analysis? Will you use specific theoretical perspectives to help you analyze a text or explain observed behaviors?
Provide background and rationale for methodologies that are unfamiliar for your readers . Very often in the social sciences, research problems and the methods for investigating them require more explanation/rationale than widely accepted rules governing the natural and physical sciences. Be clear and concise in your explanation.
Provide a rationale for subject selection and sampling procedure . For instance, if you propose to conduct interviews, how do you intend to select the sample population? If you are analyzing texts, which texts have you chosen, and why? If you are using statistics, why is this set of statisics being used? If other data sources exist, explain why the data you chose is most appropriate.
Address potential limitations . Are there any practical limitations that could affect your data collection? How will you attempt to control for potential confounding variables and errors? If your methodology may lead to problems you can anticipate, state this openly and show why pursuing this methodology outweighs the risk of these problems cropping up.

NOTE : Once you have written all of the elements of the methods section, subsequent revisions should focus on how to present those elements as clearly and as logically as possibly. The description of how you prepared to study the research problem, how you gathered the data, and the protocol for analyzing the data should be organized chronologically. For clarity, when a large amount of detail must be presented, information should be presented in sub-sections according to topic.

III. Problems to Avoid

Irrelevant Detail The methodology section of your paper should be thorough but to the point. Don’t provide any background information that doesn’t directly help the reader to understand why a particular method was chosen, how the data was gathered or obtained, and how it was analyzed. Unnecessary Explanation of Basic Procedures Remember that you are not writing a how-to guide about a particular method. You should make the assumption that readers possess a basic understanding of how to investigate the research problem on their own and, therefore, you do not have to go into great detail about specific methodological procedures. The focus should be on how you applied a method , not on the mechanics of doing a method. NOTE: An exception to this rule is if you select an unconventional approach to doing the method; if this is the case, be sure to explain why this approach was chosen and how it enhances the overall research process. Problem Blindness It is almost a given that you will encounter problems when collecting or generating your data. Do not ignore these problems or pretend they did not occur. Often, documenting how you overcame obstacles can form an interesting part of the methodology. It demonstrates to the reader that you can provide a cogent rationale for the decisions you made to minimize the impact of any problems that arose. Literature Review Just as the literature review section of your paper provides an overview of sources you have examined while researching a particular topic, the methodology section should cite any sources that informed your choice and application of a particular method [i.e., the choice of a survey should include any citations to the works you used to help construct the survey].

It’s More than Sources of Information! A description of a research study's method should not be confused with a description of the sources of information. Such a list of sources is useful in itself, especially if it is accompanied by an explanation about the selection and use of the sources. The description of the project's methodology complements a list of sources in that it sets forth the organization and interpretation of information emanating from those sources.

Azevedo, L.F. et al. How to Write a Scientific Paper: Writing the Methods Section. Revista Portuguesa de Pneumologia 17 (2011): 232-238; Butin, Dan W. The Education Dissertation A Guide for Practitioner Scholars . Thousand Oaks, CA: Corwin, 2010; Carter, Susan. Structuring Your Research Thesis . New York: Palgrave Macmillan, 2012; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008. Methods Section . The Writer’s Handbook. Writing Center. University of Wisconsin, Madison; Writing the Experimental Report: Methods, Results, and Discussion . The Writing Lab and The OWL. Purdue University; Methods and Materials . The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College.

Writing Tip

Statistical Designs and Tests? Do Not Fear Them!

Don't avoid using a quantitative approach to analyzing your research problem just because you fear the idea of applying statistical designs and tests. A qualitative approach, such as conducting interviews or content analysis of archival texts, can yield exciting new insights about a research problem, but it should not be undertaken simply because you have a disdain for running a simple regression. A well designed quantitative research study can often be accomplished in very clear and direct ways, whereas, a similar study of a qualitative nature usually requires considerable time to analyze large volumes of data and a tremendous burden to create new paths for analysis where previously no path associated with your research problem had existed.

Another Writing Tip

Knowing the Relationship Between Theories and Methods

There can be multiple meaning associated with the term "theories" and the term "methods" in social sciences research. A helpful way to delineate between them is to understand "theories" as representing different ways of characterizing the social world when you research it and "methods" as representing different ways of generating and analyzing data about that social world. Framed in this way, all empirical social sciences research involves theories and methods, whether they are stated explicitly or not. However, while theories and methods are often related, it is important that, as a researcher, you deliberately separate them in order to avoid your theories playing a disproportionate role in shaping what outcomes your chosen methods produce.

Introspectively engage in an ongoing dialectic between theories and methods to help enable you to use the outcomes from your methods to interrogate and develop new theories, or ways of framing conceptually the research problem. This is how scholarship grows and branches out into new intellectual territory.

Reynolds, R. Larry. Ways of Knowing. Alternative Microeconomics. Part 1, Chapter 3. Boise State University; The Theory-Method Relationship . S-Cool Revision. United Kingdom.

<< Previous: What Is Scholarly vs. Popular?
Next: Qualitative Methods >>
Last Updated: Jul 18, 2023 11:58 AM
URL: https://library.sacredheart.edu/c.php?g=29803
QuickSearch
Library Catalog
Databases A-Z
Publication Finder
Course Reserves
Citation Linker
Digital Commons

Research Support

Ask a Librarian
Appointments
Interlibrary Loan (ILL)
Research Guides
Databases by Subject
Citation Help

Using the Library

Reserve a Group Study Room
Renew Books
Honors Study Rooms
Off-Campus Access
Library Policies
Library Technology

User Information

Grad Students
Online Students
COVID-19 Updates
Staff Directory
News & Announcements
Library Newsletter

My Accounts

Interlibrary Loan
Staff Site Login

FIND US ON

USC Libraries
Research Guides

Organizing Your Social Sciences Research Paper

Quantitative Methods
Purpose of Guide
Design Flaws to Avoid
Independent and Dependent Variables
Glossary of Research Terms
Reading Research Effectively
Narrowing a Topic Idea
Broadening a Topic Idea
Extending the Timeliness of a Topic Idea
Academic Writing Style
Choosing a Title
Making an Outline
Paragraph Development
Research Process Video Series
Executive Summary
The C.A.R.S. Model
Background Information
The Research Problem/Question
Theoretical Framework
Citation Tracking
Content Alert Services
Evaluating Sources
Primary Sources
Secondary Sources
Tiertiary Sources
Scholarly vs. Popular Publications
Qualitative Methods
Insiderness
Using Non-Textual Elements
Limitations of the Study
Common Grammar Mistakes
Writing Concisely
Avoiding Plagiarism
Footnotes or Endnotes?
Further Readings
Generative AI and Writing
USC Libraries Tutorials and Other Guides
Bibliography

Quantitative methods emphasize objective measurements and the statistical, mathematical, or numerical analysis of data collected through polls, questionnaires, and surveys, or by manipulating pre-existing statistical data using computational techniques . Quantitative research focuses on gathering numerical data and generalizing it across groups of people or to explain a particular phenomenon.

Babbie, Earl R. The Practice of Social Research . 12th ed. Belmont, CA: Wadsworth Cengage, 2010; Muijs, Daniel. Doing Quantitative Research in Education with SPSS . 2nd edition. London: SAGE Publications, 2010.

Need Help Locating Statistics?

Resources for locating data and statistics can be found here:

Statistics & Data Research Guide

Characteristics of Quantitative Research

Your goal in conducting quantitative research study is to determine the relationship between one thing [an independent variable] and another [a dependent or outcome variable] within a population. Quantitative research designs are either descriptive [subjects usually measured once] or experimental [subjects measured before and after a treatment]. A descriptive study establishes only associations between variables; an experimental study establishes causality.

Quantitative research deals in numbers, logic, and an objective stance. Quantitative research focuses on numeric and unchanging data and detailed, convergent reasoning rather than divergent reasoning [i.e., the generation of a variety of ideas about a research problem in a spontaneous, free-flowing manner].

Its main characteristics are :

The data is usually gathered using structured research instruments.
The results are based on larger sample sizes that are representative of the population.
The research study can usually be replicated or repeated, given its high reliability.
Researcher has a clearly defined research question to which objective answers are sought.
All aspects of the study are carefully designed before data is collected.
Data are in the form of numbers and statistics, often arranged in tables, charts, figures, or other non-textual forms.
Project can be used to generalize concepts more widely, predict future results, or investigate causal relationships.
Researcher uses tools, such as questionnaires or computer software, to collect numerical data.

The overarching aim of a quantitative research study is to classify features, count them, and construct statistical models in an attempt to explain what is observed.

Things to keep in mind when reporting the results of a study using quantitative methods :

Explain the data collected and their statistical treatment as well as all relevant results in relation to the research problem you are investigating. Interpretation of results is not appropriate in this section.
Report unanticipated events that occurred during your data collection. Explain how the actual analysis differs from the planned analysis. Explain your handling of missing data and why any missing data does not undermine the validity of your analysis.
Explain the techniques you used to "clean" your data set.
Choose a minimally sufficient statistical procedure ; provide a rationale for its use and a reference for it. Specify any computer programs used.
Describe the assumptions for each procedure and the steps you took to ensure that they were not violated.
When using inferential statistics , provide the descriptive statistics, confidence intervals, and sample sizes for each variable as well as the value of the test statistic, its direction, the degrees of freedom, and the significance level [report the actual p value].
Avoid inferring causality , particularly in nonrandomized designs or without further experimentation.
Use tables to provide exact values ; use figures to convey global effects. Keep figures small in size; include graphic representations of confidence intervals whenever possible.
Always tell the reader what to look for in tables and figures .

NOTE: When using pre-existing statistical data gathered and made available by anyone other than yourself [e.g., government agency], you still must report on the methods that were used to gather the data and describe any missing data that exists and, if there is any, provide a clear explanation why the missing data does not undermine the validity of your final analysis.

Babbie, Earl R. The Practice of Social Research . 12th ed. Belmont, CA: Wadsworth Cengage, 2010; Brians, Craig Leonard et al. Empirical Political Analysis: Quantitative and Qualitative Research Methods . 8th ed. Boston, MA: Longman, 2011; McNabb, David E. Research Methods in Public Administration and Nonprofit Management: Quantitative and Qualitative Approaches . 2nd ed. Armonk, NY: M.E. Sharpe, 2008; Quantitative Research Methods. Writing@CSU. Colorado State University; Singh, Kultar. Quantitative Social Research Methods . Los Angeles, CA: Sage, 2007.

Basic Research Design for Quantitative Studies

Before designing a quantitative research study, you must decide whether it will be descriptive or experimental because this will dictate how you gather, analyze, and interpret the results. A descriptive study is governed by the following rules: subjects are generally measured once; the intention is to only establish associations between variables; and, the study may include a sample population of hundreds or thousands of subjects to ensure that a valid estimate of a generalized relationship between variables has been obtained. An experimental design includes subjects measured before and after a particular treatment, the sample population may be very small and purposefully chosen, and it is intended to establish causality between variables. Introduction The introduction to a quantitative study is usually written in the present tense and from the third person point of view. It covers the following information:

Identifies the research problem -- as with any academic study, you must state clearly and concisely the research problem being investigated.
Reviews the literature -- review scholarship on the topic, synthesizing key themes and, if necessary, noting studies that have used similar methods of inquiry and analysis. Note where key gaps exist and how your study helps to fill these gaps or clarifies existing knowledge.
Describes the theoretical framework -- provide an outline of the theory or hypothesis underpinning your study. If necessary, define unfamiliar or complex terms, concepts, or ideas and provide the appropriate background information to place the research problem in proper context [e.g., historical, cultural, economic, etc.].

Methodology The methods section of a quantitative study should describe how each objective of your study will be achieved. Be sure to provide enough detail to enable the reader can make an informed assessment of the methods being used to obtain results associated with the research problem. The methods section should be presented in the past tense.

Study population and sampling -- where did the data come from; how robust is it; note where gaps exist or what was excluded. Note the procedures used for their selection;
Data collection – describe the tools and methods used to collect information and identify the variables being measured; describe the methods used to obtain the data; and, note if the data was pre-existing [i.e., government data] or you gathered it yourself. If you gathered it yourself, describe what type of instrument you used and why. Note that no data set is perfect--describe any limitations in methods of gathering data.
Data analysis -- describe the procedures for processing and analyzing the data. If appropriate, describe the specific instruments of analysis used to study each research objective, including mathematical techniques and the type of computer software used to manipulate the data.

Results The finding of your study should be written objectively and in a succinct and precise format. In quantitative studies, it is common to use graphs, tables, charts, and other non-textual elements to help the reader understand the data. Make sure that non-textual elements do not stand in isolation from the text but are being used to supplement the overall description of the results and to help clarify key points being made. Further information about how to effectively present data using charts and graphs can be found here .

Statistical analysis -- how did you analyze the data? What were the key findings from the data? The findings should be present in a logical, sequential order. Describe but do not interpret these trends or negative results; save that for the discussion section. The results should be presented in the past tense.

Discussion Discussions should be analytic, logical, and comprehensive. The discussion should meld together your findings in relation to those identified in the literature review, and placed within the context of the theoretical framework underpinning the study. The discussion should be presented in the present tense.

Interpretation of results -- reiterate the research problem being investigated and compare and contrast the findings with the research questions underlying the study. Did they affirm predicted outcomes or did the data refute it?
Description of trends, comparison of groups, or relationships among variables -- describe any trends that emerged from your analysis and explain all unanticipated and statistical insignificant findings.
Discussion of implications – what is the meaning of your results? Highlight key findings based on the overall results and note findings that you believe are important. How have the results helped fill gaps in understanding the research problem?
Limitations -- describe any limitations or unavoidable bias in your study and, if necessary, note why these limitations did not inhibit effective interpretation of the results.

Conclusion End your study by to summarizing the topic and provide a final comment and assessment of the study.

Summary of findings – synthesize the answers to your research questions. Do not report any statistical data here; just provide a narrative summary of the key findings and describe what was learned that you did not know before conducting the study.
Recommendations – if appropriate to the aim of the assignment, tie key findings with policy recommendations or actions to be taken in practice.
Future research – note the need for future research linked to your study’s limitations or to any remaining gaps in the literature that were not addressed in your study.

Black, Thomas R. Doing Quantitative Research in the Social Sciences: An Integrated Approach to Research Design, Measurement and Statistics . London: Sage, 1999; Gay,L. R. and Peter Airasain. Educational Research: Competencies for Analysis and Applications . 7th edition. Upper Saddle River, NJ: Merril Prentice Hall, 2003; Hector, Anestine. An Overview of Quantitative Research in Composition and TESOL . Department of English, Indiana University of Pennsylvania; Hopkins, Will G. “Quantitative Research Design.” Sportscience 4, 1 (2000); "A Strategy for Writing Up Research Results. The Structure, Format, Content, and Style of a Journal-Style Scientific Paper." Department of Biology. Bates College; Nenty, H. Johnson. "Writing a Quantitative Research Thesis." International Journal of Educational Science 1 (2009): 19-32; Ouyang, Ronghua (John). Basic Inquiry of Quantitative Research . Kennesaw State University.

Strengths of Using Quantitative Methods

Quantitative researchers try to recognize and isolate specific variables contained within the study framework, seek correlation, relationships and causality, and attempt to control the environment in which the data is collected to avoid the risk of variables, other than the one being studied, accounting for the relationships identified.

Among the specific strengths of using quantitative methods to study social science research problems:

Allows for a broader study, involving a greater number of subjects, and enhancing the generalization of the results;
Allows for greater objectivity and accuracy of results. Generally, quantitative methods are designed to provide summaries of data that support generalizations about the phenomenon under study. In order to accomplish this, quantitative research usually involves few variables and many cases, and employs prescribed procedures to ensure validity and reliability;
Applying well established standards means that the research can be replicated, and then analyzed and compared with similar studies;
You can summarize vast sources of information and make comparisons across categories and over time; and,
Personal bias can be avoided by keeping a 'distance' from participating subjects and using accepted computational techniques .

Babbie, Earl R. The Practice of Social Research . 12th ed. Belmont, CA: Wadsworth Cengage, 2010; Brians, Craig Leonard et al. Empirical Political Analysis: Quantitative and Qualitative Research Methods . 8th ed. Boston, MA: Longman, 2011; McNabb, David E. Research Methods in Public Administration and Nonprofit Management: Quantitative and Qualitative Approaches . 2nd ed. Armonk, NY: M.E. Sharpe, 2008; Singh, Kultar. Quantitative Social Research Methods . Los Angeles, CA: Sage, 2007.

Limitations of Using Quantitative Methods

Quantitative methods presume to have an objective approach to studying research problems, where data is controlled and measured, to address the accumulation of facts, and to determine the causes of behavior. As a consequence, the results of quantitative research may be statistically significant but are often humanly insignificant.

Some specific limitations associated with using quantitative methods to study research problems in the social sciences include:

Quantitative data is more efficient and able to test hypotheses, but may miss contextual detail;
Uses a static and rigid approach and so employs an inflexible process of discovery;
The development of standard questions by researchers can lead to "structural bias" and false representation, where the data actually reflects the view of the researcher instead of the participating subject;
Results provide less detail on behavior, attitudes, and motivation;
Researcher may collect a much narrower and sometimes superficial dataset;
Results are limited as they provide numerical descriptions rather than detailed narrative and generally provide less elaborate accounts of human perception;
The research is often carried out in an unnatural, artificial environment so that a level of control can be applied to the exercise. This level of control might not normally be in place in the real world thus yielding "laboratory results" as opposed to "real world results"; and,
Preset answers will not necessarily reflect how people really feel about a subject and, in some cases, might just be the closest match to the preconceived hypothesis.

Research Tip

Finding Examples of How to Apply Different Types of Research Methods

SAGE publications is a major publisher of studies about how to design and conduct research in the social and behavioral sciences. Their SAGE Research Methods Online and Cases database includes contents from books, articles, encyclopedias, handbooks, and videos covering social science research design and methods including the complete Little Green Book Series of Quantitative Applications in the Social Sciences and the Little Blue Book Series of Qualitative Research techniques. The database also includes case studies outlining the research methods used in real research projects. This is an excellent source for finding definitions of key terms and descriptions of research design and practice, techniques of data gathering, analysis, and reporting, and information about theories of research [e.g., grounded theory]. The database covers both qualitative and quantitative research methods as well as mixed methods approaches to conducting research.

SAGE Research Methods Online and Cases

<< Previous: Qualitative Methods
Next: Insiderness >>
Last Updated: Oct 10, 2023 1:30 PM
URL: https://libguides.usc.edu/writingguide

Social Anxiety Disorder
Bipolar Disorder
Kids Mental Health
Therapy Center
When To See a Therapist
Types of Therapy
Best Online Therapy
Best Couples Therapy
Best Family Therapy
Managing Stress
Sleep and Dreaming
Understanding Emotions
Self-Improvement
Healthy Relationships
Relationships in 2023
Student Resources
Personality Types
Verywell Mind Insights
2023 Verywell Mind 25
Mental Health in the Classroom
Editorial Process
Meet Our Review Board
Crisis Support

How to Write a Methods Section for a Psychology Paper

Tips and Examples of an APA Methods Section

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Emily is a board-certified science editor who has worked with top digital publishing brands like Voices for Biodiversity, Study.com, GoodTherapy, Vox, and Verywell.

Verywell / Brianna Gilmartin

The methods section of an APA format psychology paper provides the methods and procedures used in a research study or experiment . This part of an APA paper is critical because it allows other researchers to see exactly how you conducted your research.

Method refers to the procedure that was used in a research study. It included a precise description of how the experiments were performed and why particular procedures were selected. While the APA technically refers to this section as the 'method section,' it is also often known as a 'methods section.'

The methods section ensures the experiment's reproducibility and the assessment of alternative methods that might produce different results. It also allows researchers to replicate the experiment and judge the study's validity.

This article discusses how to write a methods section for a psychology paper, including important elements to include and tips that can help.

What to Include in a Method Section

So what exactly do you need to include when writing your method section? You should provide detailed information on the following:

Research design
Participants
Participant behavior

The method section should provide enough information to allow other researchers to replicate your experiment or study.

Components of a Method Section

The method section should utilize subheadings to divide up different subsections. These subsections typically include participants, materials, design, and procedure.

Participants

In this part of the method section, you should describe the participants in your experiment, including who they were (and any unique features that set them apart from the general population), how many there were, and how they were selected. If you utilized random selection to choose your participants, it should be noted here.

For example: "We randomly selected 100 children from elementary schools near the University of Arizona."

At the very minimum, this part of your method section must convey:

Basic demographic characteristics of your participants (such as sex, age, ethnicity, or religion)
The population from which your participants were drawn
Any restrictions on your pool of participants
How many participants were assigned to each condition and how they were assigned to each group (i.e., randomly assignment , another selection method, etc.)
Why participants took part in your research (i.e., the study was advertised at a college or hospital, they received some type of incentive, etc.)

Information about participants helps other researchers understand how your study was performed, how generalizable the result might be, and allows other researchers to replicate the experiment with other populations to see if they might obtain the same results.

In this part of the method section, you should describe the materials, measures, equipment, or stimuli used in the experiment. This may include:

Testing instruments
Technical equipment
Any psychological assessments that were used
Any special equipment that was used

For example: "Two stories from Sullivan et al.'s (1994) second-order false belief attribution tasks were used to assess children's understanding of second-order beliefs."

For standard equipment such as computers, televisions, and videos, you can simply name the device and not provide further explanation.

Specialized equipment should be given greater detail, especially if it is complex or created for a niche purpose. In some instances, such as if you created a special material or apparatus for your study, you might need to include an illustration of the item in the appendix of your paper.

In this part of your method section, describe the type of design used in the experiment. Specify the variables as well as the levels of these variables. Identify:

The independent variables
Dependent variables
Control variables
Any extraneous variables that might influence your results.

Also, explain whether your experiment uses a within-groups or between-groups design.

For example: "The experiment used a 3x2 between-subjects design. The independent variables were age and understanding of second-order beliefs."

The next part of your method section should detail the procedures used in your experiment. Your procedures should explain:

What the participants did
How data was collected
The order in which steps occurred

For example: "An examiner interviewed children individually at their school in one session that lasted 20 minutes on average. The examiner explained to each child that he or she would be told two short stories and that some questions would be asked after each story. All sessions were videotaped so the data could later be coded."

Keep this subsection concise yet detailed. Explain what you did and how you did it, but do not overwhelm your readers with too much information.

Tips for How to Write a Methods Section

In addition to following the basic structure of an APA method section, there are also certain things you should remember when writing this section of your paper. Consider the following tips when writing this section:

Use the past tense : Always write the method section in the past tense.
Be descriptive : Provide enough detail that another researcher could replicate your experiment, but focus on brevity. Avoid unnecessary detail that is not relevant to the outcome of the experiment.
Use an academic tone : Use formal language and avoid slang or colloquial expressions. Word choice is also important. Refer to the people in your experiment or study as "participants" rather than "subjects."
Use APA format : Keep a style guide on hand as you write your method section. The Publication Manual of the American Psychological Association is the official source for APA style.
Make connections : Read through each section of your paper for agreement with other sections. If you mention procedures in the method section, these elements should be discussed in the results and discussion sections.
Proofread : Check your paper for grammar, spelling, and punctuation errors.. typos, grammar problems, and spelling errors. Although a spell checker is a handy tool, there are some errors only you can catch.

After writing a draft of your method section, be sure to get a second opinion. You can often become too close to your work to see errors or lack of clarity. Take a rough draft of your method section to your university's writing lab for additional assistance.

A Word From Verywell

The method section is one of the most important components of your APA format paper. The goal of your paper should be to clearly detail what you did in your experiment. Provide enough detail that another researcher could replicate your study if they wanted.

Finally, if you are writing your paper for a class or for a specific publication, be sure to keep in mind any specific instructions provided by your instructor or by the journal editor. Your instructor may have certain requirements that you need to follow while writing your method section.

Frequently Asked Questions

While the subsections can vary, the three components that should be included are sections on the participants, the materials, and the procedures.

Describe who the participants were in the study and how they were selected.
Define and describe the materials that were used including any equipment, tests, or assessments
Describe how the data was collected

To write your methods section in APA format, describe your participants, materials, study design, and procedures. Keep this section succinct, and always write in the past tense. The main heading of this section should be labeled "Method" and it should be centered, bolded, and capitalized. Each subheading within this section should be bolded, left-aligned and in title case.

The purpose of the methods section is to describe what you did in your experiment. It should be brief, but include enough detail that someone could replicate your experiment based on this information. Your methods section should detail what you did to answer your research question. Describe how the study was conducted, the study design that was used and why it was chosen, and how you collected the data and analyzed the results.

Erdemir F. How to write a materials and methods section of a scientific article ? Turk J Urol . 2013;39(Suppl 1):10-5. doi:10.5152/tud.2013.047

Kallet RH. How to write the methods section of a research paper . Respir Care . 2004;49(10):1229-32. PMID: 15447808.

American Psychological Association. Publication Manual of the American Psychological Association (7th ed.). Washington DC: The American Psychological Association; 2019.

American Psychological Association. APA Style Journal Article Reporting Standards . Published 2020.

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts.

Methodology
Open access
Published: 11 October 2016

Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research

Stephen J. Gentles 1 , 4 ,
Cathy Charles 1 ,
David B. Nicholas 2 ,
Jenny Ploeg 3 &
K. Ann McKibbon 1

Systematic Reviews volume 5 , Article number: 172 ( 2016 ) Cite this article

46k Accesses

49 Citations

13 Altmetric

Metrics details

Overviews of methods are potentially useful means to increase clarity and enhance collective understanding of specific methods topics that may be characterized by ambiguity, inconsistency, or a lack of comprehensiveness. This type of review represents a distinct literature synthesis method, although to date, its methodology remains relatively undeveloped despite several aspects that demand unique review procedures. The purpose of this paper is to initiate discussion about what a rigorous systematic approach to reviews of methods, referred to here as systematic methods overviews , might look like by providing tentative suggestions for approaching specific challenges likely to be encountered. The guidance offered here was derived from experience conducting a systematic methods overview on the topic of sampling in qualitative research.

The guidance is organized into several principles that highlight specific objectives for this type of review given the common challenges that must be overcome to achieve them. Optional strategies for achieving each principle are also proposed, along with discussion of how they were successfully implemented in the overview on sampling. We describe seven paired principles and strategies that address the following aspects: delimiting the initial set of publications to consider, searching beyond standard bibliographic databases, searching without the availability of relevant metadata, selecting publications on purposeful conceptual grounds, defining concepts and other information to abstract iteratively, accounting for inconsistent terminology used to describe specific methods topics, and generating rigorous verifiable analytic interpretations. Since a broad aim in systematic methods overviews is to describe and interpret the relevant literature in qualitative terms, we suggest that iterative decision making at various stages of the review process, and a rigorous qualitative approach to analysis are necessary features of this review type.

Conclusions

We believe that the principles and strategies provided here will be useful to anyone choosing to undertake a systematic methods overview. This paper represents an initial effort to promote high quality critical evaluations of the literature regarding problematic methods topics, which have the potential to promote clearer, shared understandings, and accelerate advances in research methods. Further work is warranted to develop more definitive guidance.

Peer Review reports

While reviews of methods are not new, they represent a distinct review type whose methodology remains relatively under-addressed in the literature despite the clear implications for unique review procedures. One of few examples to describe it is a chapter containing reflections of two contributing authors in a book of 21 reviews on methodological topics compiled for the British National Health Service, Health Technology Assessment Program [ 1 ]. Notable is their observation of how the differences between the methods reviews and conventional quantitative systematic reviews, specifically attributable to their varying content and purpose, have implications for defining what qualifies as systematic. While the authors describe general aspects of “systematicity” (including rigorous application of a methodical search, abstraction, and analysis), they also describe a high degree of variation within the category of methods reviews itself and so offer little in the way of concrete guidance. In this paper, we present tentative concrete guidance, in the form of a preliminary set of proposed principles and optional strategies, for a rigorous systematic approach to reviewing and evaluating the literature on quantitative or qualitative methods topics. For purposes of this article, we have used the term systematic methods overview to emphasize the notion of a systematic approach to such reviews.

The conventional focus of rigorous literature reviews (i.e., review types for which systematic methods have been codified, including the various approaches to quantitative systematic reviews [ 2 – 4 ], and the numerous forms of qualitative and mixed methods literature synthesis [ 5 – 10 ]) is to synthesize empirical research findings from multiple studies. By contrast, the focus of overviews of methods, including the systematic approach we advocate, is to synthesize guidance on methods topics. The literature consulted for such reviews may include the methods literature, methods-relevant sections of empirical research reports, or both. Thus, this paper adds to previous work published in this journal—namely, recent preliminary guidance for conducting reviews of theory [ 11 ]—that has extended the application of systematic review methods to novel review types that are concerned with subject matter other than empirical research findings.

Published examples of methods overviews illustrate the varying objectives they can have. One objective is to establish methodological standards for appraisal purposes. For example, reviews of existing quality appraisal standards have been used to propose universal standards for appraising the quality of primary qualitative research [ 12 ] or evaluating qualitative research reports [ 13 ]. A second objective is to survey the methods-relevant sections of empirical research reports to establish current practices on methods use and reporting practices, which Moher and colleagues [ 14 ] recommend as a means for establishing the needs to be addressed in reporting guidelines (see, for example [ 15 , 16 ]). A third objective for a methods review is to offer clarity and enhance collective understanding regarding a specific methods topic that may be characterized by ambiguity, inconsistency, or a lack of comprehensiveness within the available methods literature. An example of this is a overview whose objective was to review the inconsistent definitions of intention-to-treat analysis (the methodologically preferred approach to analyze randomized controlled trial data) that have been offered in the methods literature and propose a solution for improving conceptual clarity [ 17 ]. Such reviews are warranted because students and researchers who must learn or apply research methods typically lack the time to systematically search, retrieve, review, and compare the available literature to develop a thorough and critical sense of the varied approaches regarding certain controversial or ambiguous methods topics.

While systematic methods overviews , as a review type, include both reviews of the methods literature and reviews of methods-relevant sections from empirical study reports, the guidance provided here is primarily applicable to reviews of the methods literature since it was derived from the experience of conducting such a review [ 18 ], described below. To our knowledge, there are no well-developed proposals on how to rigorously conduct such reviews. Such guidance would have the potential to improve the thoroughness and credibility of critical evaluations of the methods literature, which could increase their utility as a tool for generating understandings that advance research methods, both qualitative and quantitative. Our aim in this paper is thus to initiate discussion about what might constitute a rigorous approach to systematic methods overviews. While we hope to promote rigor in the conduct of systematic methods overviews wherever possible, we do not wish to suggest that all methods overviews need be conducted to the same standard. Rather, we believe that the level of rigor may need to be tailored pragmatically to the specific review objectives, which may not always justify the resource requirements of an intensive review process.

The example systematic methods overview on sampling in qualitative research

The principles and strategies we propose in this paper are derived from experience conducting a systematic methods overview on the topic of sampling in qualitative research [ 18 ]. The main objective of that methods overview was to bring clarity and deeper understanding of the prominent concepts related to sampling in qualitative research (purposeful sampling strategies, saturation, etc.). Specifically, we interpreted the available guidance, commenting on areas lacking clarity, consistency, or comprehensiveness (without proposing any recommendations on how to do sampling). This was achieved by a comparative and critical analysis of publications representing the most influential (i.e., highly cited) guidance across several methodological traditions in qualitative research.

The specific methods and procedures for the overview on sampling [ 18 ] from which our proposals are derived were developed both after soliciting initial input from local experts in qualitative research and an expert health librarian (KAM) and through ongoing careful deliberation throughout the review process. To summarize, in that review, we employed a transparent and rigorous approach to search the methods literature, selected publications for inclusion according to a purposeful and iterative process, abstracted textual data using structured abstraction forms, and analyzed (synthesized) the data using a systematic multi-step approach featuring abstraction of text, summary of information in matrices, and analytic comparisons.

For this article, we reflected on both the problems and challenges encountered at different stages of the review and our means for selecting justifiable procedures to deal with them. Several principles were then derived by considering the generic nature of these problems, while the generalizable aspects of the procedures used to address them formed the basis of optional strategies. Further details of the specific methods and procedures used in the overview on qualitative sampling are provided below to illustrate both the types of objectives and challenges that reviewers will likely need to consider and our approach to implementing each of the principles and strategies.

Organization of the guidance into principles and strategies

For the purposes of this article, principles are general statements outlining what we propose are important aims or considerations within a particular review process, given the unique objectives or challenges to be overcome with this type of review. These statements follow the general format, “considering the objective or challenge of X, we propose Y to be an important aim or consideration.” Strategies are optional and flexible approaches for implementing the previous principle outlined. Thus, generic challenges give rise to principles, which in turn give rise to strategies.

We organize the principles and strategies below into three sections corresponding to processes characteristic of most systematic literature synthesis approaches: literature identification and selection ; data abstraction from the publications selected for inclusion; and analysis , including critical appraisal and synthesis of the abstracted data. Within each section, we also describe the specific methodological decisions and procedures used in the overview on sampling in qualitative research [ 18 ] to illustrate how the principles and strategies for each review process were applied and implemented in a specific case. We expect this guidance and accompanying illustrations will be useful for anyone considering engaging in a methods overview, particularly those who may be familiar with conventional systematic review methods but may not yet appreciate some of the challenges specific to reviewing the methods literature.

Results and discussion

Literature identification and selection.

The identification and selection process includes search and retrieval of publications and the development and application of inclusion and exclusion criteria to select the publications that will be abstracted and analyzed in the final review. Literature identification and selection for overviews of the methods literature is challenging and potentially more resource-intensive than for most reviews of empirical research. This is true for several reasons that we describe below, alongside discussion of the potential solutions. Additionally, we suggest in this section how the selection procedures can be chosen to match the specific analytic approach used in methods overviews.

Delimiting a manageable set of publications

One aspect of methods overviews that can make identification and selection challenging is the fact that the universe of literature containing potentially relevant information regarding most methods-related topics is expansive and often unmanageably so. Reviewers are faced with two large categories of literature: the methods literature , where the possible publication types include journal articles, books, and book chapters; and the methods-relevant sections of empirical study reports , where the possible publication types include journal articles, monographs, books, theses, and conference proceedings. In our systematic overview of sampling in qualitative research, exhaustively searching (including retrieval and first-pass screening) all publication types across both categories of literature for information on a single methods-related topic was too burdensome to be feasible. The following proposed principle follows from the need to delimit a manageable set of literature for the review.

Principle #1:

Considering the broad universe of potentially relevant literature, we propose that an important objective early in the identification and selection stage is to delimit a manageable set of methods-relevant publications in accordance with the objectives of the methods overview.

Strategy #1:

To limit the set of methods-relevant publications that must be managed in the selection process, reviewers have the option to initially review only the methods literature, and exclude the methods-relevant sections of empirical study reports, provided this aligns with the review’s particular objectives.

We propose that reviewers are justified in choosing to select only the methods literature when the objective is to map out the range of recognized concepts relevant to a methods topic, to summarize the most authoritative or influential definitions or meanings for methods-related concepts, or to demonstrate a problematic lack of clarity regarding a widely established methods-related concept and potentially make recommendations for a preferred approach to the methods topic in question. For example, in the case of the methods overview on sampling [ 18 ], the primary aim was to define areas lacking in clarity for multiple widely established sampling-related topics. In the review on intention-to-treat in the context of missing outcome data [ 17 ], the authors identified a lack of clarity based on multiple inconsistent definitions in the literature and went on to recommend separating the issue of how to handle missing outcome data from the issue of whether an intention-to-treat analysis can be claimed.

In contrast to strategy #1, it may be appropriate to select the methods-relevant sections of empirical study reports when the objective is to illustrate how a methods concept is operationalized in research practice or reported by authors. For example, one could review all the publications in 2 years’ worth of issues of five high-impact field-related journals to answer questions about how researchers describe implementing a particular method or approach, or to quantify how consistently they define or report using it. Such reviews are often used to highlight gaps in the reporting practices regarding specific methods, which may be used to justify items to address in reporting guidelines (for example, [ 14 – 16 ]).

It is worth recognizing that other authors have advocated broader positions regarding the scope of literature to be considered in a review, expanding on our perspective. Suri [ 10 ] (who, like us, emphasizes how different sampling strategies are suitable for different literature synthesis objectives) has, for example, described a two-stage literature sampling procedure (pp. 96–97). First, reviewers use an initial approach to conduct a broad overview of the field—for reviews of methods topics, this would entail an initial review of the research methods literature. This is followed by a second more focused stage in which practical examples are purposefully selected—for methods reviews, this would involve sampling the empirical literature to illustrate key themes and variations. While this approach is seductive in its capacity to generate more in depth and interpretive analytic findings, some reviewers may consider it too resource-intensive to include the second step no matter how selective the purposeful sampling. In the overview on sampling where we stopped after the first stage [ 18 ], we discussed our selective focus on the methods literature as a limitation that left opportunities for further analysis of the literature. We explicitly recommended, for example, that theoretical sampling was a topic for which a future review of the methods sections of empirical reports was justified to answer specific questions identified in the primary review.

Ultimately, reviewers must make pragmatic decisions that balance resource considerations, combined with informed predictions about the depth and complexity of literature available on their topic, with the stated objectives of their review. The remaining principles and strategies apply primarily to overviews that include the methods literature, although some aspects may be relevant to reviews that include empirical study reports.

Searching beyond standard bibliographic databases

An important reality affecting identification and selection in overviews of the methods literature is the increased likelihood for relevant publications to be located in sources other than journal articles (which is usually not the case for overviews of empirical research, where journal articles generally represent the primary publication type). In the overview on sampling [ 18 ], out of 41 full-text publications retrieved and reviewed, only 4 were journal articles, while 37 were books or book chapters. Since many books and book chapters did not exist electronically, their full text had to be physically retrieved in hardcopy, while 11 publications were retrievable only through interlibrary loan or purchase request. The tasks associated with such retrieval are substantially more time-consuming than electronic retrieval. Since a substantial proportion of methods-related guidance may be located in publication types that are less comprehensively indexed in standard bibliographic databases, identification and retrieval thus become complicated processes.

Principle #2:

Considering that important sources of methods guidance can be located in non-journal publication types (e.g., books, book chapters) that tend to be poorly indexed in standard bibliographic databases, it is important to consider alternative search methods for identifying relevant publications to be further screened for inclusion.

Strategy #2:

To identify books, book chapters, and other non-journal publication types not thoroughly indexed in standard bibliographic databases, reviewers may choose to consult one or more of the following less standard sources: Google Scholar, publisher web sites, or expert opinion.

In the case of the overview on sampling in qualitative research [ 18 ], Google Scholar had two advantages over other standard bibliographic databases: it indexes and returns records of books and book chapters likely to contain guidance on qualitative research methods topics; and it has been validated as providing higher citation counts than ISI Web of Science (a producer of numerous bibliographic databases accessible through institutional subscription) for several non-biomedical disciplines including the social sciences where qualitative research methods are prominently used [ 19 – 21 ]. While we identified numerous useful publications by consulting experts, the author publication lists generated through Google Scholar searches were uniquely useful to identify more recent editions of methods books identified by experts.

Searching without relevant metadata

Determining what publications to select for inclusion in the overview on sampling [ 18 ] could only rarely be accomplished by reviewing the publication’s metadata. This was because for the many books and other non-journal type publications we identified as possibly relevant, the potential content of interest would be located in only a subsection of the publication. In this common scenario for reviews of the methods literature (as opposed to methods overviews that include empirical study reports), reviewers will often be unable to employ standard title, abstract, and keyword database searching or screening as a means for selecting publications.

Principle #3:

Considering that the presence of information about the topic of interest may not be indicated in the metadata for books and similar publication types, it is important to consider other means of identifying potentially useful publications for further screening.

Strategy #3:

One approach to identifying potentially useful books and similar publication types is to consider what classes of such publications (e.g., all methods manuals for a certain research approach) are likely to contain relevant content, then identify, retrieve, and review the full text of corresponding publications to determine whether they contain information on the topic of interest.

In the example of the overview on sampling in qualitative research [ 18 ], the topic of interest (sampling) was one of numerous topics covered in the general qualitative research methods manuals. Consequently, examples from this class of publications first had to be identified for retrieval according to non-keyword-dependent criteria. Thus, all methods manuals within the three research traditions reviewed (grounded theory, phenomenology, and case study) that might contain discussion of sampling were sought through Google Scholar and expert opinion, their full text obtained, and hand-searched for relevant content to determine eligibility. We used tables of contents and index sections of books to aid this hand searching.

Purposefully selecting literature on conceptual grounds

A final consideration in methods overviews relates to the type of analysis used to generate the review findings. Unlike quantitative systematic reviews where reviewers aim for accurate or unbiased quantitative estimates—something that requires identifying and selecting the literature exhaustively to obtain all relevant data available (i.e., a complete sample)—in methods overviews, reviewers must describe and interpret the relevant literature in qualitative terms to achieve review objectives. In other words, the aim in methods overviews is to seek coverage of the qualitative concepts relevant to the methods topic at hand. For example, in the overview of sampling in qualitative research [ 18 ], achieving review objectives entailed providing conceptual coverage of eight sampling-related topics that emerged as key domains. The following principle recognizes that literature sampling should therefore support generating qualitative conceptual data as the input to analysis.

Principle #4:

Since the analytic findings of a systematic methods overview are generated through qualitative description and interpretation of the literature on a specified topic, selection of the literature should be guided by a purposeful strategy designed to achieve adequate conceptual coverage (i.e., representing an appropriate degree of variation in relevant ideas) of the topic according to objectives of the review.

Strategy #4:

One strategy for choosing the purposeful approach to use in selecting the literature according to the review objectives is to consider whether those objectives imply exploring concepts either at a broad overview level, in which case combining maximum variation selection with a strategy that limits yield (e.g., critical case, politically important, or sampling for influence—described below) may be appropriate; or in depth, in which case purposeful approaches aimed at revealing innovative cases will likely be necessary.

In the methods overview on sampling, the implied scope was broad since we set out to review publications on sampling across three divergent qualitative research traditions—grounded theory, phenomenology, and case study—to facilitate making informative conceptual comparisons. Such an approach would be analogous to maximum variation sampling.

At the same time, the purpose of that review was to critically interrogate the clarity, consistency, and comprehensiveness of literature from these traditions that was “most likely to have widely influenced students’ and researchers’ ideas about sampling” (p. 1774) [ 18 ]. In other words, we explicitly set out to review and critique the most established and influential (and therefore dominant) literature, since this represents a common basis of knowledge among students and researchers seeking understanding or practical guidance on sampling in qualitative research. To achieve this objective, we purposefully sampled publications according to the criterion of influence , which we operationalized as how often an author or publication has been referenced in print or informal discourse. This second sampling approach also limited the literature we needed to consider within our broad scope review to a manageable amount.

To operationalize this strategy of sampling for influence , we sought to identify both the most influential authors within a qualitative research tradition (all of whose citations were subsequently screened) and the most influential publications on the topic of interest by non-influential authors. This involved a flexible approach that combined multiple indicators of influence to avoid the dilemma that any single indicator might provide inadequate coverage. These indicators included bibliometric data (h-index for author influence [ 22 ]; number of cites for publication influence), expert opinion, and cross-references in the literature (i.e., snowball sampling). As a final selection criterion, a publication was included only if it made an original contribution in terms of novel guidance regarding sampling or a related concept; thus, purely secondary sources were excluded. Publish or Perish software (Anne-Wil Harzing; available at http://www.harzing.com/resources/publish-or-perish ) was used to generate bibliometric data via the Google Scholar database. Figure 1 illustrates how identification and selection in the methods overview on sampling was a multi-faceted and iterative process. The authors selected as influential, and the publications selected for inclusion or exclusion are listed in Additional file 1 (Matrices 1, 2a, 2b).

Literature identification and selection process used in the methods overview on sampling [ 18 ]

In summary, the strategies of seeking maximum variation and sampling for influence were employed in the sampling overview to meet the specific review objectives described. Reviewers will need to consider the full range of purposeful literature sampling approaches at their disposal in deciding what best matches the specific aims of their own reviews. Suri [ 10 ] has recently retooled Patton’s well-known typology of purposeful sampling strategies (originally intended for primary research) for application to literature synthesis, providing a useful resource in this respect.

Data abstraction

The purpose of data abstraction in rigorous literature reviews is to locate and record all data relevant to the topic of interest from the full text of included publications, making them available for subsequent analysis. Conventionally, a data abstraction form—consisting of numerous distinct conceptually defined fields to which corresponding information from the source publication is recorded—is developed and employed. There are several challenges, however, to the processes of developing the abstraction form and abstracting the data itself when conducting methods overviews, which we address here. Some of these problems and their solutions may be familiar to those who have conducted qualitative literature syntheses, which are similarly conceptual.

Iteratively defining conceptual information to abstract

In the overview on sampling [ 18 ], while we surveyed multiple sources beforehand to develop a list of concepts relevant for abstraction (e.g., purposeful sampling strategies, saturation, sample size), there was no way for us to anticipate some concepts prior to encountering them in the review process. Indeed, in many cases, reviewers are unable to determine the complete set of methods-related concepts that will be the focus of the final review a priori without having systematically reviewed the publications to be included. Thus, defining what information to abstract beforehand may not be feasible.

Principle #5:

Considering the potential impracticality of defining a complete set of relevant methods-related concepts from a body of literature one has not yet systematically read, selecting and defining fields for data abstraction must often be undertaken iteratively. Thus, concepts to be abstracted can be expected to grow and change as data abstraction proceeds.

Strategy #5:

Reviewers can develop an initial form or set of concepts for abstraction purposes according to standard methods (e.g., incorporating expert feedback, pilot testing) and remain attentive to the need to iteratively revise it as concepts are added or modified during the review. Reviewers should document revisions and return to re-abstract data from previously abstracted publications as the new data requirements are determined.

In the sampling overview [ 18 ], we developed and maintained the abstraction form in Microsoft Word. We derived the initial set of abstraction fields from our own knowledge of relevant sampling-related concepts, consultation with local experts, and reviewing a pilot sample of publications. Since the publications in this review included a large proportion of books, the abstraction process often began by flagging the broad sections within a publication containing topic-relevant information for detailed review to identify text to abstract. When reviewing flagged text, the reviewer occasionally encountered an unanticipated concept significant enough to warrant being added as a new field to the abstraction form. For example, a field was added to capture how authors described the timing of sampling decisions, whether before (a priori) or after (ongoing) starting data collection, or whether this was unclear. In these cases, we systematically documented the modification to the form and returned to previously abstracted publications to abstract any information that might be relevant to the new field.

The logic of this strategy is analogous to the logic used in a form of research synthesis called best fit framework synthesis (BFFS) [ 23 – 25 ]. In that method, reviewers initially code evidence using an a priori framework they have selected. When evidence cannot be accommodated by the selected framework, reviewers then develop new themes or concepts from which they construct a new expanded framework. Both the strategy proposed and the BFFS approach to research synthesis are notable for their rigorous and transparent means to adapt a final set of concepts to the content under review.

Accounting for inconsistent terminology

An important complication affecting the abstraction process in methods overviews is that the language used by authors to describe methods-related concepts can easily vary across publications. For example, authors from different qualitative research traditions often use different terms for similar methods-related concepts. Furthermore, as we found in the sampling overview [ 18 ], there may be cases where no identifiable term, phrase, or label for a methods-related concept is used at all, and a description of it is given instead. This can make searching the text for relevant concepts based on keywords unreliable.

Principle #6:

Since accepted terms may not be used consistently to refer to methods concepts, it is necessary to rely on the definitions for concepts, rather than keywords, to identify relevant information in the publication to abstract.

Strategy #6:

An effective means to systematically identify relevant information is to develop and iteratively adjust written definitions for key concepts (corresponding to abstraction fields) that are consistent with and as inclusive of as much of the literature reviewed as possible. Reviewers then seek information that matches these definitions (rather than keywords) when scanning a publication for relevant data to abstract.

In the abstraction process for the sampling overview [ 18 ], we noted the several concepts of interest to the review for which abstraction by keyword was particularly problematic due to inconsistent terminology across publications: sampling , purposeful sampling , sampling strategy , and saturation (for examples, see Additional file 1 , Matrices 3a, 3b, 4). We iteratively developed definitions for these concepts by abstracting text from publications that either provided an explicit definition or from which an implicit definition could be derived, which was recorded in fields dedicated to the concept’s definition. Using a method of constant comparison, we used text from definition fields to inform and modify a centrally maintained definition of the corresponding concept to optimize its fit and inclusiveness with the literature reviewed. Table 1 shows, as an example, the final definition constructed in this way for one of the central concepts of the review, qualitative sampling .

We applied iteratively developed definitions when making decisions about what specific text to abstract for an existing field, which allowed us to abstract concept-relevant data even if no recognized keyword was used. For example, this was the case for the sampling-related concept, saturation , where the relevant text available for abstraction in one publication [ 26 ]—“to continue to collect data until nothing new was being observed or recorded, no matter how long that takes”—was not accompanied by any term or label whatsoever.

This comparative analytic strategy (and our approach to analysis more broadly as described in strategy #7, below) is analogous to the process of reciprocal translation —a technique first introduced for meta-ethnography by Noblit and Hare [ 27 ] that has since been recognized as a common element in a variety of qualitative metasynthesis approaches [ 28 ]. Reciprocal translation, taken broadly, involves making sense of a study’s findings in terms of the findings of the other studies included in the review. In practice, it has been operationalized in different ways. Melendez-Torres and colleagues developed a typology from their review of the metasynthesis literature, describing four overlapping categories of specific operations undertaken in reciprocal translation: visual representation, key paper integration, data reduction and thematic extraction, and line-by-line coding [ 28 ]. The approaches suggested in both strategies #6 and #7, with their emphasis on constant comparison, appear to fall within the line-by-line coding category.

Generating credible and verifiable analytic interpretations

The analysis in a systematic methods overview must support its more general objective, which we suggested above is often to offer clarity and enhance collective understanding regarding a chosen methods topic. In our experience, this involves describing and interpreting the relevant literature in qualitative terms. Furthermore, any interpretative analysis required may entail reaching different levels of abstraction, depending on the more specific objectives of the review. For example, in the overview on sampling [ 18 ], we aimed to produce a comparative analysis of how multiple sampling-related topics were treated differently within and among different qualitative research traditions. To promote credibility of the review, however, not only should one seek a qualitative analytic approach that facilitates reaching varying levels of abstraction but that approach must also ensure that abstract interpretations are supported and justified by the source data and not solely the product of the analyst’s speculative thinking.

Principle #7:

Considering the qualitative nature of the analysis required in systematic methods overviews, it is important to select an analytic method whose interpretations can be verified as being consistent with the literature selected, regardless of the level of abstraction reached.

Strategy #7:

We suggest employing the constant comparative method of analysis [ 29 ] because it supports developing and verifying analytic links to the source data throughout progressively interpretive or abstract levels. In applying this approach, we advise a rigorous approach, documenting how supportive quotes or references to the original texts are carried forward in the successive steps of analysis to allow for easy verification.

The analytic approach used in the methods overview on sampling [ 18 ] comprised four explicit steps, progressing in level of abstraction—data abstraction, matrices, narrative summaries, and final analytic conclusions (Fig. 2 ). While we have positioned data abstraction as the second stage of the generic review process (prior to Analysis), above, we also considered it as an initial step of analysis in the sampling overview for several reasons. First, it involved a process of constant comparisons and iterative decision-making about the fields to add or define during development and modification of the abstraction form, through which we established the range of concepts to be addressed in the review. At the same time, abstraction involved continuous analytic decisions about what textual quotes (ranging in size from short phrases to numerous paragraphs) to record in the fields thus created. This constant comparative process was analogous to open coding in which textual data from publications was compared to conceptual fields (equivalent to codes) or to other instances of data previously abstracted when constructing definitions to optimize their fit with the overall literature as described in strategy #6. Finally, in the data abstraction step, we also recorded our first interpretive thoughts in dedicated fields, providing initial material for the more abstract analytic steps.

Summary of progressive steps of analysis used in the methods overview on sampling [ 18 ]

In the second step of the analysis, we constructed topic-specific matrices , or tables, by copying relevant quotes from abstraction forms into the appropriate cells of matrices (for the complete set of analytic matrices developed in the sampling review, see Additional file 1 (matrices 3 to 10)). Each matrix ranged from one to five pages; row headings, nested three-deep, identified the methodological tradition, author, and publication, respectively; and column headings identified the concepts, which corresponded to abstraction fields. Matrices thus allowed us to make further comparisons across methodological traditions, and between authors within a tradition. In the third step of analysis, we recorded our comparative observations as narrative summaries , in which we used illustrative quotes more sparingly. In the final step, we developed analytic conclusions based on the narrative summaries about the sampling-related concepts within each methodological tradition for which clarity, consistency, or comprehensiveness of the available guidance appeared to be lacking. Higher levels of analysis thus built logically from the lower levels, enabling us to easily verify analytic conclusions by tracing the support for claims by comparing the original text of publications reviewed.

Integrative versus interpretive methods overviews

The analytic product of systematic methods overviews is comparable to qualitative evidence syntheses, since both involve describing and interpreting the relevant literature in qualitative terms. Most qualitative synthesis approaches strive to produce new conceptual understandings that vary in level of interpretation. Dixon-Woods and colleagues [ 30 ] elaborate on a useful distinction, originating from Noblit and Hare [ 27 ], between integrative and interpretive reviews. Integrative reviews focus on summarizing available primary data and involve using largely secure and well defined concepts to do so; definitions are used from an early stage to specify categories for abstraction (or coding) of data, which in turn supports their aggregation; they do not seek as their primary focus to develop or specify new concepts, although they may achieve some theoretical or interpretive functions. For interpretive reviews, meanwhile, the main focus is to develop new concepts and theories that integrate them, with the implication that the concepts developed become fully defined towards the end of the analysis. These two forms are not completely distinct, and “every integrative synthesis will include elements of interpretation, and every interpretive synthesis will include elements of aggregation of data” [ 30 ].

The example methods overview on sampling [ 18 ] could be classified as predominantly integrative because its primary goal was to aggregate influential authors’ ideas on sampling-related concepts; there were also, however, elements of interpretive synthesis since it aimed to develop new ideas about where clarity in guidance on certain sampling-related topics is lacking, and definitions for some concepts were flexible and not fixed until late in the review. We suggest that most systematic methods overviews will be classifiable as predominantly integrative (aggregative). Nevertheless, more highly interpretive methods overviews are also quite possible—for example, when the review objective is to provide a highly critical analysis for the purpose of generating new methodological guidance. In such cases, reviewers may need to sample more deeply (see strategy #4), specifically by selecting empirical research reports (i.e., to go beyond dominant or influential ideas in the methods literature) that are likely to feature innovations or instructive lessons in employing a given method.

In this paper, we have outlined tentative guidance in the form of seven principles and strategies on how to conduct systematic methods overviews, a review type in which methods-relevant literature is systematically analyzed with the aim of offering clarity and enhancing collective understanding regarding a specific methods topic. Our proposals include strategies for delimiting the set of publications to consider, searching beyond standard bibliographic databases, searching without the availability of relevant metadata, selecting publications on purposeful conceptual grounds, defining concepts and other information to abstract iteratively, accounting for inconsistent terminology, and generating credible and verifiable analytic interpretations. We hope the suggestions proposed will be useful to others undertaking reviews on methods topics in future.

As far as we are aware, this is the first published source of concrete guidance for conducting this type of review. It is important to note that our primary objective was to initiate methodological discussion by stimulating reflection on what rigorous methods for this type of review should look like, leaving the development of more complete guidance to future work. While derived from the experience of reviewing a single qualitative methods topic, we believe the principles and strategies provided are generalizable to overviews of both qualitative and quantitative methods topics alike. However, it is expected that additional challenges and insights for conducting such reviews have yet to be defined. Thus, we propose that next steps for developing more definitive guidance should involve an attempt to collect and integrate other reviewers’ perspectives and experiences in conducting systematic methods overviews on a broad range of qualitative and quantitative methods topics. Formalized guidance and standards would improve the quality of future methods overviews, something we believe has important implications for advancing qualitative and quantitative methodology. When undertaken to a high standard, rigorous critical evaluations of the available methods guidance have significant potential to make implicit controversies explicit, and improve the clarity and precision of our understandings of problematic qualitative or quantitative methods issues.

A review process central to most types of rigorous reviews of empirical studies, which we did not explicitly address in a separate review step above, is quality appraisal . The reason we have not treated this as a separate step stems from the different objectives of the primary publications included in overviews of the methods literature (i.e., providing methodological guidance) compared to the primary publications included in the other established review types (i.e., reporting findings from single empirical studies). This is not to say that appraising quality of the methods literature is not an important concern for systematic methods overviews. Rather, appraisal is much more integral to (and difficult to separate from) the analysis step, in which we advocate appraising clarity, consistency, and comprehensiveness—the quality appraisal criteria that we suggest are appropriate for the methods literature. As a second important difference regarding appraisal, we currently advocate appraising the aforementioned aspects at the level of the literature in aggregate rather than at the level of individual publications. One reason for this is that methods guidance from individual publications generally builds on previous literature, and thus we feel that ahistorical judgments about comprehensiveness of single publications lack relevance and utility. Additionally, while different methods authors may express themselves less clearly than others, their guidance can nonetheless be highly influential and useful, and should therefore not be downgraded or ignored based on considerations of clarity—which raises questions about the alternative uses that quality appraisals of individual publications might have. Finally, legitimate variability in the perspectives that methods authors wish to emphasize, and the levels of generality at which they write about methods, makes critiquing individual publications based on the criterion of clarity a complex and potentially problematic endeavor that is beyond the scope of this paper to address. By appraising the current state of the literature at a holistic level, reviewers stand to identify important gaps in understanding that represent valuable opportunities for further methodological development.

To summarize, the principles and strategies provided here may be useful to those seeking to undertake their own systematic methods overview. Additional work is needed, however, to establish guidance that is comprehensive by comparing the experiences from conducting a variety of methods overviews on a range of methods topics. Efforts that further advance standards for systematic methods overviews have the potential to promote high-quality critical evaluations that produce conceptually clear and unified understandings of problematic methods topics, thereby accelerating the advance of research methodology.

Hutton JL, Ashcroft R. What does “systematic” mean for reviews of methods? In: Black N, Brazier J, Fitzpatrick R, Reeves B, editors. Health services research methods: a guide to best practice. London: BMJ Publishing Group; 1998. p. 249–54.

Google Scholar

Cochrane handbook for systematic reviews of interventions. In. Edited by Higgins JPT, Green S, Version 5.1.0 edn: The Cochrane Collaboration; 2011.

Centre for Reviews and Dissemination: Systematic reviews: CRD’s guidance for undertaking reviews in health care . York: Centre for Reviews and Dissemination; 2009.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700–0.

Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9(1):59.

Article PubMed PubMed Central Google Scholar

Kastner M, Tricco AC, Soobiah C, Lillie E, Perrier L, Horsley T, Welch V, Cogo E, Antony J, Straus SE. What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Med Res Methodol. 2012;12(1):1–1.

Article Google Scholar

Booth A, Noyes J, Flemming K, Gerhardus A. Guidance on choosing qualitative evidence synthesis methods for use in health technology assessments of complex interventions. In: Integrate-HTA. 2016.

Booth A, Sutton A, Papaioannou D. Systematic approaches to successful literature review. 2nd ed. London: Sage; 2016.

Hannes K, Lockwood C. Synthesizing qualitative research: choosing the right approach. Chichester: Wiley-Blackwell; 2012.

Suri H. Towards methodologically inclusive research syntheses: expanding possibilities. New York: Routledge; 2014.

Campbell M, Egan M, Lorenc T, Bond L, Popham F, Fenton C, Benzeval M. Considering methodological options for reviews of theory: illustrated by a review of theories linking income and health. Syst Rev. 2014;3(1):1–11.

Cohen DJ, Crabtree BF. Evaluative criteria for qualitative research in health care: controversies and recommendations. Ann Fam Med. 2008;6(4):331–9.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reportingqualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Article PubMed Google Scholar

Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.

Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78.

Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005;365(9465):1159–62.

Alshurafa M, Briel M, Akl EA, Haines T, Moayyedi P, Gentles SJ, Rios L, Tran C, Bhatnagar N, Lamontagne F, et al. Inconsistent definitions for intention-to-treat in relation to missing outcome data: systematic review of the methods literature. PLoS One. 2012;7(11):e49163.

Article CAS PubMed PubMed Central Google Scholar

Gentles SJ, Charles C, Ploeg J, McKibbon KA. Sampling in qualitative research: insights from an overview of the methods literature. Qual Rep. 2015;20(11):1772–89.

Harzing A-W, Alakangas S. Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison. Scientometrics. 2016;106(2):787–804.

Harzing A-WK, van der Wal R. Google Scholar as a new source for citation analysis. Ethics Sci Environ Polit. 2008;8(1):61–73.

Kousha K, Thelwall M. Google Scholar citations and Google Web/URL citations: a multi‐discipline exploratory analysis. J Assoc Inf Sci Technol. 2007;58(7):1055–65.

Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A. 2005;102(46):16569–72.

Booth A, Carroll C. How to build up the actionable knowledge base: the role of ‘best fit’ framework synthesis for studies of improvement in healthcare. BMJ Quality Safety. 2015;24(11):700–8.

Carroll C, Booth A, Leaviss J, Rick J. “Best fit” framework synthesis: refining the method. BMC Med Res Methodol. 2013;13(1):37.

Carroll C, Booth A, Cooper K. A worked example of “best fit” framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol. 2011;11(1):29.

Cohen MZ, Kahn DL, Steeves DL. Hermeneutic phenomenological research: a practical guide for nurse researchers. Thousand Oaks: Sage; 2000.

Noblit GW, Hare RD. Meta-ethnography: synthesizing qualitative studies. Newbury Park: Sage; 1988.

Book Google Scholar

Melendez-Torres GJ, Grant S, Bonell C. A systematic review and critical appraisal of qualitative metasynthetic practice in public health to develop a taxonomy of operations of reciprocal translation. Res Synthesis Methods. 2015;6(4):357–71.

Article CAS Google Scholar

Glaser BG, Strauss A. The discovery of grounded theory. Chicago: Aldine; 1967.

Dixon-Woods M, Agarwal S, Young B, Jones D, Sutton A. Integrative approaches to qualitative and quantitative evidence. In: UK National Health Service. 2004. p. 1–44.

Download references

Acknowledgements

Not applicable.

There was no funding for this work.

Availability of data and materials

The systematic methods overview used as a worked example in this article (Gentles SJ, Charles C, Ploeg J, McKibbon KA: Sampling in qualitative research: insights from an overview of the methods literature. The Qual Rep 2015, 20(11):1772-1789) is available from http://nsuworks.nova.edu/tqr/vol20/iss11/5 .

Authors’ contributions

SJG wrote the first draft of this article, with CC contributing to drafting. All authors contributed to revising the manuscript. All authors except CC (deceased) approved the final draft. SJG, CC, KAB, and JP were involved in developing methods for the systematic methods overview on sampling.

Authors’ information

Competing interests.

The authors declare that they have no competing interests.

Consent for publication

Ethics approval and consent to participate, author information, authors and affiliations.

Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada

Stephen J. Gentles, Cathy Charles & K. Ann McKibbon

Faculty of Social Work, University of Calgary, Alberta, Canada

David B. Nicholas

School of Nursing, McMaster University, Hamilton, Ontario, Canada

Jenny Ploeg

CanChild Centre for Childhood Disability Research, McMaster University, 1400 Main Street West, IAHS 408, Hamilton, ON, L8S 1C7, Canada

Stephen J. Gentles

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen J. Gentles .

Additional information

Cathy Charles is deceased

Additional file

Additional file 1:.

Submitted: Analysis_matrices. (DOC 330 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and Permissions

About this article

Cite this article.

Gentles, S.J., Charles, C., Nicholas, D.B. et al. Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research. Syst Rev 5 , 172 (2016). https://doi.org/10.1186/s13643-016-0343-0

Download citation

Received : 06 June 2016

Accepted : 14 September 2016

Published : 11 October 2016

DOI : https://doi.org/10.1186/s13643-016-0343-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Systematic review
Literature selection
Research methods
Research methodology
Overview of methods
Systematic methods overview
Review methods

Systematic Reviews

ISSN: 2046-4053

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

What is Research Methodology? Definition, Types, and Examples

Research methodology 1,2 is a structured and scientific approach used to collect, analyze, and interpret quantitative or qualitative data to answer research questions or test hypotheses. A research methodology is like a plan for carrying out research and helps keep researchers on track by limiting the scope of the research. Several aspects must be considered before selecting an appropriate research methodology, such as research limitations and ethical concerns that may affect your research.

The research methodology section in a scientific paper describes the different methodological choices made, such as the data collection and analysis methods, and why these choices were selected. The reasons should explain why the methods chosen are the most appropriate to answer the research question. A good research methodology also helps ensure the reliability and validity of the research findings. There are three types of research methodology—quantitative, qualitative, and mixed-method, which can be chosen based on the research objectives.

What is research methodology ?

A research methodology describes the techniques and procedures used to identify and analyze information regarding a specific research topic. It is a process by which researchers design their study so that they can achieve their objectives using the selected research instruments. It includes all the important aspects of research, including research design, data collection methods, data analysis methods, and the overall framework within which the research is conducted. While these points can help you understand what is research methodology, you also need to know why it is important to pick the right methodology.

Why is research methodology important?

Helps other researchers who may want to replicate your research; the explanations will be of benefit to them.
You can easily answer any questions about your research if they arise at a later stage.
A research methodology provides a framework and guidelines for researchers to clearly define research questions, hypotheses, and objectives.
It helps researchers identify the most appropriate research design, sampling technique, and data collection and analysis methods.
A sound research methodology helps researchers ensure that their findings are valid and reliable and free from biases and errors.
It also helps ensure that ethical guidelines are followed while conducting research.
A good research methodology helps researchers in planning their research efficiently, by ensuring optimum usage of their time and resources.

Having a good research methodology in place has the following advantages: 3

Types of research methodology

There are three types of research methodology based on the type of research and the data required. 1

Quantitative research methodology focuses on measuring and testing numerical data. This approach is good for reaching a large number of people in a short amount of time. This type of research helps in testing the causal relationships between variables, making predictions, and generalizing results to wider populations.
Qualitative research methodology examines the opinions, behaviors, and experiences of people. It collects and analyzes words and textual data. This research methodology requires fewer participants but is still more time consuming because the time spent per participant is quite large. This method is used in exploratory research where the research problem being investigated is not clearly defined.
Mixed-method research methodology uses the characteristics of both quantitative and qualitative research methodologies in the same study. This method allows researchers to validate their findings, verify if the results observed using both methods are complementary, and explain any unexpected results obtained from one method by using the other method.

What are the types of sampling designs in research methodology?

Sampling 4 is an important part of a research methodology and involves selecting a representative sample of the population to conduct the study, making statistical inferences about them, and estimating the characteristics of the whole population based on these inferences. There are two types of sampling designs in research methodology—probability and nonprobability.

Probability sampling

In this type of sampling design, a sample is chosen from a larger population using some form of random selection, that is, every member of the population has an equal chance of being selected. The different types of probability sampling are:

Systematic —sample members are chosen at regular intervals. It requires selecting a starting point for the sample and sample size determination that can be repeated at regular intervals. This type of sampling method has a predefined range; hence, it is the least time consuming.
Stratified —researchers divide the population into smaller groups that don’t overlap but represent the entire population. While sampling, these groups can be organized, and then a sample can be drawn from each group separately.
Cluster —the population is divided into clusters based on demographic parameters like age, sex, location, etc.
Convenience —selects participants who are most easily accessible to researchers due to geographical proximity, availability at a particular time, etc.
Purposive —participants are selected at the researcher’s discretion. Researchers consider the purpose of the study and the understanding of the target audience.
Snowball —already selected participants use their social networks to refer the researcher to other potential participants.
Quota —while designing the study, the researchers decide how many people with which characteristics to include as participants. The characteristics help in choosing people most likely to provide insights into the subject.

What are data collection methods?

During research, data are collected using various methods depending on the research methodology being followed and the research methods being undertaken. Both qualitative and quantitative research have different data collection methods, as listed below.

Qualitative research 5

One-on-one interviews: Helps the interviewers understand a respondent’s subjective opinion and experience pertaining to a specific topic or event
Document study/literature review/record keeping: Researchers’ review of already existing written materials such as archives, annual reports, research articles, guidelines, policy documents, etc.
Focus groups: Constructive discussions that usually include a small sample of about 6-10 people and a moderator, to understand the participants’ opinion on a given topic.
Qualitative observation : Researchers collect data using their five senses (sight, smell, touch, taste, and hearing).

Quantitative research 6

Sampling: The most common type is probability sampling.
Interviews: Commonly telephonic or done in-person.
Observations: Structured observations are most commonly used in quantitative research. In this method, researchers make observations about specific behaviors of individuals in a structured setting.
Document review: Reviewing existing research or documents to collect evidence for supporting the research.
Surveys and questionnaires. Surveys can be administered both online and offline depending on the requirement and sample size.

What are data analysis methods?

The data collected using the various methods for qualitative and quantitative research need to be analyzed to generate meaningful conclusions. These data analysis methods 7 also differ between quantitative and qualitative research.

Quantitative research involves a deductive method for data analysis where hypotheses are developed at the beginning of the research and precise measurement is required. The methods include statistical analysis applications to analyze numerical data and are grouped into two categories—descriptive and inferential.

Descriptive analysis is used to describe the basic features of different types of data to present it in a way that ensures the patterns become meaningful. The different types of descriptive analysis methods are:

Measures of frequency (count, percent, frequency)
Measures of central tendency (mean, median, mode)
Measures of dispersion or variation (range, variance, standard deviation)
Measure of position (percentile ranks, quartile ranks)

Inferential analysis is used to make predictions about a larger population based on the analysis of the data collected from a smaller population. This analysis is used to study the relationships between different variables. Some commonly used inferential data analysis methods are:

Correlation: To understand the relationship between two or more variables.
Cross-tabulation: Analyze the relationship between multiple variables.
Regression analysis: Study the impact of independent variables on the dependent variable.
Frequency tables: To understand the frequency of data.
Analysis of variance: To test the degree to which two or more variables differ in an experiment.

Qualitative research involves an inductive method for data analysis where hypotheses are developed after data collection. The methods include:

Content analysis: For analyzing documented information from text and images by determining the presence of certain words or concepts in texts.
Narrative analysis: For analyzing content obtained from sources such as interviews, field observations, and surveys. The stories and opinions shared by people are used to answer research questions.
Discourse analysis: For analyzing interactions with people considering the social context, that is, the lifestyle and environment, under which the interaction occurs.
Grounded theory: Involves hypothesis creation by data collection and analysis to explain why a phenomenon occurred.
Thematic analysis: To identify important themes or patterns in data and use these to address an issue.

How to choose a research methodology?

Here are some important factors to consider when choosing a research methodology: 8

Research objectives, aims, and questions —these would help structure the research design.
Review existing literature to identify any gaps in knowledge.
Check the statistical requirements —if data-driven or statistical results are needed then quantitative research is the best. If the research questions can be answered based on people’s opinions and perceptions, then qualitative research is most suitable.
Sample size —sample size can often determine the feasibility of a research methodology. For a large sample, less effort- and time-intensive methods are appropriate.
Constraints —constraints of time, geography, and resources can help define the appropriate methodology.

How to write a research methodology ?

A research methodology should include the following components: 3,9

Research design —should be selected based on the research question and the data required. Common research designs include experimental, quasi-experimental, correlational, descriptive, and exploratory.
Research method —this can be quantitative, qualitative, or mixed-method.
Reason for selecting a specific methodology —explain why this methodology is the most suitable to answer your research problem.
Research instruments —explain the research instruments you plan to use, mainly referring to the data collection methods such as interviews, surveys, etc. Here as well, a reason should be mentioned for selecting the particular instrument.
Sampling —this involves selecting a representative subset of the population being studied.
Data collection —involves gathering data using several data collection methods, such as surveys, interviews, etc.
Data analysis —describe the data analysis methods you will use once you’ve collected the data.
Research limitations —mention any limitations you foresee while conducting your research.
Validity and reliability —validity helps identify the accuracy and truthfulness of the findings; reliability refers to the consistency and stability of the results over time and across different conditions.
Ethical considerations —research should be conducted ethically. The considerations include obtaining consent from participants, maintaining confidentiality, and addressing conflicts of interest.

Frequently Asked Questions

Q1. What are the key components of research methodology?

A1. A good research methodology has the following key components:

Research design
Data collection procedures
Data analysis methods
Ethical considerations

Q2. Why is ethical consideration important in research methodology?

A2. Ethical consideration is important in research methodology to ensure the readers of the reliability and validity of the study. Researchers must clearly mention the ethical norms and standards followed during the conduct of the research and also mention if the research has been cleared by any institutional board. The following 10 points are the important principles related to ethical considerations: 10

Participants should not be subjected to harm.
Respect for the dignity of participants should be prioritized.
Full consent should be obtained from participants before the study.
Participants’ privacy should be ensured.
Confidentiality of the research data should be ensured.
Anonymity of individuals and organizations participating in the research should be maintained.
The aims and objectives of the research should not be exaggerated.
Affiliations, sources of funding, and any possible conflicts of interest should be declared.
Communication in relation to the research should be honest and transparent.
Misleading information and biased representation of primary data findings should be avoided.

Q3. What is the difference between methodology and method?

A3. Research methodology is different from a research method, although both terms are often confused. Research methods are the tools used to gather data, while the research methodology provides a framework for how research is planned, conducted, and analyzed. The latter guides researchers in making decisions about the most appropriate methods for their research. Research methods refer to the specific techniques, procedures, and tools used by researchers to collect, analyze, and interpret data, for instance surveys, questionnaires, interviews, etc.

Research methodology is, thus, an integral part of a research study. It helps ensure that you stay on track to meet your research objectives and answer your research questions using the most appropriate data collection and analysis tools based on your research design.

Research methodologies. Pfeiffer Library website. Accessed August 15, 2023. https://library.tiffin.edu/researchmethodologies/whatareresearchmethodologies
Types of research methodology. Eduvoice website. Accessed August 16, 2023. https://eduvoice.in/types-research-methodology/
The basics of research methodology: A key to quality research. Voxco. Accessed August 16, 2023. https://www.voxco.com/blog/what-is-research-methodology/
Sampling methods: Types with examples. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/types-of-sampling-for-social-research/
What is qualitative research? Methods, types, approaches, examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-qualitative-research-methods-types-examples/
What is quantitative research? Definition, methods, types, and examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-quantitative-research-types-and-examples/
Data analysis in research: Types & methods. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/data-analysis-in-research/#Data_analysis_in_qualitative_research
Factors to consider while choosing the right research methodology. PhD Monster website. Accessed August 17, 2023. https://www.phdmonster.com/factors-to-consider-while-choosing-the-right-research-methodology/
What is research methodology? Research and writing guides. Accessed August 14, 2023. https://paperpile.com/g/what-is-research-methodology/
Ethical considerations. Business research methodology website. Accessed August 17, 2023. https://research-methodology.net/research-methodology/ethical-considerations/

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Scientific writing style guides explained

Language and grammar rules for academic writing, climatic vs. climactic: difference and examples, you may also like, dangling modifiers and how to avoid them in....

Privacy Policy
SignUp/Login

Home » Research Methods – Types, Examples and Guide

Research Methods – Types, Examples and Guide

Table of Contents

Research Methods

Definition:

Research Methods refer to the techniques, procedures, and processes used by researchers to collect , analyze, and interpret data in order to answer research questions or test hypotheses. The methods used in research can vary depending on the research questions, the type of data that is being collected, and the research design.

Types of Research Methods

Types of Research Methods are as follows:

Qualitative research Method

Qualitative research methods are used to collect and analyze non-numerical data. This type of research is useful when the objective is to explore the meaning of phenomena, understand the experiences of individuals, or gain insights into complex social processes. Qualitative research methods include interviews, focus groups, ethnography, and content analysis.

Quantitative Research Method

Quantitative research methods are used to collect and analyze numerical data. This type of research is useful when the objective is to test a hypothesis, determine cause-and-effect relationships, and measure the prevalence of certain phenomena. Quantitative research methods include surveys, experiments, and secondary data analysis.

Mixed Method Research

Mixed Method Research refers to the combination of both qualitative and quantitative research methods in a single study. This approach aims to overcome the limitations of each individual method and to provide a more comprehensive understanding of the research topic. This approach allows researchers to gather both quantitative data, which is often used to test hypotheses and make generalizations about a population, and qualitative data, which provides a more in-depth understanding of the experiences and perspectives of individuals.

Key Differences Between Research Methods

The following Table shows the key differences between Quantitative, Qualitative and Mixed Research Methods

Examples of Research Methods

Examples of Research Methods are as follows:

Qualitative Research Example:

A researcher wants to study the experience of cancer patients during their treatment. They conduct in-depth interviews with patients to gather data on their emotional state, coping mechanisms, and support systems.

Quantitative Research Example:

A company wants to determine the effectiveness of a new advertisement campaign. They survey a large group of people, asking them to rate their awareness of the product and their likelihood of purchasing it.

Mixed Research Example:

A university wants to evaluate the effectiveness of a new teaching method in improving student performance. They collect both quantitative data (such as test scores) and qualitative data (such as feedback from students and teachers) to get a complete picture of the impact of the new method.

Applications of Research Methods

Research methods are used in various fields to investigate, analyze, and answer research questions. Here are some examples of how research methods are applied in different fields:

Psychology : Research methods are widely used in psychology to study human behavior, emotions, and mental processes. For example, researchers may use experiments, surveys, and observational studies to understand how people behave in different situations, how they respond to different stimuli, and how their brains process information.
Sociology : Sociologists use research methods to study social phenomena, such as social inequality, social change, and social relationships. Researchers may use surveys, interviews, and observational studies to collect data on social attitudes, beliefs, and behaviors.
Medicine : Research methods are essential in medical research to study diseases, test new treatments, and evaluate their effectiveness. Researchers may use clinical trials, case studies, and laboratory experiments to collect data on the efficacy and safety of different medical treatments.
Education : Research methods are used in education to understand how students learn, how teachers teach, and how educational policies affect student outcomes. Researchers may use surveys, experiments, and observational studies to collect data on student performance, teacher effectiveness, and educational programs.
Business : Research methods are used in business to understand consumer behavior, market trends, and business strategies. Researchers may use surveys, focus groups, and observational studies to collect data on consumer preferences, market trends, and industry competition.
Environmental science : Research methods are used in environmental science to study the natural world and its ecosystems. Researchers may use field studies, laboratory experiments, and observational studies to collect data on environmental factors, such as air and water quality, and the impact of human activities on the environment.
Political science : Research methods are used in political science to study political systems, institutions, and behavior. Researchers may use surveys, experiments, and observational studies to collect data on political attitudes, voting behavior, and the impact of policies on society.

Purpose of Research Methods

Research methods serve several purposes, including:

Identify research problems: Research methods are used to identify research problems or questions that need to be addressed through empirical investigation.
Develop hypotheses: Research methods help researchers develop hypotheses, which are tentative explanations for the observed phenomenon or relationship.
Collect data: Research methods enable researchers to collect data in a systematic and objective way, which is necessary to test hypotheses and draw meaningful conclusions.
Analyze data: Research methods provide tools and techniques for analyzing data, such as statistical analysis, content analysis, and discourse analysis.
Test hypotheses: Research methods allow researchers to test hypotheses by examining the relationships between variables in a systematic and controlled manner.
Draw conclusions : Research methods facilitate the drawing of conclusions based on empirical evidence and help researchers make generalizations about a population based on their sample data.
Enhance understanding: Research methods contribute to the development of knowledge and enhance our understanding of various phenomena and relationships, which can inform policy, practice, and theory.

When to Use Research Methods

Research methods are used when you need to gather information or data to answer a question or to gain insights into a particular phenomenon.

Here are some situations when research methods may be appropriate:

To investigate a problem : Research methods can be used to investigate a problem or a research question in a particular field. This can help in identifying the root cause of the problem and developing solutions.
To gather data: Research methods can be used to collect data on a particular subject. This can be done through surveys, interviews, observations, experiments, and more.
To evaluate programs : Research methods can be used to evaluate the effectiveness of a program, intervention, or policy. This can help in determining whether the program is meeting its goals and objectives.
To explore new areas : Research methods can be used to explore new areas of inquiry or to test new hypotheses. This can help in advancing knowledge in a particular field.
To make informed decisions : Research methods can be used to gather information and data to support informed decision-making. This can be useful in various fields such as healthcare, business, and education.

Advantages of Research Methods

Research methods provide several advantages, including:

Objectivity : Research methods enable researchers to gather data in a systematic and objective manner, minimizing personal biases and subjectivity. This leads to more reliable and valid results.
Replicability : A key advantage of research methods is that they allow for replication of studies by other researchers. This helps to confirm the validity of the findings and ensures that the results are not specific to the particular research team.
Generalizability : Research methods enable researchers to gather data from a representative sample of the population, allowing for generalizability of the findings to a larger population. This increases the external validity of the research.
Precision : Research methods enable researchers to gather data using standardized procedures, ensuring that the data is accurate and precise. This allows researchers to make accurate predictions and draw meaningful conclusions.
Efficiency : Research methods enable researchers to gather data efficiently, saving time and resources. This is especially important when studying large populations or complex phenomena.
Innovation : Research methods enable researchers to develop new techniques and tools for data collection and analysis, leading to innovation and advancement in the field.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

Assignment – Types, Examples and Writing Guide

References in Research – Types, Examples and...

Figures in Research Paper – Examples and Guide

Delimitations in Research – Types, Examples and...

Qualitative Research – Methods, Analysis Types...

Research Findings – Types Examples and Writing...

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base

Methodology

Primary Research | Definition, Types, & Examples

Primary Research | Definition, Types, & Examples

Published on January 14, 2023 by Tegan George . Revised on June 22, 2023.

Primary research is a research method that relies on direct data collection , rather than relying on data that’s already been collected by someone else. In other words, primary research is any type of research that you undertake yourself, firsthand, while using data that has already been collected is called secondary research .

Primary research is often used in qualitative research , particularly in survey methodology, questionnaires, focus groups, and various types of interviews . While quantitative primary research does exist, it’s not as common.

When to use primary research, types of primary research, examples of primary research, advantages and disadvantages of primary research, other interesting articles, frequently asked questions.

Primary research is any research that you conduct yourself. It can be as simple as a 2-question survey, or as in-depth as a years-long longitudinal study . The only key is that data must be collected firsthand by you.

Primary research is often used to supplement or strengthen existing secondary research. It is usually exploratory in nature, concerned with examining a research question where no preexisting knowledge exists. It is also sometimes called original research for this reason.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

Academic style
Vague sentences
Style consistency

See an example

Primary research can take many forms, but the most common types are:

Surveys and questionnaires
Observational studies
Interviews and focus groups

Surveys and questionnaires collect information about a group of people by asking them questions and analyzing the results. They are a solid choice if your research topic seeks to investigate something about the characteristics, preferences, opinions, or beliefs of a group of people.

Surveys and questionnaires can take place online, in person, or through the mail. It is best to have a combination of open-ended and closed-ended questions, and how the questions are phrased matters. Be sure to avoid leading questions, and ask any related questions in groups, starting with the most basic ones first.

Observational studies are an easy and popular way to answer a research question based purely on what you, the researcher, observes. If there are practical or ethical concerns that prevent you from conducting a traditional experiment , observational studies are often a good stopgap.

There are three types of observational studies: cross-sectional studies , cohort studies, and case-control studies. If you decide to conduct observational research, you can choose the one that’s best for you. All three are quite straightforward and easy to design—just beware of confounding variables and observer bias creeping into your analysis.

Similarly to surveys and questionnaires, interviews and focus groups also rely on asking questions to collect information about a group of people. However, how this is done is slightly different. Instead of sending your questions out into the world, interviews and focus groups involve two or more people—one of whom is you, the interviewer, who asks the questions.

There are 3 main types of interviews:

Structured interviews ask predetermined questions in a predetermined order.
Unstructured interviews are more flexible and free-flowing, proceeding based on the interviewee’s previous answers.
Semi-structured interviews fall in between, asking a mix of predetermined questions and off-the-cuff questions.

While interviews are a rich source of information, they can also be deceptively challenging to do well. Be careful of interviewer bias creeping into your process. This is best mitigated by avoiding double-barreled questions and paying close attention to your tone and delivery while asking questions.

Alternatively, a focus group is a group interview, led by a moderator. Focus groups can provide more nuanced interactions than individual interviews, but their small sample size means that external validity is low.

Primary research can often be quite simple to pursue yourself. Here are a few examples of different research methods you can use to explore different topics.

Primary research is a great choice for many research projects, but it has distinct advantages and disadvantages.

Advantages of primary research

Advantages include:

The ability to conduct really tailored, thorough research, down to the “nitty-gritty” of your topic . You decide what you want to study or observe and how to go about doing that.
You maintain control over the quality of the data collected, and can ensure firsthand that it is objective, reliable , and valid .
The ensuing results are yours, for you to disseminate as you see fit. You maintain proprietary control over what you find out, allowing you to share your findings with like-minded individuals or those conducting related research that interests you for replication or discussion purposes.

Disadvantages of primary research

Disadvantages include:

In order to be done well, primary research can be very expensive and time consuming. If you are constrained in terms of time or funding, it can be very difficult to conduct your own high-quality primary research.
Primary research is often insufficient as a standalone research method, requiring secondary research to bolster it.
Primary research can be prone to various types of research bias . Bias can manifest on the part of the researcher as observer bias , Pygmalion effect , or demand characteristics . It can occur on the part of participants as a Hawthorne effect or social desirability bias .

A faster, more affordable way to improve your paper

Scribbr’s new AI Proofreader checks your document and corrects spelling, grammar, and punctuation mistakes with near-human accuracy and the efficiency of AI!

Proofread my paper

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

Chi square goodness of fit test
Degrees of freedom
Null hypothesis
Discourse analysis
Control groups
Mixed methods research
Non-probability sampling
Quantitative research
Inclusion and exclusion criteria

Research bias

Rosenthal effect
Implicit bias
Cognitive bias
Selection bias
Negativity bias
Status quo bias

The 3 main types of primary research are:

Exploratory research aims to explore the main aspects of an under-researched problem, while explanatory research aims to explain the causes and consequences of a well-defined problem.

There are several methods you can use to decrease the impact of confounding variables on your research: restriction, matching, statistical control and randomization.

In restriction , you restrict your sample by only including certain subjects that have the same values of potential confounding variables.

In matching , you match each of the subjects in your treatment group with a counterpart in the comparison group. The matched subjects have the same values on any potential confounding variables, and only differ in the independent variable .

In statistical control , you include potential confounders as variables in your regression .

In randomization , you randomly assign the treatment (or independent variable) in your study to a sufficiently large number of subjects, which allows you to control for all potential confounding variables.

A questionnaire is a data collection tool or instrument, while a survey is an overarching research method that involves collecting and analyzing data from people using questionnaires.

When conducting research, collecting original data has significant advantages:

You can tailor data collection to your specific research aims (e.g. understanding the needs of your consumers or user testing your website)
You can control and standardize the process for high reliability and validity (e.g. choosing appropriate measurements and sampling methods )

However, there are also some drawbacks: data collection can be time-consuming, labor-intensive and expensive. In some cases, it’s more efficient to use secondary data that has already been collected by someone else, but the data might be less reliable.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). Primary Research | Definition, Types, & Examples. Scribbr. Retrieved November 13, 2023, from https://www.scribbr.com/methodology/primary-research/

Is this article helpful?

Tegan George

Other students also liked, data collection | definition, methods & examples, observer bias | definition, examples, prevention, what is qualitative research | methods & examples, what is your plagiarism score.

Essential components of methods papers

Affiliations.

1 In vitro Toxicology and Biomedicine, Dept inaugurated by the Doerenkamp-Zbinden Foundation, University of Konstanz, Konstanz, Germany.
2 Konstanz Research School Chemical Biology (KoRS-CB) and Co-operative research training school 'Advanced in-vitro test systems for the analysis of cell-chemical interactions in drug discovery and environmental safety (inviTe)', University of Konstanz, Konstanz, Germany.
3 CAAT-Europe, University of Konstanz, Konstanz, Germany.
4 Leibniz Research Centre for Working Environment and Human Factors (IfADo), Technical University of Dortmund, Dortmund, Germany.
5 Archives of Toxicology, editor-in-chief, Heidelberg, Germany.
PMID: 30008013
DOI: 10.14573/altex.1807031

Methods papers are important for the progress of biomedical research, as they provide the essential tools to explore new questions and help to better answer old ones. However, it is often not clear how a methods paper differs from a methods protocol. Confusion between these two very different types of publication is widespread. The resultant misunderstanding contributes to a relatively poor reputation of methods research in biology despite the fact that many Nobel prizes have been awarded specifically for method development. Here, the key components of a methods paper are summarized: (i) methods description, (ii) performance standards, (iii) applicability domains, (iv) evidence for advances compared to the state-of-the-art, (v) exemplification of the method by practical application. In addition, information domains are discussed that are desirable but may be provided on a case-by-case basis or over the course of a series of papers: (vi) method robustness, (vii) accuracy and (viii) precision measures, including various quantifications of method performance, and (ix) measures of uncertainty, including a sensitivity analysis. Finally, elements of the overall framing of the method description are highlighted. These include the scientific, technical and, e.g., toxicological rationale for the method, and also the prediction model, i.e., the procedure used to transform primary data into new information.

Keywords: BenchMarks series.

Publication types

Biomedical Research*
Publications / standards*
Research Design / standards*

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
Explore content
About the journal
Publish with us
Sign up for alerts
Published: 08 July 2022

What makes a Nature Methods paper

Nature Methods volume 19 , pages 771–772 ( 2022 ) Cite this article

7948 Accesses

10 Altmetric

Metrics details

Biological techniques

We explain what our editorial team looks for when considering a methods paper for publication.

Our editorial team members are often asked by potential authors about what we are looking for in a paper. Occasionally, authors express frustration that the editorial ‘triage’ or ‘desk reject’ stage, as it is referred to, feels subject to the whims of an editor or perhaps even their mood that day. However, we can assure you that we have long implemented a robust, multifactor editorial process for selecting papers to send out for peer review.

When a new manuscript arrives in our submission system, the chief editor assigns it to the team member whose expertise most closely matches the paper subject. More information about our editors and their scientific backgrounds is available on our website ; we also invite you to read a summary of how we handle papers in our February 2019 Editorial .

No matter the field of research, there are some common elements that we always look for when judging methods paper submissions. To pass the editorial triage stage, a paper generally needs to check the boxes described below. (Note that here we focus on our Article and Brief Communication content types; other content types we publish are described in our August 2020 Editorial .)

Nature Methods’ mission is to champion method and tool development research in the basic life sciences. Therefore, we consider methods papers with a primarily clinical, diagnostic or therapeutic focus to be out of scope. Methods papers in other fields such as chemistry or physics are also out of scope — unless the authors can make a strong case as to why the paper will make an impact on a broad life sciences community. If uncertain about whether a paper fits our scope, you are welcome to submit a presubmission inquiry via our submission system .

Though our broad scope covers all of the basic life sciences, there are certain major fields where we are particularly interested in publishing papers — for example, single cell analysis, genomics and transcriptomics, microscopy and imaging, structural biology, proteomics, metabolomics, genome engineering, stem cell biology, neuroscience and immunology. Within these fields, there are particular areas that represent the frontiers of methods development where we are most keen to receive papers. To learn more about what areas are piquing our interest, we invite you to read our recent Methods to Watch features, published in every January issue.

Novelty is a key element of a Nature Methods paper. In a paper’s introduction, we look for a clear explanation as to why a method or tool is a substantial advance over the state of the art. We also assess the paper in the context of the peer-reviewed literature to be confident that elements of the method or a very similar approach have not been previously reported by the authors or by another lab. That is not to say that we will not consider a strong paper in an area where there are already other published similar methods — we weigh the timeliness of the topic, the practical value of the method, the performance compared to other approaches, and whether the paper makes a strong case that the method will enable new applications (more on this below). We aim to publish a mix of papers with high conceptual novelty and high immediate practical value.

Method description

We want to publish methods that will be useable by others. It should therefore go without saying that method must be described in detail. The focus on the paper should be on the method and its characterization, not on new biological findings obtained. To enable method reuse, we often require that authors describing a complex experimental workflow provide a step-by-step protocol as a supplementary item (or better yet, deposit it in a protocol repository); authors must also describe their plan to distribute any unique experimental materials. To ensure that software tools are useable, we require a detailed description of the underlying algorithms, the code (ideally hosted in a code repository and assigned a DOI), a license and a user guide.

Validation and benchmarking

Strong validation of a method’s performance is an essential ingredient of a Nature Methods paper. Whether an experimental approach or computational tool, authors should always take care to follow established field standards for best practices when validating its performance.

Experimental methods should be applied to at least one well-characterized system to demonstrate that the method produces expected results. Computational tools should be validated on a ground truth or gold standard dataset if available in the field. Simulated datasets, ideally with noise added to make the data more realistic, are also useful for validation, but we nearly always also want to see tests on real experimental datasets.

If similar methods have already been published, we also expect to see some performance benchmarking. This process can be somewhat fraught, as authors do not always have the technical expertise or access required to utilize different technologies or may not be sufficiently knowledgeable about the ins and outs of running another group’s software. We rely on our expert reviewers to help us judge whether the new method has been appropriately compared to existing methods.

General applicability

We aim to publish methods that will be broadly applicable to life science researchers. Methods that are limited to studying a particular biological process may be too narrow in focus for our journal to consider. We also typically want to see data showing that good method performance is not just a one-off for a well-behaved system, but that it performs well with a diversity of systems or datasets. Just how many examples need to be shown to prove general applicability is field dependent, but two distinct applications is typically the minimum.

Challenging demonstration

Though our editorial focus is on the method itself, a ‘killer application’ can go a long way in showing readers why they should care about a method and perhaps consider adopting it in their own research. However, we are flexible about this – not every paper we publish has a killer application; it really depends on how well the paper has checked the other boxes. New biological results are not required for publication in Nature Methods — though it certainly doesn’t hurt. Preliminary biological findings are often okay by us as long as conclusions are not overhyped and limitations are stated.

If a paper fails to check all of these boxes, does that mean it will be desk-rejected? Not necessarily. We often will send papers missing a few key elements for peer review, but we will let the authors know as early as possible that the paper will need some work to bring it up to our editorial standards before potential publication.

Many journals publish methods papers, but we think that our editorial standards are what makes a Nature Methods paper unique. As always, we welcome your feedback about how to better serve our authors and readers.

Rights and permissions

Reprints and Permissions

About this article

Cite this article.

What makes a Nature Methods paper. Nat Methods 19 , 771–772 (2022). https://doi.org/10.1038/s41592-022-01558-4

Download citation

Published : 08 July 2022

Issue Date : July 2022

DOI : https://doi.org/10.1038/s41592-022-01558-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

Explore articles by subject
Guide to authors
Editorial policies

This paper is in the following e-collection/theme issue:

Published on 8.11.2023 in Vol 25 (2023)

The Status of Data Management Practices Across German Medical Data Integration Centers: Mixed Methods Study

Authors of this article:

Original Paper

Kerstin Gierend 1 , Dipl Inf (FH) ;
Sherry Freiesleben 2 , MSc ;
Dennis Kadioglu 3, 4 , MSc ;
Fabian Siegel 1 , MD ;
Thomas Ganslandt 5 * , MD ;
Dagmar Waltemath 2 * , Dr -lng

1 Department of Biomedical Informatics at the Center for Preventive Medicine and Digital Health, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany

2 Core Unit Data Integration Center and Medical Informatics Laboratory, University Medicine Greifswald, Greifswald, Germany

3 Institute for Medical Informatics (IMI), Goethe University Frankfurt, University Hospital, Frankfurt am Main, Germany

4 Department for Information and Communication Technology (DICT), Data Integration Center (DIC), Goethe University Frankfurt, University Hospital, Frankfurt am Main, Germany

5 Chair of Medical Informatics, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

*these authors contributed equally

Corresponding Author:

Kerstin Gierend, Dipl Inf (FH)

Department of Biomedical Informatics at the Center for Preventive Medicine and Digital Health

Medical Faculty Mannheim

Heidelberg University

Theodor-Kutzer-Ufer 1-3

Mannheim, 68167

Phone: 49 621383 ext 8087

Email: [email protected]

Background: In the context of the Medical Informatics Initiative, medical data integration centers (DICs) have implemented complex data flows to transfer routine health care data into research data repositories for secondary use. Data management practices are of importance throughout these processes, and special attention should be given to provenance aspects. Insufficient knowledge can lead to validity risks and reduce the confidence and quality of the processed data. The need to implement maintainable data management practices is undisputed, but there is a great lack of clarity on the status.

Objective: Our study examines the current data management practices throughout the data life cycle within the Medical Informatics in Research and Care in University Medicine (MIRACUM) consortium. We present a framework for the maturity status of data management practices and present recommendations to enable a trustful dissemination and reuse of routine health care data.

Methods: In this mixed methods study, we conducted semistructured interviews with stakeholders from 10 DICs between July and September 2021. We used a self-designed questionnaire that we tailored to the MIRACUM DICs, to collect qualitative and quantitative data. Our study method is compliant with the Good Reporting of a Mixed Methods Study (GRAMMS) checklist.

Results: Our study provides insights into the data management practices at the MIRACUM DICs. We identify several traceability issues that can be partially explained with a lack of contextual information within nonharmonized workflow steps, unclear responsibilities, missing or incomplete data elements, and incomplete information about the computational environment information. Based on the identified shortcomings, we suggest a data management maturity framework to reach more clarity and to help define enhanced data management strategies.

Conclusions: The data management maturity framework supports the production and dissemination of accurate and provenance-enriched data for secondary use. Our work serves as a catalyst for the derivation of an overarching data management strategy, abiding data integrity and provenance characteristics as key factors. We envision that this work will lead to the generation of fairer and maintained health research data of high quality.

Introduction

Data integration centers (DICs) within the German Medical Informatics Initiative (MII) have evolved rapidly in the past years [ 1 - 4 ]. DICs process and provide digital medical data for the secondary use in research. The foundation of data sharing (DS) and interoperability within the MII is an agreed-upon common core data set (CDS). The basic modules are generic and include data items encoding laboratory results, diagnosis, procedures, or medication data. The extension modules contain domain-specific data such as oncology or microbiology data [ 5 ]. The CDS data items are processed using a standardized extract-transform-load (ETL) development process that follows the data life cycle ( Figure 1 ). Specific testing measures throughout the data processing chain are implemented to ensure accuracy and high quality. The architecture of every Medical Informatics in Research and Care in University Medicine (MIRACUM) DIC (see also Figure 1 ) is built upon the medical informatics reusable ecosystem of open source linkable and interoperable software tools [ 6 ]. Data requests by researchers are limited to and based on generic institutional policies and a defined legal framework. The concrete status of the DICs with respect to enabling the findable, accessible, interoperable, reusable (FAIR) principles still needs to be determined [ 7 ]. However, several initiatives have already outlined the importance of applying the FAIR principles for both input and output data [ 8 - 10 ].

The data life cycle describes the journey of biomedical data from data collection to final analysis and publication ( Figure 1 ). Particularly when working with (sensitive) patient data, the understanding of the data’s origin and the relationship between an element and its predecessors, also called traceability (see Textbox 1 ), is highly relevant for legal requirements and a fundamental prerequisite of data quality. “Black box” processing and reporting of findings based on routine data should no longer be acceptable [ 11 ] since it may lead to loss of data and contextual knowledge about the data [ 12 ]. This is a reason why the DICs faces an increasing pressure to implement thorough data management concepts, in particular provenance. An option is the adoption of generic provenance concepts from the World Wide Web Consortium (W3C) [ 13 ]. However, the application of these concepts requires insights and understanding of the data management tasks in the given context.

Insufficient information about data formation processes and metadata (see Textbox 1 ) pose validity risks and can impede the quality assessment of extracted clinical data and related processes. Data with unknown provenance and lack of traceability endure from a confidence deficiency and therefore minimize the acceptance for secondary use.

Provenance (World Wide Web Consortium [W3C] working definition): “Provenance of a resource is a record that describes entities and processes involved in producing and delivering or otherwise influencing that resource. Provenance provides a critical foundation for assessing authenticity, enabling trust, and allowing reproducibility. Provenance assertions are a form of contextual metadata and can themselves become important records with their own provenance” [ 13 ].
W3C provenance: is a family of specifications for provenance with a generic concept to express specific meta-information (or metadata) about data and its related artifacts. Provenance records contain the agents (eg, people and institutions), entities (eg, data sources and data elements), and activities (eg, extract, load, and transform), involved in producing, influencing, or delivering a piece of data or a thing. The granularity of the W3C provenance concepts influences the level of traceable data management activities [ 13 ]. Provenance can be distinguished as data and workflow provenance [ 14 ].
Meta-information (or metadata): machine understandable information for the web [ 15 ]. Metadata contain substantial characteristics to express (provenance) information for any kind of artifacts during data managing and play a crucial role in the implementation of the findable, accessible, interoperable, reusable (FAIR) principles [ 7 ].
Traceability: ability to retain the identity of the product and its origin [ 11 ]. Traceability is essential to ensure data integrity and trust in the data [ 16 ]. In our study traceability is the ability to trace (identify and measure) all the steps that led to a particular point in a data transformation process. Traceability assumes enrichment of data with proper meta-information.

In this work, we seek clarification about the data management processes in German DICs. We aim to facilitate a comprehensive understanding and transparency of these processes to boost data reliability and integrity. We therefore ran a mixed method study across all MIRACUM DICs to get a picture of current traceability and verifiability of patient data and metadata processed from heterogeneous clinical data sources. We expect that DICs would benefit from an increased focus on governance of data management practices rather than random or only partly managed data processing. To support the change, we offer a maturity framework which can be implemented in DICs for self-evaluation. We hypothesize that the framework will foster the implementation of improved data management processes, transparency, traceability, and better provenance tracing.

Study Design

This study uses a mixed methods design [ 17 ] and associated best practices [ 18 ]. A mixed methods design leads to more plausible and comprehensible quantitative outcomes if combined with qualitative statements. The design involved the collection of qualitative and quantitative data in a single interview and subsequent analysis to strengthen the study’s conclusions. The collection of quantitative and qualitative data was performed concurrently on the same survey and with the same priority. The study has been reported according to the Good Reporting of a Mixed Methods Study (GRAMMS) checklist [ 19 ] ( Multimedia Appendix 1 ). Based on the survey results and discussions among the authors, a maturity framework was developed, following the capability maturity model (CMM) [ 20 ].

Study Settings and Participants

The study was performed as a semistructured interview. The interview questions cover clinical data processing and provenance practices within the DICs. The results from a MIRACUM workshop on FAIR data management and discussions with data experts from different DICs contributed to the design of the questionnaires. In addition, we build the questions upon insights from a survey on the research field of provenance [ 14 ].

For this work, we distinguished data management operations that concern the data integration (DI) phase (blue items in Figure 1 ) from operations concerning the DS phase (orange items in Figure 1 ). Thus, the interview questions were split into 2 separate questionnaires, containing 16 questions (DI) and 38 questions (DS), respectively. The DI questions covered data management activities during the extraction, transformation, and loading of electronic health records. DS questions comprised available documentation of resources, activities for DS processes, and organizational information. The interview does not cover the management of patients’ consent since it is a precondition for data processing and release from the DICs [ 3 ].

The questions were numbered and grouped by subject. A mixture of open and closed questions was chosen to get a more comprehensive insight into the respective fields. The questionnaires were created in German language and pilot-tested internally with data experts.

Stakeholders from each MIRACUM site participated in the interview. We provided the questionnaires in advance with the option to delegate the task to accountable staff members. This kept both the interviewer and the participant in line, avoided distractions, and encouraged an open communication. Participants consent was obtained in written form ahead of the actual interview.

A total of 10 DICs (all MIRACUM sites) were invited to participate. We subsequently collected data from all sites with 22 participants, thereof 4 women and 18 men contributors. Due to the COVID-19 pandemic, all interviews were conducted virtually. The interviewing person shared the screen with the questionnaire displayed on it while the interview was conducted. Qualitative and quantitative data were collected in German language based on the participant’s answers during the interview phase. All data were concurrently entered into a database (Research Electronic Data Capture [REDCap; Vanderbilt University]) by the interviewing person during the interview [ 21 ]. The data collection took between 1.5 hours and 4 hours per DIC. Overall, the data collection period lasted over 3 months. Due to the interview technique no missing data occurred.

Data Collection

The data collection method relied on asking questions within the predetermined thematic framework. Even for closed questions, there was always the option to ask additional questions and to store the answers.

The data collection included quantitative and qualitative data with equal emphasis ( Multimedia Appendix 2 ). The qualitative data were collected in free-text fields and during the interviews with stakeholder professionals ( Multimedia Appendix 2 ). The data collection took between 1.5 hours and 4 hours per DIC. Before starting the data analysis, all collected data were translated into English and covalidated.

Data Analysis

After performing the semistructured interview, we conducted a thematic analysis. We converted or transformed qualitative data into quantitative scores or constructs by “coding” the qualitative responses into different groups. We identified common topics or patterns and ensured that these patterns appropriately represent the participants’ responses using the 4-eyes-principle.

The analyses were conducted anonymously without identifying the respective DIC. The tables and figures outline the individual characteristics and frequency counts were calculated. The categorical variables are described using counts and percentages, if applicable. The data were described using median and range for the continuous variables, if applicable. The figures were created with R (version 4.2.0; The R Foundation) [ 22 ]. Qualitative, free-text data were read, analyzed, and coded, if necessary. The narratives representing the coded themes were produced from the data material. The data analysis was reviewed by all authors.

Integration

Qualitative data were combined with quantitative data whenever possible. Thus, the qualitative results were integrated with the corresponding quantitative results and then presented numerically. The outcome was reported as descriptive statistical results. Whenever integration was not possible, we reported qualitative results instead. After analysis of the qualitative and quantitative data, the preliminary findings were presented and discussed among the authors.

Ethical Considerations

The ethics approval was waived by the University of Heidelberg or Mannheim University Medicine Ethics Committee II. Informed consent was obtained from all subjects (the stakeholders) to participate in the interview about the status of their data processing pipelines. All study data are deidentified. The participants did not receive any compensation.

In our study, we seek clarification about the data management processes in German DICs. We aim to facilitate a comprehensive understanding and transparency of the prevailing practices for data extractions, data transformations, data storage, and data provision to boost data reliability and integrity. We first present the main survey outcomes, and then we introduce a maturity framework.

Results Overview

All 10 DICs of the MIRACUM consortium participated in the survey between July and October 2021. All 22 participants, either the head of a DIC or a member of the technical staff, responded to a total of 66 questions, thereof 16 questions about the DI phase and 12 questions about the locally used data elements and catalogs from the MII CDS. A total of 9 DICs answered the 38 DS specific question ( Table 1 ); data from the Core Unit Data Integration Center at the University Medicine Greifswald is missing.

General and Organizational Matters

Expectation regarding provenance.

The interview revealed considerable expectations regarding the collection and use of provenance and metadata information, also beyond the W3C provenance definition ( Table 2 ). Interestingly, the most common expectations were associated with the assessment of data quality (n=7), with traceability and information capability (n=7), and with the transparency in processing steps, workflows, or data sets (n=2). Other frequently named expectations were linked to technical reasons (n=4) such as debugging or performance evaluation. Less frequent terms included compliance with regulations (n=2), reproducibility, support of scientific usage process, or increased confidence in data. Expectations like clear regulation of responsible parties, interoperability, and increased acceptance were mentioned once (n=1). In this, 1 DIC stated no usage of provenance information at all.

Self-Assessment of Provenance Experience

When analyzing the data in Figure 2 , we observed a low provenance experience. More than half of the DICs ranked their provenance experience as a starter level with a score 0-3 (n=6). The 3 sites reported an advanced level with a score 4-7. Just 1 site rated their experience with a score of 8 (corresponding to expert level).

Organizational Structure

Consistent with the W3C provenance model [ 13 ] and Herschel et al [ 14 ], the organizational component of a DIC represents a core unit at many German medical faculties. When asked for the organizational prerequisites, all DICs reported that specifications of the manufacturer systems and standard operating procedures (SOPs) were available. However, the degree of maturity varies across DICs.

At the time of the interview, all DICs (n=10, 100%) were in a continuous development process with drafted SOPs at different levels. However, some DICs (n=3) already reported gaps in their SOPs, preventing the full coverage of process flows for DI and DS. Nearly half of the sites (n=4, 40%) used already approved SOPs. Roles and responsibilities, as central parts of the SOP, had been defined in most DICs (n=8, 80%). Only a few DICs (n=3, 30%) had a dedicated role concept (Figure S1 in Multimedia Appendix 3 ).

Availability of Metadata and Related Tool Usage

No consequent and targeted practice for provenance capture could be determined. We hypothesize that it might be difficult to develop a standardized, structured, and machine-readable metadata schema across all German university hospitals. Similar results regarding insufficient availability of (semantic) metadata for provenance were observed [ 23 ]. Detailed results for the individual questions are given in Figure S2 in Multimedia Appendix 3 .

Metadata Exploitation During Data Management

The development of metadata schemata is an important factor for high traceability [ 16 ]. Hence, we were interested in learning how organizational and document resources might help to generate metadata and to embed metadata within the digital object itself. This analysis section targeted the annotation status such as labeling of data elements, data sets, or tagging of files. Detailed results are available in Figures S3-S5 in Multimedia Appendix 3 .

Documentation Matters

All interviewees declared that data management activities were not subjected to specific data management planning or tools. Any planning or preparational documentation was collectively performed using tools such as JIRA (Atlassian) [ 24 ] or Confluence (Atlassian) [ 25 ]. During the DS phase, most DICs (n=8) follow internal SOPs for the documentation of methods, or data management plans, respectively. All other DICs reported that internal, project-specific tools were applied. Processes were partially under construction.

Documentation Artifacts From Data Elements and Coding in Data Integration Phase

Appreciably all sites (n=10) reported about their level of documentation for accessing the source systems, for the maintenance of the data elements, for code development and execution, as well as the content of log files as part of their ETL-process as described below.

Annotation of Data Elements

As expected, and in line with the literature [ 26 ], preliminary attempts for data annotation exist. However, these attempts do not yet cover the whole processing pipeline in all DICs. The applied annotation approaches vary, too. It is noteworthy that the best, and partially automatic annotation was yielded on the joint segment Fast Healthcare Interoperability Resources (FHIR) to the research data repository (RDR; n=10), since this pipeline is part of the MIRACUM standard ETL process [ 3 ]. Detailed results are available in Figure S3 in Multimedia Appendix 3 .

Log Files for Improved Traceability

Log files are text-based files, which include timestamps, store events, processes, and transactions. Thus, log files provide valuable provenance information. However, a direct access to log files was not possible during this study due to the risk of disclosing critical or sensitive information.

Most DICs (n=9) already established log files to trace environment and execution information, particularly during DI. In most cases, the log files contain configurable parameters and elements, mostly generated within the respective infrastructure framework. Some frameworks comprised self-defined information and messages for error, warning, and execution statistics. Depending on the actual process, short- or long-term retention could be observed. Long-term retention was applied for data transfer logs and short-term for application logs, for example, throughout the ETL life cycle. Half of the DICs (n=5) reported that the access to source systems is automatically logged with user information and time stamps, but without relationship to the particular data elements. In general, access to the source-application itself is not possible. More than two-thirds of the centers (n=4) have manual logging features in place. Only 1 DIC does not perform any logging (n=1).

Only sparse information was provided about the computational environment and execution workflows during runtime of scripts in productive operation (Figure S4 in Multimedia Appendix 3 ). A small number of sites (n=2) reported that automated and collaboratively accessible information were created. The 3 centers (n=3) said that no such information was generated. All other survey participants (up to n=7) asserted that the logging protocols were either compiled manually or generated automatically. Based on the survey data no systematic approach was deducible.

However, half of the DICs reported that scripts which are executed during the data requesting phase often do not produce log files. If log files were produced by the scripts, they contained information about execution and error history (n=4). Many DICs emphasized their capability of access-logging to data pools, computational environments, and execution history. The recorded information includes details about Docker containers such as the software status of the environment. However, some data seems to be missing in the logs, including the date of execution or the user account. Logs from the RDR Informatics for Integrating Biology and the Bedside indicated access logins (who and when) and querying of data elements. Extraction protocols were produced for some source systems.

Versioning Information Status

Version information, an important element for reproducible research, creates a history for each file. Based on the annotation of the source code and artifacts, for example, the used programming language, provenance data can create relationships between individual elements or documents. Figure S5 in Multimedia Appendix 3 illustrates that the generated code was in general subjected to textual documentation, and code was mostly versioned in a DI pipeline (n=10). GitHub is mainly used as version control tool. Importantly, the DI segment FHIR to the RDR provided full versioning capability in all DICs (n=10). The reason is that MIRACUM developed and delivered a centralized component for this workflow step [ 3 ]. Also, code for the ETL segment for data processing from staging servers to a FHIR repository is highly version controlled. Lower coverage was observed on the initial stretch source system to staging (n=7). In this, 1 interviewee explained that this circumstance was due the code being in the responsibility of the manufacturer. Another expert said that code was managed manually (n=1). In 1 case, no version control was implemented at all. The situation is similar when data is queried by scripts for research purposes since code versioning was tool-guided by the most DICs (n=7). Overall, the results suggest that version information is available, but needs to be prepared in more detail to be useful for provenance processing.

Documentation Artifacts of Testing Procedures and Script Validation

A considerably high number of DICs confirmed the implementation of test procedures (n=8) and data quality measures (n=9; Figure 3 ).

Notably, different test documentation strategies were reported by the stakeholders (Figure S6 in Multimedia Appendix 3 ). Most sites (n=4) mentioned the provision of an automated testing documentation which was collaboratively accessible for the authorized staff during the data integrity measurements pipeline.

Data quality is mainly assessed using the Data Quality Assessment reporting tool, which has been developed within MIRACUM (n=6) [ 27 ]. A total of 2 DICs used self-developed assessment and documentation of data quality.

All DICs validate their data querying scripts. Evidence for validation is provided by manually documenting the queries in a structured and permanently accessible way on GitLab (GitLab Inc) [ 28 ] or JIRA [ 24 ] (n=5). Unstructured evidence such as the four-eyes principle was practiced in 4 DICs.

Documentation Artifacts From Final Review and Facts About Research Result Objects

All previous processes and individual outcomes contribute to the history of the so-called result object. We anticipate that the result object should contain all provenance-related metadata. As shown in Figure S7 in Multimedia Appendix 3 , most participants (n=5) examine all the documentation and artifacts for traceability. Applied examination methods included the 4-eyes-principle, random sample checks, or ETL checklists with defined examination criteria. Approximately one-third of the respondents (n=3) indicated that the traceability of documentation and related artifacts was not checked. Only 1 DIC has plans to check traceability systematically and automated. Remarkably, examination of the result object regarding adherence to FAIR principles and provenance assessment was not performed in any DIC. These findings indicate a lack of awareness for FAIR data management, as has also been observed in a recent survey within the German Network University Medicine [ 29 ].

Derivation of a Maturity Framework

On the basis of our study results, we derived a data integration center toward maturity framework (DIC2MF), which incorporates the specific needs and metadata items of German DICs ( Figure 4 ). The DIC2MF indicates a DIC’s readiness status for provenance tracking (“provenance power”) and can be used as a benchmarking tool.

Dimensions and Categories of the Framework

The DIC2MF concept is based on the CMM. Unlike the already published maturity model for provenance management, which was established in the hydro- and geoscientists’ field [ 30 ], our approach comprises 6 dimensions and related categories ( Figure 4 ) which together constitute provenance characteristics that a DIC requires to be effective in delivering traceable and reliable patient data for secondary use. The dimensions and categories were influenced by the grouping of key interview findings from (1) related organizational, legal, and technical conditions, (2) the metadata exploitation based on data annotation and documentation degree and associated operations, and (3) including the measures to ensure quality during the different operations. Textbox 2 elucidates the proposed framework and the associated characteristics.

Each dimension is represented by a specific ability level. Figure 5 depicts the different gradations of the 5 ability levels “unmanaged,” “incipient,” “controlled,” “operational,” and “optimized.” Each level describes a degree of traceability fulfillment and is an indicator for the provenance power in the DIC. The completeness and quality of traceability goes hand in hand with the levels of maturity. An instantiation of the framework is shown in Figure 6 .

Roles and responsibilities (staff, roles, and training)
Standard operating procedures (quality management)
Regulations (eg, general data protection regulation and patient consent)
Risk management (controlling risks)
Input sources
Output sets
Data elements
Considers information from log files created during data conversion, for example, to cover the facets of provenance according to the World Wide Web Consortium provenance recommendation (see Textbox 1 )
Common data model
Domain specific catalogs
Testing procedures
Validation approach
Organization and reporting
Long-term archiving
Data security of patient data
Data accessibility of patient data
Infrastructure and computation environment
Tools and software
It should be mentioned that relationships exist between the dimensions, for example, data processing must adhere to given data governance rules

Instantiation of the Framework

The inner circle ( Figure 6 ) represents the grouped 6 data management dimensions for provenance tracking (following the specification in Figure 4 ). Each dimension contains multiple categories and each category reflects the substantial characteristics for the expression of provenance. The quality of provenance expression can be derived from the ability scale (between 0 and 4) which defines how reliably and maintainably the implemented practices within a DIC can produce the required outcomes. The higher the bar the more provenance information is available. Thus, the height of the bar is an indicator for the need to improve data management practices given on the description of the ability level. For example, progress from 1 maturity level to the next one may be reached by adding fine-granular metadata in compliance with the W3C provenance components agent, activity, and entity in a second step. The presented concepts are a first step toward identifying the requirements for traceability within a DIC.

Principal Findings

We successfully performed a mixed method study and gained deep insight into the status of data management processes in the German medical DICs. Our work facilitates understanding and traceability and will potentially boost the reliability and integrity of data for secondary use. We derived a maturity framework and applied it as a benchmark to measure the degree of traceability and deriving from this the provenance power of individual data elements in MIRACUM German DICs. The proposed maturity framework for provenance readiness helps DICs to identify their conceptual bottlenecks in provenance tracking and increases trustful dissemination of clinical data.

We hypothesize that our work could serve as a catalyst for an overarching data management strategy for DICs. The beneficial approach presented here could be implemented widely as a common assessment tool, within the MII structure and in the medical research field itself.

Framework Applicability

The framework can be used for critical systematic self-evaluation. It can guide the identification of relevant components for provenance tracking and thus facilitate traceability of patient’s data processing. The information obtained from the framework dimensions A to F help to develop the necessary metadata, and consequently enhance traceability on process and element level.

Establishing Traceability and Best Practices

Establishing traceability is one of the biggest challenges associated with any data conversion. A combination of several aspects may lead to the condition that traceability has not been implemented effectively at the DICs. Predominantly, a lack of awareness and provenance expertise could be a key finding from the self-assessment of provenance experience ( Figure 2 ) and indicates a subordinate role of provenance to date. A lack of technological framework may furthermore hinder the uptake of provenance in the data processing pipelines. Here, the traceability issue can be linked to a lack of granularity including details about workflow steps and about the processed data elements themselves. ETL pipelines are mostly implemented individually by the DICs. Practices in the highly ranked centers for provenance expertise revealed that these include annotation and metadata documentation, even if it is not always machine-readable and automatically recorded.

A tentative explanation is that there is no systematic approach for gathering provenance data of individual data items ( Table 2 ). The procedure of tracking data set or data items is neither formalized nor sufficiently standardized. Consequently, no targeted provenance collection and metadata concept has been established as of now. In addition, sparsely developed traceability decreases the reliability and thus the quality of single data elements for secondary use ( Figure 3 ). Even if general testing procedures are available in the DI pipelines, there is a lack in quality traceability.

The following examples showcase how DICs may increase their maturity level by using the proposed framework dimensions and categories while connecting metadata to the associated artifact: (1) dimension A foresees (a) guidance on data managing activities, like define operations by SOPs, introduce data management plans, and consider legal restrictions and (b) regular data management training for the responsible staff. Connect both topics at least on data and process level. (2) The challenge of dimension B could be passed step by step (a) while gaining and deriving targeted annotation from log files for building and filling the maturity framework on a data element level, log files are configurable and enable the traceable storage of events so that these can be analyzed and optimized. In this way, log files thus help to track data and their processes, and to reconstruct transactions. Elements of log files could be selected as in the proposed framework, for example, source and target system, information about type of event or logged action, version or actor; (b) by having appropriate, clear, and complete documentation for all measured data in place, if possible, in machine-actionable way and connect this information to the data; (c) by making metadata accessible and adding richer prospective and retrospective provenance metadata. These actions will allow for fine-grained versioning workflows linking to outputs produced during the distinct executions of ETL pipelines. The metadata approach should consider information derived from the W3C components agents (such as developer and data owner), activities (such as different programming scripts), entities (such as data sources or data elements). (3) Convert the extracted data into common and interoperable health care standards as defined in dimension C and connect the associated metadata information to your processed data as described in dimension B. (4) Testing and validation (dimension D) approaches add quality information to the processed data itself. Collect available metadata on applied activities to ensure data quality as given in dimension B. (5) Dimension E, dedicated to the DS phase enriches a data element with information from the data requesting, reporting, and archiving phase. (6) Dimension F intends to collect meta-information about the operational environment in which the data were collected and processed.

Related Work

Provenance tracking and granularity issues were addressed in different papers [ 31 , 32 ]. Gierend et al [ 33 ] performed a scoping review on provenance in biomedical research and offered comprehensive results concerning the practical application of provenance and the associated challenges, including aspects like completeness and validation and provenance granularity issues. Curcin et al [ 34 ] reported that both data and processes need provenance, gathered in consistent, interoperable manner to make research results verifiable and reproducible. These works directed our study approach to examine the traceability aspect. Johns et al [ 35 ] tried to figure out knowledge on provenance methods in a more general way. Regarding the term provenance, Herschel et al [ 14 ] pointed to the definition of provenance, which leaves room for many different interpretations of and approaches to provenance and investigated the question why capturing provenance is useful. This led us to clearly define the goal of our study and give clear expectations regarding provenance accomplishment. Furthermore, this might give clear expectations regarding provenance accomplishment and provide the framework for the scope and the extent of implementation measures. In the same way, the outcome of our study can be used by the recently launched community-driven project which aims to define a “MInimal Requirements for Automated Provenance Information Enrichment” guideline [ 36 ]. The projects’ goal is to build a general data model and semantics for provenance in the biomedical community.

Training issues were addressed as a challenge of poor data management practice [ 26 ]. Better health informatics training and permanent data manager and software architect positions are demanded in health research groups. This indicates that our maturity framework needs an iterative and interdisciplinary approach to implement traceability in data processing pipelines.

Lessons Learnt

During the conduction of the semistructured interview and the implementation of our framework, we learnt that the extent of the complex processing steps requires interdisciplinary team work to come to a proper level of provenance granularity. We are convinced that the community will benefit from a consequent exchange with stakeholders from different areas of expertise, like medical experts, data owners, and computer scientist. In addition, we encountered a major increase of transparency and traceability since we started with a consequent application of the maturity framework approach in our DIC. Moreover, having data governance in place, would facilitate the FAIR oriented data management planning and as such boost the data asset to be more reliable and trustful for or in the research field. Another recommendation is to spend more time on training in this field.

Ongoing Processes

Changing conditions in clinical routine, in granularity of requirements (decision-making, identifier management, and legal matters) demand continuous adaptation of the framework. We foresee extensions for provenance representation and storage, provenance retrieval, and usability along discussion for risk and benefit.

There are recent advancements to transform the dimension and categories into the W3C provenance concepts. We introduce a first provenance implementation in our DIC in Mannheim (University Medicine Mannheim DIC) in a proof-of-concept study in peer review phase.

Limitations

Our investigation is limited to the MIRACUM DICs, to their current service profiles and development stages as well as to the experience of the involved staff. Since provenance data are sporadically available, we were not able to consider maintainability aspects of provenance. Derivation of qualitative and quantitative results to the framework levels was performed by means of an evaluative description of metadata availability and the ability of traceable data. Integration of pseudonymization and consent management are external processes and not in primary scope for this study.

Conclusions

Implementing traceable data life cycles and transparent data management processes are sophisticated and challenging tasks, not only for the MIRACUM DICs. Notwithstanding, sufficient traceability would enable data to be a trusted asset in the medical DIC. Our paper provides insights on how institutions (attempt to) implement data management principles to provide clinical routine data for secondary use. However, to implement traceability, explainability of the relationships and the order between the data and process elements are required. We discussed the extensive transformations, curations, and linked artifacts of collected data elements and workflows during the entire data life cycle. The obtained insights led us to identify possible improvements and actions. One such action is the introduction of a maturity framework which visualizes the specific traceability challenges on a technical and organizational level observed at each DIC. In future, we seek to derive a generic provenance model and common data provenance strategy based on the traceability findings. To this end, we will investigate how complete provenance, as part of a FAIR data management strategy, can be delivered and what the limitations are in this regard. We envision that this work will lead to FAIR and maintained health and research data of high quality.

Acknowledgments

We would like to thank Achim Michel-Backofen, Albert Vass, Christian Haverkamp, Christian Seidemann, Daniel Schmitz, Dennis Kadioglu, Detlef Kraska, Jakub Lidke, Jan Maluche, Jan Sparenberg, Jonathan Mang, Kim Hee Eun, Noemi Deppenwiese, Robert Gött, Sebastian Stöcker, Sherry Freiesleben, Stephan Wojke, Tim Herrmann, and Thomas Litschko, who are the participants in the interview. We thank the reviewers for their constructive comments and valuable suggestions. This work was part of the precondition for obtaining the degree Dr sc. hum. from the University Heidelberg. This research is funded by the German Federal Ministry of Education and Research within the German Medical Informatics Initiative (FKZ 01ZZ1801A, 01ZZ1801E and 01ZZ1801M) and within the Network University Medicine (FKZ 01KX2121). For the publication fee we acknowledge financial support by Deutsche Forschungsgemeinschaft within the funding programme „Open Access Publikationskosten“ as well as by Heidelberg University.

Authors' Contributions

KG developed the questionnaires and conducted the semistructured interview, prepared data for analysis, performed the graphical analysis for the maturity framework, drafted all sections of the paper, coordinated reviewing, incorporated the comments from the coauthors, and submitted the paper. SF did the graphical analysis of the survey data and worked on the paper. DK contributed to editorial revision, finalization of the paper, and content support, especially in general on the topic of metadata. FS contributed to the discussion of the maturity framework and to the interpretation of data. DW contributed to the discussion of the concept, advisory support during preparation of the interview, editorial revision, and finalization of the paper. TG contributed to the discussion of the concept and finalization of the paper. All authors reviewed and approved the final version of the paper.

Conflicts of Interest

None declared.

Good reporting of a mixed methods study checklist [ 19 ].

Data collection items (in English and German language).

Additional results from the survey.

Semler SC, Wissing F, Heyder R. German medical informatics initiative. Methods Inf Med 2018;57(S 01):e50-e56 [ https://www.thieme-connect.de/products/ejournals/html/10.3414/ME18-03-0003 ] [ CrossRef ] [ Medline ]
Data Integration Centers. MIRACUM. URL: https://www.miracum.org/en/das-konsortium/datenintegrationszentren/ [accessed 2022-09-22]
Prokosch HU, Acker T, Bernarding J, Binder H, Boeker M, Boerries M, et al. MIRACUM: medical informatics in research and care in university medicine. Methods Inf Med 2018;57(S 01):e82-e91 [ https://www.thieme-connect.de/products/ejournals/html/10.3414/ME17-02-0025 ] [ CrossRef ] [ Medline ]
Data integration centres. Medical Informatics Initiative Germany. URL: https://www.medizininformatik-initiative.de/en/consortia/data-integration-centres [accessed 2022-09-22]
The Medical Informatics Initiative's core data set. Medical Informatics Initiative Germany. URL: https://www.medizininformatik-initiative.de/index.php/en/medical-informatics-initiatives-core-data-set [accessed 2022-09-22]
MIRACOLIX-Tools. MIRACUM. URL: https://www.miracum.org/en/das-konsortium/datenintegrationszentren/miracolix-tools/ [accessed 2022-09-22]
Jacobsen A, de Miranda Azevedo R, Juty N, Batista D, Coles S, Cornet R, et al. FAIR principles: interpretations and implementation considerations. Data Intell 2020;2(1-2):10-29 [ https://direct.mit.edu/dint/article/2/1-2/10/10017/FAIR-Principles-Interpretations-and-Implementation ] [ CrossRef ]
Vasilevsky NA, Brush MH, Paddock H, Ponting L, Tripathy SJ, Larocca GM, et al. On the reproducibility of science: unique identification of research resources in the biomedical literature. PeerJ 2013;1:e148 [ https://peerj.com/articles/148/ ] [ CrossRef ] [ Medline ]
Mehrabi N, Morstatter F, Saxena N, Lerman K, Galstyan A. A survey on bias and fairness in machine learning. ACM Comput Surv 2021;54(6):1-35 [ CrossRef ]
Hasselbring W, Carr L, Hettrick S, Packer H, Tiropanis T. From FAIR research data toward FAIR and open research software. IT 2020;62(1):39-47 [ https://www.degruyter.com/document/doi/10.1515/itit-2019-0040/html ] [ CrossRef ]
de Lusignan S, Liaw ST, Krause P, Curcin V, Vicente MT, Michalakidis G, et al. Key concepts to assess the readiness of data for international research: data quality, lineage and provenance, extraction and processing errors, traceability, and curation: contribution of the IMIA primary health care informatics working group. Yearb Med Inform 2018;20(01):112-120 [ https://www.thieme-connect.de/products/ejournals/pdf/10.1055/s-0038-1638748.pdf ] [ CrossRef ]
Weng C. Clinical data quality: a data life cycle perspective. Biostat Epidemiol 2020;4(1):6-14 [ https://europepmc.org/abstract/MED/32258941 ] [ CrossRef ] [ Medline ]
PROV-overview. W3C Working Group Note. URL: https://www.w3.org/TR/prov-overview/ [accessed 2022-09-22]
Herschel M, Diestelkämper R, Lahmar HB. A survey on provenance: what for? What form? What from? VLDB J 2017;26(6):881-906 [ CrossRef ]
Metadata and resource description. W3C Technology and Society domain. URL: https://www.w3.org/Metadata/ [accessed 2023-09-01]
Hume S, Sarnikar S, Noteboom C. Enhancing traceability in clinical research data through a metadata framework. Methods Inf Med 2020;59(2-03):75-85 [ CrossRef ] [ Medline ]
Kelley K, Clark B, Brown V, Sitzia J. Good practice in the conduct and reporting of survey research. Int J Qual Health Care 2003;15(3):261-266 [ https://academic.oup.com/intqhc/article/15/3/261/1856193?login=false ] [ CrossRef ] [ Medline ]
Dowding D. Best practices for mixed methods research in the health sciences John W. Creswell, Ann Carroll Klassen, Vicki L. Plano Clark, Katherine Clegg Smith for the office of behavioral and social sciences research; qualitative methods overview Jo Moriarty. Qual Soc Work 2013;12(4):541-545 [ CrossRef ]
O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy 2008;13(2):92-98 [ CrossRef ] [ Medline ]
Humphrey WS. Characterizing the software process: a maturity framework. IEEE Softw 1988;5(2):73-79 [ CrossRef ]
Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O'Neal L, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform 2019;95:103208 [ https://www.sciencedirect.com/science/article/pii/S1532046419301261?via%3Dihub ] [ CrossRef ] [ Medline ]
The R project for statistical computing. R. 2020. URL: https://www.R-project.org/ [accessed 2023-10-18]
Razick S, Močnik R, Thomas LF, Ryeng E, Drabløs F, Sætrom P. The eGenVar data management system--cataloguing and sharing sensitive data and metadata for the life sciences. Database (Oxford) 2014;2014:bau027 [ https://academic.oup.com/database/article/doi/10.1093/database/bau027/2634026?login=false ] [ CrossRef ] [ Medline ]
Jira software features. ATLASSIAN. URL: https://www.atlassian.com/software/jira/features [accessed 2022-09-22]
ATLASSIAN. URL: https://www.atlassian.com/software/confluence/features [accessed 2022-09-22]
Curcin V, Soljak M, Majeed A. Managing and exploiting routinely collected NHS data for research. Inform Prim Care 2012;20(4):225-231 [ https://access.portico.org/Portico/auView?auId=ark:%2F27927%2Fphw1p05j244 ] [ CrossRef ] [ Medline ]
Kapsner LA, Mang JM, Mate S, Seuchter SA, Vengadeswaran A, Bathelt F, et al. Linking a consortium-wide data quality assessment tool with the MIRACUM metadata repository. Appl Clin Inform 2021;12(4):826-835 [ https://www.thieme-connect.de/products/ejournals/html/10.1055/s-0041-1733847 ] [ CrossRef ] [ Medline ]
Kamoun C, Roméjon J, de Soyres H, Gallois A, Girard E, Hupé P. Biogitflow: development workflow protocols for bioinformatics pipelines with git and GitLab. F1000Res 2020;9:632 [ https://f1000research.com/articles/9-632/v3 ] [ CrossRef ] [ Medline ]
Michaelis L, Poyraz RA, Muzoora MR, Gierend K, Bartschke A, Dieterich C, et al. Insights into the FAIRness of the German network university medicine: a survey. Stud Health Technol Inform 2023;302:741-742 [ CrossRef ] [ Medline ]
Taylor K, Woodcock R, Cuddy S, Thew P, Lemon D. A provenance maturity model. In: Denzer R, Argent RM, Schimak G, Hrebícek J, editors. Environmental Software Systems Infrastructures, Services and Applications. Germany: Springer International Publishing; 2015:1-18
Groth P, Moreau L. Representing distributed systems using the open provenance model. Future Gener Comput Syst 2011;27(6):757-765 [ https://www.sciencedirect.com/science/article/abs/pii/S0167739X10001949?via%3Dihub ] [ CrossRef ]
Guedes T, Martins LB, Falci MLF, Silva V, Ocaña KACS, Mattoso M, et al. Capturing and analyzing provenance from spark-based scientific workflows with SAMbA-RaP. Future Gener Comput Syst 2020;112:658-669 [ https://www.sciencedirect.com/science/article/abs/pii/S0167739X1931742X?via%3Dihub ] [ CrossRef ]
Gierend K, Krüger F, Waltemath D, Fünfgeld M, Ganslandt T, Zeleke AA. Approaches and criteria for provenance in biomedical data sets and workflows: protocol for a scoping review. JMIR Res Protoc 2021;10(11):e31750 [ https://www.researchprotocols.org/2021/11/e31750 ] [ CrossRef ] [ Medline ]
Curcin V, Miles S, Danger R, Chen Y, Bache R, Taweel A. Implementing interoperable provenance in biomedical research. Future Gener Comput Syst 2014;34:1-16 [ https://www.sciencedirect.com/science/article/abs/pii/S0167739X13002653?via%3Dihub ] [ CrossRef ]
Johns M, Meurers T, Wirth FN, Haber AC, Müller A, Halilovic M, et al. Data provenance in biomedical research: scoping review. J Med Internet Res 2023;25:e42289 [ https://www.jmir.org/2023/1/e42289 ] [ CrossRef ] [ Medline ]
Gierend K, Wodke JAH, Genehr S, Gött R, Henkel R, Krüger F, et al. TAPP: Defining standard provenance information for clinical research data and workflows—Obstacles and opportunities. New York, NY, United States: Association for Computing Machinery; 2023 Presented at: WWW '23 Companion: Companion Proceedings of the ACM Web Conference 2023; 30 April 2023-4 May 2023; Austin TX USA p. 1551-1554 URL: https://dl.acm.org/doi/10.1145/3543873.3587562 [ CrossRef ]

Abbreviations

Edited by A Mavragani; submitted 08.05.23; peer-reviewed by M Sicilia, T Miksa, N Howe; comments to author 19.08.23; revised version received 09.09.23; accepted 29.09.23; published 08.11.23

©Kerstin Gierend, Sherry Freiesleben, Dennis Kadioglu, Fabian Siegel, Thomas Ganslandt, Dagmar Waltemath. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 08.11.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

IMAGES

Research Methods
Research Methodology Sample Paper : 5 An Example
🎉 Research paper methodology example. Methology Sample for You. 2019-01-21
Example Method Paper / How to Write Research Methodology: 13 Steps
Qualitative Research Paper Introduction / Http Fampra Oxfordjournals
Example Of Methodology In Research Paper

VIDEO

Ethics in Clinical Research Part I
Techniques of Somatometry & Somatoscopy
BUSINESS RESEARCH METHODS I QUESTION PAPER ANSWERING
Research Paper Methodology
How to Write a Problem Statement for Research
Caste-Tribe Interaction

COMMENTS

Research Methods
Research methods are specific procedures for collecting and analyzing data. Developing your research methods is an integral part of your research design. When planning your methods, there are two key decisions you will make. First, decide how you will collect data. Your methods depend on what type of data you need to answer your research question:
What Is a Research Methodology?
Step 1: Explain your methodological approach Step 2: Describe your data collection methods Step 3: Describe your analysis method Step 4: Evaluate and justify the methodological choices you made Tips for writing a strong methodology chapter Other interesting articles Frequently asked questions about methodology How to write a research methodology
How to write the Methods section of a research paper
How to write the Methods section of a research paper Popular This article is in Methods Dr. Dhriti Bhattacharyya Sep 18, 2018 Reading time 8 mins The Methods section of a research article is like a roadmap leading to the core of the research, guiding the readers through the actual journey the authors took to reach their destination.
How to Write an APA Methods Section
The methods section of an APA style paper is where you report in detail how you performed your study. Research papers in the social and natural sciences often follow APA style. This article focuses on reporting quantitative research methods.
How to Write Your Methods
A methods section that enables other researchers to understand and replicate your results is a constant principle of rigorous, transparent, and Open Science. Aim to be thorough, even if a particular journal doesn't require the same level of detail . Reproducibility is all of our responsibility.
How to Write a Research Paper
A research paper is a piece of academic writing that provides analysis, interpretation, and argument based on in-depth independent research. Research papers are similar to academic essays, but they are usually longer and more detailed assignments, designed to assess not only your writing skills but also your skills in scholarly research.
What Is Qualitative Research?
Common approaches include grounded theory, ethnography, action research, phenomenological research, and narrative research. They share some similarities, but emphasize different aims and perspectives.
Writing the Research Paper
Compile surveys, collect data, gather materials for quantitative analysis (if these are good methods to investigate the topic more deeply). Come up with new ideas about the topic. Try to formulate your ideas in a few sentences. Write a short outline of your future paper. Review your notes and other materials and enrich the outline.
Organizing Your Social Sciences Research Paper
The methods section describes actions taken to investigate a research problem and the rationale for the application of specific procedures or techniques used to identify, select, process, and analyze information applied to understanding the problem, thereby, allowing the reader to critically evaluate a study's overall validity and reliability.
Research Methods: What are research methods?
What are research methods. Research methods are the strategies, processes or techniques utilized in the collection of data or evidence for analysis in order to uncover new information or create better understanding of a topic. There are different types of research methods which use different tools for data collection.
How to Write a Research Methodology in 4 Steps
Learn how to write a strong methodology chapter that allows readers to evaluate the reliability and validity of the research. A good methodology chapter incl...
Organizing Academic Research Papers: 6. The Methodology
I. Groups of Research Methods. There are two main groups of research methods in the social sciences: The empirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences. This type of research focuses on objective knowledge, research questions that can be answered yes or no, and operational definitions of variables to be measured.
Quantitative Methods
Quantitative methods emphasize objective measurements and the statistical, mathematical, or numerical analysis of data collected through polls, questionnaires, and surveys, or by manipulating pre-existing statistical data using computational techniques.Quantitative research focuses on gathering numerical data and generalizing it across groups of people or to explain a particular phenomenon.
How to write the methods section of a research paper
The methods section of a research paper provides the information by which a study's validity is judged. Therefore, it requires a clear and precise description of how an experiment was done, and the rationale for why specific experimental procedures were chosen.
How to Write Research Methodology: Overview, Tips, and Techniques
Methodology in research is defined as the systematic method to resolve a research problem through data gathering using various techniques, providing an interpretation of data gathered and drawing conclusions about the research data. Essentially, a research methodology is the blueprint of a research or study (Murthy & Bhojanna, 2009, p. 32).
How to Write a Methods Section of an APA Paper
The methods section of an APA format psychology paper provides the methods and procedures used in a research study or experiment. This part of an APA paper is critical because it allows other researchers to see exactly how you conducted your research. Method refers to the procedure that was used in a research study.
Reviewing the research methods literature: principles and strategies
Background Overviews of methods are potentially useful means to increase clarity and enhance collective understanding of specific methods topics that may be characterized by ambiguity, inconsistency, or a lack of comprehensiveness. This type of review represents a distinct literature synthesis method, although to date, its methodology remains relatively undeveloped despite several aspects that ...
What is Research Methodology? Definition, Types, and Examples
Research methodology 1,2 is a structured and scientific approach used to collect, analyze, and interpret quantitative or qualitative data to answer research questions or test hypotheses. A research methodology is like a plan for carrying out research and helps keep researchers on track by limiting the scope of the research.
Research Methods
Definition: Research Methods refer to the techniques, procedures, and processes used by researchers to collect, analyze, and interpret data in order to answer research questions or test hypotheses. The methods used in research can vary depending on the research questions, the type of data that is being collected, and the research design.
Papers on research methods: The hidden gems of the research literature
A research methods paper that presents a data analysis software is the contribution of Stiglic, Watson, and Cilar . The researchers present R, a package in the public domain, and provide the code in R for a confirmatory factor analysis. Our last special issue paper is not a research methods paper. Nor is it any of our traditional paper types.
Primary Research
Revised on June 22, 2023. Primary research is a research method that relies on direct data collection, rather than relying on data that's already been collected by someone else. In other words, primary research is any type of research that you undertake yourself, firsthand, while using data that has already been collected is called secondary ...
Essential components of methods papers
Here, the key components of a methods paper are summarized: (i) methods description, (ii) performance standards, (iii) applicability domains, (iv) evidence for advances compared to the state-of-the-art, (v) exemplification of the method by practical application.
PDF Research Papers in the Sciences
Research Papers in the Sciences, Fall 2019. 1 of 7 Research Papers in the Sciences (Undergraduate) Scientific research is shared through scientific research papers. It's considered part of the duty of ... One method to address this issue is to remove the oxidized copper. As vinegar is known to remove rust from iron, there seemed to be a ...
What makes a Nature Methods paper
Scope. Nature Methods' mission is to champion method and tool development research in the basic life sciences. Therefore, we consider methods papers with a primarily clinical, diagnostic or ...
Essay Sample on Research Method and Design
Free download. This research design intends to provide a suitable framework for studying life skills as a buffer to the negative effects of adverse childhood experiences. The study will employ a mixed-type of method. It will consist of a set of well-structured questionnaires and an interview with experts on the topic of the subject.
Pedagogy of Experimental Design: Scientific research methods in
Semantic Scholar extracted view of "Pedagogy of Experimental Design: Scientific research methods in architectural education" by Yaara Schattner. ... Semantic Scholar's Logo. Search 214,771,017 papers from all fields of science. Search. Sign In Create Free Account. DOI: 10.21606/iasdr.2023.833; Corpus ID: 263724904;
CRediT author statement
Methodology. Development or design of methodology; creation of models. Software. Programming, software development; designing computer programs; implementation of the computer code and supporting algorithms; testing of existing code components. Validation. Verification, whether as a part of the activity or separate, of the overall replication ...
Journal of Medical Internet Research
Background: In the context of the Medical Informatics Initiative, medical data integration centers (DICs) have implemented complex data flows to transfer routine health care data into research data repositories for secondary use. Data management practices are of importance throughout these processes, and special attention should be given to provenance aspects.

How to write the Methods section of a research paper

Related Reading

The correct way to report p values

How to choose the research methodology best suited for your study

Manuscript structure: How to convey your most important ideas through…

Trending Searches

Recent Searches

Ensure understanding, reproducibility and replicability

Why Methods Matter

A clear methods section impacts editorial evaluation and readers’ understanding, and is also the backbone of transparency and replicability.

What to include in your methods section

A constant principle of rigorous science

Imagine replicating your own work, years in the future

Visual aids for methods help when reading the whole paper

Ethical Considerations

Existing standards, checklists, guidelines, partners

Summary Writing tips

Don’t

Advanced Research Methods

Before Writing the Paper

Academic Expression

Research Methods: What are research methods?

What are research methods

Types of research

Data collection tools

Organizing Academic Research Papers: 6. The Methodology

Importance of a Good Methodology Section

Structure and Writing Style

Writing Tip

Another Writing Tip

Research Support

Using the Library

User Information

My Accounts

Organizing Your Social Sciences Research Paper

Need Help Locating Statistics?

Characteristics of Quantitative Research

Basic Research Design for Quantitative Studies

Strengths of Using Quantitative Methods

Limitations of Using Quantitative Methods

Research Tip

How to Write a Methods Section for a Psychology Paper

What to Include in a Method Section

Components of a Method Section

Participants

Tips for How to Write a Methods Section

A Word From Verywell

Frequently Asked Questions

Reviewing the research methods literature: principles and strategies illustrated by a systematic overview of sampling in qualitative research

Conclusions

The example systematic methods overview on sampling in qualitative research

Organization of the guidance into principles and strategies

Results and discussion

Delimiting a manageable set of publications

Principle #1:

Strategy #1:

Searching beyond standard bibliographic databases

Principle #2:

Strategy #2:

Searching without relevant metadata

Principle #3:

Strategy #3:

Purposefully selecting literature on conceptual grounds

Principle #4:

Strategy #4:

Data abstraction

Iteratively defining conceptual information to abstract

Principle #5:

Strategy #5:

Accounting for inconsistent terminology

Principle #6:

Strategy #6:

Generating credible and verifiable analytic interpretations

Principle #7:

Strategy #7:

Integrative versus interpretive methods overviews

Acknowledgements

Availability of data and materials

Authors’ contributions

Authors’ information