Advertisement

Advertisement

Mathematical modeling for theory-oriented research in educational technology

  • Development Article
  • Published: 29 November 2021
  • Volume 70 , pages 149–167, ( 2022 )

Cite this article

  • Elena Novak   ORCID: orcid.org/0000-0003-0873-2081 1  

759 Accesses

2 Altmetric

Explore all metrics

Mathematical modeling describes how events, concepts, and systems of interest behave in the world using mathematical concepts. This research approach can be applied to theory construction and testing by using empirical data to evaluate whether the specific theory can explain the empirical data or whether the theory fits the data available. Although extensively used in the physical sciences and engineering, as well as some social and behavioral sciences to examine theoretical claims and form predictions of future events and behaviors, theory-oriented mathematical modeling is less common in educational technology research. This article explores the potential of using theory-oriented mathematical modeling for theory construction and testing in the field of educational technology. It presents examples of how this approach was used in social, behavioral, and educational disciplines, and provides rationale for why educational technology research can benefit from a theory-oriented model-testing approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

research paper on mathematical modeling pdf

Adapted from Bessière et al. ( 2006 )

research paper on mathematical modeling pdf

Adapted from Anderson ( 2011 )

research paper on mathematical modeling pdf

Adapted from Picciano ( 2017 )

research paper on mathematical modeling pdf

Adapted from Means et al. ( 2014 )

Anderson, T. (2008). The theory and practice of online learning (2nd ed.). AU Press.

Google Scholar  

Anderson, T. (2011). The theory and practice of online learning . AU Press.

Atkinson, R. C., & Schiffrin, R. M. (1971). The control of short-term memory. Scientific American, 225 , 82–90.

Atkinson, R. C., & Shiffrin, R. M. (1968a). Human memory: A proposed system and its control processes. In K. W. Spence & J. T. Spence (Eds.), The psychology of learning and motivation (Volume 2) (pp. 89–195). Academic Press.

Atkinson, R. C., & Shiffrin, R. M. (1968b). Human memory: A proposed system and its control processes. In K. W. Spence & J. T. Spence (Eds.), The psychology of learning and motivation: Advances in research and theory (Vol 2) (pp. 89–195). Academic Press.

Bessière, K., Newhagen, J. E., Robinson, J. P., & Shneiderman, B. (2006). A model for computer frustration: The role of instrumental and dispositional factors on incident, session, and post-session frustration and mood. Computers in Human Behavior, 22 (6), 941–961. https://doi.org/10.1016/j.chb.2004.03.015

Article   Google Scholar  

Blömeke, S., & Kaiser, G. (2011). Homogeneity or heterogeneity? Profiles of opportunities to learn in primary teacher education and their relationship to cultural context and outcomes. ZDM Mathematics Education, 44 (3), 249–264. https://doi.org/10.1007/s11858-011-0378-6

Boland, L. A. (2014). Model building in economics: Its purposes and limitations . Cambridge University Press.

Bulfin, S., Henderson, M., Johnson, N. F., & Selwyn, N. (2014). Methodological capacity within the field of “educational technology” research: An initial investigation. British Journal of Educational Technology, 45 (3), 403–414.

Cilesiz, S., & Spector, J. M. (2014). The philosophy of science and educational technology research. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (pp. 875–884). New York: Springer.

Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 20 , 116–117.

Dabbaghian, V., & Mago, V. K. (2014). Theories and simulations of complex social systems . Springer.

Dym, C. (2004). Principles of mathematical modeling . Academic Press.

Ertmer, P. A. (1999). Addressing first- and second-order barriers to change: Strategies for technology integration. Educational Technology Research and Development, 47 (4), 47–61.

Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education model. The Internet and Higher Education, 2 (2–3), 87–105.

Goe, L. (2007). The link between teacher quality and student outcomes: A research synthesis . Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved from http://ncctq.learningpt.org/publications/LinkBetweenTQandStudentOutcomes.pdf

Gollwitzer, P. M. (1999). Implementation intentions. Strong effects of simple plans. American Psychologist, 54 (7), 493–503.

Harasim, L. (2012). Learning theory and online technologies . Routledge/Taylor & Francis.

Hew, K. F., & Brush, T. (2007). Integrating technology into K-12 teaching and learning: Current knowledge gaps and recommendations for future research. Education Technology Research and Development, 55 , 223–252. https://doi.org/10.1007/s11423-006-9022-5

Hew, K. F., Lan, M., Tang, Y., Jia, C., & Lo, C. K. (2019). Where is the “theory” within the field of educational technology research? British Journal of Educational Technology . https://doi.org/10.1111/bjet.12770

Hoffman, B. (2010). “I think I can, but I’m afraid to try”: The role of self-efficacy beliefs and mathematics anxiety in mathematics problem-solving efficiency. Learning and Individual Differences, 20 (3), 276–283.

Hoffman, R. (2003). Why buy that theory? In O. Sacks (Ed.), The best American science writing: 2003 (pp. 222–227). Harper-Collins.

Hollander, E. P. (1967). Principles and methods of social psychology . Oxford University Press.

Holmberg (1985).  The feasibility of t heory of teaching for distance education and a proposed theory (ZIFF Paiere 60). Hagen, West Germany: Fern Universitat, Zentrales Institute fur Fernstudienforscgung Arbeitsbereich. (ERIC Document Reproduction Service No. ED290013).

Huang, W.-H., Huang, W.-Y., & Tschopp, J. (2010). Sustaining iterative game playing processes in DGBL: The relationship between motivational processing and outcome processing. Computers & Education, 55 (2), 789–797.

Jaccard, J., & Jacoby, J. (2009). Theory construction and model building skills: A practical guide for social scientists . Guilford Press.

Johns, G. (2006). The essential impact of context on organizational behavior. Academy of Management Review, 31 (2), 386–408.

Jones, C., & Czerniewicz, L. (2011). Theory in learning technology. Research in Learning Technology, 19 (3), 173–177.

Keller, J. M. (1999). Using the ARCS motivational process in computer-based instruction and distance education. New Directions for Teaching and Learning, 78 , 39–48.

Keller, J. M. (2008). An integrative theory of motivation, volition, and performance. Technology, Instruction, Cognition and Learning, 16 , 79–104.

Knowles, M. S., Holton, E. F., & Swanson, R. A. (1998). The adult learner (5th ed.). Butterworth-Heinemann Publishers.

Kuhl, J. (1987). Action control: The maintenance of motivational states. In F. Halisch & J. Kuhl (Eds.), Motivation, intention and volition (pp. 279–291). Springer.

Kuhn, T. S. (1996). The structure of scientific revolutions . University of Chicago Press.

Malone, T. W. (1985). Designing organizational interfaces. In L. Borman & R. Smith (Eds.), Proceedings of the CHI’85 Conference on Human Factors in Computing Systems (pp. 66–71). New York, NY: ACM Press.

Markauskaite, L., & Reimann, P. (2014). Editorial: e-Research for education: Applied, methodological and critical perspectives. British Journal of Educational Technology, 45 (3), 385–391.

Mayer, R. E. (2001). Multimedia learning . Cambridge University Press.

McDonnell, L. M. (1995). Opportunity to learn as a research concept and a policy instrument. Educational Evaluation and Policy Analysis, 17 (3), 305–322.

McKenney, S., & Reeves, T. C. (2014). Educational design research. In J. M. Spector, M. D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (pp. 131–140). Springer.

Means, B., Bakia, M., & Murphy, R. (2014). Learning online: What research tells us about whether, when and how . Routledge.

Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115 , 1–47.

Mintzberg, H. (2005). Developing theory about the development of theory. In M. Hitt & K. Smith (Eds.), Minds in management: The process of theory development (pp. 355–372). Oxford University Press.

Moore, M. G. (1997). Theory of transactional distance. In D. Keegan (Ed.), Theoretical principles of distance education (pp. 22–38). Routledge.

Moore, M. G., & Diehl, W. C. (2018). Handbook of distance education . Routledge.

Morgan, C., & Wildemuth, B. M. (2009). Questions related to theory. Applications of social research methods to questions in information and library science (pp. 40–50). Libraries Unlimited.

Nelson, R., & Winter, S. (1974). Neoclassical vs. evolutionary theories of economic growth: Critique and prospectus. Economic Journal, 84 (336), 886–905.

Novak, E. (2014). Toward a mathematical model of motivation, volition, and performance. Computers & Education, 74 , 73–80. https://doi.org/10.1016/j.compedu.2014.01.009

Novak, E., Daday, J., & McDaniel, K. (2018). Using a mathematical model of motivation, volition, and performance to examine students’ e-text learning experiences. Educational Technology Research & Development, 66 (5), 1189–1209. https://doi.org/10.1007/s11423-018-9599-5

Novak, E., McDaniel, K., Daday, J., & Soyturk, I. (2021). Understanding student frustration with e-learning materials: Development and validation of an E-Text Frustration scale . Featured Research Paper presented at the Association for Educational Communications and Technology (AECT), Chicago, IL. November 2021.

Opp, K.-D. (1970). Theories of the middle range as a strategy for the construction of a general sociological theory. Quality and Quantity, 4 (2), 243–253. https://doi.org/10.1007/BF00199565

Pearl, J. (2000). Causality: Models, reasoning, and inference . Cambridge University Press.

Picciano, A. G. (2017). Theories and frameworks for online education: Seeking an integrated model. Online Learning, 21 (3), 166–190. https://doi.org/10.24059/olj.v21i3.1225

Puntambekar, S., et al. (2018). Design-based research. In F. Fisher (Ed.), International handbook of the learning sciences (pp. 383–392). Routledge.

Qian, H., & Youngs, P. (2016). The effect of teacher education programs on future elementary mathematics teachers’ knowledge: A five-country analysis using TEDS-M data. Journal of Mathematics Teacher Education, 19 (4), 371–396. https://doi.org/10.1007/s10857-014-9297-0

Reeves, T. C. (2006). Design research from the technology perspective. In J. V. Akker, K. Gravemeijer, S. McKenney, & N. Nieveen (Eds.), Educational design research (pp. 86–109). Routledge.

Rodgers, J. L. (2003). EMOSA sexuality models, memes, and the tipping point: Policy and program implications. In D. Romer (Ed.), Reducing adolescent risk: Toward an integrated approach (pp. 185–192). Sage.

Rodgers, J. L. (2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65 (1), 1–12.

Rodgers, J. L., & Doughty, D. (2001). Does having boys or girls run in the family? Chance, 14 , 8–13.

Rodgers, J. L., & Rowe, D. C. (1993). Social contagion and adolescent sexual behavior: A developmental EMOSA model. Psychological Review, 100 , 479–510. https://doi.org/10.1037/0033-295X.100.3.479

Rowe, D. C., & Rodgers, J. L. (1991). Adolescent smoking and drinking: Are they epidemics? Journal of Studies on Alcohol, 52 , 110–117.

Scheerens, J., & Blömeke, S. (2016). Integrating teacher education effectiveness research into educational effectiveness models. Educational Research Review, 18 , 70–87. https://doi.org/10.1016/j.edurev.2016.03.002

Seel, N. M. (2009). Bonjour tristesse: Why don’t we research as we have been taught? Methodological considerations on instructional technology research. Technology, Instrustion, Cognition and Learning, 6 , 151–176.

Shadish, W., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference . Houghton Mifflin.

Shepard, R. N. (1982). Geometrical approximations to the structure of musical pitch. Psychological Review, 89 , 305–333.

Siemens, G. (2004). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 2 , 1–8.

Simon, H. A., & Newell, A. (1956). Models: Their uses and limitations. In D. White (Ed.), The state of the social sciences (pp. 61–83). University of Chicago Press.

Straub, D. W. (2009). Editor’s comments: Why top journals accept your paper. MIS Quarterly, 33 (3), iii–x.

Suppes, P. (1978). Impact of research on education: Some case studies . National Academy of Education.

Tatto, M. T., Schwille, J., Senk, S., Ingvarson, L., Peck, R., & Rowley, G. (2008). Teacher education and development study in mathematics (TEDS-M): Policy, practice, and readiness to teach primary and secondary mathematics. Conceptual framework . Teacher Education and Development International, Study Center, College of Education, Michigan State University.

Tipton, E., & Olsen, R. B. (2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 47 (8), 516–524. https://doi.org/10.3102/0013189x18781522

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27 (3), 425–478.

Venkatesh, V., Thong, J. Y. L., & Xu, X. (2016). Unified theory of acceptance and use of technology: A synthesis and the road ahead. Journal of the Association for Information Systems, 17 (5), 328–376.

Wedemeyer, C. A. (1981). Learning at the back door: Reflections on non-traditional learning in the lifespan . The University of Wisconsin Press.

Wenger, E. (1998). Communities of practice: Learning, meaning, and identity . Cambridge University Press.

Wenger, E., & Lave, J. (1991). Learning in doing: Social, cognitive and computational perspectives. Situated learning: Legitimate peripheral participation . Cambridge University Press.

Willingham, D. (2008). What is developmentally appropriate? American Educator, 32 (2), 34–39.

Zawacki-Richter, O., Bäcker, E., & Vogt, S. (2009). Review of distance education research (2000 to 2008): Analysis of research areas, methods, and authorship patterns. International Review of Research in Open and Distributed Learning, 10 (6), 21–50. https://doi.org/10.19173/irrodl.v10i6.741

Zimmerman, B. J. (2001). Theories of self-regulated learning and academic achievement: An overview and analysis. In B. J. Zimmerman & D. H. Schunk (Eds.), Self-regulated learning and academic achievement. Theoretical perspectives (pp. 1–38). Mahwah.

Download references

Author information

Authors and affiliations.

School of Teaching, Learning and Curriculum Studies, Kent State University, 150 Terrace Drive, P.O. Box 5190, Kent, OH, 44242-0001, USA

Elena Novak

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Elena Novak .

Ethics declarations

Conflict of interest.

The author declares that she has no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Novak, E. Mathematical modeling for theory-oriented research in educational technology. Education Tech Research Dev 70 , 149–167 (2022). https://doi.org/10.1007/s11423-021-10069-6

Download citation

Accepted : 21 November 2021

Published : 29 November 2021

Issue Date : February 2022

DOI : https://doi.org/10.1007/s11423-021-10069-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Model-testing
  • Mathematical modeling
  • Educational technology
  • Find a journal
  • Publish with us
  • Track your research

Loading metrics

Open Access

Ten simple rules for tackling your first mathematical models: A guide for graduate students by graduate students

Roles Conceptualization, Investigation, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Department of Biological Sciences, University of Toronto Scarborough, Toronto, Ontario, Canada, Department of Ecology and Evolution, University of Toronto, Toronto, Ontario, Canada

ORCID logo

Affiliation Department of Ecology and Evolution, University of Toronto, Toronto, Ontario, Canada

Affiliation Department of Physical and Environmental Sciences, University of Toronto Scarborough, Toronto, Ontario, Canada

Affiliation Department of Biology, Memorial University of Newfoundland, St John’s, Newfoundland, Canada

  • Korryn Bodner, 
  • Chris Brimacombe, 
  • Emily S. Chenery, 
  • Ariel Greiner, 
  • Anne M. McLeod, 
  • Stephanie R. Penk, 
  • Juan S. Vargas Soto

PLOS

Published: January 14, 2021

  • https://doi.org/10.1371/journal.pcbi.1008539
  • Reader Comments

Fig 1

Citation: Bodner K, Brimacombe C, Chenery ES, Greiner A, McLeod AM, Penk SR, et al. (2021) Ten simple rules for tackling your first mathematical models: A guide for graduate students by graduate students. PLoS Comput Biol 17(1): e1008539. https://doi.org/10.1371/journal.pcbi.1008539

Editor: Scott Markel, Dassault Systemes BIOVIA, UNITED STATES

Copyright: © 2021 Bodner et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Biologists spend their time studying the natural world, seeking to understand its various patterns and the processes that give rise to them. One way of furthering our understanding of natural phenomena is through laboratory or field experiments, examining the effects of changing one, or several, variables on a measured response. Alternatively, one may conduct an observational study, collecting field data and comparing a measured response along natural gradients. A third and complementary way of understanding natural phenomena is through mathematical models. In the life sciences, more scientists are incorporating these quantitative methods into their research. Given the vast utility of mathematical models, ranging from providing qualitative predictions to helping disentangle multiple causation (see Hurford [ 1 ] for a more complete list), their increased adoption is unsurprising. However, getting started with mathematical models may be quite daunting for those with traditional biological training, as in addition to understanding new terminology (e.g., “Jacobian matrix,” “Markov chain”), one may also have to adopt a different way of thinking and master a new set of skills.

Here, we present 10 simple rules for tackling your first mathematical models. While many of these rules are applicable to basic scientific research, our discussion relates explicitly to the process of model-building within ecological and epidemiological contexts using dynamical models. However, many of the suggestions outlined below generalize beyond these disciplines and are applicable to nondynamic models such as statistical models and machine-learning algorithms. As graduate students ourselves, we have created rules we wish we had internalized before beginning our model-building journey—a guide by graduate students, for graduate students—and we hope they prove insightful for anyone seeking to begin their own adventures in mathematical modelling.

thumbnail

  • PPT PowerPoint slide
  • PNG larger image
  • TIFF original image

Boxes represent susceptible, infected, and recovered compartments, and directed arrows represent the flow of individuals between these compartments with the rate of flow being controlled by the contact rate, c , the probability of infection, γ , and the recovery rate, θ .

https://doi.org/10.1371/journal.pcbi.1008539.g001

research paper on mathematical modeling pdf

Rule 1: Know your question

“All models are wrong, some are useful” is a common aphorism, generally attributed to statistician George Box, but determining which models are useful is dependent upon the question being asked. The practice of clearly defining a research question is often drilled into aspiring researchers in the context of selecting an appropriate research design, interpreting statistical results, or when outlining a research paper. Similarly, the practice of defining a clear research question is important for mathematical models as their results are only as interesting as the questions that motivate them [ 5 ]. The question defines the model’s main purpose and, in all cases, should extend past the goal of merely building a model for a system (the question can even answer whether a model is even necessary). Ultimately, the model should provide an answer to the research question that has been proposed.

When the research question is used to inform the purpose of the model, it also informs the model’s structure. Given that models can be modified in countless ways, providing a purpose to the model can highlight why certain aspects of reality were included in the structure while others were ignored [ 6 ]. For example, when deciding whether we should adopt a more realistic model (i.e., add more complexity), we can ask whether we are trying to inform general theory or whether we are trying to model a response in a specific system. For example, perhaps we are trying to predict how fast an epidemic will grow based on different age-dependent mixing patterns. In this case, we may wish to adapt our basic SIR model to have age-structured compartments if we suspect this factor is important for the disease dynamics. However, if we are exploring a different question, such as how stochasticity influences general SIR dynamics, the age-structured approach would likely be unnecessary. We suggest that one of the first steps in any modelling journey is to choose the processes most relevant to your question (i.e., your hypothesis) and the direct and indirect causal relationships among them: Are the relationships linear, nonlinear, additive, or multiplicative? This challenge can be aided with a good literature review. Depending on your model purpose, you may also need to spend extra time getting to know your system and/or the data before progressing forward. Indeed, the more background knowledge acquired when forming your research question, the more informed your decision-making when selecting the structure, parameters, and data for your model.

Rule 2: Define multiple appropriate models

Natural phenomena are complicated to study and often impossible to model in their entirety. We are often unsure about the variables or processes required to fully answer our research question(s). For example, we may not know how the possibility of reinfection influences the dynamics of a disease system. In cases such as these, our advice is to produce and sketch out a set of candidate models that consider alternative terms/variables which may be relevant for the phenomena under investigation. As in Fig 2 , we construct 2 models, one that includes the ability for recovered individuals to become infected again, and one that does not. When creating multiple models, our general objective may be to explore how different processes, inputs, or drivers affect an outcome of interest or it may be to find a model or models that best explain a given set of data for an outcome of interest. In our example, if the objective is to determine whether reinfection plays an important role in explaining the patterns of a disease, we can test our SIR candidate models using incidence data to determine which model receives the most empirical support. Here we consider our candidate models to be alternative hypotheses, where the candidate model with the least support is discarded. While our perspective of models as hypotheses is a view shared by researchers such as Hilborn and Mangel [ 7 ], and Penk and colleagues [ 8 ], please note that others such as Oreskes and colleagues [ 9 ] believe that models are not subject to proof and hence disagree with this notion. We encourage modellers who are interested in this debate to read the provided citations.

thumbnail

(A) A susceptible/infected/recovered model where individuals remain immune (gold) and (B) a susceptible/infected/recovered model where individuals can become susceptible again (blue). Arrows indicate the direction of movement between compartments, c is the contact rate, γ is the infection rate given contact, and θ is the recovery rate. The text below each conceptual model are the hypotheses ( H1 and H2 ) that represent the differences between these 2 SIR models.

https://doi.org/10.1371/journal.pcbi.1008539.g002

Finally, we recognize that time and resource constraints may limit the ability to build multiple models simultaneously; however, even writing down alternative models on paper can be helpful as you can always revisit them if your primary model does not perform as expected. Of course, some candidate models may not be feasible or relevant for your system, but by engaging in the activity of creating multiple models, you will likely have a broader perspective of the potential factors and processes that fundamentally shape your system.

Rule 3: Determine the skills you will need (and how to get them)

Equipping yourself with the necessary analytical tools that form the basis of all quantitative techniques is essential. As Darwin said, those that have knowledge of mathematics seem to be endowed with an extra sense [ 10 ], and having a background in calculus, linear algebra, and statistics can go a long way. Thus, make it a habit to set time for yourself to learn these mathematical skills, and do not treat all your methods like a black box. For instance, if you plan to use ODEs, consider brushing up on your calculus, e.g., using Stewart [ 11 ]. If you are working with a system of ODEs, also read up on linear algebra, e.g., using Poole [ 12 ]. Some universities also offer specialized math biology courses that combine topics from different math courses to teach the essentials of mathematical modelling. Taking these courses can help save time, and if they are not available, their syllabi can help focus your studying. Also note that while narrowing down a useful skillset in the early stages of model-building will likely spare you from some future headaches, as you progress in your project, it is inevitable that new skills will be required. Therefore, we advise you to check in at different stages of your modelling journey to assess the skills that would be most relevant for your next steps and how best to acquire them. Hopefully, these decisions can also be made with the help of your supervisor and/or a modelling mentor. Building these extra skills can at first seem daunting but think of it as an investment that will pay dividends in improving your future modelling work.

When first attempting to tackle a specific problem, find relevant research that accomplishes the same tasks and determine if you understand the processes and techniques that are used in that study. If you do, then you can implement similar techniques and methods, and perhaps introduce new methods. If not, then determine which tools you need to add to your toolbox. For instance, if the problem involves a system of ODEs (e.g., SIR models, see above), can you use existing symbolic software (e.g., Maple, Matlab, Mathematica) to determine the systems dynamics via a general solution, or is the complexity too great that you will need to create simulations to infer the dynamics? Figuring out questions like these is key to understanding what skills you will need to work with the model you develop. While there is a time and a place for involving collaborators to help facilitate methods that are beyond your current reach, we strongly advocate that you approach any potential collaborator only after you have gained some knowledge of the methods first. Understanding the methodology, or at least its foundation, is not only crucial for making a fruitful collaboration, but also important for your development as a scientist.

Rule 4: Do not reinvent the wheel

While we encourage a thorough understanding of the methods researchers employ, we simultaneously discourage unnecessary effort redoing work that has already been done. Preventing duplication can be ensured by a thorough review of the literature (but note that reproducing original model results can advance your knowledge of how a model functions and lead to new insights in the system). Often, we are working from established theory that provides an existing framework that can be applied to different systems. Adapting these frameworks can help advance your own research while also saving precious time. When digging through articles, bear in mind that most modelling frameworks are not system-specific. Do not be discouraged if you cannot immediately find a model in your field, as the perfect model for your question may have been applied in a different system or be published only as a conceptual model. These models are still useful! Also, do not be shy about reaching out to authors of models that you think may be applicable to your system. Finally, remember that you can be critical of what you find, as some models can be deceptively simple or involve assumptions that you are not comfortable making. You should not reinvent the wheel, but you can always strive to build a better one.

Rule 5: Study and apply good coding practices

The modelling process will inevitably require some degree of programming, and this can quickly become a challenge for some biologists. However, learning to program in languages commonly adopted by the scientific community (e.g., R, Python) can increase the transparency, accessibility, and reproducibility of your models. Even if you only wish to adopt preprogrammed models, you will likely still need to create code of your own that reads in data, applies functions from a collection of packages to analyze the data, and creates some visual output. Programming can be highly rewarding—you are creating something after all—but it can also be one of the most frustrating parts of your research. What follows are 3 suggestions to avoid some of the frustration.

Organization is key, both in your workflow and your written code. Take advantage of existing software and tools that facilitate keeping things organized. For example, computational notebooks like Jupyter notebooks or R-Markdown documents allow you to combine text, commands, and outputs in an easily readable and shareable format. Version control software like Git makes it simple to both keep track of changes as well as to safely explore different model variants via branches without worrying that the original model has been altered. Additionally, integrating with hosting services such as Github allows you to keep your changes safely stored in the cloud. For more details on learning to program, creating reproducible research, programming with Jupyter notebooks, and using Git and Github, see the 10 simple rules by Carey and Papin [ 13 ], Sandve and colleagues [ 14 ], Rule and colleagues [ 15 ], and Perez-Riverol and colleagues [ 16 ], respectively.

Comment your code and comment it well (see Fig 3 ). These comments can be the pseudocode you have written on paper prior to coding. Assume that when you revisit your code weeks, months, or years later, you will have forgotten most of what you did and why you did it. Good commenting can also help others read and use your code, making it a critical part of increasing scientific transparency. It is always good practice to write your comments before you write the code, explaining what the code should do. When coding a function, include a description of its inputs and outputs. We also encourage you to publish your commented model code in repositories such that they are easily accessible to others—not only to get useful feedback for yourself but to provide the modelling foundation for others to build on.

thumbnail

Two functionally identical codes in R [ 17 ] can look very different without comments (left) and with descriptive comments (right). Writing detailed comments will help you and others understand, adapt, and use your code.

https://doi.org/10.1371/journal.pcbi.1008539.g003

When writing long code, test portions of it separately. If you are writing code that will require a lot of processing power or memory to run, use a simple example first, both to estimate how long the project will take, and to avoid waiting 12 hours to see if it works. Additionally, when writing code, try to avoid too many packages and “tricks” as it can make your code more difficult to understand. Do not be afraid of writing 2 separate functions if it will make your code more intuitive. As with writing, your skill as a writer is not dependent on your ability to use big words, but instead about making sure your reader understands what you are trying to communicate.

Rule 6: Sweat the “right” small stuff

By “sweat the ‘right’ small stuff,” we mean considering the details and assumptions that can potentially make or break a mathematical model. A good start would be to ensure your model follows the rules of mass and energy conservation. In a closed system, mass and energy cannot be created nor destroyed, and thus, the left side of the mathematical equation must equal the right under all circumstances. For example, in Eq 2 , if the number of susceptible individuals decreases due to infection, we must include a negative term in this equation (− cγIS ) to indicate that loss and its conjugate (+ cγIS ) to the infected individuals equation, Eq 3 , to represent that gain. Similarly, units of all terms must also be balanced on both sides of the equation. For example, if we wish to add or subtract 2 values, we must ensure their units are equivalent (e.g., cannot add day −1 and year −1 ). Simple oversights in units can lead to major setbacks and create bizarre dynamics, so it is worth taking the time to ensure the units match up.

Modellers should also consider the fundamental boundary conditions of each parameter to determine if there are some values that are illogical. Logical constraints and boundaries can be developed for each parameter using prior knowledge and assumptions (e.g., Huntley [ 18 ]). For example, when considering an SIR model, there are 2 parameters that comprise the transmission rate—the contact rate, c , and the probability of infection given contact, γ . Using our intuition, we can establish some basic rules: (1) the contact rate cannot be negative; (2) the number of susceptible, infected, and recovered individuals cannot be below 0; and (3) the probability of infection given contact must fall between 0 and 1. Keeping these in mind as you test your model’s dynamics can alert you to problems in your model’s structure. Finally, simulating your model is an excellent method to obtain more reasonable bounds for inputs and parameters and ensure behavior is as expected. See Otto and Day [ 5 ] for more information on the “basic ingredients” of model-building.

Rule 7: Simulate, simulate, simulate

Even though there is a lot to be learned from analyzing simple models and their general solutions, modelling a complex world sometimes requires complex equations. Unfortunately, the cost of this complexity is often the loss of general solutions [ 19 ]. Instead, many biologists must calculate a numerical solution, an approximate solution, and simulate the dynamics of these models [ 20 ]. Simulations allow us to explore model behavior, given different structures, initial conditions, and parameters ( Fig 4 ). Importantly, they allow us to understand the dynamics of complex systems that may otherwise not be ethical, feasible, or economically viable to explore in natural systems [ 21 ].

thumbnail

Gold lines represent the SIR structure ( Fig 2A ) where lifelong immunity of individuals is inferred after infection, and blue lines represent an SIRS structure ( Fig 2B ) where immunity is lost over time. The solid lines represent model dynamics assuming a recovery rate ( θ ) of 0.05, while dotted lines represent dynamics assuming a recovery rate of 0.1. All model runs assume a transmission rate, cγ , of 0.2 and an immunity loss rate, ψ , of 0.01. By using simulations, we can explore how different processes and rates change the system’s dynamics and furthermore determine at what point in time these differences are detectable. SIR, Susceptible-Infected-Recovered; SIRS, Susceptible-Infected-Recovered-Susceptible.

https://doi.org/10.1371/journal.pcbi.1008539.g004

One common method of exploring the dynamics of complex systems is through sensitivity analysis (SA). We can use this simulation-based technique to ascertain how changes in parameters and initial conditions will influence the behavior of a system. For example, if simulated model outputs remain relatively similar despite large changes in a parameter value, we can expect the natural system represented by that model to be robust to similar perturbations. If instead, simulations are very sensitive to parameter values, we can expect the natural system to be sensitive to its variation. Here in Fig 4 , we can see that both SIR models are very sensitive to the recovery rate parameter ( θ ) suggesting that the natural system would also be sensitive to individuals’ recovery rates. We can therefore use SA to help inform which parameters are most important and to determine which are distinguishable (i.e., identifiable). Additionally, if observed system data are available, we can use SA to help us establish what are the reasonable boundaries for our initial conditions and parameters. When adopting SA, we can either vary parameters or initial conditions one at a time (local sensitivity) or preferably, vary multiple of them in tandem (global sensitivity). We recognize this topic may be overwhelming to those new to modelling so we recommend reading Marino and colleagues [ 22 ] and Saltelli and colleagues [ 23 ] for details on implementing different SA methods.

Simulations are also a useful tool for testing how accurately different model fitting approaches (e.g., Maximum Likelihood Estimation versus Bayesian Estimation) can recover parameters. Given that we know the parameter values for simulated model outputs (i.e., simulated data), we can properly evaluate the fitting procedures of methods when used on that simulated data. If your fitting approach cannot even recover simulated data with known parameters, it is highly unlikely your procedure will be successful given real, noisy data. If a procedure performs well under these conditions, try refitting your model to simulated data that more closely resembles your own dataset (i.e., imperfect data). If you know that there was limited sampling and/or imprecise tools used to collect your data, consider adding noise, reducing sample sizes, and adding temporal and spatial gaps to see if the fitting procedure continues to return reasonably correct estimates. Remember, even if your fitting procedures continue to perform well given these additional complexities, issues may still arise when fitting to empirical data. Models are approximations and consequently their simulations are imperfect representations of your measured outcome of interest. However, by evaluating procedures on perfectly known imperfect data, we are one step closer to having a fitting procedure that works for us even when it seems like our data are against us.

Rule 8: Expect model fitting to be a lengthy, arduous but creative task

Model fitting requires an understanding of both the assumptions and limitations of your model, as well as the specifics of the data to be used in the fitting. The latter can be challenging, particularly if you did not collect the data yourself, as there may be additional uncertainties regarding the sampling procedure, or the variables being measured. For example, the incidence data commonly adopted to fit SIR models often contain biases related to underreporting, selective reporting, and reporting delays [ 24 ]. Taking the time to understand the nuances of the data is critical to prevent mismatches between the model and the data. In a bad case, a mismatch may lead to a poor-fitting model. In the worst case, a model may appear well-fit, but will lead to incorrect inferences and predictions.

Model fitting, like all aspects of modelling, is easier with the appropriate set of skills (see Rule 2). In particular, being proficient at constructing and analyzing mathematical models does not mean you are prepared to fit them. Fitting models typically requires additional in-depth statistical knowledge related to the characteristics of probability distributions, deriving statistical moments, and selecting appropriate optimization procedures. Luckily, a substantial portion of this knowledge can be gleaned from textbooks and methods-based research articles. These resources can range from covering basic model fitting, such as determining an appropriate distribution for your data and constructing a likelihood for that distribution (e.g., Hilborn and Mangel [ 7 ]), to more advanced topics, such as accounting for uncertainties in parameters, inputs, and structures during model fitting (e.g., Dietze [ 25 ]). We find these sources among others (e.g., Hobbs and Hooten [ 26 ] for Bayesian methods; e.g., Adams and colleagues [ 27 ] for fitting noisy and sparse datasets; e.g., Sirén and colleagues [ 28 ] for fitting individual-based models; and Williams and Kendall [ 29 ] for multiobject optimization—to name a few) are not only useful when starting to fit your first models, but are also useful when switching from one technique or model to another.

After you have learned about your data and brushed up on your statistical knowledge, you may still run into issues when model fitting. If you are like us, you will have incomplete data, small sample sizes, and strange data idiosyncrasies that do not seem to be replicated anywhere else. At this point, we suggest you be explorative in the resources you use and accept that you may have to combine multiple techniques and/or data sources before it is feasible to achieve an adequate model fit (see Rosenbaum and colleagues [ 30 ] for parameter estimation with multiple datasets). Evaluating the strength of different techniques can be aided by using simulated data to test these techniques, while SA can be used to identify insensitive parameters which can often be ignored in the fitting process (see Rule 7).

Model accuracy is an important metric but “good” models are also precise (i.e., reliable). During model fitting, to make models more reliable, the uncertainties in their inputs, drivers, parameters, and structures, arising due to natural variability (i.e., aleatory uncertainty) or imperfect knowledge (i.e., epistemic uncertainty), should be identified, accounted for, and reduced where feasible [ 31 ]. Accounting for uncertainty may entail measurements of uncertainties being propagated through a model (a simple example being a confidence interval), while reducing uncertainty may require building new models or acquiring additional data that minimize the prioritized uncertainties (see Dietze [ 25 ] and Tsigkinopoulou and colleagues [ 32 ] for a more thorough review on the topic). Just remember that although the steps outlined in this rule may take a while to complete, when you do achieve a well-fitted reliable model, it is truly something to be celebrated.

Rule 9: Give yourself time (and then add more)

Experienced modellers know that it often takes considerable time to build a model and that even more time may be required when fitting to real data. However, the pervasive caricature of modelling as being “a few lines of code here and there” or “a couple of equations” can lead graduate students to hold unrealistic expectations of how long finishing a model may take (or when to consider a model “finished”). Given the multiple considerations that go into selecting and implementing models (see previous rules), it should be unsurprising that the modelling process may take weeks, months, or even years. Remembering that a published model is the final product of long and hard work may help reduce some of your time-based anxieties. In reality, the finished product is just the tip of the iceberg and often unseen is the set of failed or alternative models providing its foundation. Note that taking time early on to establish what is “good enough” given your objective, and to instill good modelling practices, such as developing multiple models, simulating your models, and creating well-documented code, can save you considerable time and stress.

Rule 10: Care about the process, not just the endpoint

As a graduate student, hours of labor coupled with relative inexperience may lead to an unwillingness to change to a new model later down the line. But being married to one model can restrict its efficacy, or worse, lead to incorrect conclusions. Early planning may mitigate some modelling problems, but many issues will only become apparent as time goes on. For example, perhaps model parameters cannot be estimated as you previously thought, or assumptions made during model formulation have since proven false. Modelling is a dynamic process, and some steps will need to be revisited many times as you correct, refine, and improve your model. It is also important to bear in mind that the process of model-building is worth the effort. The process of translating biological dynamics into mathematical equations typically forces us to question our assumptions, while a misspecified model often leads to novel insights. While we may wish we had the option to skip to a final finished product, in the words of Drake, “sometimes it’s the journey that teaches you a lot about your destination”.

There is no such thing as a failed model. With every new error message or wonky output, we learn something useful about modelling (mostly begrudgingly) and, if we are lucky, perhaps also about the study system. It is easy to cave in to the ever-present pressure to perform, but as graduate students, we are still learning. Luckily, you are likely surrounded by other graduate students, often facing similar challenges who can be an invaluable resource for learning and support. Finally, remember that it does not matter if this was your first or your 100th mathematical model, challenges will always present themselves. However, with practice and determination, you will become more skilled at overcoming them, allowing you to grow and take on even greater challenges.

Acknowledgments

We thank Marie-Josée Fortin, Martin Krkošek, Péter K. Molnár, Shawn Leroux, Carina Rauen Firkowski, Cole Brookson, Gracie F.Z. Wild, Cedric B. Hunter, and Philip E. Bourne for their helpful input on the manuscript.

  • 1. Hurford A. Overview of mathematical modelling in biology II. 2012 [cite 2020 October 25]. Available: https://theartofmodelling.wordpress.com/2012/01/04/overview-of-mathematical-modelling-in-biology-ii/
  • View Article
  • PubMed/NCBI
  • Google Scholar
  • 3. Maki Y, Hirose H, ADSIR M. Infectious Disease Spread Analysis Using Stochastic Differential Equations for SIR Model. International Conference on Intelligent Systems, Modelling and Simulation. IEEE. 2013.
  • 5. Otto SP, Day T. A biologist’s guide to mathematical modeling in ecology and evolution. Princeton, NJ: Princeton University Press; 2007.
  • 7. Hilborn R, Mangel M. The ecological detective: Confronting models with data. Princeton, NJ: Princeton University Press; 1997.
  • 10. Darwin C. The autobiography of Charles Darwin. Darwin F, editor. 2008. Available: https://www.gutenberg.org/files/2010/2010-h/2010-h.htm
  • 11. Stewart J. Calculus: Early transcendentals. Eighth. Boston, MA: Cengage Learning; 2015.
  • 12. Poole D. Linear algebra: A modern introduction. Fourth. Stamford, CT: Cengage Learning; 2014.
  • 17. R Core Team. R: A language and environment for statistical computing (version 3.6.0, R foundation for statistical computing). 2020.
  • 18. Huntley HE. Dimensional analysis. First. New York, NY: Dover Publications; 1967.
  • 19. Corless RM, Fillion N. A graduate introduction to numerical methods. New York, NY: Springer; 2016.
  • 23. Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, et al. Global sensitivity analysis: The primer. Chichester: Wiley; 2008.
  • 25. Dietze MC. Ecological forecasting. Princeton, NJ: Princeton University Press; 2017.
  • 26. Hobbs NT, Hooten MB. Bayesian models: A statistical primer for ecologists. Princeton, NJ: Princeton University Press
  • Research article
  • Open access
  • Published: 19 November 2019

Mathematical modelling for health systems research: a systematic review of system dynamics and agent-based models

  • Rachel Cassidy   ORCID: orcid.org/0000-0002-4824-0260 1 ,
  • Neha S. Singh 1 ,
  • Pierre-Raphaël Schiratti 2 , 3 ,
  • Agnes Semwanga 4 ,
  • Peter Binyaruka 5 ,
  • Nkenda Sachingongu 6 ,
  • Chitalu Miriam Chama-Chiliba 7 ,
  • Zaid Chalabi 8 ,
  • Josephine Borghi 1 &
  • Karl Blanchet 1  

BMC Health Services Research volume  19 , Article number:  845 ( 2019 ) Cite this article

20k Accesses

43 Citations

5 Altmetric

Metrics details

Mathematical modelling has been a vital research tool for exploring complex systems, most recently to aid understanding of health system functioning and optimisation. System dynamics models (SDM) and agent-based models (ABM) are two popular complementary methods, used to simulate macro- and micro-level health system behaviour. This systematic review aims to collate, compare and summarise the application of both methods in this field and to identify common healthcare settings and problems that have been modelled using SDM and ABM.

We searched MEDLINE, EMBASE, Cochrane Library, MathSciNet, ACM Digital Library, HMIC, Econlit and Global Health databases to identify literature for this review. We described papers meeting the inclusion criteria using descriptive statistics and narrative synthesis, and made comparisons between the identified SDM and ABM literature.

We identified 28 papers using SDM methods and 11 papers using ABM methods, one of which used hybrid SDM-ABM to simulate health system behaviour. The majority of SDM, ABM and hybrid modelling papers simulated health systems based in high income countries. Emergency and acute care, and elderly care and long-term care services were the most frequently simulated health system settings, modelling the impact of health policies and interventions such as those targeting stretched and under resourced healthcare services, patient length of stay in healthcare facilities and undesirable patient outcomes.

Conclusions

Future work should now turn to modelling health systems in low- and middle-income countries to aid our understanding of health system functioning in these settings and allow stakeholders and researchers to assess the impact of policies or interventions before implementation. Hybrid modelling of health systems is still relatively novel but with increasing software developments and a growing demand to account for both complex system feedback and heterogeneous behaviour exhibited by those who access or deliver healthcare, we expect a boost in their use to model health systems.

Peer Review reports

Introduction

Health systems are complex adaptive systems [ 1 ]. As such, they are characterised by extraordinary complexity in relationships among highly heterogeneous groups of stakeholders and the processes they create [ 2 ]. Systems phenomena of massive interdependencies, self-organising and emergent behaviour, non-linearity, time lags, feedback loops, path dependence and tipping points make health system behaviour difficult and sometimes impossible to predict or manage [ 3 ]. Conventional reductionist approaches using epidemiological and implementation research methods are inadequate for tackling the problems health systems pose [ 4 ]. It is increasingly recognised that health systems and policy research need a special set of approaches, methods and tools that derive from systems thinking perspectives [ 5 ]. Health systems encompass a many tiered system providing services to local, district and national populations, from community health centres to tertiary hospitals. Attempting to evaluate the performance of such a multi-faceted organisation presents a daunting task. Mathematical modelling, capable of simulating the behaviour of complex systems, is therefore a vital research tool to aid our understanding of health system functioning and optimisation.

System dynamics model (SDM)

System dynamics models (SDM) and agent-based models (ABM) are the two most popular mathematical modelling methods for evaluating complex systems; while SDM are used to study macro-level system behaviour such as the movement of resources or quantities in a system over time, ABM capture micro-level system behaviour, such as human decision-making and heterogeneous interactions between humans.

While use of SDM began in business management [ 6 , 7 ] it now has wide spread application from engineering to economics, from environmental science to waste and recycling research [ 8 , 9 , 10 , 11 , 12 , 13 ]. A SDM simulates the movement of entities in a system, using differential equations to model over time changes to system state variables. A stock and flow diagram can be used to provide a visual representation of a SDM, describing the relationships between system variables using stocks, rates and influencing factors. The diagram can be interpreted as mimicking the flow of water in and out of a bath tub [ 7 ]; the rates control how much ‘water’ (some quantifiable entity, resource) can leave or enter a ‘bath tub’ (a stock, system variable) which changes over time depending on what constraints or conditions (e.g. environmental or operational) are placed on the system. Often before the formulation of a stock and flow diagram, a causal loop diagram is constructed which can be thought of as a ‘mental model’ of the system [ 14 ], representing key dynamic hypotheses.

Agent-based model (ABM)

Unlike SDM, ABM is a ground-up representation of a system, simulating the changing states of individual ‘agents’ in a system rather than the broad entities or aggregate behaviour modelled in SDM. Aggregate system behaviour can however be inferred from ABM. Use of ABM to model system behaviour has been trans-disciplinary, with application in economics to ecology, from social sciences to engineering [ 15 , 16 , 17 , 18 , 19 ]. There can be multiple types of agent modelled, each assigned their own characteristics and pattern of behaviour [ 20 , 21 ]. Agents can learn from their own experiences, make decisions and perform actions based on set rules (e.g. heuristics), informed by their interactions with other agents, their own assigned attributes or based on their interaction with the modelled environment [ 22 ]. The interactions between agents can result in three levels of communication between agents; one-to-one communication between agents, one-to-many communication between agents and one-to-location communication where an agent can influence other agents contained in a particular location [ 22 ].

Why use SDM and ABM to model health systems?

ABM and SDM, with their ability to simulate micro- and macro-level behaviour, are complementary instruments for examining the mechanisms in complex systems and are being recognised as crucial tools for exploratory analysis. Their use in mapping health systems, for example, has steadily risen over the last three decades. ABM is well-suited to explore systems with dynamic patient or health worker activity, a limitation of other differential equation or event-based simulation tools [ 23 , 24 , 25 ]. Unlike discrete-event simulation (DES) for example, which simulates a queue of events and agent attributes over time [ 26 ], the agents modelled in ABM are decision makers rather than passive individuals. Closer to the true system modelled, ABM can also incorporate ongoing learning from events whereby patients can be influenced by their interactions with other patients or health workers and by their own personal experience with the health system [ 21 ]. SDM has also been identified as a useful tool for simulating feedback and activity across the care continuum [ 27 , 28 , 29 , 30 ] and is highly adept at capturing changes to the system over time [ 31 ]. This is not possible with certain ‘snapshot in time’ modelling approaches such as DES [ 32 ]. SDM is best implemented where the aim of the simulation is to examine aggregate flows, trends and sub-system behaviour as opposed to intricate individual flows of activity which are more suited to ABM or DES [ 33 ].

There are also models that can accommodate two or more types of simulation, known as hybrid models. Hybrid models produce results closer to true system behaviour by drawing on the strengths of one or more modelling methods while reducing the limitations associated with using a single simulation type [ 27 ]. The activity captured in such models emulates the individual variability of patients and health professionals while retaining the complex, aggregate behaviour exhibited in health systems.

Health scientists and policy makers alike have recognised the potential of using SDM and ABM to model all aspects of health systems in support of decision making from emergency department (ED) optimisation [ 34 ] to policies that support prevention or health promotion [ 35 ]. Before implementing or evaluating costly health policy interventions or health service re-structuring in the real world, modelling provides a relatively risk-free and low budget method of examining the likely impact of potential health system policy changes. They allow the simulation of ‘what if’ scenarios to optimise an intervention [ 36 ]. They can help identify sensitive parameters in the system that can impede the success of initiatives and point to possible spill-over effects of these initiatives to other departments, health workers or patients. Perhaps most important of all, these modelling methods allow researchers to produce simulations, results and a graphical-user interface in relation to alternative policy options that are communicable to stakeholders in the health system [ 37 ], those responsible for implementing system-wide initiatives and changes.

Study aim and objectives

Given the increasing amount of literature in this field, the main aim of the study was to examine and describe the use of SDM and ABM to model health systems. The specific objectives were as follows: (1) Determine the geographical, and healthcare settings in which these methods have been used (2) Identify the purpose of the research, particularly the health policies or interventions tested (3) Evaluate the limitations of these methods and study validation, and (4) Compare the use of SDM and ABM in health system research.

Although microsimulation, DES and Markov models have been widely used in disease health modelling and health economic evaluation, our aim in this study was to review the literature on mathematical methods which are used to model complex dynamic systems, SDM and ABM. These models represent two tenants of modelling: macroscopic (top-level) and microscopic (individual-level) approaches. Although microsimulation and DES are individual-based models like ABM, individuals in ABM are “active agents” i.e. decision-makers rather than “passive agents” which are the norm in microsimulation and DES models. Unlike Markov models which are essentially one-dimensional, unidirectional and linear, SDM are multi-dimensional, nonlinear with feedback mechanisms. We have therefore focussed our review on SDM and ABM because they are better suited to characterise the complexity of health systems. This study reviews the literature on the use of SDM and ABM in modelling health systems, and identifies and compares the key characteristics of both modelling approaches in unwrapping the complexity of health systems. In identifying and summarising this literature, this review will shed light on the types of health system research questions that these methods can be used to explore, and what they add to more traditional methods of health system research. By providing an over overview of how these models can be used within health system research, this paper is also expected to encourage wider use and uptake of these methods by health system researchers and policy makers.

The review was conducted in compliance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement [ 38 ].

Search strategy and information sources

The literature on ABM and SDM of health systems has not been confined to a single research discipline, making it necessary to widen the systematic review to capture peer-reviewed articles found in mathematical, computing, medicine and health databases. Accordingly, we searched MEDLINE, EMBASE, Cochrane Library, MathSciNet, ACM Digital Library, HMIC, Econlit and Global Health databases for literature. The search of health system literature was narrowed to identify articles that were concerned with modelling facility-based healthcare, services and related healthcare financing agreements which had been excluded or were not the focus of previous reviews [ 34 , 35 , 39 , 40 , 41 ]. The search criteria used for MEDLINE was as follows, with full search terms for each database and search terms used to locate SDM and ABM literature found in Additional file  1 :

(health system* OR health care OR healthcare OR health service* OR health polic* OR health facil* OR primary care OR secondary care OR tertiary care OR hospital*).ab,ti. AND (agent-based OR agent based).ab,ti. AND (model*).ab,ti.

In addition, the reference list of papers retained in the final stage of the screening process, and systematic reviews identified in the search, were reviewed for relevant literature.

Data extraction and synthesis

The screening process for the review is given in Fig.  1 (adapted from [ 38 ]). All search results were uploaded to Mendeley reference software where duplicate entries were removed. The remaining records were screened using their titles and abstracts, removing entries based on eligibility criteria given in Table  1 . Post-abstract review, the full text of remaining articles was screened. Papers retained in final stage of screening were scrutinised, with data imported to Excel based on the following categories; publication date, geographical and healthcare setting modelled, purpose of research in addition to any policies or interventions tested, rationale for modelling method and software platform, validation and limitations of model. The results were synthesised using descriptive statistics and analysis of paper content that were used to answer the objectives.

figure 1

a Flow-chart for systematic review of SDMs and b ABMs of health systems (Database research discipline is identified by colour; mathematical and computing (red), medicine (blue) and health (green) databases). Adapted from PRISMA [ 38 ]

The studies were first described by three characteristics: publication date, geographical setting, and what aspect of the health system was modelled and why. These characteristics were chosen for the following reasons. Publication date (Fig.  2 ) allows us to examine the quantity of SDM and ABM studies over time. Geographical settings (Fig. 2 , top) allows us to see which health systems have been studied, as health systems in LMIC are very different from those in developed countries. Studies are classified as modelling health systems in high, upper middle, lower middle and low income countries as classified by The World Bank based on economy, July 2018 [ 42 ]. Finally, we examined which aspects of the health system have been modelled and the types of research/policy questions that the models were designed to address, to shed light on the range of potential applications of these models, and also potential gaps in their application to date.

figure 2

Number of articles in the final review by year of publication and economic classification

The analysis of paper content was split into three sections; SDM use in health system research (including hybrid SDM-DES), ABM use in health system research (including hybrid ABM-DES) and hybrid SDM-ABM use in health system research. The quality of selected studies will not be presented as our aim was to compare and summarise the application of SDM and ABM in modelling health systems rather than a quality appraisal of studies.

Study selection

The search initially yielded 535 citations for ABM and 996 citations for SDM of facility-based healthcare and services (see Fig. 1 ). Post-full text screening 11 ABM and 28 SDM papers were retained for analysis, six of which utilised hybrid modelling methods. Three of the hybrid modelling papers integrated SDM with DES [ 43 , 44 , 45 ], two integrated ABM with DES [ 24 , 46 ] and one integrated SDM with ABM [ 47 ]. A summary table of selected papers is given in Table  2 .

Descriptive statistics

Publication date.

The first SDM paper to model health systems was published in 1998 [ 56 ] whilst the first publication [ 66 ] utilising ABM came almost a decade later (Fig. 2 ). We found an increasing trend in publications for both modelling approaches, with 90.9% (10/11) and 71.4% (20/28) of all ABM and SDM articles, respectively, having been published in the last decade. The first hybrid modelling article was published in 2010 [ 43 ], using SDM and DES to model the impact of an intervention to aid access to social care services for elderly patients in Hampshire, England.

Geographical setting

The proportion of papers that modelled health systems in high, upper middle, lower middle and low income countries is presented in Fig. 2 . Eighteen (18/28) papers that employed SDM simulated health systems in high income countries including England [ 33 , 36 , 43 , 45 , 50 , 54 , 56 , 57 ] and Canada [ 28 , 51 , 62 ]. Four SDM papers simulated upper middle income country health systems, including Turkey [ 52 , 59 ] and China [ 64 ], with a nominal number of papers (5/28) focussing on lower middle or low income countries (West Bank and Gaza [ 48 , 55 ], Indonesia [ 37 ], Afghanistan [ 30 ] and Uganda [ 60 ]). Almost all ABM papers (9/11) modelled a high income country health system, including the US [ 20 , 23 , 25 ] and Austria [ 65 ]. Two (2/11) ABM papers described an upper-middle income based health system (Brazil [ 22 , 67 ]). All six articles that implemented a hybrid SDM or ABM simulated health systems based in high income countries, including Germany [ 44 ] and Poland [ 47 ].

Healthcare setting and purpose of research

The healthcare settings modelled in the SDM, ABM and hybrid simulation papers are presented in Fig.  3 . Healthcare settings modelled using SDM included systems that were concerned with delivering emergency or acute care (11/28) [ 28 , 31 , 36 , 45 , 47 , 50 , 56 , 57 , 58 , 61 , 62 ], elderly or long-term care services (LTC)(12/28) [ 28 , 31 , 36 , 43 , 44 , 45 , 49 , 50 , 51 , 54 , 61 , 62 ] and hospital waste management (4/28) [ 37 , 48 , 52 , 55 ]. Twenty of the SDM papers selected in this review assessed the impact of health policy or interventions on the modelled system. Common policy targets included finding robust methods to relieve stretched healthcare services, ward occupancy and patient length of stay [ 28 , 31 , 36 , 43 , 49 , 50 , 54 , 58 , 62 ], reducing the time to patient admission [ 33 , 53 , 61 ], targeting undesirable patient health outcomes [ 47 , 58 , 60 , 63 ], optimising performance-based incentive health system policies [ 30 , 59 ] and reducing the total cost of care [ 33 , 54 , 61 ]. The remaining eight papers explored factors leading to undesirable emergency care system behaviour [ 56 , 57 ], simulating hospital waste management systems and predicting future waste generation [ 37 , 48 , 55 ], estimating future demand for cardiac care [ 44 ], exploring the impact of patient admission on health professionals stress level in an integrated care system [ 45 ], and variation in physician decision-making [ 32 ].

figure 3

The health system sector locations modelled in the SDM, ABM and hybrid modelling literature. Long-term care (LTC); Accountable care organisation (ACO); Maternal, newborn and child health (MNCH)

ABM papers modelled systems focussed on delivering emergency or acute care (4/11) [ 21 , 22 , 47 , 67 ] and accountable care organisations (ACO) or health insurance reimbursement schemes (3/11) [ 23 , 25 , 65 ]. Nine of the ABM papers assessed the impact of health policy or interventions on the modelled system. Common policy targets included decreasing the time agents spent performing tasks, waiting for a service or residing in parts of the system [ 20 , 22 , 24 , 67 ], reducing undesirable patient outcomes [ 23 , 25 , 47 , 67 ], reducing the number of patients who left a health facility without being seen by a physician [ 22 , 67 ] and optimising resource utility (beds and healthcare staff) [ 46 , 66 , 67 ]. The remaining two papers described simulation tools capable of comparing health insurance reimbursement schemes [ 65 ] and assessing risk, allocation of resources and identifying weaknesses in emergency care services [ 21 ].

Papers that utilised hybrid simulation, combining the strengths of two modelling approaches to capture detailed individual variability, agent-decision making and patient flow, modelled systems focussed on delivering elderly care or LTC services [ 43 , 44 , 45 ] and emergency or acute care [ 45 , 47 ]. Four of the hybrid simulation papers assessed the impact of policy or intervention on the modelled system. Policy targets included improving access to social support and care services [ 43 ], reducing undesirable patient outcomes [ 47 ], decreasing patient waiting time to be seen by a physician [ 24 ] and improving patient flow through the system by optimising resource allocation [ 46 ]. The remaining two papers used hybrid simulation to estimate the future demand for health care from patients with cardiac disease [ 44 ] and model patient flow through an integrated care system to estimate impact of patient admission on health care professionals wellbeing [ 45 ].

SDM use in health systems research (including hybrid SDM-DES)

Rationale for using model.

Gaining a holistic system perspective to facilitate the investigation of delays and bottlenecks in health facility processes, exploring counter-intuitive behaviour and monitoring inter-connected processes between sub-systems was cited frequently as reasons for using SDM to model health systems [ 28 , 36 , 37 , 48 , 56 ]. SDM was also described as a useful tool for predicting future health system behaviour and demand for care services, essential for health resource and capacity planning [ 48 , 60 ]. Configuration of the model was not limited by data availability [ 28 , 52 , 64 ] and could integrate data from various sources when required [ 51 ].

SDM was described as a tool for health policy exploration and optimising system interventions [ 33 , 36 , 51 , 54 , 58 , 64 ], useful for establishing clinical and financial ramifications on multiple groups (such as patients and health care providers) [ 63 ], identifying policy resistance or unintended system consequences [ 59 , 61 ] and quantifying the impact of change to the health system before real world implementation [ 62 ]. The modelling platform also provided health professionals, stakeholders and decision makers with an accessible visual learning environment that enabled engagement with experts necessary for model conception and validation [ 48 , 50 , 55 , 57 ]. The model interface could be utilised by decision makers to develop and test alternative policies in a ‘real-world’ framework that strengthened their understanding of system-wide policy impact [ 31 , 49 , 58 , 61 ].

SDM-DES hybrid models enabled retention of deterministic and stochastic system variability and preservation of unique and valuable features of both methods [ 44 ], capable of describing the flow of entities through a system and rapid insight without the need for large data collection [ 43 ], while simulating individual variability and detailed interactions that influence system behaviour [ 43 ]. SDM-DES offered dual model functionality [ 44 ] vital for simulating human-centric activity [ 45 ], reducing the practical limitations that come with using either SDM or DES to model health systems such as attempting to use SDM to model elements which have non-aggregated values (e.g. patient arrival time) [ 45 ] which is better suited for DES.

Healthcare setting

Sixteen papers that utilised SDM modelled systems that were concerned with the delivery of emergency or acute care, or elderly care or LTC services.

Ten of the reviewed papers primarily modelled sectors of the health system that delivered emergency or acute care Footnote 1 , Footnote 2 . Brailsford et al. [ 50 ], Lane et al. [ 56 ], Lane et al. [ 57 ] and Lattimer et al. [ 36 ] simulated the delivery of emergency care in English cities, specifically in Nottingham and London. Brailsford et al. [ 50 ] and Lattimer et al. [ 36 ] created models that replicated the entire emergency care system for the city of Nottingham, from primary care (i.e. General Practice surgeries) to secondary care (i.e. hospital admissions wards), to aid understanding of how emergency care was delivered and how the system would need to adapt to increasing demand. Lane et al. [ 56 ] and Lane et al. [ 57 ] modelled the behaviour of an ED in an inner-London teaching hospital, exploring the knock on effects of ED performance to hospital ward occupancy and elective admissions. Esensoy et al. [ 28 ] and Wong et al. [ 62 ] both modelled emergency care in Canada, Esensoy et al. [ 28 ] focussing on six sectors of the Ontario health system that cared for stroke patients while Wong et al. [ 62 ] simulated the impact of delayed transfer of General Internal Medicine patients on ED occupancy. Rashwan et al. [ 31 ], Walker et al. [ 61 ] and Mahmoudian-Dehkordi et al. [ 58 ] modelled patient flow through a generic emergency care facility with six possible discharge locations in Ireland, a sub-acute extended care hospital with patient flow from feeder facilities in Australia and an intensive care unit, ED and general wards in a generic facility.

Five of the SDM papers primarily simulated the behaviour of LTC facilities or care services for elderly patients Footnote 3 . Ansah et al. [ 49 ] modelled the demand and supply of general LTC services in Singapore with specific focus on the need for LTC and acute health care professionals. Desai et al. [ 54 ] developed a SDM that investigated future demand of care services for older people in Hampshire, England which simulated patient flow through adult social care services offering 13 different care packages. In modelling complex care service demand, Cepoiu-Martin et al. [ 51 ] explored patient flow within the Alberta continuing care system in Canada which offered supportive living and LTC services for patients with dementia. Brailsford et al. [ 43 ] used a hybrid SDM-DES model to investigate how local authorities could improve access to services and support for older people, in particular the long term impact of a new contact centre for patients. The SDM replicated the whole system for long term care, simulating the future demography and demand for care services and the nested DES model simulated the operational issues and staffing of the call centre in anticipation of growing demand for services. Zulkepli et al. [ 45 ] also used SDM-DES to model the behaviour of an integrated care system in the UK, modelling patient flow (DES) and intangible variables (SDM) related to health professionals such as motivation and stress levels.

Policy impact evaluation/testing

Twenty papers that utilised SDM tested the impact of policy or interventions on key health system performance or service indicators. The intended target of these policies ranged from relieving strained and under resourced healthcare services, decreasing healthcare costs to reducing patient mortality rates.

Ansah et al. [ 49 ], Brailsford et al. [ 50 ] and Desai et al. [ 54 ] aimed to reduce occupancy in acute or emergency care departments through policies that targeted elderly utilisation of these services. While demand for LTC services is expected to exponentially increase in Singapore, focus has been placed on expanding the acute care sector. Ansah et al. [ 49 ] simulated various LTC service expansion policies (static ‘current’ policy, slow adjustment, quick adjustment, proactive adjustment) and identified that proactive expansion of LTC services stemmed the number of acute care visits by elderly patients over time and required only a modest increase in the number of health professionals when compared with other policies. In Brailsford et al. [ 50 ] simulation of the entire emergency care system for Nottingham, England, policy testing indicated that while the emergency care system is operating near full capacity, yearly total occupancy of hospital beds could be reduced by re-directing emergency admissions from patients over 60 years of age (who make up around half of all admissions) to more appropriate services, such as those offered by community care facilities. To explore challenges that accompany providing care for an ageing population subject to budget restraints, Desai et al. [ 54 ] simulated the delivery and demand for social care services in Hampshire over a projected 5 year period. In offering care packages to only critical need clients and encouraging extra care services at home rather than offering residential care, the number of patients accessing acute care services reduced over the observed period.

Desai et al. [ 54 ], in addition to Taylor et al. [ 33 ] and Walker et al. [ 61 ], also examined policies that could reduce the total cost of care. Increasing the proportion of hired unqualified care workers (over qualified care workers who are employed at a higher cost rate) resulted in savings which could be fed back into care funding, although Desai et al. [ 54 ] remarked on the legal and practical limitations to this policy. Taylor et al. [ 33 ] examined the impact of shifting cardiac catheterization services from tertiary to secondary level hospitals for low risk investigations and explored how improvements could be made to services. Significant and stable improvements in service, including reduced waiting list and overall cost of service, were achieved with the implementation of strict (appropriate referral) guidelines for admitting patients. Walker et al. [ 61 ] modelled patient flow from feeder hospitals to a single sub-acute extended care facility in Victoria, Australia, to assess the impact of local rules used by the medical registrar for admission. The local admission policy which prioritised admissions from patients under the care of private doctors pushed the total cost of care over the facility budget by 6% whereas employing no prioritisation rule reduced the total cost of care to 3% under budget.

Semwanga et al. [ 60 ], Mahmoudian-Dehkordi et al. [ 58 ] and Worni et al. [ 63 ] evaluated the impact of health policy on undesirable patient outcomes (mortality and post-treatment complication rates). Semwanga et al. [ 60 ] tested the effectiveness of policies designed to promote maternal and neonatal care in Uganda, established from the literature. Policies that enabled service uptake, such as community health education, free delivery kits and motorcycle coupons were significant in reducing neonatal death over the simulated period. Mahmoudian-Dehkordi et al. [ 58 ] explored the intended and unintended consequences of intensive care unit resource and bed management policies on system performance indicators, including patient mortality. During a simulated crisis scenario, prioritising intensive care unit patient admission to general wards over emergency admissions was found to be the most effective policy in reducing total hospital mortality. Worni et al. [ 63 ] estimated the impact of a policy to reduce venous thromboembolism rates post-total knee arthroplasty surgery and identified unintentional consequences of the strategy. The policy prevented the reimbursement of patient care fees in the event that a patient was not taking the recommended prophylaxis medication and consequently develops venous thromboembolism. Simulation results indicated a positive 3-fold decrease in venous thromboembolism rates but an unintended 6-fold increase in the number of patients who develop bleeding complications as a result of compulsory prophylaxis treatment.

Validation (including sensitivity analysis)

Statistically-based models are usually used in quantitative data rich environments where model parameters are estimated through maximum likelihood or least-squares estimation methods. Bayesian methods can also be used to compare alternative statistical model structures. SDMs and ABMs on the other hand are not fitted to data observations in the traditional statistical sense. The data are used to inform model development. Both quantitative data and qualitative data (e.g. from interviews) can be used to inform the structure of the model and the parameters of the model. Furthermore, model structure and parameter values can also be elicited from expert opinion. This means that the nature of validation of ABMs and SDMs requires more scrutiny than that of other types of models.

With increasing complexity of such models, and to strengthen confidence in their use particularly for decision support, models are often subjected to sensitivity analysis and validation tests. Twenty-two papers that utilised SDM undertook model validation, the majority having performed behavioural validity tests (see Additional file  2 for details of validation methods for each model). Key model output such as bed occupancy [ 36 , 50 ], department length of stay [ 62 ] and number of department discharges [ 31 ] were compared with real system performance data from hospitals [ 32 , 33 , 36 , 48 , 50 , 54 , 58 , 59 , 61 , 62 ], local councils [ 54 ], nationally reported figs [ 31 , 64 ]. as well being reviewed by experts [ 57 , 60 ] as realistic. Others performed more structure orientated validity tests. Model conception [ 28 , 60 ], development [ 30 , 36 , 50 , 53 , 54 , 57 , 62 ] and formulation [ 54 , 56 , 59 ] were validated by a variety of experts including health professionals [ 47 , 53 , 54 , 57 , 59 , 62 ], community groups [ 56 ] and leaders [ 60 ], steering committees [ 36 ], hospital and care representatives [ 50 , 56 , 59 ], patient groups [ 60 ] and healthcare policy makers [ 60 ]. Further tests for structural validity included checking model behaviour when subjected to extreme conditions or extreme values of parameters [ 30 , 31 , 52 , 57 , 59 , 60 , 64 ], model dimensional consistency [ 31 , 52 , 57 , 59 , 60 ], model boundary adequacy [ 31 ] and mass balance [ 54 ] and integration error checks [ 31 , 52 ]. Sensitivity analysis was performed to assess how sensitive model output was to changes in key parameters [ 49 , 51 , 57 , 60 , 64 ], to test the impact of parameters that had been based on expert opinion on model output [ 28 ] and varying key system parameters to test the robustness and effectiveness of policies [ 28 , 30 , 52 , 53 , 58 ] (on the assumption of imperfect policy implementation [ 28 ]).

Limitations of research

Most of the model limitations reported were concerned with missing parameters, feedback or inability to simulate all possible future health system innovations. Mielczarek et al. [ 44 ], Cepoiu-Martin et al. [ 51 ], Ansah et al. [ 49 ] and Rashwan et al. [ 31 ] did not take into account how future improvements in technology or service delivery may have impacted results, such as the possibility of new treatment improving patient health outcomes [ 51 ] and how this could impact the future utilisation of acute care services [ 49 ]. Walker et al. [ 61 ] and Alonge et al. [ 30 ] described how the models may not simulate all possible actions or interactions that occurred in the real system, such as all proactive actions taken by hospital managers to achieve budget targets [ 61 ] or all unintended consequences of a policy on the system [ 30 ]. De Andrade et al. [ 53 ] and Rashwan et al. [ 31 ] discussed the reality of model boundaries, that SDMs cannot encapsulate all health sub-sector behaviour and spill-over effects. Although these have been listed here as limitations, not accounting for possible future improvements in healthcare service or not simulating all possible actions in the modelled system did not prevent authors from fulfilling study objectives. When developing a SDM, it is not possible to account for all possible spill-over effects to other healthcare departments and this should not be attempted; model boundaries are set to only include variables and feedback that are pertinent to exploring the defined problem.

Simplification of model parameters was another common limitation. Wong et al. [ 62 ] stated that this would result in some model behaviour not holding in the real system, such as using weekly hospital admission and discharge averages in place of hourly rates due to the hospital recording aggregated data. This aggregation of model parameters may not have reflected real system complexity; Eleyan et al. [ 55 ] did not differentiate between service level and type of hospital when modelling health care waste production (described as future work) and Worni et al. [ 63 ] refrained from stratifying post-surgery complications by severity, potentially combining lethal and less harmful complications within the same stock (although this did not detract from the study conclusion that the rate of complications would increase as a result of the tested policy).

Data availability, lack of costing analysis and short time horizons were also considered credible limitations. Models that had been calibrated with real data were at risk of using datasets that contained measurement errors or incomplete datasets lacking information required to inform model structure or feedback [ 32 ]. Routine facility data required for model conception and formulation was unavailable which restricted the replication of facility behaviour in the model [ 36 ] and restricted validation of model behaviour [ 59 ], although it should be noted that this is only one method among many for SDM validation and the author was able to use other sources of data for this purpose. Lack of costing or cost effectiveness analysis when testing policies [ 60 ], particularly policies that required significant investment or capacity expansion [ 58 ], limited discussion on their feasibility in the real system. Models that simulated events over short time scales did not evaluate long term patient outcomes [ 33 ] or the long term effects of facility policies on certain groups of patient [ 57 ].

ABM use in health system research (including hybrid ABM-DES)

The model’s ability to closely replicate human behaviour that exists in the real system was frequently cited [ 20 , 21 , 22 , 25 , 66 ], providing a deeper understanding of multiple agent decision-making [ 23 , 67 ], agent networks [ 25 ] and interactions [ 21 , 22 ]. The modelling method was described as providing a flexible framework capable of conveying intricate system structures [ 20 ], where simulations captured agent capacity for learning and adaptive behaviour [ 20 , 25 ] and could incorporate stochastic processes that mimicked agent transition between states [ 25 ]. ABM took advantage of key individual level agent data [ 25 ] and integrated information from various sources including demographic, epidemiological and health service data [ 65 ]. The visualisation of systems and interface available with ABM software packages facilitated stakeholder understanding of how tested policies could impact financial and patient health outcomes [ 23 ], particularly those experts in the health industry with minimal modelling experience [ 67 ].

Integrating DES and ABM within a single model ensured an intelligent and flexible approach for simulating complex systems, such as the outpatient clinic described in Kittipittayakorn et al. [ 24 ]. The hybrid model captured both orthopaedic patient flow and agent decision-making that enabled identification of health care bottlenecks and optimum resource allocation [ 24 ].

Seven papers that utilised ABM modelled systems that were either concerned with delivering emergency or acute care 2 , ACOs or health insurance reimbursement schemes.

Liu et al. [ 21 ] and Yousefi et al. [ 22 ] modelled behaviour in EDs in Spanish and Brazilian tertiary hospitals. Liu et al. [ 21 ] simulated the behaviour of eleven key agents in the ED including patients, admission staff, doctors, triage nurses and auxiliary staff. Patients were admitted to the ED and triaged before tests were requested and a diagnosis issued. Over time, agent states changed based on their interaction with other agents such as when a doctor decided upon a course of action for a patient (sending the patient home, to another ward, or continue with diagnosis and treatment). For further details of agent type and model rules for each paper, see Additional file  3 .

Yousefi et al. [ 22 ] modelled the activities of patients, doctors, nurses and receptionists in a ED. Agents could communicate with each other, to a group of other agents or could send a message to an area of the ED where other agents reside. They made decisions based on these interactions and the information available to them at the time. The main focus of the simulation was on patients who left the ED without being seen by a physician; patients decided whether to leave the ED based on a ‘tolerance’ time extracted from the literature, which changed based on their interaction with other agents. In an additional paper, Yousefi et al. [ 67 ] simulated decision-making by patients, doctors, nurses and lab technicians within a generic ED informed from the literature. Group decision-making was employed, whereby facility staff could interact with each other and reach a common solution for improving the efficacy of the department such as re-allocating staff where needed. Yousefi et al. [ 67 ], Yousefi et al. [ 22 ] and Liu et al. [ 21 ] each used a finite state machine (a computational model which describes an entity that can be in one of a finite number of states) to model interactions between agents and their states.

Liu et al. [ 25 ] and Alibrahim et al. [ 23 ] modelled the behaviour of patients, health providers and payers using series of conditional probabilities, where health providers had participated in an ACO in the United States. Liu et al. [ 25 ] presented a model where health providers within an ACO network worked together to reduce congestive heart failure patient healthcare costs and were consequently rewarded a portion of the savings from the payer agent (hypothetically, the Centers for Medicare and Medicaid Services). Patients were Medicare beneficiaries over the age of 65 who developed diabetes, hypertension and/or congestive heart failure and sought care within the network of health providers formed of three hospitals and 15 primary care physician clinics. Alibrahim et al. [ 23 ] adapted Liu et al. [ 25 ] ACO network model to allow patients to bypass their nearest medical provider in favour of an alternative provider. The decision for a patient to bypass their nearest health centre was influenced by patient characteristics, provider characteristics and the geographical distance between health providers. Providers were also given a choice on whether to participate in an ACO network, where they would then need to implement a comprehensive congestive heart failure disease management programme.

Einzinger et al. [ 65 ] created a tool that could be used to compare different health insurance reimbursement schemes in the Austrian health sector. The ABM utilised anonymous routine data from practically all persons with health insurance in Austria, pertaining to medical services accessed in the outpatient sector. In the simulation, patients developed a chronic medical issue (such as coronary heart disease) that required medical care and led to the patient conducting a search of medical providers through the health market. The patient then accessed care at their chosen provider where the reimbursement system, notified of the event via a generic interface, reimbursed the medical provider for patients care.

Nine papers tested the impact of policy on key health system performance or service indicators. The intended target of these policies ranged from decreasing patient length of stay, to reducing the number of patients who leave without being seen by a physician to reducing patient mortality and hospitalisation rates.

Huynh et al. [ 20 ], Yousefi et al. [ 22 ], Yousefi et al. [ 67 ] and Kittipittayakorn et al. [ 24 ] tested policies to reduce the time agents spent performing tasks, waiting for a service or residing in parts of the system. Huynh et al. [ 20 ] modelled the medication administration workflow for registered nurses at an anonymous medical centre in the United States and simulated changes to the workflow to improve medication administration safety. Two policies were tested; establishing a rigid order for tasks to be performed and for registered nurses to perform tasks in the most frequently observed order (observed in a real medical centre) to see if this improved the average amount of time spent on tasks. Yousefi et al. [ 67 ] modelled the effects of group decision-making in ED compared with the standard approach for resource allocation (where a single supervisor allocates resources) to assess which policy resulted in improved ED performance. Turning ‘on’ group decision-making and starting the simulation with a higher number of triage staff and receptionists resulted in the largest reduction of average patient length of stay and number of patients who left without being seen. This last performance indicator was the subject of an additional paper [ 22 ], with focus on patient-to-patient interactions and how this impacted their decision to leave the ED before being seen by a physician. Four policies adapted from case studies were simulated to reduce the number of patients leaving the ED without being seen and average patient length of stay. The policy of fast-tracking patients who were not acutely unwell during triage performed well as opposed to baseline, where acutely ill patients were always given priority. Kittipittayakorn et al. [ 24 ] used ABM-DES to identify optimal scheduling for appointments in an orthopaedic outpatient clinic, with average patient waiting time falling by 32% under the tested policy.

Liu et al. [ 25 ], Alibrahim et al. [ 23 ] and Yousefi et al. [ 67 ] tested the impact of health policy on undesirable patient outcomes (patient mortality and hospitalisation rates). Liu et al. [ 25 ] modelled health care providers who operated within an ACO network and outside of the network and compared patient outcomes. Providers who operated within the ACO network worked together to reduce congestive heart failure patient healthcare costs and were then rewarded with a portion of the savings. As part of their membership, providers implemented evidence-based interventions for patients, including comprehensive discharge planning with post-discharge follow-up; this intervention was identified in the literature as key to reducing congestive heart failure patient hospitalisation and mortality, leading to a reduction in patient care fees without compromising the quality of care. The ACO network performed well, with a 10% reduction observed in hospitalisation compared with the standard care network. In another study [ 23 ] six scenarios were simulated with combinations of patient bypass capability (turned “on” or “off”) and provider participation in the ACO network (no ACO present, optional participation in ACO or compulsory participation in ACO). Provider participation in the ACO, in agreement with Liu et al. [ 25 ], led to reduced mortality and congestive heart failure patient hospitalisation, with patient bypass capability marginally increasing provider ACO participation. Yousefi et al. [ 67 ] also modelled the impact of group decision-making in ED on the number of patient deaths and number of wrong discharges i.e. patients sent to the wrong sector for care after triage and are then discharged before receiving correct treatment.

Nine of the 11 papers that utilised ABM undertook model validation, consisting almost exclusively of behavioural validity tests. Model output, such as patient length of stay and mortality rates, was reviewed by health professionals [ 46 , 66 ] and compared with data extracted from pilot studies [ 20 ], health facilities (historical) [ 22 , 24 , 46 , 65 , 66 ], national health surveys [ 65 ] and relevant literature [ 23 , 25 ]. Papers presented the results of tests to determine the equivalence of variance [ 20 ] and difference in mean [ 20 , 24 ] between model output and real data. Structural validity tests included extreme condition testing [ 23 , 46 ] and engaging health care experts to ensure the accuracy of model framework [ 22 , 47 ]. Sensitivity analysis was performed to determine how variations or uncertainty in key parameters (particularly where they had not been derived from historical or care data [ 65 ]) affected model outcomes [ 23 , 25 ].

The majority of model limitations reported were concerned the use or availability of real system or case data. Huynh et al. [ 20 ], Yousefi et al. [ 67 ] and Liu et al. [ 25 ] formulated their models using data that was obtainable, such as limited sample data extracted from a pilot study [ 20 ], national average trends [ 25 ] and data from previous studies [ 67 ]. Yousefi et al. [ 22 ] case study dataset did not contain key system feedback, such as the tolerance time of patients waiting to be seen by a physician in the ED, although authors were able to extract this data from a comparable study identified in the literature.

Missing model feedback or parameters, strict model boundaries and simplification of system elements were also considered limitations. Huynh et al. [ 20 ], Hutzschenreuter et al. [ 66 ] and Einzinger et al. [ 65 ] did not model all the realistic complexities of their system, such as all possible interruptions to tasks that occur in patient care units [ 20 ], patient satisfaction of admission processes [ 66 ] (which will be addressed in future work), how treatment influences the course of disease or that morbid patients are at higher risk of developing co-morbidity than healthier patients, which would affect the service needs and consumption needs of the patient [ 65 ]. To improve the accuracy of the model, Huynh et al. stated that further research is taking place to obtain real, clinical data (as opposed to clinical simulation lab results) to assess the impact of interruptions on workflow. Liu et al.’s [ 21 ] model boundary did not include other hospital units that may have been affected by ED behaviour and they identify this as future work, for example to include hospital wards that are affected by ED behaviour. Alibrahim et al. [ 23 ] and Einzinger et al. [ 65 ] made simplifications to the health providers and networks that were modelled, such as assuming equal geographical distances and identical care services between health providers in observed networks [ 23 ], limiting the number of factors that influenced a patients decision to bypass their nearest health provider [ 65 ] and not simulating changes to health provider behaviour based on service utilisation or reimbursement scheme in place [ 23 ]. Alibrahim et al. [ 23 ] noted that although the model was constrained by such assumptions, the focus of future work would be to improve the capability of the model to accurately study the impact of patient choice on economic, health and health provider outcomes.

SDM-ABM use in health system research

A single paper used hybrid SDM-ABM to model health system behaviour. Djanatliev et al. [ 47 ] developed a tool that could be used to assess the impact of new health technology on performance indicators such as patient health and projected cost of care. A modelling method that could reproduce detailed, high granularity system elements in addition to abstract, aggregate health system variables was sought and a hybrid SDM-ABM was selected. The tool nested an agent-based human decision-making module (regarding healthcare choices) within a system dynamics environment, simulating macro-level behaviour such as health care financing and population dynamics. A case study was presented to show the potential impact of Mobile Stroke Units (MSU) on patient morbidity in Berlin, where stroke diagnosis and therapy could be initiated quickly as opposed to standard care. The model structure was deemed credible after evaluation by experts, including doctors and health economists.

Comparison of SDM and ABM papers

The similarities and differences among the SDM and ABM body of literature are described in this section and shown in Table  3 . A high proportion of papers across both modelling methods simulated systems that were concerned with emergency or acute care. A high number of SDM papers (11/28) simulated patient flow and pathways through emergency care [ 28 , 31 , 36 , 45 , 47 , 50 , 56 , 57 , 58 , 61 , 62 ] with a subset evaluating the impact of policies that relieved pressure on at capacity ED’s [ 28 , 36 , 50 , 58 , 62 ]. ABM papers simulated micro-level behaviour associated with emergency care, such as health professional and patient behaviour in EDs and what impact agent interactions have on actions taken over time [ 21 , 22 , 47 , 67 ]. ACOs and health insurance reimbursement schemes, a common modelled healthcare setting among the ABM papers [ 23 , 25 , 65 ] was the focus of a single SDM paper [ 63 ] while health care waste management, a popular healthcare setting for SDM application [ 37 , 48 , 52 , 55 ] was entirely absent among the selected ABM literature. SDM and ABM were both used to test the impact of policy on undesirable patient outcomes, including patient mortality [ 23 , 25 , 58 , 60 , 67 ] and hospitalisation rates [ 23 , 25 ]. Interventions for reducing patient waiting time for services [ 24 , 33 , 53 , 61 , 67 ] and patient length of stay [ 22 , 31 , 67 ] were also tested using these methods, while policy exploration to reduce the total cost of care was more frequent among SDM studies [ 33 , 54 , 61 ].

SDM and ABM software platforms provide accessible, user-friendly visualisations of systems that enable engagement with health experts necessary for model validation [ 48 , 50 , 55 , 57 ] and facilitate stakeholder understanding of how alternative policies can impact health system performance under a range conditions [ 31 , 49 , 58 , 61 ]. The ability to integrate information and data from various sources was also cited as rationale for using SDM and ABM [ 51 ]. Reasons for using SDM to model health systems, as opposed to other methods, included gaining a whole-system perspective crucial for investigating undesirable or counter-intuitive system behaviour across sub-systems [ 28 , 36 , 37 , 48 , 56 ] and identifying unintended consequences or policy resistance with tested health policies [ 59 , 61 ]. The ability to replicate human behaviour [ 20 , 21 , 22 , 25 , 66 ] and capacity for learning and adaptive behaviour [ 20 , 25 ] was frequently cited as rationale for using ABM to simulate health systems.

Validation of SDMs and ABMs consisted mostly of behavioural validity tests where model output was reviewed by experts and compared to real system performance data or to relevant literature. Structural validity tests were uncommon among ABM papers while expert consultation on model development [ 30 , 36 , 50 , 53 , 54 , 57 , 62 , 63 ], extreme condition [ 30 , 31 , 52 , 57 , 59 , 60 , 64 ] and dimensional consistency tests [ 31 , 52 , 57 , 59 , 60 ] were frequently reported in the SDM literature. The inability to simulate all actions or interactions that occur in the real system [ 20 , 30 , 61 , 65 , 66 ] and simplification of model parameters [ 23 , 55 , 62 , 63 , 65 ] were described as limitations in both SDM and ABM papers. Data availability for model conception and formulation [ 20 , 22 , 25 , 32 , 36 , 67 ] and the impact of model boundaries (restricting exploration of interconnected sub-system behaviour [ 21 , 31 , 53 ]) were also cited limitations common to both sets of literature. Lack of costing analysis [ 58 , 60 ], short time horizons [ 33 , 57 ] and an inability to model future improvements in technology or service delivery [ 31 , 44 , 49 , 51 ] were additionally cited among the SDM papers.

Statement of principal findings

Our review has confirmed that there is a growing body of research demonstrating the use of SDM and ABM to model health care systems to inform policy in a range of settings. While the application of SDM has been more widespread (with 28 papers identified) there are also a growing number of ABM being used (11), just over half of which used hybrid simulation. A single paper used hybrid SDM-ABM to model health system behaviour. To our knowledge this is the first review to identify and compare the application of both SDM and ABM to model health systems. The first ABM article identified in this review was published almost a decade after the first SDM paper; this reflects to a certain extent the increasing availability of SDM and ABM dedicated software tools with the developments in ABM software lagging behind their SDM modelling counterparts.

Emergency and acute care, and elderly care and LTC services were the most frequently simulated health system setting. Both sets of services are facing exponential increases in demand with constraints on resources, presenting complex issues ideal for evaluation through simulation. Models were used to explore the impact and potential spill over effects of alternative policy options, prior to implementation, on patient outcomes, service use and efficiency under various structural and financial constraints.

Strengths and weaknesses of the study

To ensure key papers were identified, eight databases across four research areas were screened for relevant literature. Unlike other reviews in the field [ 39 , 40 ], there was no restriction placed on publication date. The framework for this review was built to provide a general overview of the SDM and ABM of healthcare literature, capturing papers excluded in other published reviews as a result of strict inclusion criteria. These include reviews that have focussed specifically on compiling examples of modelled health policy application in the literature [ 35 ] or have searched for papers with a particular health system setting, such as those that solely simulate the behaviour of emergency departments [ 34 ]. One particularly comprehensive review of the literature had excluded papers that simulated hospital systems, which we have explicitly included as part of our search framework [ 39 ].

The papers presented in this review, with selection restricted by search criteria, provide a broad picture of the current health system modelling landscape. The focus of this review was to identify models of facility-based healthcare, purposely excluding literature where the primary focus is on modelling disease progression, disease transmission or physiological disorders which can be found in other reviews such as Chang et al. [ 39 ] and Long et al. [ 41 ]. The data sources or details of how data was used to conceptualise and formulate models are not presented in this paper; this could on its own be the focus of another study and we hope to publish these results as future work. This information would be useful for researchers who want to gain an understanding of the type and format of data used to model health systems and best practice for developing and validating such models.

Literature that was not reported in English was excluded from the review which may have resulted in a small proportion of relevant papers being missed. Papers that described DES models, the other popular modelling method for simulating health system processes, were not included in this review (unless DES methods are presented as part of a hybrid model integrated with SDM or ABM) but have been compiled elsewhere [ 68 , 69 , 70 ]. Finally, the quality of the papers was not assessed.

Implications for future research

A nominal number of SDM papers (9/28), an even lower proportion of ABM papers (2/11) and none of the hybrid methods papers simulated health systems based in low- or middle-income countries (LMICs). The lower number of counterpart models in LMICs can be attributed to a lack of capacity in modelling methods and perhaps the perceived scarcity of suitable data; however, the rich quantitative and qualitative primary data collected in these countries for other types of evaluation could be used to develop such models. Building capacity for using these modelling methods in LMICs should be a priority and generating knowledge of how and which secondary data to use in these settings for this purpose. In this review, we observed that it is feasible to use SDM to model low-income country health systems, including those in Uganda [ 60 ] and Afghanistan [ 30 ]. The need to increase the use of these methods within LMICs is paramount; even in cases where there is an absence of sufficient data, models can be formulated for LMICs and used to inform on key data requirements through sensitivity analysis, considering the resource and healthcare delivery constraints experienced by facilities in these settings. This research is vital for our understanding of health system functioning in LMICs, and given the greater resource constraints, to allow stakeholders and researchers to assess the likely impact of policies or interventions before their costly implementation, and to shed light on optimised programme design.

Health system professionals can learn greatly from using modelling tools, such as ABM, SDM and hybrid models, developed originally in non-health disciplines to understand complex dynamic systems. Understanding the complexity of health systems therefore require collaboration between health scientists and scientists from other disciplines such as engineering, mathematics and computer science. Discussion and application of hybrid models is not a new phenomenon in other fields but their utilisation in exploring health systems is still novel; the earliest article documenting their use in this review was published in 2010 [ 43 ]. Five of the six hybrid modelling papers [ 43 , 44 , 45 , 46 , 47 ] were published as conference proceedings (the exception Kittipittayakorn et al. [ 24 ]), demonstrating the need to include conference articles in systematic reviews of the literature in order to capture new and evolving applications of modelling for health systems research.

The configuration and extent to which two distinct types of models are combined has been described in the literature [ 71 , 72 , 73 , 74 , 75 ]. The hybrid modelling papers selected in this review follow what is described as ‘hierarchical’ or ‘process environment’ model structures, the former where two distinct models pass information to each other and the latter where one model simulates system processes within the environment of another model [ 72 ]. Truly ‘integrated’ models, considered the ‘holy grail’ [ 43 ] of hybrid simulation, where elements of the system are simulated by both methods of modelling with no clear distinction, were not identified in this review and in the wider literature remain an elusive target. In a recent review of hybrid modelling in operational research only four papers were identified to have implemented truly integrated hybrid simulation and all used bespoke software, unrestricted by the current hybrid modelling environments [ 76 ].

Of the six hybrid modelling papers, only Djanatliev et al. [ 47 ] presented a model capable of both ABM and SDM simulation. The crucial macro- and micro- level activity captured in such models represent feedback in the wider, complex system while retaining the variable behaviour exhibited by those who access or deliver healthcare. With increasing software innovation and growing demand for multi-method modelling in not only in healthcare research but in the wider research community, we need to increase their application to modelling health systems and progress towards the ‘holy grail’ of hybrid modelling.

We identified 28 papers using SDM methods and 11 papers using ABM methods to model health system behaviour, six of which implemented hybrid model structures with only a single paper using SDM-ABM. Emergency and acute care, and elderly care and LTC services were the most frequently simulated health system settings, modelling the impact of health policies and interventions targeting at-capacity healthcare services, patient length of stay in healthcare facilities and undesirable patient outcomes. A high proportion of articles modelled health systems in high income countries; future work should now turn to modelling healthcare settings in LMIC to support policy makers and health system researchers alike. The utilisation of hybrid models in healthcare is still relatively new but with an increasing demand to develop models that can simulate the macro- and micro-level activity exhibited by health systems, we will see an increase in their use in the future.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

One of the elderly or LTC services papers also modelled emergency or acute care but it was not the primary focus and is therefore not discussed here.

The single SDM-ABM paper that modelled the delivery of emergency or acute care is discussed in section ‘SDM-ABM use in health system research’.

Six of the emergency or acute care review papers and one of the cardiology care papers also modelled elderly or LTC services but it was not the primary focus and are therefore not discussed here.

Abbreviations

Accountable care organisation

Agent-based model

Discrete-event simulation

Emergency Department

Long-term care

Low- and middle-income countries

System dynamics model

Kitson A, Brook A, Harvey G, Jordan Z, Marshall R, O’Shea R, et al. Using Complexity and Network Concepts to Inform Healthcare Knowledge Translation. Int J Heal Policy Manag. 2017;7:231–43. https://doi.org/10.15171/ijhpm.2017.79 .

Article   Google Scholar  

Paina L, Peters DH. Understanding pathways for scaling up health services through the lens of complex adaptive systems. Health Policy Plan. 2012;27:365–73. https://doi.org/10.1093/heapol/czr054 .

Article   PubMed   Google Scholar  

Lipsitz LA. Understanding health care as a complex system: the foundation for unintended consequences. JAMA. 2012;308:243–4. https://doi.org/10.1001/jama.2012.7551 .

Article   PubMed   PubMed Central   CAS   Google Scholar  

Adam T, de Savigny D. Systems thinking for strengthening health systems in LMICs: need for a paradigm shift. Health Policy Plan. 2012;27:iv1–3. https://doi.org/10.1093/heapol/czs084 .

de Savigny D, Blanchet K, Adam T. Applied systems thinking for health systems research : a methodological handbook: McGraw-Hill Education; 2017.

Shepherd SP. A review of system dynamics models applied in transportation. Transp B Transp Dyn. 2014;2:83–105. https://doi.org/10.1080/21680566.2014.916236 .

Sterman JD. Business dynamics: systems thinking and modeling for a complex world: McGraw-Hill Companies Inc; 2000.

Kunc M, Mortenson MJ, Vidgen R. A computational literature review of the field of system dynamics from 1974 to 2017. J Simul. 2018;12:115–27. https://doi.org/10.1080/17477778.2018.1468950 .

System Dynamics for Engineering Students. Elsevier; 2018. doi: https://doi.org/10.1016/C2011-0-05346-2 .

Radzicki MJ. System Dynamics and Its Contribution to Economics and Economic Modeling. Encycl. Complex. Syst. Sci. New York: Springer New York; 2009. p. 8990–9000. https://doi.org/10.1007/978-0-387-30440-3_539 .

Book   Google Scholar  

Ford A. Global warming and system dynamics. Int Conf Syst Dyn Soc. 2007.

Fiddaman TS. Exploring policy options with a behavioral climate-economy model. Syst Dyn Rev. 2002;18:243–67. https://doi.org/10.1002/sdr.241 .

Popli K, Sudibya GL, Kim S. A review of solid waste management using system dynamics modeling. J Environ Sci Int. 2017;26:1185–200. https://doi.org/10.5322/JESI.2017.26.10.1185 .

Groesser SN, Schaffernicht M. Mental models of dynamic systems: taking stock and looking ahead. Syst Dyn Rev. 2012;28:46–68. https://doi.org/10.1002/sdr.476 .

Epstein JM. Generative social science: studies in agent-based computational modeling: STU-Stud. Princeton University Press; 2006.

Wilensky U, Rand W. An Introduction to Agent-Based Modeling: Mit Press; 2015.

Miller JH, Page SE. Complex Adaptive Systems: An Introduction to Computational Models of Social Life: STU-Stud. Princeton University Press; 2007.

Railsback SF, Grimm V. Agent-Based and Individual-Based Modeling: Princeton University Press; 2011.

Abar S, Theodoropoulos G, Lemarinier P, O’Hare G. Agent based Modelling and simulation tools: a review of the state-of-art software. Comput Sci Rev. 2017;24:13–33. https://doi.org/10.1016/j.cosrev.2017.03.001 .

Huynh N, Snyder R, Vidal J, Tavakoli A, Cai B. Application of computer simulation modeling to medication administration process redesign. J Healthc Eng. 2012;3:649–62. https://doi.org/10.1260/2040-2295.3.4.649 .

Liu Z, Cabrera E, Rexachs D, Luque E. A generalized agent-based model to simulate emergency departments. Sixth Int Conf Adv Syst Simul. 2014:65–70.

Yousefi M, Yousefi M, Fogliatto FS, Ferreira RPM, Kim JH. Simulating the behavior of patients who leave a public hospital emergency department without being seen by a physician: a cellular automaton and agent-based framework. Brazilian J Med Biol Res. 2018;51:e6961. https://doi.org/10.1590/1414-431X20176961 .

Alibrahim A, Wu S. An agent-based simulation model of patient choice of health care providers in accountable care organizations. Health Care Manag Sci. 2018;21:131–43. https://doi.org/10.1007/s10729-016-9383-1 .

Kittipittayakorn C, Ying K-C. Using the integration of discrete event and agent-based simulation to enhance outpatient service quality in an orthopedic department. J Healthc Eng. 2016;2016:4189206. https://doi.org/10.1155/2016/4189206 .

Article   PubMed Central   Google Scholar  

Liu P, Wu S. An agent-based simulation model to study accountable care organizations. Health Care Manag Sci. 2016;19:89–101. https://doi.org/10.1007/s10729-014-9279-x .

Allen M, Spencer A, Gibson A, Matthews J, Allwood A, Prosser S, et al. Right cot, right place, right time: improving the design and organisation of neonatal care networks – a computer simulation study. Heal Serv Deliv Res. 2015;3:9. https://doi.org/10.3310/hsdr03200 .

Brailsford SC. Tutorial: Advances and challenges in healthcare simulation modeling: 2007 Winter Simul. Conf., IEEE; 2007. p. 1436–48. https://doi.org/10.1109/WSC.2007.4419754 .

Esensoy AV, Carter MW. High-Fidelity whole-system patient flow modeling to assess health care transformation policies. Eur J Oper Res. 2018;266:221–37. https://doi.org/10.1016/j.ejor.2017.09.019 .

Homer JB, Hirsch GB. System dynamics modeling for public health: background and opportunities. Am J Public Health. 2006;96:452–8. https://doi.org/10.2105/AJPH.2005.062059 .

Article   PubMed   PubMed Central   Google Scholar  

Alonge O, Lin S, Igusa T, Peters DH. Improving health systems performance in low- and middle-income countries: a system dynamics model of the pay-for-performance initiative in Afghanistan. Health Policy Plan. 2017;32:1417–26. https://doi.org/10.1093/heapol/czx122 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Rashwan W, Abo-Hamad W, Arisha A. A system dynamics view of the acute bed blockage problem in the Irish healthcare system. Eur J Oper Res. 2015;247:276–93. https://doi.org/10.1016/j.ejor.2015.05.043 .

Ghaffarzadegan N, Epstein AJ, Martin EG. Practice variation, bias, and experiential learning in cesarean delivery: A data-based system dynamics approach. Health Serv Res. 2013;48:713–34. https://doi.org/10.1111/1475-6773.12040 .

Taylor K, Dangerfield B. Modelling the feedback effects of reconfiguring health services. J Oper Res Soc. 2005;56:659–75. https://doi.org/10.1057/palgrave.jors.2601862 .

Mohiuddin S, Busby J, Savovic J, Richards A, Northstone K, Hollingworth W, et al. Patient flow within UK emergency departments: a systematic review of the use of computer simulation modelling methods. BMJ Open. 2017;7:e015007. https://doi.org/10.1136/bmjopen-2016-015007 .

Atkinson J-A, Wells R, Page A, Dominello A, Haines M, Wilson A. Applications of system dynamics modelling to support health policy. Public Heal Res Pract. 2015;25. https://doi.org/10.17061/phrp2531531 .

Lattimer V, Brailsford S, Turnbull J, Tarnaras P, Smith H, George S, et al. Reviewing emergency care systems I: insights from system dynamics modelling. Emerg Med J. 2004;21:685–91. https://doi.org/10.1136/emj.2002.003673 .

Chaerul M, Tanaka M, Shekdar AV. A system dynamics approach for hospital waste management. Waste Manag. 2008;28:442–9. https://doi.org/10.1016/j.wasman.2007.01.007 .

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. https://doi.org/10.1136/bmj.b2535 .

Chang AY, Ogbuoji O, Atun R, Verguet S. Dynamic modeling approaches to characterize the functioning of health systems: A systematic review of the literature. Soc Sci Med. 2017;194:160–7. https://doi.org/10.1016/j.socscimed.2017.09.005 .

Rusoja E, Haynie D, Sievers J, Mustafee N, Nelson F, Reynolds M, et al. Thinking about complexity in health: a systematic review of the key systems thinking and complexity ideas in health. J Eval Clin Pract. 2018;24:600–6. https://doi.org/10.1111/jep.12856 .

Long KM, Meadows GN. Simulation modelling in mental health: a systematic review. J Simul. 2017. https://doi.org/10.1057/s41273-017-0062-0 .

Banco Mundial. World Bank: country and lending groups - DataBank. World Bank Gr. 2018.

Brailsford SC, Desai SM, Viana J. Towards the holy grail: Combining system dynamics and discrete-event simulation in healthcare: Proc. 2010 Winter Simul. Conf., IEEE; 2010. p. 2293–303. https://doi.org/10.1109/WSC.2010.5678927 .

Mielczarek B, Zabawa J. Modeling Healthcare Demand Using a Hybrid Simulation Approach, vol. 2016: Proc. 2016 Winter Simul. Conf., IEEE Press. p. 1535–46.

Zulkepli J. Hybrid simulation for modelling large systems: an example of integrated care model: Proc. 2012 Winter Simul. Conf; 2012.

Viana J, Simonsen TB, Dahl FA, Flo K. A Hybrid Discrete Event Agent Based Overdue Pregnancy Outpatient Clinic Simulation Model. Proc. 2018 Winter Simul. Conf. Piscataway: IEEE Press; 2018. p. 1488–99.

Google Scholar  

Djanatliev A, German R, Kolominsky-Rabas P, Hofmann BM. Hybrid simulation with loosely coupled system dynamics and agent-based models for prospective health technology assessments: Proc. Winter Simul. Conf., Winter Simulation Conference; 2012. p. 69:1–69:12.

Al-Khatib IA, Eleyan D, Garfield J. A system dynamics approach for hospital waste management in a city in a developing country: the case of Nablus, Palestine. Environ Monit Assess. 2016;188:503. https://doi.org/10.1007/s10661-016-5487-9 .

Ansah JP, Eberlein RL, Love SR, Bautista MA, Thompson JP, Malhotra R, et al. Implications of long-term care capacity response policies for an aging population: a simulation analysis. Health Policy. 2014;116:105–13. https://doi.org/10.1016/j.healthpol.2014.01.006 .

Brailsford SC. Emergency and on-demand health care: modelling a large complex system. J Oper Res Soc. 2004;55.

Cepoiu-Martin M, Bischak DP. Policy choices in dementia care - an exploratory analysis of the Alberta Continuing Care System (ACCS) using system dynamics. Spec Issue Complex Forum “real” World Heal Syst Implic Complex. Theor Sci. 2018;24:278–84.

Ciplak N, Barton JR. A system dynamics approach for healthcare waste management: A case study in Istanbul Metropolitan City, Turkey. Waste Manag Res. 2012;30:576–86. https://doi.org/10.1177/0734242X12443405 .

de Andrade L, Lynch C, Carvalho E, Rodrigues CG, Vissoci JRN, Passos GF, et al. System dynamics modeling in the evaluation of delays of care in ST-segment elevation myocardial infarction patients within a tiered health system. PLoS One. 2014;9:e103577. https://doi.org/10.1371/journal.pone.0103577 .

Desai MS, Penn ML, Brailsford S, Chipulu M. Modelling of Hampshire adult services—gearing up for future demands. Health Care Manag Sci. 2008;11:167–76. https://doi.org/10.1007/s10729-007-9049-0 .

Eleyan D, Al-Khatib IA, Garfield J. System dynamics model for hospital waste characterization and generation in developing countries. Waste Manag Res. 2013;31:986–95. https://doi.org/10.1177/0734242X13490981 .

Lane D. Emergency but no accident: a system dynamics study of casualty waiting times in the British NHS. Eurohealth (Lond). 1998;4.

Lane DC. Looking in the wrong place for healthcare improvements: a system dynamics study of an accident and emergency department. J Oper Res Soc. 2000;51.

Mahmoudian-Dehkordi A, Sadat S. Sustaining critical care: using evidence-based simulation to evaluate ICU management policies. Health Care Manag Sci. 2017;20:532–47. https://doi.org/10.1007/s10729-016-9369-z .

Meker T, Barlas Y. Dynamic consequences of performance-based payment Systems in Public Hospitals. Syst Res Behav Sci. 2015;32:459–80. https://doi.org/10.1002/sres.2338 .

Semwanga AR, Nakubulwa S, Adam T. Applying a system dynamics modelling approach to explore policy options for improving neonatal health in Uganda. Heal Res Policy Syst. 2016;14:35. https://doi.org/10.1186/s12961-016-0101-8 .

Walker BC, Haslett T. The dynamics of local rules in hospital admission processes. Aust Health Rev. 2003;26:98–106.

Wong HJ, Wu RC, Caesar M, Abrams H, Morra D. Smoothing inpatient discharges decreases emergency department congestion: a system dynamics simulation model. Emerg Med J. 2010;27:593–8. https://doi.org/10.1136/emj.2009.078543 .

Worni M, Pietrobon R, Zammar GR, Shah J, Yoo B, Maldonato M, et al. System dynamics to model the unintended consequences of denying payment for venous thromboembolism after total knee arthroplasty. PLoS One. 2012;7:e30578. https://doi.org/10.1371/journal.pone.0030578 .

Yu W, Li M, Ge Y, Li L, Zhang Y, Liu Y, et al. Transformation of potential medical demand in China: A system dynamics simulation model. J Biomed Inform. 2015;57:399–414. https://doi.org/10.1016/j.jbi.2015.08.015 .

Einzinger P, Popper N, Breitenecker F, Pfeffer N, Jung R, Endel G. The GAP-DRG Model: Simulation of Outpatient Care for Comparison of Different Reimbursement Schemes. Proc. 2013 Winter Simul. Conf. Simul. Mak. Decis. A complex world. Piscataway: IEEE Press; 2013. p. 2299–308.

Hutzschenreuter AK, Bosman PAN, Blonk-Altena I, van Aarle J, La Poutré H. Agent-based patient admission scheduling in hospitals. Belgian/Netherlands Artif Intell Conf. 2008:315–6. https://doi.org/10.1007/3-540-32062-8 .

Yousefi M, Ferreira R. An agent-based simulation combined with group decision-making technique for improving the performance of an emergency department. Brazilian J Med Biol Res. 2017;50:1–10. https://doi.org/10.1590/1414-431X20175955 .

Günal MM, Pidd M. Discrete event simulation for performance modelling in health care: a review of the literature. J Simul. 2010;4:42–51. https://doi.org/10.1057/jos.2009.25 .

Salleh S, Thokala P, Brennan A, Hughes R, Dixon S. Discrete event simulation-based resource Modelling in health technology assessment. Pharmacoeconomics. 2017;35:989–1006. https://doi.org/10.1007/s40273-017-0533-1 .

Zhang X. Application of discrete event simulation in health care: a systematic review. BMC Health Serv Res. 2018;18:687. https://doi.org/10.1186/s12913-018-3456-4 .

Bennett PG. On linking approaches to decision-aiding: issues and prospects. J Oper Res Soc. 1985;36:659. https://doi.org/10.2307/2582261 .

Chahal K, Eldabi T. Applicability of hybrid simulation to different modes of governance in UK healthcare: 2008 Winter Simul. Conf., IEEE; 2008. p. 1469–77. https://doi.org/10.1109/WSC.2008.4736226 .

Lättilä L, Hilletofth P, Lin B. Hybrid simulation models – when, why, how? Expert Syst Appl. 2010;37:7969–75. https://doi.org/10.1016/j.eswa.2010.04.039 .

Brailsford SC. Hybrid simulation in healthcare: New concepts and new tools: 2015 Winter Simul. Conf., IEEE; 2015. p. 1645–53. https://doi.org/10.1109/WSC.2015.7408284 .

Morgan JS, Howick S, Belton V. A toolkit of designs for mixing discrete event simulation and system dynamics. Eur J Oper Res. 2017;257:907–18. https://doi.org/10.1016/j.ejor.2016.08.016 .

Brailsford SC, Eldabi T, Kunc M, Mustafee N, Osorio AF. Hybrid simulation modelling in operational research: a state-of-the-art review. Eur J Oper Res. 2018. https://doi.org/10.1016/j.ejor.2018.10.025 .

Download references

Acknowledgements

Not applicable.

The work described in this paper was funded by the Health Systems Research Initiative (HSRI). MRC Grant Reference Number: MR/R013454/1.

Author information

Authors and affiliations.

Department of Global Health and Development, London School of Hygiene and Tropical Medicine, 15-17 Tavistock Place, London, WC1H 9SH, UK

Rachel Cassidy, Neha S. Singh, Josephine Borghi & Karl Blanchet

Department of Mathematics, University College London, London, UK

Pierre-Raphaël Schiratti

Sia Partners UK, London, UK

Information Systems Department, College of Computing and Information Sciences, Makerere University, P.O. Box 7062, Kampala, Uganda

Agnes Semwanga

Ifakara Health Institute, PO Box 78373, Dar es Salaam, Tanzania

Peter Binyaruka

Department of Gender Studies, School of Humanities and Social Sciences, University of Zambia, 10101, Lusaka, Zambia

Nkenda Sachingongu

Economic and Business Research Programme, University of Zambia, Institute of Economic and Social Research, P O Box 30900, 10101, Lusaka, Zambia

Chitalu Miriam Chama-Chiliba

Department of Public Health, Environments and Society, London School of Hygiene and Tropical, London, UK

Zaid Chalabi

You can also search for this author in PubMed   Google Scholar

Contributions

RC, KB, ZC specified the search criteria and selection of databases. RC screened titles, evaluated full text articles and was responsible for full text extraction. RC is lead author, NSS, KB, ZC, PS, JB, AS, PB, CC, NS provided guidance on the draft and final manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Rachel Cassidy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1..

Search criteria used for each database.

Additional file 2.

Descriptive table of validation methods used in SDM and ABM literature.

Additional file 3.

Descriptive table of ABM model rules.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Cassidy, R., Singh, N.S., Schiratti, PR. et al. Mathematical modelling for health systems research: a systematic review of system dynamics and agent-based models. BMC Health Serv Res 19 , 845 (2019). https://doi.org/10.1186/s12913-019-4627-7

Download citation

Received : 14 June 2019

Accepted : 11 October 2019

Published : 19 November 2019

DOI : https://doi.org/10.1186/s12913-019-4627-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • System dynamics
  • Agent-based
  • Health systems
  • Systematic review

BMC Health Services Research

ISSN: 1472-6963

research paper on mathematical modeling pdf

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Int J Mol Sci

Logo of ijms

Mathematical Modelling in Biomedicine: A Primer for the Curious and the Skeptic

1 Laboratory of Systems Tumor Immunology, Comprehensive Cancer Center Erlangen and Deutsches Zentrum Immuntherapie (DZI), Department of Dermatology, FAU Erlangen-Nürnberg, Universitätsklinikum Erlangen, 91054 Erlangen, Germany; [email protected] (C.L.); [email protected] (X.L.); [email protected] (M.E.)

Christopher Lischer

Momchil nenov.

2 Institute of Mechanics, Bulgarian Academy of Sciences, Acad. G. Bonchev Str., bl. 4, 1113 Sofia, Bulgaria; [email protected] (M.N.); [email protected] (S.N.)

Svetoslav Nikolov

Martin eberhardt, associated data.

In most disciplines of natural sciences and engineering, mathematical and computational modelling are mainstay methods which are usefulness beyond doubt. These disciplines would not have reached today’s level of sophistication without an intensive use of mathematical and computational models together with quantitative data. This approach has not been followed in much of molecular biology and biomedicine, however, where qualitative descriptions are accepted as a satisfactory replacement for mathematical rigor and the use of computational models is seen by many as a fringe practice rather than as a powerful scientific method. This position disregards mathematical thinking as having contributed key discoveries in biology for more than a century, e.g., in the connection between genes, inheritance, and evolution or in the mechanisms of enzymatic catalysis. Here, we discuss the role of computational modelling in the arsenal of modern scientific methods in biomedicine. We list frequent misconceptions about mathematical modelling found among biomedical experimentalists and suggest some good practices that can help bridge the cognitive gap between modelers and experimental researchers in biomedicine. This manuscript was written with two readers in mind. Firstly, it is intended for mathematical modelers with a background in physics, mathematics, or engineering who want to jump into biomedicine. We provide them with ideas to motivate the use of mathematical modelling when discussing with experimental partners. Secondly, this is a text for biomedical researchers intrigued with utilizing mathematical modelling to investigate the pathophysiology of human diseases to improve their diagnostics and treatment.

1. Science: A World of Systems (and Models)

A good portion of science relies on the cascades of models that represent our reality at different spatial and temporal scales. A model is a simplified and often abstract representation of a complex natural system. Models are used in science to help understand, hypothesize about, or simulate the behavior of the natural systems they represent. Since a model is an abstraction of the natural system, it often includes only those elements, interactions, and processes of the system that are required to investigate the hypotheses in question. By exploiting the innate ability of the human brain to work with abstractions, models make highly complex phenomena accessible to study. In this sense, they can be used as a tool to develop hypotheses and to conceive and execute experiments to test those hypotheses. Models also provide an overview of the current knowledge on the natural system in question, facilitating the exchange of up-to-date information between researchers working on the same topic. Moreover, models can be used to simulate aspects of the natural system. A simulation is a rule-based recapitulation of the natural system’s behavior under relevant conditions using the model.

Since the features of a model are tightly linked to the purpose we want to give to it, there are multiple types of models. Models that are common to any branch of science are semantic models ; these consist in verbalization of the natural system’s features and the hypothesis using natural language. This is the type of model that underlies the results and discussion sections in most scientific papers.

More important for our discussion are the “lab bench” models and the “mathematical” models ( Figure 1 ). A lab bench model is a simplified or analogous version of a natural system employed in experiments under controlled conditions. In molecular and cell biology, lab bench models are usually units of life that can be conveniently propagated and studied in experiments to understand a given biological phenomenon. The consensus is that discoveries made via experimentation with the lab bench model provide insights into the behavior of similar phenomena in other organisms, especially humans. This is the case for cell lines, organoids, and mouse or rat strains with given genotypes or phenotypes, which have been used consistently in biomedicine as models for many human diseases.

An external file that holds a picture, illustration, etc.
Object name is ijms-22-00547-g001.jpg

Science employs different types of models to represent natural systems. Let us suppose that we are interested in investigating the properties and potential vulnerabilities of a melanoma metastasis (here, the “ natural system ”, visualized with MELC microscopy as in Ostalecki et al. 2017 [ 1 ]). One can represent the natural system with a semantic model , that is, the verbalization in natural language of the key compounds and processes as well as the hypotheses about a melanoma micrometastasis. Under some simplifying assumptions, a melanoma metastasis can be studied with lab bench models . For example, if one is interested in the interplay between cancer and immune cells, it is possible to co-culture tumor cells with relevant types of immune cells in vitro, like in Vescovi et al. 2019 [ 2 ]. In most cases, an alternative option is mathematical models , that is, sets of parametric equations that encode the key properties of metastasis and the hypotheses. The mathematical model is the basis for computational simulations to design experiments or to formulate or explore hypotheses like in Santos et al. 2016 [ 3 ].

A mathematical model is a set of parametric equations or other mathematical entities that encode the basic properties of the investigated natural system and that can be used to perform computational simulations. The same way that there are lab bench models with different features and purpose, there are different classes of mathematical models. One can classify them based on their treatment of the system’s dynamics as static models , which describe the system’s state at a point in time; comparative static models , which compare the properties of the system at different points in time; and dynamic models , which follow changes in the system over time. One can also classify models based on the mathematical apparatus they employ or the knowledge they exploit. There are mathematical models grounded in statistics that are used to process, analyze, and impute quantitative data generated in lab bench experiments or obtained from patient samples. All hypothesis tests and estimators of statistical correlation or inference between biological data sets are essentially (bio)statistical models.

However, there are also mathematical models that encode a mechanistic description of the natural system. This means the model equations and variables account for the interactions between the system’s key biological components and can be used for computational simulation of molecular and cellular processes. There are many subclasses of these models. A significant number of them reside in biophysics, developed to allow physically accurate simulations of atomic movement and molecular interactions in biomolecules including protein–protein interactions, DNA folding, or biomembrane dynamics. Other mechanistic models describe biochemical processes that shape cell phenotypes and cell-to-cell interactions. These are inspired by the mathematical models built in chemistry to understand and predict the kinetics of chemical reactions. In any case, just as one cannot elucidate all the mysteries of modern biomedicine with a single experimental technique, say confocal microscopy, a single subclass of mathematical model is not useful for every purpose. Every problem or hypothesis requires a carefully selected mathematical modelling approach.

2. The Scientific Method and the Role of Mathematical Modelling in It

Contemporary science consists in application of the scientific method and careful examination of the results one obtains with it. In the classical view, the scientific method is composed of three steps: observation of a natural phenomenon, elaboration of a hypothesis based on the observation, and design of an adequate experiment to test the hypothesis. If this is the way one would conceive scientific work nowadays, we would be strictly following the approaches employed by Galileo Galilei, Johannes Kepler, and Isaac Newton at the inception of modern science [ 4 ].

However, if you are a 21st century scientist, you are probably implementing a subtle variation of this method, which we will call the Einstein-grade scientific method. Here, the work is not performed in long-term isolation of a scientific ivory tower. Rather, the scientist continuously interacts with a community of peers. We retain the three basic steps mentioned above, but after the experimental test, one (more or less) immediately communicates the results in a booklet-length scientific publication. Early communication allows for your peers to try and reproduce your results. When experimental evidence accumulates and the underlying hypotheses are accepted, they are integrated in the scientific corpus of the field, that is, the set of theories, data, and hypotheses commonly accepted by the majority of researchers. Furthermore, new observations and experiments are continuously cross-checked against the scientific corpus in a way such that the method has an actual cyclic structure.

In the Einstein-grade scientific method, mathematical and computational modelling plays a pivotal role in many different manners ( Figure 2 ). Firstly, today’s “observations” in the majority of fields of natural sciences, including biomedicine, come as quantitative data that have to be processed, assessed, and analyzed with statistical models. In biomedicine, this is especially true in the case of sequencing data for the detection of common patterns in the sequence or the expression of genes in large cohorts of patients, a task impossible without sophisticated statistical and bioinformatics methods [ 5 ]. The features of these special mathematical models have a significant impact on the conclusions one can derive from sequencing data.

An external file that holds a picture, illustration, etc.
Object name is ijms-22-00547-g002.jpg

Sketch of the Einstein-grade scientific method (grey boxes) and the place that mathematical modelling occupies in it (blue boxes): the photograph of Albert Einstein is modified from the photo “Albert Einstein colorized” by Michael W. Gorth as stored in Wikimedia (CC-BY-SA-4.0, accession date: 10.08.2020; https://commons.wikimedia.org/wiki/File:Albert_Einstein_colourised_portrait.jpg ).

Secondly, computational models can be utilized to derive, ponder on, or substantiate hypotheses. Mathematical modelling has been employed to design validation experiments or to evaluate whether the results of the experiment agree with the expectations derived from the hypothesis and the scientific corpus . Finally, models can be used to achieve a formalized and unambiguous description of a field’s accepted knowledge in which theories, data, and hypotheses are organized and interconnected through equations. A remarkable example of this power is contained in a recent study by Meyer-Hermann and colleagues, in which a comprehensive mathematical model was used to summarize and elaborate around the contemporary knowledge on B-cell differentiation and maturation in germinal centers [ 6 ]. In one sense, Meyer-Hermann’s model is conceptually close to the equations that make up large physical theories, such as electromagnetism. In Supplementary Table S1 , we include and comment on select publications in which the different manners to deploy models are illustrated for biomedicine.

There are a few more issues to mention and discuss about the role of mathematical models in the scientific method. The formal language of a mathematical model enhances precision and clarity compared with natural-language descriptions in the semantic models usually preferred by biomedical experimentalists. Everybody with adequate training can develop an identical understanding of a statement formulated by an equation, for example, v = v max * S /( K M + S ). However, statements made in natural languages can be vague and can provoke misunderstandings. For example, is there a difference between “transcription factor X activates the expression of gene Y” and “transcription factor X promotes the expression of gene Y”?

The Einstein-grade method constitutes a collective endeavor. This means that it is common to find several researchers from different research institutions and locations and even with backgrounds complementing each other completing one cycle of the scientific method. For example, a German theoretical biologist, a sort of mathematical modeler, can publish a hypothesis based on his modelling efforts which will years later pique the interest of an Indonesian molecular biologist to perform experimental validation. Such occurrences are rather common in physics or chemistry, with a world record in some predictions made by Albert Einstein hundreds of years ago that have been experimentally validated only recently (which is why we use the term Einstein-grade here). However, they can also happen in biomedicine: Hodgkin and Huxley established in the first half of the 20th century a long-lasting scientific collaboration to elucidate the biophysical and biochemical mechanisms behind the initiation and propagation of action potential in nerve cell axons, a key phenomenon in understanding the ability of the nervous system to process and store information. The remarkable culmination of this collaboration was the amalgamation of their discoveries into a mathematical model which formalizes, explains, and quantifies the electrical excitability in nerve cells (see their seminal paper in Alan Hodgkin and Andrew Huxley 1952 [ 7 ] and a historical analysis in Schwiening 2012 [ 8 ]). Interestingly, some of the assumptions and simulations in the Hodgkin–Huxley model led Hodgkin’s team to predict already in 1955 that potassium ion channels can be occupied by multiple ions simultaneously. This prediction was confirmed using X-ray crystallography only in 1998, more than 40 years later, in a paradigmatic instance of the Einstein-grade scientific method in neurobiology [ 9 ].

Lastly, the scientific method is nowadays not monolithic and there are multiple variations of the general workflow commented on here [ 10 ]. This plurality of scientific methods may correspond to the different phases of discovery in diverse disciplines, the nature of the problems tackled, or even the styles of thinking belonging to different scientific communities. This notion applies also to mathematical modelling in biology and biomedicine [ 11 ].

3. The Love-and-Hate Relationship of Biology and Mathematics

The story of mathematics and modelling in biology is rather long, which might come as a surprise to many researchers. Biostatistical modelling and analysis have always been a key method in fields like evolutionary biology. In the beginning of the 20th century and right after the rediscovery of the Mendelian inheritance theory, there were many fundamental contributions to biology from statisticians that tried to bridge the gap between Mendelian genetics and Darwin’s theory of evolution. For example, the journal Biometrika [ 12 ] was established as early as 1901 by several founding fathers of modern statistics like Francis Galton and Karl Pearson with the intention of promoting biometrics, that is, the application of statistics to the analysis of biological data. Nowadays, it is difficult to dive into advanced concepts of evolutionary biology and population genetics without an understanding of mathematical modelling [ 13 ]. We also point out the seminal works of K.L. von Bertalanffy, P.A. Weiss, or M.D. Mesarovic in the application of the general systems theory to organisms dating back to the first part of 20th century (see Drack and Wolkenhauer [ 14 ] for detailed discussion).

In parallel to the search for the link between genetics and evolution theory, a new branch of science originating in physical chemistry was created by researchers interested in the dynamics of chemical reactions in living cells. This field became known as biochemistry, and already in 1913, biochemists made use of mathematical modelling to understand the mechanisms behind enzymatic catalysis. The Michaelis–Menten equation, taught in biochemistry courses throughout the world, is the first-ever mathematical model describing the dynamics of a biochemical reaction (see an updated translation of the original paper in Michaelis et al. 2011 [ 15 ]). We find in this field scientists like Jacques Monod and his team, who used mathematical modelling to understand sophisticated features of enzyme activity, such as allosteric regulation (see the seminal contribution in Monod et al. 1965 [ 16 ] and a 50-year retrospective analysis in Changeaux 2012 [ 17 ]). Fascinated by the elegance of these results, for several decades, a myriad of biochemists devoted their efforts to enzymology, that is, elucidation of the mechanisms of reaction, quantification, and modelling of the enzyme catalysis.

If mathematical modelling is necessary to immerse oneself in the connection between genes, phenotypes, and evolution as well as to understand the inner workings of catalytic proteins, how is it that certain modern biomedical researchers are reluctant to incorporate mathematical and theoretical approaches into their work? In the 1970s and 1980s, new experimental techniques were invented allowing the targeted mutation of selected genes [ 18 ]. These techniques became so fundamental in the hunt for the link between genes, proteins, and cellular functions that they led to the inception of modern molecular biology. An experimental approach based on targeted mutation followed by assessment of their effects has dominated the field since then. It is an approach that relies on advanced experimental skills, trial and error, intuition, and small-scale studies, and perhaps, that was what taught several generations of biomedical researchers that mathematics and formalized systems were somehow unnecessary in biology.

However, nowadays, two developments are turning this trend. On the one hand, newly discovered techniques for producing quantitative high-throughput data on whole classes of biomolecules (the omics revolution) required statistical methodologies for processing and analyzing these massive amounts of data. Additionally, mathematical and computational methods are indispensable in finding insights and connections between genes in these data [ 19 ]. This is the rationale behind medical genomics, the field that scans quantitative high-throughput data to find the genetic causation of diseases. On the other hand, there is mounting evidence that proteins and genes in cells do not work in isolation but rather organize into tightly interconnected networks which are often disturbed in pathological conditions [ 20 ]. These networks contain feedback loops, feedforward loops, or network hubs and gene regulatory circuits that can induce nonlinear behavior like homeostasis, self-sustained oscillations, or biostability [ 21 ]. There features, rather than anecdotic, are intrinsic and necessary to many vital cellular processes, e.g., the cell cycle [ 22 ].

The situation in 2020 is peculiar. We are rapidly conjuring biomedicine that necessarily relies more and more on quantitative high-throughput data, advanced statistics, bioinformatics, and computational modelling [ 23 ]. However, a significant fraction of its practitioners have insufficient mathematical and computational skills, probably ones worse than the generation that was initiated into the quantification of enzyme catalysis through kinetic equations and has recently retired. In line with this and to substantiate further discussion, we will now introduce the basics of mathematical modelling of biochemical networks.

4. A Primer on Mechanistic Modelling of Biochemical Systems

There are mathematical and computational models that encode a mechanistic description of the natural system in which equations and variables account for the interactions between the key biological components of the system, for example, signaling proteins and transcription factors. Among them, there is a family of models that are conceived to integrate the topology of biochemical networks and the kinetics of their molecular interactions with their ability to control cell phenotypes. These models are inspired by the models built in chemistry to understand and predict the kinetics of chemical reactions. These mathematical and computational models are formulated, characterized, and utilized following a well-established procedure common to several branches of physics, chemistry, and engineering, which we name here as the modelling workflow . In a nutshell, the workflow includes the following sequential operations ( Figure 3 ):

An external file that holds a picture, illustration, etc.
Object name is ijms-22-00547-g003.jpg

The modelling workflow in biomedicine: during model derivation, the biological knowledge and hypotheses about the studied system are encoded in a mathematical model. In model calibration, quantitative experimental data are added to characterize the mathematical model and to give values to the model parameters. In model validation, the ability of the model to make predictions is assessed by judging the agreement between new quantitative data and equivalent simulations of the calibrated model. In model analysis, a validated model is used to investigate the system using computer simulations or other tools like stability analysis.

Model derivation: Biomedical information from scientific literature is surveyed to select relevant biomolecules and interactions for the investigated hypothesis. With this information, a graphical depiction of the network of interacting molecules or cells is sketched. Under some formal or heuristic rules, a mathematical model is derived from the network graph. The mathematical model consists of mathematical equations (i.e., ordinary, partial differential, or integrodifferential equations) or other computational entities (i.e., Boolean-logic networks or Petri nets).

Model calibration: To ensure that the model mimics the behavior of the natural system in a given biological scenario, one has to attribute values to free parameters in the model. In some cases, it is possible to discern these from published quantitative data. More often, however, one has to design and perform biological experiments that produce adequate quantitative data. Later, the mathematical model and the quantitative data are integrated using a computational process which assigns optimal values to the model parameters while minimizing the mismatch between experimental observations and corresponding model simulations.

Model validation: The ability of the model to predict the system’s behavior is judged based on the alignment between quantitative data from a different experiment not used for calibration and the corresponding simulations of the calibrated model. A mismatch between data and simulation leads to reformulation of the hypothesis or the model’s structure, which is reflected in a modification of its mathematical equations and a re-iteration of the entire procedure.

Model analysis: A validated model can be used to design and perform predictive simulations, that is, simulations of the system’s behavior under new biological scenarios. This type of simulations has been successfully deployed to detect potential drug targets or to identify biomarkers for diagnosis in cancer and other multifactorial diseases (see Table S1 for selected examples). Furthermore, tools like stability analysis and bifurcation analysis can uncover nonlinear properties of the investigated network, delineating regions in the system’s phase space with distinctive stability or critical values of the model parameters provoking qualitative changes in the system’s behavior. Despite all the power that mathematical models bring to the table, however, it goes without saying that any prediction will require further experimental validation with lab bench models.

5. Frequent Unfounded Criticisms to Mathematical Modelling in Biomedicine

Now, we will list and discuss misconceptions about mathematical modelling in biomedicine which one can hear rather often when talking to experimental researchers in seminars and conferences (see also Table 1 for a summary).

Top misunderstandings on modelling and how to fight them.

5.1. Mathematical Models Cannot Reproduce the Complexity of Biology

When applied to biology in contrast to other natural sciences like physics or chemistry, this statement is a rather elaborate instance of magical thinking. Cells and physicochemical systems are governed by the same thermodynamic laws as any other natural system, laws that have been formulated in mathematical terms. The complexity of stellar systems’ dynamics is at least comparable with that of physiological systems, yet mathematical modelling is the standard tool to postulate hypotheses, to design experiments, and to formulate theories in astrophysics. The dynamics of the Earth’s atmosphere and climate are governed by the same laws of chemical kinetics and reaction-diffusion that apply to biochemical reactions. However, sophisticated mathematical modelling is behind the daily weather forecast or the recommendations of the Intergovernmental Panel on Climate Change. Besides nineteenth-century holistic thinking, there is not a single solid argument to support the notion that mathematical models cannot reproduce the complexity of biochemical and physiological systems.

5.2. Your Model Is Not Physiological. The Real System Is More Complex Than Your Mathematical Model

It may look like a softer version of the previous statement, but here, the emphasis is different. The idea is that modelling may be a valid option, but the current model does not contain sufficient detail to meaningfully represent the physiological context. When inspected more closely, this statement does not actually criticize mathematical modelling in particular but rather the fundamental act of utilizing any sort of model in biology. As we said above, any model, whether mathematical or experimental, is an abstraction that, rather than contemplating every detail of the system, includes the elements, interactions, and processes necessary to investigate the natural system under the hypothesis in question. This implies an intentional attempt at simplification by the researcher that is a shared feature of mathematical models and cell lines, organoids, or mouse models. This criticism can be countered by stating that the model must be as complex as necessary to capture the hypothesis, a notion equally valid for mathematical and lab bench models.

5.3. You Should Employ Data in Your Mathematical Model

A mathematical model in the sense discussed here always relies on quantitative data. As indicated above, model calibration is only possible with experimental data. Thus, a well-formulated mathematical model is based on quantitative data, which makes the above objection moot. Moreover, the interplay between model and quantitative data can come in different flavors. While usually one both calibrates and validates the model with data, one can also forego calibration in favor of analytical tools like stability analysis and can derive qualitative predictions about the system’s dynamics which are accessible to further experimental validation. Bar-or et al. 2000 [ 24 ] is a classic example of the ability of mathematical models to make qualitative predictions about the regulation of gene circuits from data. In the paper, the authors collected and synthesized all the information available at the time about the interplay between the TF p53 and its transcriptional target and repressor Mdm2 to hypothesize a) that they form a negative feedback loop gene circuit and b) that, under DNA damage, the system displayed oscillations in the expression of its components. They derived a qualitative mathematical model based on these hypotheses and found out that the model simulations predicted actual oscillations in p53 and Mdm2 levels in some experimental scenarios. They further validated this model-based prediction utilizing in vitro experiments. Interestingly, this “design principle” associated to oscillating gene circuits with TFs and their targets and repressors has been found in other master regulator TFs also by integrating quantitative data and mathematical modelling (see the case of NFkB in inflammation in Nelson et al. 2004 [ 25 ]).

5.4. Your Predictions Are Not Experimentally Validated

As explained above, once you perform an Einstein-grade version of the scientific method, it is not mandatory that a single paper conveys all the steps of the method. One team of researchers can formulate and investigate a hypothesis with mathematical modelling and simulations, and another team can follow up with validation experiments once technology and effort allow it (see the case of the neuronal action potential above). This is not to say that modelers are generally exempt from the need to validate, though. They should attempt to engage with experimental collaborators to see their own work come to fruition. An important aspect here is to provide ways of facilitating communication between modelers and experimentalists. In this sense, scientific papers on modelling and simulation should be written in a manner that allows the design of experiments to validate their hypotheses and predictions. Ultimately, this requires using common scientific language understandable for both mathematical modelers and experimental researchers.

5.5. I Do Not See the Clinical Relevance of Your Predictions

Very little of the scientific research in biomedicine delivers immediate clinical relevance, be it results obtained via mathematical modelling or through experimentation. However, at the same time, all basic research has an unpredictable long-term potential for enhancing clinical practice. To illustrate its potential, let us analyze a series of results obtained in the context of miRNA regulation and cancer. MicroRNAs exert posttranscriptional repression of selected gene targets [ 26 ] and play a pivotal role in some cell phenotypes subverted in cancer [ 27 ]. Lai et al. [ 28 ] utilized mathematical modelling to investigate the overarching hypothesis that different miRNAs can cooperate in the repression of some of their targets, a prediction that they experimentally validated for miR-572′s and miR-93′s joint repression of CDKNA1, a key cell cycle protein deregulated in cancer. In a continuation of this work, Schmitz et al. [ 29 ] used hybridization and molecular dynamics simulations of the binding of two cooperative miRNAs on their target mRNA to illustrate the general biophysical feasibility of this mechanism and to elucidate how it works at the molecular level. Moreover, they performed a human genome-wide exploration to systematically look for this type of joint miRNA regulation. Based on these results and further computational simulations, they hypothesized that miRNA cooperativity and its modelling can predict drug targets in cancer. They tested the hypothesis in a case study on cooperativity between miR-205-5p and miR-342-3p and its capacity to repress E2F1-mediated chemoresistance in cancer [ 30 ]. Their model-based prediction was confirmed experimentally. Finally, Lai et al. [ 31 ] extended this approach to the whole genome and systematically identified pairs of miRNAs that cooperatively target upregulated genes in metastatic melanoma. In summary, this series of interconnected papers illustrates how mathematical modelling can lead the way from hypothesis formulation and basic research to identifying potential clinical applications.

6. Rules to Build Mathematical Models That Can Be Understood by Experimentalists

To conclude, we elaborate on a few recommendations for biomedical modelers for when they conceive and implement their mathematical models, which hopefully will help bridge the cognitive chasm between them and experimentalists.

6.1. Know Your Problem

A good modeler should become an expert in the biomedical system that they plan to model. This is the best warranty that the structure and the hypotheses behind the mathematical model make sense and are consistent with the current biomedical knowledge. In addition, becoming an expert on the topic will help in choosing the right model assumptions, data, and hypotheses to be tested. Complementary to this, the best models emerge from constant interaction between competent biomodelers and experimental researchers. Remember, though, that collaboration is productive when communication is fluent, and this is only possible when a common language is spoken. In the present, this common language is the one that modelers need to learn when diving into the biology of the system they want to model. We refer again to Meyer-Hermann et al. 2012 [ 6 ] and their ability as modelers to acquire a deep understanding of B-cell biology and how they translated it into their mathematical model. However, we postulate that, in the long-term, biomedical researchers need in turn to rediscover the more precise language that math offers, which will also help them to quickly grasp advances in their own field of interest. There is an in-between methodology that could rekindle the growth of mathematical skills in experimentalists: network biology. In silico reconstruction, visualization, and modelling of intracellular biochemical networks provide a framework for connecting genes and molecules quantitatively to phenotypes and hence understanding the function and dynamics of cellular systems [ 32 ]. The network biology approach relies on mathematical concepts from graph theory, statistics, and mathematical modelling but is yet intuitive enough to allow a fluent discussion between wet- and dry-lab biologists. This has been, for example, advantageously used by yeast biologists to connect their experimentally detected biological interactions with their effect in cell phenotypes.

6.2. Select the Right Type of Mathematical Model, and Select It Early

The features of the mathematical model largely depend on the aim of the study; the scale and structural complexity of the investigated system; and the quantity, quality, and nature of the available experimental data. A model to investigate the nonlinearity associated with a feedback loop circuit has completely different requirements than a model of quantitative drug dosage in humans. This affects in particular the selection of the mathematical framework in which the model is derived and simulated. There is no single best modelling framework for every biomedical system or purpose, and therefore, the choice of model often relies on a trade-off between several requirements. We want to mention here (a) the computational demand and scalability, (b) the nature and necessary amount of calibration data, and (c) the way time and space are handled in the simulations. Sometimes, standard modelling frameworks are not suited for the problem in question and hybrid computational models of different types need to be considered (Chiam et al. 2006 illustrate how this type of hybridization can be done in the context of signaling pathways [ 33 ]; a discussion of this issue in the context of bacterial infection can be found in Cantone et al. 2017 [ 34 ]). When trying to facilitate the communication between modelers and experimentalists, one interesting approach is rule-based modelling. Compared to more math-heavy methods, rule-based modelling allows compact representations of reaction networks with a language-oriented structure; this makes them similar to semantic models and hence closer to the way of thinking of experimentalists (as a case study, see the epidermal growth factor receptor signaling network built in [ 35 ]).

6.3. Build on Preceding Efforts

To start the derivation of a mathematical model totally from scratch makes sense only when there is no alternative. A prudent modeler should reuse, extend, and adapt preexisting models when possible. In some cases, this will not be possible because the available models are based on different experimental conditions, formulated for a different biological scenario, or derived using an unsuitable modelling framework. In these cases, even if the model is found to be partially incompatible, its assessment will help judge the validity and portability of its assumptions and hypotheses in the context of one’s own modelling effort. If the problem lies in the modelling framework, it is sometimes worth translating the model into one’s chosen framework. In the ideal case, the model authors should have uploaded a fully annotated version of their model to a public repository (e.g., Biomodels [ 36 ]), which facilitates the work of incorporating the model into one’s own. Some tools even allow the semiautomatic translation of models from one formalism to another (see for example OdiFy [ 37 ]). As an example of this idea of building on preceding modelling efforts, in Csikász-Nagy 2009, one can find a comprehensive overview of the cascade of increasingly detailed mathematical models constructed since 1991 to understand the regulation of the cell cycle and how mathematical models are based on or have benefitted from the results obtained with previously developed models [ 38 ].

6.4. The Size Does Not Always Matter

There is nowadays a tendency to moon-shoot everything in biomedicine. This has translated also to biomodelling, and some researchers think that the quality of a mathematical model is measured in terms of the number of model variables as well as the required computational power and the complexity of the simulation algorithms. However, quality in modelling is primarily achieved through the biological precision of the assumptions encoded in the model equations. Thus, models can look simple and be small in terms of their number of equations but can actually possess the right features for the purpose of the specific modelling effort. This is somewhat similar to a BIC ballpoint pen. This is the simplest and cheapest ballpoint pen one can buy, but it actually displays a number of easy-to-overlook sophisticated features conceived to optimize its design in economic, ergonomic, and safety terms. One can also formulate BIC-like mathematical models, in which one gives priority to the description of the biological context and hypothesis and its planned utilization rather than to unnecessary size, complexity, or levels of detail ( Figure 4 ).

An external file that holds a picture, illustration, etc.
Object name is ijms-22-00547-g004.jpg

BIC pen-like mathematical models: prioritizing the purpose of the model instead of its detailedness . BIC pens look like the simplest and cheapest ballpoint pen one can buy, but their apparent simplicity conceals features conceived to optimize them in economic, ergonomic, and safety terms. In Santos and coworkers 2016 [ 3 ], a similar strategy was followed to build a mathematical model accounting for anticancer dendritic cell (DC) vaccination composed of only two ordinary differential equations, far simpler than other published models [ 41 ]. ( A ) Simplicity in design: BIC pens have a characteristically simple hexagonal structure; this apparently naïve choice significantly reduces the material consumption of the pen and minimizes the required space for storage. An important aspect to consider in DC vaccine modelling is the bioavailability of the cells after their injection. There are much elaborated models describing this process [ 42 ], but for our purpose, it was sufficient to model DC bioavailability with a cyclic piecewise linear function that mimics the known overall behavior of injected DCs. ( B ) Mathematics behind design principles: compared to standard circular pens, BIC pens hardly roll on the surface of a table. This feature was explicitly desired when drafting their design. In Santos et al., we wanted a simple enough model that was still able to mimic the interaction between the tumor and both innate and adaptive immunity; to this end, the model contained two nonlinear kinetic rates in a single equation, which are still able to mimic the basics of the interplay between the tumor and the two branches of immunity. ( C ) Ability to solve problems: in the end, simplicity has to be reconciled with effectiveness. The design of a BIC pen, for example, integrates more characteristics like minimizing the risk of suffocation when swallowing the cap. The predictions made in Santos et al. (2016) in terms of which phenotypic features sensitize the tumor to the therapy were aligned with patient data from clinical trials; furthermore, the model predicted alternative phenotypes that promote therapy resistance. The figures about DC vaccine modelling are adapted from Santos et al. 2016 under the conditions of an open access publication (CC BY 4.0). The figures about the BIC pen were inspired by the content of the webpage www.bicworld.com . Fight by doing: A route map to good mathematical modelling in biomedicine.

6.5. Set Your Results in Stone

One should prepare one’s own models in such a way that they can be understood and reused by other researchers, thereby closing the loop inherent to the scientific method. This is a frequently disregarded aspect in many fields in mathematical biomedicine, in which the number of modeler teams can be counted on two hands, and consequently, any model should have a high potential for reuse, adaptation, or integration. A model that is difficult to reproduce or understand by other modelers is guaranteed to collect virtual dust on its publication shelf. Thus, it is advantageous to implement equations and simulations in standard formats [ 39 ]. Further, whenever possible, the models and their supporting experimental data should be uploaded to repositories [ 36 ]. In closing, we emphasize the importance of the careful manual curation and annotation of mathematical models. In line with this, whenever possible, one should give preference to frameworks that allow for fast, simple, and standardized dissemination and model exchange [ 40 ].

7. The Best of Both Worlds—A Final Note on Mathematics, Models, Big Data, and Experimental Biology

The approach based on semantic and lab bench models has been very successful in past decades in describing many fundamental regulatory pathways in molecular biology. However, due to the massive amount of data produced nowadays, we are quickly reaching its limits. This is especially true (and dramatic) in the description of dynamic systems like heterogeneous cell populations, genomic regulation, or multifactorial diseases like cancer, for which this approach is clearly inadequate. There is no semantic solution to properly characterize multi-variable states, events, or diseases.

There is a clear understanding in the wet-lab biology community that advanced high throughput data analysis approaches are an urgent necessity for them, but in the opinion of many wet-lab biologists, data analysis and modelling are often conceived as a mere press of the button on computers that leads to publication-ready figures and plots within minutes. This has led to an entire industry producing “clickable” data analysis and modelling tools that make little demands on the knowledge of the operator. We think that this can lead to many problems, like reproducibility issues or the inability to critically judge the results produced by those tools, especially since most of them use proprietary algorithms. Additionally, the ability to generate an output from a purchased software or analysis suite might not automatically mean that this output is correct or sensible. Hence, an understanding of the underlying data and modelling theory is important and is often neglected in some wet-lab settings.

In the future, we as a community of biomedical researchers should strive to recognize that we need both sides of the coin, giving them equal weight in considerations like funding or time investment. Biology should orient itself towards the other two major natural science disciplines, physics and chemistry, and try to give their students a well-rounded education in mathematical understanding, data analysis, and modelling as well as in computer programming. Until that time, we need to make sure that we find ways to communicate between the two areas in a way that furthers productive collaboration.

Acknowledgments

Some of the misconceptions about modelling come from actual discussions with experimental and clinical researchers at the Department of Dermatology of the FAU Erlangen-Nürnberg, to whom we are grateful for their honest though at times scathing disagreement. Some parts of the text are based on a presentation given by J.V. at the Heidelberger Institut für Theoretische Studien (HITS), and others are from exchanges of ideas within the laboratory of Systems Tumor Immunology. J.V. also thanks Olaf Wolkenhauer (University of Rostock) for 15 years of continuous discussion on the foundations of systems biology.

Supplementary Materials

The following are available online at https://www.mdpi.com/1422-0067/22/2/547/s1 , Table S1: Real cases of mathematical modeling in biomedicine with additional bibliography and further reading.

Author Contributions

Initial draft: J.V.; Figures: J.V.; Final draft: J.V., M.E., C.L., S.N., M.N.; revision of the draft: J.V., X.L. All authors have read and agreed to the published version of the manuscript.

J.V.’s work in mathematical modelling applied to biomedicine has been generously funded over the years by the German Ministry of Education and Research (BMBF) through the initiatives FORSYS [CALSYS-FORSYS 0315264], e:Bio [e:Bio-miRSys 0316175A, e:Bio-MelEVIR 031L0073A], and e:Med [e:Med-CAPSyS 01ZX1304F, e:Med-MelAutim 01ZX1905A] and targeted initiatives in artificial intelligence [KI-VesD, 031L0244A] as well as by the German Research Foundation (DFG) [Ve642/1-1 in SPP1757] and the Bavarian Government [Gaminfection-UKER]. We also acknowledge support by Deutsche Forschungsgemeinschaft and Friedrich-Alexander-Universität Erlangen-Nürnberg within the funding program Open Access Publishing.

Conflicts of Interest

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Help | Advanced Search

Computer Science > Computation and Language

Title: large language models: a survey.

Abstract: Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \cite{kaplan2020scaling,hoffmann2022training}. The research area of LLMs, while very recent, is evolving rapidly in many different ways. In this paper, we review some of the most prominent LLMs, including three popular LLM families (GPT, LLaMA, PaLM), and discuss their characteristics, contributions and limitations. We also give an overview of techniques developed to build, and augment LLMs. We then survey popular datasets prepared for LLM training, fine-tuning, and evaluation, review widely used LLM evaluation metrics, and compare the performance of several popular LLMs on a set of representative benchmarks. Finally, we conclude the paper by discussing open challenges and future research directions.

Submission history

Access paper:.

  • Download PDF
  • HTML (experimental)
  • Other Formats

license icon

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

BibSonomy logo

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

  • Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

UP Board Class 10 Maths Model Paper 2024 with Answers_00.1

UP Board Class 10 Maths Model Paper 2024 with Answer

The UP Board Class 10 Maths Model Paper 2024 PDF is given in English and Hindi on this page. Students can also view the UP Board Class 10 Maths Model Paper 2024 solutions here.

UP Board Class 10 Maths Model Paper 2024 with Answers_20.1

Table of Contents

The UP Board Class 10 Maths paper is compulsory for all the Class 10 board students of the UP Board. So, the marks scored in the Mathematics paper is highly important for further education. Students must therefore prepare with full dedication for the Maths paper class 10 UP Board. To help students in acing the exam, below we are providing the UP Board 10 Maths model paper 2024 along with its solutions. The students can also get the actual board paper of maths class 10 along with its answer key from here.

UP Board Class 10 Maths Model Paper 2024

The UP Board Class 10 Maths model paper 2024 is the best source to prepare for the exam in the final days of the exam. As exam is just a few days ahead, students must solve the questions present in the model paper and match them with the standard answers provided in the article. The UP Board Class 10 model question paper for the mathematics subject is prepared by the experienced faculty. The model paper questions have high chances of appearing in the actual exam paper.

The model paper provided below is based on the latest exam pattern. The paper consists of questions that are trending in nature and are being asked in the board exams frequently. The paper provided below covers the entire syllabus of the UP Board class 10 Math subject.

Get UP Board Class 12 Revision Notes in Hindi

Class 10 Maths Model Paper 2024 UP Board: Overview

The UP Board Class 10 Math paper is being conducted on 27 February 2024. The examination is being held in the first shift from 8:30 AM to 11:45 AM. The real board question paper and its solutions will be updated on that day by our expert faculty. The Mathematics exam paper of the UP Board class 10 is held in offline mode for 3 hours and 15 minutes duration. The highlights of the Up Board Class 10 Maths model paper 2024 is given hereunder.

UP Board Maths Paper 2024 Class 10 Pattern

The UP Board Mathematics paper 2024 contains two different sections: A and B. The section A consists of objective type questions. The section A has 20 multiple choice questions with one mark each. The section B of the UP Board Class 10 Maths exam paper has descriptive type questions. The question length in this section varies from short answer type questions to long answer type questions. The first fifteen minutes are given to read the question paper thoroughly by the UPMSP.

UP Board Class 10 Maths Model Paper 2024 with Answers_30.1

Class 10 Maths Board Paper 2024 UP Marking Scheme

Students who are going to take the class 10 maths board exam must know the marking scheme in advance in order to score better marks in the exam. The marking scheme for the UP Board Class 10 Maths paper is shown below.

Check: UP Board Exam Centers List 2024

Maths Model Paper 2024 Class 10 UP Board Questions with Solutions

The model questions of the maths paper along with the solutions is provided below.

1. L.C.M. of any two numbers is 60 and H.C.F is 3. If one number is 12 then other number will be

(i) 20       (ii) 15 (iii) 180       (iv) 36

Ans: (ii) 15

2. The product of a non-zero rational number and an irrational number is 

(i) Always irrational number      (ii) Always rational number (iii) Rational or Irrational number       (iv) One

Ans: (i) Always irrational number

3. Solutions of linear equations x + 2y – 5 = 0 and 4x + 8y – 20 = 0 will be 

(i) Unique Solution       (ii) Infinitely many solutions (iii) No Solution       (iv) Two Solutions

Ans: (ii) Infinitely many solutions

4. The largest number by which dividing 70 and 125 gives remainders 5 and 8 respectively is 

(i) 13       (ii) 65 (iii) 875       (iv) 1750

Ans: (i) 13 

5. The sum of the first 5 multiples of 3 is 

(i) 45       (ii) 55 (iii) 65       (iv) 75

Ans: (i) 45 

6. Consider the following statements about the quadratic question 2x²– 4x + 3 = 0

(a) The discriminant of the given equation is less than 0. (b) The equation has no real roots. (c) The discriminant of the equation is zero. (d) The roots of given equation are real.

Select the correct option from the following. (i) a and b are correct       (ii) a and d are correct (iii) c and d are correct       (iv) only a is correct

Ans: (i) a and b are correct

7. AOBC is a rectangle with three vertices points A (0, 3), O (0, 0) and B (5, 0). Its diagonal is

(i) 5       (ii) 3 (iii) √34       (iv) 4

Ans: (iii) √34

8. Two triangles are similar

(i) If their corresponding angles are equal. (ii) Their corresponding sides are in the same ratio (iii) Both of the above. (iv) None of these

Ans: (iii) Both of the above.

Check: UP Board Date Sheet 2024

9. If in a right angled ∆ ABC ∠C = 90° AC = 3cm and BC = 4 cm then the measure of the median passing through point C is

(i) 2.5 cm       (ii) 3 cm (iii) 3.5 cm       (iv) 4 cm

Ans: (i) 2.5 cm

10. If (sinΘ − cosΘ ) = 1 then the value of (sin^4Θ + cos^4 Θ) is

(i) 1       (ii) 3/4 (iii) 1/2       (iv) 1/4

11. sin2A = 2sinA is true when A is equal to 

(i) 0°       (ii) 30° (iii) 45°       (iv) 60°

Ans: (i) 0°

12. The value of (secA + tanA)(1 – sinA) will be

(i) secA       (ii) sinA (iii) cosecA       (iv) cosA

Ans: (iv) cosA

13. A pot having long neck is a combination of

(i) a sphere and a cylinder (ii) a hemi-sphere and a cylinder (iii) two hemi-spheres (iv) a cylinder and a cone

Ans: (ii) a hemi-sphere and a cylinder

14. The mean of positive odd numbers from 1 to 10 will be

(i) 2      (ii) 3

(iii) 4      (iv) 5

Ans: (iv) 5

15. When a dice thrown the probability of getting an odd number less than 3 is

(i) 1/6      (ii) 1/3 (iii) 1/2     (iv) 0

Ans: (i) 1/6

UP Board Model Paper 2024 Class 10 Maths PDF

The model paper pdf for both the English and Hindi medium candidates is given below.

UP Board Class 10 Maths Mode Paper 2024 in Hindi

UP Board Class 10 Maths Mode Paper 2024 in English

Mathematics Board Paper 2024 Class 10 UP Board

The Mathematics board paper for the UP Board class 10 Maths exam will be updated on 27 February after 11:45 AM on this page. Students should keep checking the page to get the board paper as early as possible.

UP Board Class 10 Maths Model Paper 2024 with Answers_40.1

Math Board Paper 2024 Class 10 Solutions UP Board

The UP Board Class 10 maths solutions for the real exam paper is provided by our expert faculty for mathematics. The answers are always hundred percent accurate without any error. Students can use the solution key to compare their exam performance and predict their marks. The solutions for the board paper will be update below after the conduction of the board exam.

Sharing is caring!

How much time is given to solve the UP Board Class 10 Maths Paper 2024?

Students get 3 hours and 15 minutes to solve the UP Board Class 10 Maths paper 2024.

How many sections are there in the UP Board Class 10 Maths Paper?

The UP Board Class 10 Maths paper has two sections: Section A and Section B. The question type in both the sections are different.

How many multiple choice questions are there in the UP Board 10th Maths paper 2024?

The UP Board Maths 10th Paper 2024 contains 20 multiple choice questions.

  • Question Paper

Karnataka 2nd PUC Physics Model Question Paper 2024 with Solutions

Leave a comment

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

UP Board Class 10 Maths Model Paper 2024 with Answers_50.1

Trending Articles

  • CBSE Admit Card 2024
  • CBSE Date Sheet 2024 for Class 10 and 12
  • ICSE Date Sheet 2024
  • JEE Mains Syllabus 2024 [Reduced]
  • NEET Syllabus 2024 [Reduced]
  • NEET 2024 Exam Date
  • JEE Mains 2024 Exam Date
  • CUET 2024 Exam Date
  • SSC GD Admit Card 2024

CBSE Board Exam 2024

  • CBSE Class 10 Answer Key All Subjects 
  • CBSE Class 12 Answer key All Subjects 
  • CBSE Class 10 Syllabus 2024 
  • CBSE Class 12 Syllabus 2024 
  • CBSE Previous Year Papers  
  • CUET Syllabus
  • CUET Previous Year paper
  • CUET Participating College & Universities
  • JEE Main 2024  
  • JEE Main Syllabus 2024  
  • JEE Main Exam Analysis 2023  
  • NEET 2024  
  • NEET Syllabus 2024
  • NEET State wise Cut off
  • NEET Rank Predictor  
  • NEET OMR Sheet
  • NEET College Predictor

Recent Posts

Important exams, ncert solutions.

  • NCERT Class 12
  • NCERT Class 11
  • NCERT Class 10
  • NCERT Class 9

NCERT Books

School syllabus.

  • CBSE Class 12
  • CBSE Class 11
  • CBSE Class 10
  • CBSE Class 9
  • CBSE Result
  • JEE Mains 2024

Our Other Websites

  • Teachers Adda
  • Bankers Adda
  • Adda Malayalam
  • Adda Punjab
  • Current Affairs
  • Defence Adda
  • Adda Bengali
  • Engineers Adda
  • Adda Marathi
  • Adda School

UP Board Class 10 Maths Model Paper 2024 with Answers_230.1

Get all your queries solved in one single place. We at Adda247 school strive each day to provide you the best material across the online education industry. We consider your struggle as our motivation to work each day.

Download Adda247 App

UP Board Class 10 Maths Model Paper 2024 with Answers_240.1

Follow us on

youtube

  • Responsible Disclosure Program
  • Cancellation & Refunds
  • Terms & Conditions
  • Privacy Policy

IMAGES

  1. (PDF) Mathematical Modeling and Optimization of Complex Structures

    research paper on mathematical modeling pdf

  2. Mathematical Modeling: Methods, Applications and Research

    research paper on mathematical modeling pdf

  3. MATHEMATICAL MODELING A Comprehensive Introduction / mathematical

    research paper on mathematical modeling pdf

  4. (PDF) Mathematical Modeling: Proposal of a General Methodology and

    research paper on mathematical modeling pdf

  5. (PDF) What Is Mathematical Modeling

    research paper on mathematical modeling pdf

  6. (PDF) A Mathematical Modeling in Advertising Budget

    research paper on mathematical modeling pdf

VIDEO

  1. WRITING THE CHAPTER 3|| Research Methodology (Research Design and Method)

  2. Paper Four Mathematical Modeling M. Sc Sem. First (Mathematics) Dr. RMLAU

  3. Mathematical Foundation solved question paper May/June 2022 1st sem BCA

  4. Applied Mathematics Model Paper//

  5. Research Methodology and Project Report model papers || degree 6th sem research methodology

  6. Experimental Design Masterclass- Ep. 2 Hypothesis #dataanalytics #statistics #data

COMMENTS

  1. (PDF) Mathematical Modelling

    Gilbert Greefrath University of Münster Abstract and Figures The integration of applications and mathematical modelling into mathematics education plays an important role in many national...

  2. PDF Mathematical Modeling

    In practice, modeling entails a systematic approach to problem solving that brings the techniques and structures of mathematics to bear in an effort to describe, understand, and make predictions about a wide range of empirical phenomena.

  3. PDF Mathematical Modelling Approach in Mathematics Education

    mathematics. Mathematical modeling can be defined as using mathematics to explain and define the events in real life, to test ideas and to make estimations about real life events. Theoretical basis of the mathematical modelling approach, "model", "mathematical modelling" and "modeling activity" concepts are explained in this study and

  4. PDF Math Modeling in Educational Research: An Approach to ...

    This paper focuses on investigating the dominance of math modeling in educational research through the lens of transcendental realism. We identify some challenges to the approach, the problems of math modeling, and the relation between the problems and methodological fallacies in educational research.

  5. The use of mathematical modeling studies for evidence synthesis and

    A mathematical model is a "mathematical framework representing variables and their interrelationships to describe observed phenomena or predict future events." 9 We define a mathematical modeling study as a study that uses mathematical modeling to address specific research questions, for example, the impact of interventions in health care facili...

  6. Mathematical modeling for theory-oriented research in educational

    Mathematical modeling describes how events, concepts, and systems of interest behave in the world using mathematical concepts. This research approach can be applied to theory construction and testing by using empirical data to evaluate whether the specific theory can explain the empirical data or whether the theory fits the data available.

  7. Mathematical Modeling: Models, Analysis and Applications, 2nd Edition

    Chapter 1 "About Mathematical Modeling" introduces the main concepts of mathematical description of reality and briefs on the history of mathematical thought starting from ancient civilizations and going through the medieval world to the modern developments.

  8. PDF The Effects of Mathematical Modelling on Students' Achievement ...

    applying mathematics and building mathematical models for areas and purposes that are basically extra-mathematical" (Nish et al., 2007, p.5). Developing such competencies requires putting explicitly MM activities on the agenda of teaching and learning of mathematics. Research (Nish

  9. Ten simple rules for tackling your first mathematical models: A guide

    Citation: Bodner K, Brimacombe C, Chenery ES, Greiner A, McLeod AM, Penk SR, et al. (2021) Ten simple rules for tackling your first mathematical models: A guide for graduate students by graduate students. PLoS Comput Biol 17 (1): e1008539. https://doi.org/10.1371/journal.pcbi.1008539 Editor: Scott Markel, Dassault Systemes BIOVIA, UNITED STATES

  10. Mathematical modelling for health systems research: a systematic review

    Published: 19 November 2019 Mathematical modelling for health systems research: a systematic review of system dynamics and agent-based models Rachel Cassidy, Neha S. Singh, Pierre-Raphaël Schiratti, Agnes Semwanga, Peter Binyaruka, Nkenda Sachingongu, Chitalu Miriam Chama-Chiliba, Zaid Chalabi, Josephine Borghi & Karl Blanchet

  11. Mathematical Modelling in Biomedicine: A Primer for the Curious and the

    1. Science: A World of Systems (and Models) A good portion of science relies on the cascades of models that represent our reality at different spatial and temporal scales. A model is a simplified and often abstract representation of a complex natural system.

  12. [PDF] Mathematical Modelling Approach in Mathematics Education

    Theoretical basis of the mathematical modelling approach, "model", " model", "mathematical modelling" and "modeling activity" concepts are explained in this study and examples of these concepts are given. The topic of models and modeling has come to be important for science and mathematics education in recent years. The topic of "Modeling" topic is especially important for examinations such as ...

  13. PDF Modeling Mathematical Notation Semantics in Academic Papers

    In some cases, sen-tences have very long sequences of mathematical notation, as seen in this example: "Let the data source be $ Y \text { \in } \lbrace Y _ 1 , \dots , Y _ m \rbrace $.". In this case, the model needs to pre-dict a total of 18 notation tokens between $ signs, which is quite difficult.

  14. PDF Mathematical Modeling: A way of life

    The objective of the paper is to serve the needs of the mathematical modeling community by acting as the focus for the communication of research results, reviews of progress and discussions of issues of emerging interest to those in this rapidly expanding field. 2. Methodology:

  15. [PDF] Research on Integrating Mathematical Modeling Thinking into Large

    : This study explores how mathematical modeling thinking can be integrated into large, medium, and small school teaching to enhance students' mathematical abilities and interdisciplinary thinking. Using various research methods, including literature review, surveys, teaching experiments, statistical analysis, and expert interviews, we aim to establish a localized model for integrating ...

  16. PDF Role of Statistics in Mathematical Modeling

    Received: 29/Dec/2018, Accepted: 21/Jan/2019, Online: 28/Feb/2019. Abstract— The use of numerical model dependent on measurements is broadly utilized in numerous fields, particularly in assessment subject. In this paper have three sections Mathematical Modeling, Statistics and other is job of insights in Mathematical Modeling.

  17. PDF Mathematical modelling research in Turkey: A content analysis study

    and other studies about mathematical modeling. The main purpose of research with this information is to examine the scientific studies carried out in Turkey in the field of mathematical modeling between 2004 and 2015 and to determine their trends. papersIn this respect, the answers to the following sub problems were searched:

  18. PDF Gemma:OpenModelsBasedonGemini ResearchandTechnology

    Gemma:OpenModelsBasedonGeminiResearchandTechnology tably, the 7B model uses multi-head attention whilethe2Bcheckpointsusemulti-queryatten-tion(with _ _ℎ = 1 ...

  19. Analytical modelling with experimental validation of electromagnetic

    It is found that the analytical model proposed in this research matches the simulation models, as the results are consistent with minor errors. The errors occur mainly due to the fact that the wire diameter is not considered in the analytical model and the analytical model is simplified using concentric coils. 4.2 Power transfer simulation

  20. [2402.05929] An Interactive Agent Foundation Model

    Download PDF Abstract: The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide range of applications. We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and ...

  21. [2402.06196] Large Language Models: A Survey

    Large Language Models (LLMs) have drawn a lot of attention due to their strong performance on a wide range of natural language tasks, since the release of ChatGPT in November 2022. LLMs' ability of general-purpose language understanding and generation is acquired by training billions of model's parameters on massive amounts of text data, as predicted by scaling laws \\cite{kaplan2020scaling ...

  22. UP Board Class 12 Maths Model Question Paper 2024 PDF Download

    UPMSP Math Model Paper 2024 Class 12 Questions. Some of the important model questions for the UP Maths exam 2024 is shown below. Make sure to go through each questions. Check: UP Board Exam Date 2024. UP Board Model Paper 2024 Class 12 PDF Download Maths. The model paper pdf for the UP Board class 12 Mathematics exam paper 2024 is given below.

  23. UP Board Class 10 Maths Model Paper 2024 with Answers

    The UP Board Mathematics paper 2024 contains two different sections: A and B. The section A consists of objective type questions. The section A has 20 multiple choice questions with one mark each. ... The model paper pdf for both the English and Hindi medium candidates is given below. UP Board Class 10 Maths Mode Paper 2024 in Hindi.