literature review statistical analysis

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
PMID: 30089228
DOI: 10.1146/annurev-psych-010418-102803

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

PubMed Disclaimer

LinkOut - more resources

Full text sources.

Ingenta plc
Ovid Technologies, Inc.

Other Literature Sources

scite Smart Citations

Miscellaneous

NCI CPTAC Assay Portal
Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Research Methods

Getting Started
Literature Review Research
Research Design
Research Design By Discipline
SAGE Research Methods
Teaching with SAGE Research Methods

Literature Review

What is a Literature Review?
What is NOT a Literature Review?
Purposes of a Literature Review
Types of Literature Reviews
Literature Reviews vs. Systematic Reviews
Systematic vs. Meta-Analysis

Literature Review is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

Summarizes and analyzes previous research relevant to a topic
Includes scholarly books and articles published in academic journals
Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

Help gather ideas or information
Keep up to date in current trends and findings
Help develop new questions

A literature review is important because it:

Explains the background of research on a topic.
Demonstrates why a topic is significant to a subject area.
Helps focus your own research questions or problems
Discovers relationships between research studies/ideas.
Suggests unexplored ideas or populations
Identifies major themes, concepts, and researchers on a topic.
Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University

Keep in mind the following, a literature review is NOT:

Not an essay

Not an annotated bibliography in which you summarize each article that you have reviewed. A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper where you select resources to support one side of an issue versus another. A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

provides thorough knowledge of previous studies; introduces seminal works.
helps focus one’s own research topic.
identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
helps the researcher avoid repetition of earlier research.
suggests unexplored populations.
determines whether past studies agree or disagree; identifies controversy in the literature.
tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015), Literature reviews vs systematic reviews. Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Diagram for "What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters"

Systematic review or meta-analysis?

A systematic review answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A meta-analysis is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

clearly stated objectives with pre-defined eligibility criteria for studies
explicit, reproducible methodology
a systematic search that attempts to identify all studies
assessment of the validity of the findings of the included studies (e.g. risk of bias)
systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis.

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review. More information on meta-analyses can be found in Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies. It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings. Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted. In that case, an integrative review is an appropriate strategy.

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

<< Previous: Getting Started
Next: Research Design >>
Last Updated: Jul 15, 2024 10:34 AM
URL: https://guides.lib.udel.edu/researchmethods

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

5 5. Writing your literature review

Chapter outline.

Reading results (16 minute read time)
Synthesizing information (16 minute read time)
Writing a literature review (18 minute read time)

Content warning: examples in this chapter contain references to domestic violence and details on types of abuse, drug use, poverty, mental health, sexual harassment and details on harassing behaviors, children’s mental health, LGBTQ+ oppression and suicide, obesity, anti-poverty stigma, and psychotic disorders.

5.1 Reading the results section

Learning objectives.

Learners will be able to…

Describe how statistical significance and confidence intervals demonstrate which results are most important
Differentiate between qualitative and quantitative results in an empirical journal article

If you recall from Section 3.1, empirical journal articles are those that report the results of quantitative or qualitative data analyzed by the author. They follow a set structure–introduction, methods, results, discussion/conclusions. This section is about reading the most challenging section: results.

Read beyond the abstract

At this point, I have read hundreds of literature reviews written by students. One of the challenges I have noted is that students will report the results as summarized in the abstract, rather than the detailed findings laid out in the results section of the article. This poses a problem when you are writing a literature review because you need to provide specific and clear facts that support your reading of the literature. The abstract may say something like: “we found that poverty is associated with mental health status.” For your literature review, you want the details, not the summary. In the results section of the article, you may find a sentence that states: “children living in households experiencing poverty are three times more likely to have a mental health diagnosis.” This more detailed information provides a stronger basis on which to build a literature review.

Using the summarized results in an abstract is an understandable mistake to make. The results section often contains figures and tables that may be challenging to understand. Often, without having completed more advanced coursework on statistical or qualitative analysis, some of the terminology, symbols, or diagrams may be difficult to comprehend. This section is all about how to read and interpret the results of an empirical (quantitative or qualitative) journal article. Our discussion here will be basic, and in parts three and four of the textbook, you will learn more about how to interpret results from statistical tests and qualitative data analysis.

Remember, this section only addresses empirical articles. Non-empirical articles (e.g., theoretical articles, literature reviews) don’t have results. They cite the analysis of raw data completed by other authors, not the person writing the journal article who is merely summarizing others’ work.

Quantitative results

Quantitative articles often contain tables, and scanning them is a good way to begin reading an article. A table usually provides a quick, condensed summary of the report’s key findings. Tables are a concise way to report large amounts of data. Some tables present descriptive information about a researcher’s sample (often the first in a results section). These tables will likely contain frequencies (N) and percentages (%). For example, if gender happened to be an important variable for the researcher’s analysis, a descriptive table would show how many and what percent of all study participants are of a particular gender. Frequencies or “how many” will probably be listed as N, while the percent symbol (%) might be used to indicate percentages.

In a table presenting a causal relationship, two sets of variables are represented. The independent variable , or cause, and the dependent variable , the effect. We’ll go into more detail on variables in Chapter 6. Independent variable attributes are typically presented in the table’s columns, while dependent variable attributes are presented in rows. This allows the reader to scan a table’s rows to see how values on the dependent variable change as the independent variable values change. Tables displaying results of quantitative analysis will also likely include some information about the strength and statistical significance of the relationships presented in the table. These details tell the reader how likely it is that the relationships presented will have occurred simply by chance.

Let’s look at a specific example: Table 5.1. It presents the causal relationship between gender and experiencing harassing behaviors at work. In this example, gender is the independent variable (the cause) and the harassing behaviors listed are the dependent variables (the effects). [1] Therefore, we place gender in the table’s columns and harassing behaviors in the table’s rows.

Reading across the table’s top row, we see that 2.9% of women in the sample reported experiencing subtle or obvious threats to their safety at work, while 4.7% of men in the sample reported the same. We can read across each of the rows of the table in this way. Reading across the bottom row, we see that 9.4% of women in the sample reported experiencing staring or invasion of their personal space at work while just 2.3% of men in the sample reported having the same experience. We’ll discuss p values later in this section.

Table 5.1 Percentage reporting harassing behaviors at work

Subtle or obvious threats to your safety	2.9%	4.7%	0.623
Being hit, pushed, or grabbed	2.2%	4.7%	0.480
Comments or behaviors that demean your gender	6.5%	2.3%	0.184
Comments or behaviors that demean your age	13.8%	9.3%	0.407
Staring or invasion of your personal space	9.4%	2.3%	0.039
Note: Sample size was 138 for women and 43 for men.

While you can certainly scan tables for key results, they are often difficult to understand without reading the text of the article. The article and table were meant to complement each other, and the text should provide information on how the authors interpret their findings. The table is not redundant with the text of the results section. Additionally, the first table in most results sections is a summary of the study’s sample, which provides more background information on the study than information about hypotheses and findings. It is also a good idea to look back at the methods section of the article as the data analysis plan the authors outline should walk you through the steps they took to analyze their data which will inform how they report them in the results section.

Statistical significance

These statistics represent what the researchers found in their sample, and they are using their sample to draw conclusions about the true population of all employees in the real world. The purpose of statistical analysis is usually to generalize from a the small number of people in a study’s sample to a larger population of people.

Generalizing is key to understanding statistical significance . According to Cassidy and colleagues, (2019) [2] 89% of research methods textbooks in psychology define statistical significance incorrectly. This includes an early draft of this textbook which defined statistical significance as “the likelihood that the relationships we observe could be caused by something other than chance.” If you have previously had a research methods class, this might sound familiar to you. It certainly did to me!

But statistical significance is less about “random chance” than more about the null hypothesis . Basically, at the beginning of a study a researcher develops a hypothesis about what they expect to find, usually that there is a statistical relationship between two or more variables . The null hypothesis is the opposite. The null hypothesis is that there is no relationship between the variables in a research study. Researchers then can hopefully reject the null hypothesis because they find a relationship between the variables.

For example, in Table 5.1 researchers were examining whether gender impacts harassment. Of course, researchers assumed that women were more likely to experience harassment than men. The null hypothesis, then, would be that gender has no impact on harassment. Once we conduct the study, our results will hopefully lead us to reject the null hypothesis because we find that gender impacts harassment. We would then generalize from our study’s sample to the larger population of people in the workplace.

Statistical significance is calculated using a p-value which is obtained by comparing the statistical results with a hypothetical set of results if the researchers re-ran their study a large number of times. Keeping with our example, imagine we re-ran our study with different men and women from different workplaces hundreds and hundred of times and we assume that the null hypothesis is true that gender has no impact on harassment. If results like ours come up pretty often when the null hypothesis is true, our results probably don’t mean much. “The smaller the p-value, the greater the statistical incompatibility with the null hypothesis” (Wasserstein & Lazar, 2016, p. 131) [3] Generally, researchers in the social science have used 0.05 as the value at which a result is significant (p is less than 0.05) or not significant (p is greater than 0.05). The p-value 0.05 refers to if 5% of those hypothetical results from re-running our study show the same or more extreme relationships when the null hypothesis is true. Researchers, however, may choose a stricter standard such as 0.01 in which only 1% of those hypothetical results are more extreme or a more lenient standard like 0.1 in which 10% of those hypothetical results are more extreme than what was found in the study.

Let’s look back at Table 5.1. Which one of the relationships between gender and harassing behaviors is statistically significant? It’s the last one in the table, “staring or invasion of personal space,” whose p-value is 0.039 (under the p<0.05 standard to establish statistical significance). Again, this indicates that if we re-ran our study over and over again and gender did not impact staring/invasion of space (i.e., the null hypothesis was true), only 3.9% of the time would we find similar or more extreme differences between men and women than what we observed in our study. Thus, we conclude that for staring or invasion of space only , there is a statistically significant relationship.

For contrast, let’s look at “being pushed, hit, or grabbed” and run through the same analysis to see if it is statistically significant. If we re-ran our study over and over again and the null hypothesis was true, 48% of the time (p=.48) we would find similar or more extreme differences between men and women. That means these results are not statistically significant.

This discussion should also highlight a point we discussed previously: that it is important to read the full results section, rather than simply relying on the summary in the abstract. If the abstract stated that most tests revealed no statistically significant relationships between gender and harassment, you would have missed the detail on which behaviors were and were not associated with gender. Read the full results section! And don’t be afraid to ask for help from a professor in understanding what you are reading, as results sections are often not written to be easily understood.

Statistical significance and p-values have been critiqued recently for a number of reasons, including that they are misused and misinterpreted (Wasserstein & Lazar, 2016) [4] , that researchers deliberately manipulate their analyses to have significant results (Head et al., 2015) [5] , and factor into the difficulty scientists have today in reproducing many of the results of previous social science studies (Peng, 2015). [6] For this reason, we share these principles, adapted from those put forth by the American Statistical Association, [7] for understanding and using p-values in social science:

P-values provide evidence against a null hypothesis.
P-values do not indicate whether the results were produced by random chance alone or if the researcher’s hypothesis is true, though both are common misconceptions.
Statistical significance can be detected in minuscule differences that have very little effect on the real world.
Nuance is needed to interpret scientific findings, as a conclusion does not become true or false when the p-value passes from p=0.051 to p=0.049.
Real-world decision-making must use more than reported p-values, which are subject to cherry-picking and selective reporting.
Greater confidence can be placed in studies that pre-register their hypotheses and share their data and methods openly with the public.
“By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. For example, a p-value near 0.05 taken by itself offers only weak evidence against the null hypothesis. Likewise, a relatively large p-value does not imply evidence in favor of the null hypothesis; many other hypotheses may be equally or more consistent with the observed data” (Wasserstein & Lazar, 2016, p. 132).

Confidence intervals

Because of the limitations of p-values, scientists can use other methods to determine whether their models of the world are true. One common approach is to use a confidence interval , or a range of values in which the true value is likely to be found. Confidence intervals are helpful because, as principal #5 above points out, p-values do not measure the size of an effect (Greenland et al., 2016). [8] Remember, something that has very little impact on the world can be statistically significant, and the values in a confidence interval would be helpful. In our example from Table 5.1, imagine our analysis produced a confidence interval that women are 1.2-3.4x more likely to experience “staring or invasion of personal space” than men. As with p-values, calculation for a confidence interval compares what was found in one study with a hypothetical set of results if we repeated the study over and over again. If we calculated 95% confidence intervals for all of the hypothetical set of hundreds and hundreds of studies, that would be our confidence interval.

Confidence intervals are pretty intuitive. As of this writing, my wife and are expecting our second child. The doctor told us our due date was December 11th. But the doctor also told us that December 11th was only their best estimate. They were actually 95% sure our baby might be born any time in the 30-day period between November 27th and December 25th. Confidence intervals are often listed with a percentage, like 90% or 95%, and a range of values, such as between November 27th and December 25th. You can read that as: “we are 95% sure your baby will be born between November 27th and December 25th because we’ve studied hundreds of thousands of fetuses and mothers, and we’re 95% sure your baby will be within these two dates.”

Notice that I’m hedging my bets here by using words like “best estimate.” When testing hypotheses, social scientists generally phrase their findings in a tentative way, talking about what results “indicate” or “support,” rather than making bold statements about what their results “prove.” Social scientists have humility because they understand the limitations of their knowledge. In a literature review, using a single study or fact to “prove” an argument right or wrong is often a signal to the person reading your literature review (usually your professor) that you may not have appreciated the limitations of that study or its place in the broader literature on the topic. Arguments should include multiple facts and ideas that span across studies.

You can learn more about creating tables, reading tables, and tests of statistical significance in a class focused exclusively on statistical analysis. For now, I hope this brief introduction to reading tables will improve your confidence in reading and understanding the results sections in quantitative empirical articles.

Qualitative results

Quantitative articles will contain a lot of numbers and the results of statistical tests demonstrating associations between those numbers. Qualitative articles, on the other hand, will consist mostly of quotations from participants. For most qualitative articles, the authors want to put their results in the words of their participants, as they are the experts. Articles that lack quotations make it difficult to assess whether the researcher interpreted the data in a trustworthy, unbiased manner. These types of articles may also indicate how often particular themes or ideas came up in the data, potentially reflective of how important they were to participants.

Authors often organize qualitative results by themes and subthemes. For example, see this snippet from the results section Bonanno and Veselak (2019) [9] discussion parents’ attitudes towards child mental health information sources.

Data analysis revealed four themes related to participants’ abilities to access mental health help and information for their children, and parents’ levels of trust in these sources. These themes are: others’ firsthand experiences family and friends with professional experience, protecting privacy, and uncertainty about schools as information sources. Trust emerged as an overarching and unifying concept for all of these themes. Others’ firsthand experiences. Several participants reported seeking information from other parents who had experienced mental health struggles similar to their own children. They often referenced friends or family members who had been or would be good sources of information due to their own personal experiences. The following quote from Adrienne demonstrates the importance of firsthand experience: [I would only feel comfortable sharing concerns or asking for advice] if I knew that they had been in the same situation. (Adrienne) Similarly, Michelle said:And I talked to a friend of mine who has kids who have IEPs in the district to see, kind of, how did she go about it. (Michelle) … Friends/family with professional experience . Several respondents referred to friends or family members who had professional experience with or knowledge of child mental health and suggested that these individuals would be good sources of information. For example, Hannah said: Well, what happened with me was I have an uncle who’s a psychiatrist. Sometimes if he’s up in (a city to the north), he’s retired, I can call him sometimes and get information. (Hannah) Michelle, who was in nursing school, echoed this sentiment:At this point, [if my child’s behavioral difficulties continued], I would probably call one of my [nursing] professors. That’s what I’ve done in the past when I’ve needed help with certain things…I have a professor who I would probably consider a friend who I would probably talk to first. She has a big adolescent practice. (Michelle) (p. 402-403)

The terms in bold above refer to the key themes (i.e., qualitative results) that were present in the data. Researchers will state the process by which they interpret each theme, providing a definition and usually some quotations from research participants. Researchers will also draw connections between themes, note consensus or conflict over themes, and situate the themes within the study context.

Qualitative results are specific to the time, place, and culture in which they arise, so you will have to use your best judgment to determine whether these results are relevant to your study. For example, students in my class at Radford University in Southwest Virginia may be studying rural populations. Would a study on group homes in a large urban city transfer well to group homes in a rural area?

Maybe. But even if you were using data from a qualitative study in another rural area, are all rural areas the same? How is the client population and sociocultural context in the article similar or different to the one in your study? Qualitative studies have tremendous depth, but that level of detail makes their results less applicable across situations.

Key Takeaways

The results section of empirical articles are often the most difficult to understand.
To understand a quantitative results section, look for results that were statistically significant and examine the confidence interval, if provided.
To understand a qualitative results section, look for definitions of themes or codes and use the quotations provided to understand the participants’ perspective.
Write down the results the authors identify as statistically significant in the results section.
How do the authors interpret their results in the discussion section?
Do the authors provide enough information in the introduction for you to understand their results?
Write down the key themes the authors identify and how they were defined by the participants.

5.2 Organizing information

Describe how to use summary tables to organize information from empirical articles
Describe how to use topical outlines to organize information from the literature reviews of articles you read
Create a concept map that visualizes the key concepts and relationships relevant to your working question
Use what you learn in the literature search to revise your working question

This chapter will introduce you to two tools scholars use to organize and synthesize (i.e., weave together) information from multiple sources. First, we will discuss how to build a summary table containing information from empirical articles that are highly relevant–from literature review, to methods and results–to your entire research proposal.

Second, we’ll discuss what to do with the other articles you’ve downloaded. For these articles, you should create a topical outline that organizes all relevant facts from the abstract, literature review, and conclusion along with the original, primary source from which the fact came. Our goal here is to help you utilize your reading time effectively.

Organizing empirical articles using a summary table

Your research proposal is an empirical project. You will collect raw data and analyze it to answer your question. Over the next few weeks, identify about 10 articles that are empirically similar to the study you want to conduct. If you plan on conducting surveys of practitioners, it’s a good idea for you to read in detail other articles that have used similar methods (sampling, measures, data analysis) and asked similar questions to your proposal. A summary table can help you organize these Top 10 articles: empirical articles that are highly relevant to your proposal and working question.

Using the annotations in Section 4.2 as a guide, create a spreadsheet or Word table with your annotation categories as columns and each source as new row. For example, I was searching for articles on using a specific educational technique in the literature. I wanted to know whether other researchers found positive results, how big their samples were, and whether they were conducted at a single school or across multiple schools. I looked through each empirical article on the topic and filled in a summary table. At the end, I could do an easy visual analysis and state that most studies revealed no significant results and that there were few multi-site studies. These arguments were then included in my literature review. These tables are similar to those you will find in a systematic review article.

A basic summary table is provided in Figure 5.1. A more detailed example is available from Elaine Gregersen’s blog , and you can download an Excel template from Raul Pacheco-Vega’s blog . Remember, although “the means of summarizing can vary, the key at this point is to make sure you understand what you’ve found and how it relates to your topic and research question” (Bennard et al., 2014, para. 10). [10] As you revisit and revise your working question over the next few weeks, think about those sources that are so relevant you need to understand every detail about them.

A good summary table will also ensure that when you cite these articles in your literature review, you are able to provide the necessary detail and context for readers to properly understand the results. For example, one of the common errors I see in student literature reviews is using a small, exploratory study to represent the truth about a larger population. You will also notice important differences in how variables are measured or how people are sampled, for instance, and these details are often the source of a good critical review of the literature.

A 3 by 3 table with purpose, methods, and results as columns and sources 1, 2, and 3 as rows

Using your folder of article PDFs from you’ve downloaded in previous exercises, identify which articles are likely to be most relevant to your proposed study. This may change as you revise your working question and study design over the next few weeks. Create a list of 10 articles that are highly relevant to the extent that you will need to remember key details from each section of the article.
Create a spreadsheet for your summary table and save it in your project folder on your hard drive. Using one of the templates linked in this chapter, fill in the columns of your spreadsheet. Enter the information from one of the articles you’ve read so far. As you finalize your research question over the next few weeks, fill in your summary table with the 5 most relevant empirical articles on your topic.

Organizing facts using a topical outline

If we’re only reading 10 articles in detail, what do we do with the others? Raul Pacheco-Vega recommends using the AIC approach : read the abstract, introduction, and conclusion (and the discussion section, in empirical articles). For non-empirical articles, it’s a little less clear but the first few pages and last few pages of an article usually contain the author’s reading of the relevant literature and their principal conclusions. You may also want to skim the first and last sentence of each paragraph and only reading paragraphs in which you are likely to find information relevant to your working question. Skimming like this gives you the general point of the article, though you should read in detail the most valuable resource of all–another author’s literature review.

It’s impossible to read all of the literature about your topic. You will read about 10 articles in detail, a few times more than that you will skim the abstract, introduction, and conclusion, but you will ultimately never read everything. Make the most out of the articles you do read by extracting as many facts as possible from each. You are starting your research project without a lot of knowledge of the topic you want to study, and by using the literature reviews provided in academic journal articles, you can gain a lot of knowledge about a topic in a short period of time. This way, by reading only a small number of articles, you are also reading their citations and synthesis of dozens of other articles as well.

As you read an article in detail, I suggest copying any facts you find relevant in a separate word processing document. Another idea is to copy anything you’ve annotated as background information in Section 4.2 into an outline. Copying and pasting from PDF to Word can be difficult because PDFs are image files, not documents. To make that easier, use the HTML version of the article, convert the PDF to Word in Adobe Acrobat or another PDF reader, or use the “paste special” command to paste the content into Word without formatting. If it’s an old PDF, you may have to simply type out the information you need. It can be a messy job, but having all of your facts in one place is very helpful when drafting your literature review.

You should copy and paste any fact or argument you consider important. Some good examples include definitions of concepts, statistics about the size of the social problem, and empirical evidence about the key variables in the research question, among countless others. It’s a good idea to consult with your professor and the course syllabus to understand what he or she is looking for when reading your literature review. Facts for your literature review are principally found in the introduction, results, and discussion section of an empirical article or at any point in a non-empirical article. Copy and paste into your notes anything you may want to use in your literature review.

Importantly, you must make sure you note the original source of that information. Nothing is worse than searching your articles for hours only to realize you forgot to note where your facts came from. If you found a statistic that the author used in the introduction, it almost certainly came from another source that the author cited in a footnote or internal citation. You will want to check the original source to make sure the author represented the information correctly. Moreover, you may want to read the original study to learn more about your topic and discover other sources relevant to your inquiry.

Assuming you have pulled all of the facts out of multiple articles, it’s time to start thinking about how these pieces of information relate to each other. Start grouping each fact into categories and subcategories as shown in Table 5.2. For example, a statistic stating that single adults who are homeless are more likely to be male may fit into a category of gender and homelessness. For each topic or subtopic you identify during your critical analysis of each paper, determine what those papers have in common. Likewise, determine which differ. If there are contradictory findings, you may be able to identify methodological or theoretical differences that could account for these contradictions. For example, one study may sample only high-income earners or those living in a rural area. Determine what general conclusions you can report about the topic or subtopic, based on all of the information you’ve found.

Create a separate document containing a topical outline that combines your facts from each source and organizes them by topic or category. As you include more facts and more sources in your topical outline, you will begin to see how each fact fits into a category and how categories are related to one another. Keep in mind that your category names may change over time, as may their definitions. This is a natural reflection of the learning you are doing.

Table 5.2 Topical outline

A complete topical outline is a long list of facts arranged by category. As you step back from the outline, you should assess which topic areas for which you have enough research support to allow you to draw strong conclusions. You should also assess which areas you need to do more research in before you can write a robust literature review. The topical outline should serve as a transitional document between the notes you write on each source and the literature review you submit to your professor. It is important to note that they contain plagiarized information that is copied and pasted directly from the primary sources. In this case, it is not problematic because these are just notes and are not meant to be turned in as your own ideas. For your final literature review, you must paraphrase these sources to avoid plagiarism. More importantly, you should keep your voice and ideas front-and-center in what you write as this is your analysis of the literature. Make strong claims and support them thoroughly using facts you found in the literature. We will pick up the task of writing your literature review in section 5.3.

In your folder full of article PDFs, look for the most relevant review articles. If you don’t have any, try to look for some. If there are none in your topic area, you can also use other non-empirical articles or empirical articles with long literature reviews (in the introduction and discussion sections).
Create a word processing document for your topical outline and save it in your project folder on your hard drive. Using a review article, start copying facts you identified as Background Information or Results into your topical outline. Try to organize each fact by topic or theme. Make sure to copy the internal citation for the original source of each fact. For articles that do not use internal citations, create one using the information in the footnotes and references. As you finalize your research question over the next few weeks, skim the literature reviews of the articles you download for key facts and copy them into your topical outline.

Putting the pieces together: Building a concept map

Developing a concept map or mind map around your topic can be helpful in figuring out how the facts fit together. We talked about concept mapping briefly in Chapter 2, when we were first thinking about your topic and sketching out what you already know about it. Concept mapping during the literature review stage of a research project builds on this foundation of knowledge and aims to improve the “description of the breadth and depth of literature in a domain of inquiry. It also facilitates identification of the number and nature of studies underpinning mapped relationships among concepts, thus laying the groundwork for systematic research reviews and meta-analyses” (Lesley, Floyd, & Oermann, 2002, p. 229). [11] Its purpose, like other question refinement methods, is to help you organize, prioritize, and integrate material into a workable research area – one that is interesting, answerable, feasible, objective, scholarly, original, and clear.

Think about the topics you created in your topic outline. How do they relate to one another? Within each topic, how do facts relate to one another? As you write down what you have, think about what you already know. What other related concepts do you not yet have information about? What relationships do you need to investigate further? Building a conceptual map should help you understand what you already know, what you need to learn next, and how you can organize a literature review.

This technique is illustrated in this Youtube Video . You may want to indicate which concepts and relationships you’ve already found in your review and which ones you think might be true but haven’t found evidence of yet. Once you get a sense of how your concepts are related and which relationships are important to you, it’s time to revise your working question.

Create a concept map using a pencil and paper. Identify the key ideas inside the literature, how they relate to one another, and the facts you know about them. Reflect on those areas you need to learn more about prior to writing your literature review. As you finalize your research question over the next few weeks, update your concept map and think about how you might organize it into a written literature review. Refer to the topics and headings you use in your topical outline and think about what literature you have that helps you understand each concept and relationship between them in your concept map.

Revising your working question

You should be revisiting your working question throughout the literature review process. As you continue to learn more about your topic, your question will become more specific and clearly worded. This is normal and there is no way to shorten this process. Keep revising your question in order to ensure it will contribute something new to the literature on your topic, is relevant to your target population, and is feasible for you to conduct as a student project.

For example, perhaps your initial idea or interest is how to prevent obesity. After an initial search of the relevant literature, you realize the topic of obesity is too broad to adequately cover in the time you have to do your project. You decide to narrow your focus to causes of childhood obesity. After reading some articles on childhood obesity, you further narrow your search to the influence of family risk factors on overweight children. A potential research question might then be, “What maternal factors are associated with toddler obesity in the United States?” You would then need to return to the literature to find more specific studies related to the variables in this question (e.g. maternal factors, toddler, obesity, toddler obesity).

Similarly, after an initial literature search for a broad topic such as school performance or grades, examples of a narrow research question might be:

“To what extent does parental involvement in children’s education relate to school performance over the course of the early grades?”
“Do parental involvement levels differ by family social, demographic, and contextual characteristics?”
“What forms of parent involvement are most highly correlated with children’s outcomes? What factors might influence the extent of parental involvement?” (Early Childhood Longitudinal Program, 2011). [12]

In either case, your literature search, working question, and understanding of the topic are constantly changing as your knowledge of the topic deepens. A literature review is an iterative process, one that stops, starts, and loops back on itself multiple times before completion. As research is a practice behavior of social workers, you should apply the same type of critical reflection to your inquiry as you would to your clinical or macro practice.

There are many ways to approach synthesizing literature. We’ve reviewed the following: summary tables, topical outlines, and concept maps. Other examples you may encounter include annotated bibliographies and synthesis matrices. As you are learning how to conduct research, find a method that works for you. Reviewing the literature is a core component of evidence-based practice in social work. See the resources below if you need some additional help:

Literature Reviews: Using a Matrix to Organize Research / Saint Mary’s University of Minnesota

Literature Review: Synthesizing Multiple Sources / Indiana University

Writing a Literature Review and Using a Synthesis Matrix / Florida International University

Sample Literature Reviews Grid / Complied by Lindsay Roberts

Literature review preparation: Creating a summary table . (Includes transcript) / Laura Killam

You won’t read every article all the way through. For most articles, reading the abstract, introduction, and conclusion are enough to determine its relevance.
For articles where everything seems relevant, use a summary table to keep track of details. These are particularly helpful with empirical articles.
For articles with sections relevant to your topic, copy any relevant information into a topical outline, along with the original source of that information.
Use a concept map to help you visualize the key concepts in your topic area and the relationships between them.
Revise your working question regularly. As you do, you will likely need to revise your search queries and include new articles.
Look back at the working question for your topic and consider any necessary revisions. It is important that questions become clearer and more specific over time. It is also common that your working question shift over time, sometimes drastically, as you explore new lines of inquiry in the literature. Return to your working question regularly and make sure it reflects the focus of your inquiry. You will continue to revise your working question until we formalize it into a research question at the end of Part 2 of this textbook.

5.3 Writing your literature review

Describe the components of a literature review
Begin to write your literature review
Identify the purpose of a problem statement
Apply the components of a formal argument to your topic
Use elements of formal writing style, including signposting and transitions
Recognize commons errors in literature reviews

Congratulations! By now, you should have discovered, retrieved, evaluated, synthesized, and organized the information you need for your literature review. It’s now time to turn that stack of articles, papers, and notes into a literature review–it’s time to start writing!

Writing about research is different than other types of writing. Research writing is not like a journal entry or opinion paper. The goal here is not to apply your research question to your life or growth as a practitioner. Research writing is about the provision and interpretation of facts. The tone should be objective and unbiased, and personal experiences and opinions are excluded. Particularly for students who are used to writing case notes, research writing can be a challenge. That’s why its important to normalize getting help! If your professor has not built in peer review, consider setting up a peer review group among your peers. You should also reach out to your academic advisor to see if there are writing services on your campus available to graduate students. No one should feel bad for needing help with something they haven’t done before, haven’t done in a while, or were never taught how to do.

If you’ve followed the steps in this chapter, you likely have an outline, summary table, and concept map from which you can begin the writing process. But what do you need to include in your literature review? We’ve mentioned it before, but to summarize, a literature review should:

Introduce the topic and define its key terms. Establish the importance of the topic. Provide an overview of the important literature related to the concepts found in the research question. Identify gaps or controversies in the literature. Point out consistent findings across studies. Synthesize that which is known about a topic, rather than just provide a summary of the articles you read. Discuss possible implications and directions for future research.

Do you have enough facts and sources to accomplish these tasks? It’s a good time to consult your outlines and notes on each article you plan to include in your literature review. You may also want to consult with your professor on what is expected of you. If there is something you are missing, you may want to jump back to Section 2.3 where we discussed how to search for literature. While you can always fill in material, there is the danger that you will start writing without really knowing what you are talking about or what you want to say. For example, if you don’t have a solid definition of your key concepts or a sense of how the literature has developed over time, it will be difficult to make coherent scholarly claims about your topic.

There is no magical point at which one is ready to write. As you consider whether you are ready, it may be useful to ask yourself these questions:

How will my literature review be organized?
What section headings will I be using?
How do the various studies relate to each other?
What contributions do they make to the field?
Where are the gaps in the research? What are the limitations of existing research?
And finally, but most importantly, how does my own research fit into what has already been done?

The problem statement

Scholarly works often begin with a problem statement, which serves two functions. First, it establishes why your topic is a social problem worth studying. Second, it pulls your reader into the literature review. Who would want to read about something unimportant?

A problem statement generally answers the following questions, though these are far from exhaustive:

Why is this an important problem to study?
How many people are affected by this problem?
How does this problem impact other social issues relevant to social work?
Why is your target population an important one to study?

A strong problem statement, like the rest of your literature review, should be filled with facts, theory, and arguments based on the extant literature. A research proposal differs significantly from other more reflective essays you’ve likely completed during your social work studies. If your topic were domestic violence in rural Appalachia, I’m sure you could come up with answers to the above questions without looking at a single source. However, the purpose of the literature review is not to test your intuition, personal experience, or empathy. Instead, research methods are about gaining specific and articulable knowledge to inform action. With a problem statement, you can take a “boring” topic like the color of rooms used in an inpatient psychiatric facility, transportation patterns in major cities, or the materials used to manufacture baby bottles, and help others see the topic as you see it—an important part of the social world that impacts social work practice.

The structure of a literature review

In general, the problem statement belongs at the beginning of the literature review. I usually advise students to spend no more than a paragraph or two for a problem statement. For the rest of your literature review, there is no set formula by which it needs to be organized. However, a literature review generally follows the format of any other essay—Introduction, Body, and Conclusion.

The introduction to the literature review contains a statement or statements about the overall topic. At a minimum, the introduction should define or identify the general topic, issue, or area of concern. You might consider presenting historical background, mentioning the results of a seminal study, and providing definitions of important terms. The introduction may also point to overall trends in what has been previously published on the topic or on conflicts in theory, methodology, evidence, conclusions, or gaps in research and scholarship. I also suggest putting in a few sentences that walk the reader through the rest of the literature review. Highlight your main arguments from the body of the literature review and preview your conclusion. An introduction should let the reader know what to expect from the rest of your review.

The body of your literature review is where you demonstrate your synthesis and analysis of the literature. Again, do not just summarize the literature. I would also caution against organizing your literature review by source—that is, one paragraph for source A, one paragraph for source B, etc. That structure will likely provide an adequate summary of the literature you’ve found, but it would give you almost no synthesis of the literature. That approach doesn’t tell your reader how to put those facts together, it doesn’t highlight points of agreement or contention, or how each study builds on the work of others. In short, it does not demonstrate critical thinking.

Organize your review by argument

Instead, use your outlines and notes as a guide what you have to say about the important topics you need to cover. Literature reviews are written from the perspective of an expert in that field. After an exhaustive literature review, you should feel as though you are able to make strong claims about what is true—so make them! There is no need to hide behind “I believe” or “I think.” Put your voice out in front, loud and proud! But make sure you have facts and sources that back up your claims.

I’ve used the term “ argument ” here in a specific way. An argument in writing means more than simply disagreeing with what someone else said, as this classic Monty Python sketch demonstrates. Toulman, Rieke, and Janik (1984) identify six elements of an argument:

Claim: the thesis statement—what you are trying to prove
Grounds: theoretical or empirical evidence that supports your claim
Warrant: your reasoning (rule or principle) connecting the claim and its grounds
Backing: further facts used to support or legitimize the warrant
Qualifier: acknowledging that the argument may not be true for all cases
Rebuttal: considering both sides (as cited in Burnette, 2012) [13]

Let’s walk through an example. If I were writing a literature review on a negative income tax, a policy in which people in poverty receive an unconditional cash stipend from the government each month equal to the federal poverty level, I would want to lay out the following:

Claim: the negative income tax is superior to other forms of anti-poverty assistance.
Grounds: data comparing negative income tax recipients to people receiving anti-poverty assistance in existing programs, theory supporting a negative income tax, data from evaluations of existing anti-poverty programs, etc.
Warrant: cash-based programs like the negative income tax are superior to existing anti-poverty programs because they allow the recipient greater self-determination over how to spend their money.
Backing: data demonstrating the beneficial effects of self-determination on people in poverty.
Qualifier: the negative income tax does not provide taxpayers and voters with enough control to make sure people in poverty are not wasting financial assistance on frivolous items.
Rebuttal: policy should be about empowering the oppressed, not protecting the taxpayer, and there are ways of addressing taxpayer spending concerns through policy design.

Like any effective argument, your literature review must have some kind of structure. For example, it might begin by describing a phenomenon in a general way along with several studies that provide some detail, then describing two or more competing theories of the phenomenon, and finally presenting a hypothesis to test one or more of the theories. Or, it might describe one phenomenon, then describe another that seems inconsistent with the first, then propose a theory that resolves the inconsistency, and finally present a hypothesis to test that theory. In applied research, it might describe a phenomenon or theory, then describe how that phenomenon or theory applies to some important real-world situation, and finally, may suggest a way to test whether it does, in fact, apply to that situation.

Use signposts

Another important issue is signposting . It may not be a term you are familiar with, but you are likely familiar with the concept. Signposting refers to the words used to identify the organization and structure of your literature review to your reader. The most basic form of signposting is using a topic sentence at the beginning of each paragraph. A topic sentence introduces the argument you plan to make in that paragraph. For example, you might start a paragraph stating, “There is strong disagreement in the literature as to whether psychedelic drugs cause psychotic disorders, or whether people with psychotic disorders cause people to use psychedelic drugs.” Within that paragraph, your reader would likely assume you will present evidence for both arguments. The concluding sentence of your paragraph should address the topic sentence, addressing how the facts and arguments from other authors support a specific conclusion. To continue with our example, I might say, “There is likely a reciprocal effect in which both the use of psychedelic drugs worsens pre-psychotic symptoms and worsening psychosis leads to the use of psychedelic drugs to self-medicate or escape.”

Signposting also involves using headings and subheadings. Your literature review will use APA formatting, which means you need to follow their rules for bolding, capitalization, italicization, and indentation of headings. Headings help your reader understand the structure of your literature review. They can also help if the reader gets lost and needs to re-orient themselves within the document. I often tell my students to assume I know nothing (they don’t mind) and need to be shown exactly where they are addressing each part of the literature review. It’s like walking a small child around, telling them “First we’ll do this, then we’ll do that, and when we’re done, we’ll know this!”

Another way to use signposting is to open each paragraph with a sentence that links the topic of the paragraph with the one before it. Alternatively, one could end each paragraph with a sentence that links it with the next paragraph. For example, imagine we wanted to link a paragraph about barriers to accessing healthcare with one about the relationship between the patient and physician. We could use a transition sentence like this: “Even if patients overcome these barriers to accessing care, the physician-patient relationship can create new barriers to positive health outcomes.” A transition sentence like this builds a connection between two distinct topics. Transition sentences are also useful within paragraphs. They tell the reader how to consider one piece of information in light of previous information. Even simple transitional words like ‘however’ and ‘similarly’ demonstrate critical thinking and link each building block of your argument together.

Many beginning researchers have difficulty incorporating transitions into their writing. Let’s look at an example. Instead of beginning a sentence or paragraph by launching into a description of a study, such as “Williams (2004) found that…,” it is better to start by indicating something about why you are describing this particular study. Here are some simple examples:

Another example of this phenomenon comes from the work of Williams (2004)…
Williams (2004) offers one explanation of this phenomenon…
An alternative perspective has been provided by Williams (2004)…

Now that we know to use signposts, the natural question is “What goes on the signposts?” First, it is important to start with an outline of the main points that you want to make, organized in the order you want to make them. The basic structure of your argument should then be apparent from the outline itself. Unfortunately, there is no formula I can give you that will work for everyone, but I can provide some general pointers on structuring your literature review.

The literature review tends to move from general to more specific ideas. You can build a review by identifying areas of consensus and areas of disagreement. You may choose to present historical studies—preferably seminal studies that are of significant importance—and close with the most recent research. Another approach is to start with the most distantly related facts and literature and then report on those most closely related to your research question. You could also compare and contrast valid approaches, features, characteristics, theories – that is, one approach, then a second approach, followed by a third approach.

Here are some additional tips for writing the body of your literature review:

Start broad and then narrow down to more specific information.
When appropriate, cite two or more sources for a single point, but avoid long strings of references for a single idea.
Use quotes sparingly. Quotations for definitions are okay, but reserve quotes for when something is said so well you couldn’t possible phrase it differently. Never use quotes for statistics.
Paraphrase when you need to relay the specific details within an article
Include only the aspects of the study that are relevant to your literature review. Don’t insert extra facts about a study just to take up space.
Avoid first-person like language like “I” and “we” to maintain objectivity.
Avoid informal language like contractions, idioms, and rhetorical questions.
Note any sections of your review that lack citations from the literature. Your arguments need to be based in empirical or theoretical facts. Do not approach this like a reflective journal entry.
Point out consistent findings and emphasize stronger studies over weaker ones.
Point out important strengths and weaknesses of research studies, as well as contradictions and inconsistent findings.
Implications and suggestions for further research (where there are gaps in the current literature) should be specific.

The conclusion should summarize your literature review, discuss implications, and create a space for further research needed in this area. Your conclusion, like the rest of your literature review, should make a point. What are the important implications of your literature review? How do they inform the question you are trying to answer?

You should consult with your professor and the course syllabus about the final structure your literature review should take. Here is an example of one possible structure:

Establish the importance of the topic
Number and type of people affected
Seriousness of the impact
Physical, psychological, economic, social, or spiritual consequences of the problem
Definitions of key terms
Supporting evidence
Common findings across studies, gaps in the literature
Research question(s) and hypothesis(es)

Editing your literature review

Literature reviews are more than a summary of the publications you find on a topic. As you have seen in this brief introduction, literature reviews represent a very specific type of research, analysis, and writing. We will explore these topics further in upcoming chapters. As you begin your literature review, here are some common errors to avoid:

Accepting a researcher’s finding as valid without evaluating methodology and data
Ignoring contrary findings and alternative interpretations
Using findings that are not clearly related to your own study or using findings that are too general
Dedicating insufficient time to literature searching
Reporting isolated statistical results, rather than synthesizing the results
Relying too heavily on secondary sources
Overusing quotations
Not justifying arguments using specific facts or theories from the literature

For your literature review, remember that your goal is to construct an argument for the importance of your research question. As you start editing your literature review, make sure it is balanced. Accurately report common findings, areas where studies contradict each other, new theories or perspectives, and how studies cause us to reaffirm or challenge our understanding of your topic.

It is acceptable to argue that the balance of the research supports the existence of a phenomenon or is consistent with a theory (and that is usually the best that researchers in social work can hope for), but it is not acceptable to ignore contradictory evidence. A large part of what makes a research question interesting is uncertainty about its answer (University of Minnesota, 2016). [14]

In addition to subjectivity and bias, writer’s block can obstruct the completion of your literature review. Often times, writer’s block can stem from confusing the creating and editing parts of the writing process. Many writers often start by simply trying to type out what they want to say, regardless of how good it is. Author Anne Lamott (1995) [15] terms these “shitty first drafts,” and we all write them. They are a natural and important part of the writing process.

Even if you have a detailed outline from which to work, the words are not going to fall into place perfectly the first time you start writing. You should consider turning off the editing and critiquing part of your brain for a while and allow your thoughts to flow. Don’t worry about putting the correct internal citation when you first write. Just get the information out. Only after you’ve reached a natural stopping point might you go back and edit your draft for grammar, APA style, organization, flow, and more. Divorcing the writing and editing process can go a long way to addressing writer’s block—as can picking a topic about which you have something to say!

As you are editing, keep in mind these questions adapted from Green (2012): [16]

Content: Have I clearly stated the main idea or purpose of the paper and address all the issues? Is the thesis or focus clearly presented and appropriate for the reader?
Organization: How well is it structured? Is the organization spelled out and easy to follow for the reader ?
Flow: Is there a logical flow from section to section, paragraph to paragraph, sentence to sentence? Are there transitions between and within paragraphs that link ideas together?
Development: Have I validated the main idea with supporting material? Are supporting data sufficient? Does the conclusion match the introduction?
Form: Are there any APA style issues, redundancy, problematic wording and terminology (always know the definition of any word you use!), flawed sentence constructions and selection, spelling, and punctuation?

Social workers use the APA style guide to format and structure their literature reviews. Most students know APA style only as it relates to internal and external citations. If you are confused about them, consult this amazing APA style guide from the University of Texas-Arlington library. Your university’s library likely has resources they created to help you with APA style, and you can meet with a librarian or your professor to talk about formatting questions you have. Make sure you budget in a few hours at the end of each project to build a correctly formatted references page and check your internal citations.

Of course, APA style is about much more than knowing there is a period after “et al.” or citing the location a book was published. APA style is also about what the profession considers to be good writing. If you haven’t picked up an APA publication manual because you use citation generators, know that I did the same thing when I was in school. However, I genuinely suggest picking up the newest manual for excellent guidance on writing for a professional audience. It will also help you with a common problem I hear about from students. Every professor (and every website about APA style) seems to have their own peculiar idea of “correct” APA style.

Here are some additional resources, if you would like more guidance on writing your literature review.

Doing a literature review / University of Leicester

Get lit: The literature review / Texas A&M Writing Centre

Guidebook for social work literature reviews / by Rebecca Mauldin and Matthew DeCarlo

A literature review is not a book report. Do not organize it by article, with one paragraph for each source in your references. Instead, organize it based on the key ideas and arguments.
The problem statement draws the reader into your topic by highlighting the importance of the topic to social work and to society overall.
Signposting is an important component of academic writing that helps your reader follow the structure of your argument and of your literature review.
Transitions demonstrate critical thinking and help guide your reader through your arguments.
Editing and writing are separate processes.
Consult with an APA style guide or a librarian to help you format your paper.
Write 2-3 facts you would use to address each question or component in the prompt.
Reflect on which questions you have a lot of information about and which you need to gather more information about in order to answer adequately.
Identify the key arguments you will make and how they are related to each other.
Reflect on topic sentences and concluding sentences you would use for each argument.
It wouldn’t make any sense to say that people’s workplace experiences cause their gender, so in this example, the question of which is the independent variable and which are the dependent variables has a pretty obvious answer. ↵
Cassidy, S. A., Dimova, R., Giguère, B., Spence, J. R., & Stanley, D. J. (2019). Failing grade: 89% of introduction-to-psychology textbooks that define or explain statistical significance do so incorrectly. Advances in Methods and Practices in Psychological Science , 2 (3), 233-239. ↵
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70 , p. 129-133. ↵
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS biology, 13 (3). ↵
Peng, R. (2015), The reproducibility crisis in science: A statistical counterattack. Significance , 12 , 30–32. ↵
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European journal of epidemiology , 31 (4), 337-350. ↵
Bonanno, R., & Veselak, K. (2019). A matter of trust: Parents attitudes towards child mental health information sources. Advances in Social Work , 19 (2), 397-415. ↵
Bernnard, D., Bobish, G., Hecker, J., Holden, I., Hosier, A., Jacobson, T., Loney, T., & Bullis, D. (2014). Presenting: Sharing what you’ve learned. In Bobish, G., & Jacobson, T. (eds.) The information literacy users guide: An open online textbook . https://milnepublishing.geneseo.edu/the-information-literacy-users-guide-an-open-online-textbook/chapter/present-sharing-what-youve-learned/ ↵
Leslie, M., Floyd, J., & Oermann, M. (2002). Use of MindMapper software for research domain mapping. Computers, informatics, nursing, 20(6), 229-235. ↵
Early Childhood Longitudinal Program. (2011). Example research questions . https://nces.ed.gov/ecls/researchquestions2011.asp ↵
Burnett, D. (2012). Inscribing knowledge: Writing research in social work. In W. Green & B. L. Simon (Eds.), The Columbia guide to social work writing (pp. 65-82). New York, NY: Columbia University Press. ↵
University of Minnesota Libraries Publishing. (2016). This is a derivative of Research Methods in Psychology by a publisher who has requested that they and the original author not receive attribution, which was originally released and is used under CC BY-NC-SA. This work, unless otherwise expressly stated, is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License ↵
Lamott, A. (1995). Bird by bird: Some instructions on writing and life . New York, NY: Penguin. ↵
Green, W. Writing strategies for academic papers. In W. Green & B. L. Simon (Eds.), The Columbia guide to social work writing (pp. 25-47). New York, NY: Columbia University Press. ↵

a quick, condensed summary of the report’s key findings arranged by row and column

(as in generalization) to make claims about a large population based on a smaller sample of people or items

"Assuming that the null hypothesis is true and the study is repeated an infinite number times by drawing random samples from the same populations(s), less than 5% of these results will be more extreme than the current result" (Cassidy et al., 2019, p. 233).

the assumption that no relationship exists between the variables in question

“a logical grouping of attributes that can be observed and measured and is expected to vary from person to person in a population” (Gillespie & Wagner, 2018, p. 9)

summarizes the incompatibility between a particular set of data and a proposed model for the data, usually the null hypothesis. The lower the p-value, the more inconsistent the data are with the null hypothesis, indicating that the relationship is statistically significant.

a range of values in which the true value is likely to be, to provide a more accurate description of their data

the words used to identify the organization and structure of your literature review to your reader

what a researcher hopes to accomplish with their study

Graduate research methods in social work Copyright © 2020 by Matthew DeCarlo, Cory Cummings, Kate Agnelli is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base

Methodology

How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

Search for relevant literature
Evaluate sources
Identify themes, debates, and gaps
Outline the structure
Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

Quick Run-through
Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

Demonstrate your familiarity with the topic and its scholarly context
Develop a theoretical framework and methodology for your research
Position your work in relation to other researchers and theorists
Show how your research addresses a gap or contributes to a debate
Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

Academic style
Vague sentences
Style consistency

See an example

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
Body image, self-perception, self-esteem, mental health
Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

Your university’s library catalogue
Google Scholar
Project Muse (humanities and social sciences)
Medline (life sciences and biomedicine)
EconLit (economics)
Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

What question or problem is the author addressing?
What are the key concepts and how are they defined?
What are the key theories, models, and methods?
Does the research use established frameworks or take an innovative approach?
What are the results and conclusions of the study?
How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
Themes: what questions or concepts recur across the literature?
Debates, conflicts and contradictions: where do sources disagree?
Pivotal publications: are there any influential theories or studies that changed the direction of the field?
Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

Most research has focused on young women.
There is an increasing interest in the visual aspects of social media.
But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

Look at what results have emerged in qualitative versus quantitative research
Discuss how the topic has been approached by empirical versus theoretical scholarship
Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
Critically evaluate: mention the strengths and weaknesses of your sources
Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

Sampling methods
Simple random sampling
Stratified sampling
Cluster sampling
Likert scales
Reproducibility

Statistics

Null hypothesis
Statistical power
Probability distribution
Effect size
Poisson distribution

Research bias

Optimism bias
Cognitive bias
Implicit bias
Hawthorne effect
Anchoring bias
Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

To familiarize yourself with the current state of knowledge on your topic
To ensure that you’re not just repeating what others have already done
To identify gaps in knowledge and unresolved problems that your research can address
To develop your theoretical framework and methodology
To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .

An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved September 3, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

The Literature Review, Part 1: What to Include

This blog is about what to include in your literature review. In short, the literature review is a snapshot of the current state of research on your topic, including research on study variables and major concepts or theories of your study. The literature review also helps to support your research problem and rationalize why your study is necessary by identifying gaps in the literature and the methodological weaknesses of previous studies. Below is what to include in your literature review.

Include recent, peer-reviewed studies and articles. These are really the meat of any literature review and what your literature review should primarily contain. Any historical or informational material on the topic should be included in background sections of your Introduction chapter or in a brief setup section at the beginning of the literature review.

Articles should ideally be recent within five years of the time you anticipate completing your dissertation. This five-year window, however, is not always required. Some schools allow articles to be recent within five to seven years, and some schools have no requirements. However, the intention of the literature review is to give readers a sense of the current state of research on your topic. So, in the spirit of writing an accurate and effective literature review, recent sources are recommended.

In “The Literature Review, Part 2,” we will discuss what not to include in the literature review.

[/et_pb_text][/et_pb_column][/et_pb_row]

Free Help Session: Chapters 1 and 2

During these sessions, students can get answers to introduction to the problem, background of study, statement of the problem, purpose of the study, and theoretical framework. Questions may also involve title searches, literature review, synthesis of findings, gap and critique of research. (We will not address APA style, grammar, headings, etc. If you are interested in help with the research design or nature of the study, please register for the methodology drop-in by clicking here ).

Register Here

[/et_pb_text][/et_pb_column][/et_pb_row][/et_pb_section]

We work with graduate students every day and know what it takes to get your research approved.

Address committee feedback
Roadmap to completion
Understand your needs and timeframe

Literature Reviews

What is a Literature Review?
Steps for Creating a Literature Review
Providing Evidence / Critical Analysis
Challenges when writing a Literature Review
Systematic Literature Reviews

Developing a Literature Review

1. Purpose and Scope

To help you develop a literature review, gather information on existing research, sub-topics, relevant research, and overlaps. Note initial thoughts on the topic - a mind map or list might be helpful - and avoid unfocused reading, collecting irrelevant content. A literature review serves to place your research within the context of existing knowledge. It demonstrates your understanding of the field and identifies gaps that your research aims to fill. This helps in justifying the relevance and necessity of your study.

To avoid over-reading, set a target word count for each section and limit reading time. Plan backwards from the deadline and move on to other parts of the investigation. Read major texts and explore up-to-date research. Check reference lists and citation indexes for common standard texts. Be guided by research questions and refocus on your topic when needed. Stop reading if you find similar viewpoints or if you're going off topic.

You can use a "Synthesis Matrix" to keep track of your reading notes. This concept map helps you to provide a summary of the literature and its connections is produced as a result of this study. Utilizing referencing software like RefWorks to obtain citations, you can construct the framework for composing your literature evaluation.

2. Source Selection

Focus on searching for academically authoritative texts such as academic books, journals, research reports, and government publications. These sources are critical for ensuring the credibility and reliability of your review.

Academic Books: Provide comprehensive coverage of a topic.
Journal Articles: Offer the most up-to-date research and are essential for a literature review.
Research Reports: Detailed accounts of specific research projects.
Government Publications: Official documents that provide reliable data and insights.

3. Thematic Analysis

Instead of merely summarizing sources, identify and discuss key themes that emerge from the literature. This involves interpreting and evaluating how different authors have tackled similar issues and how their findings relate to your research.

4. Critical Evaluation

Adopt a critical attitude towards the sources you review. Scrutinize, question, and dissect the material to ensure that your review is not just descriptive but analytical. This helps in highlighting the significance of various sources and their relevance to your research.

Each work's critical assessment should take into account:

Provenance: What qualifications does the author have? Are the author's claims backed up by proof, such as first-hand accounts from history, case studies, stories, statistics, and current scientific discoveries? Methodology: Were the strategies employed to locate, collect, and evaluate the data suitable for tackling the study question? Was the sample size suitable? Were the findings properly reported and interpreted? Objectivity : Is the author's viewpoint impartial or biased? Does the author's thesis get supported by evidence that refutes it, or does it ignore certain important facts? Persuasiveness: Which of the author's arguments is the strongest or weakest in terms of persuasiveness? Value: Are the author's claims and deductions believable? Does the study ultimately advance our understanding of the issue in any meaningful way?

5. Categorization

Organize your literature review by grouping sources into categories based on themes, relevance to research questions, theoretical paradigms, or chronology. This helps in presenting your findings in a structured manner.

6. Source Validity

Ensure that the sources you include are valid and reliable. Classic texts may retain their authority over time, but for fields that evolve rapidly, prioritize the most recent research. Always check the credibility of the authors and the impact of their work in the field.

7. Synthesis and Findings

Synthesize the information from various sources to draw conclusions about the current state of knowledge. Identify trends, controversies, and gaps in the literature. Relate your findings to your research questions and suggest future directions for research.

Practical Tips

Use a variety of sources, including online databases, university libraries, and reference lists from relevant articles. This ensures a comprehensive coverage of the literature.
Avoid listing sources without analysis. Use tables, bulk citations, and footnotes to manage references efficiently and make your review more readable.
Writing a literature review is an ongoing process. Start writing early and revise as you read more. This iterative process helps in refining your arguments and identifying additional sources as needed.

Brown University Library (2024) Organizing and Creating Information. Available at: https://libguides.brown.edu/organize/litreview (Accessed: 30 July 2024).

Pacheco-Vega, R. (2016) Synthesizing different bodies of work in your literature review: The Conceptual Synthesis Excel Dump (CSED) technique . Available at: http://www.raulpacheco.org/2016/06/synthesizing-different-bodies-of-work-in-your-literature-review-the-conceptual-synthesis-excel-dump-technique/ (Accessed: 30 July 2024).

Study Advice at the University of Reading (2024) Literature reviews . Available at: https://libguides.reading.ac.uk/literaturereview/developing (Accessed: 31 July 2024).

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

formulating the research question(s) and objective(s),
searching the extant literature,
screening for inclusion,
assessing the quality of primary studies,
extracting data, and
analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health (m-health) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
Analyzing data using quantitative or qualitative methods.
Presenting results in summary of findings tables.
Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed independently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guidelines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel .aisnet.org/cgi/viewcontent .cgi?article =1221&context =ecis2011 .
Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
PDF version of this title (4.5M)

In this Page

Introduction
Overview of the Literature Review Process and Steps
Types of Review Articles and Brief Illustrations
Concluding Remarks

Related information

PMC PubMed Central citations
PubMed Links to PubMed

Recent Activity

Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Information

Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

Active Journals
Find a Journal
Proceedings Series
For Authors
For Reviewers
For Editors
For Librarians
For Publishers
For Societies
For Conference Organizers
Open Access Policy
Institutional Open Access Program
Special Issues Guidelines
Editorial Process
Research and Publication Ethics
Article Processing Charges
Testimonials
Preprints.org
SciProfiles
Encyclopedia

Article Menu

Subscribe SciFeed
Recommended Articles
Google Scholar
on Google Scholar
Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Literature review on collaborative project delivery for sustainable construction: bibliometric analysis.

1. Introduction

2. literature review, 2.1. collaborative project delivery, 2.2. design build (db), 2.3. construction manager at risk (cmar), 2.4. integrated project delivery method (ipd), 2.5. sustainability, 2.6. sustainable construction, 2.7. benefits of eci comparing case studies, 2.8. collaborative delivery models, 3. methodology, 3.1. research methods, 3.2. database research, 4.1. ipd, design-build, and cmar overview, 4.1.1. yearly publication distribution of db cmar and ipd, 4.1.2. major country analysis, 4.1.3. most relevant and influential journals, 4.1.4. corresponding author countries, 4.2. keyword analysis, 4.2.1. high-frequency keyword analysis, 4.2.2. co-occurrence network analysis, 4.2.3. analysis of keywords’ frequency over time, 5. discussion, 5.1. findings of advantages and disadvantages of ipd, db, and cmar for sustainable construction, 5.1.1. advantages of ipd, 5.1.2. advantages of design-build, 5.1.3. advantages of construction manager at risk, 5.1.4. disadvantages of ipd, 5.1.5. disadvantages of design-build, 5.1.6. disadvantages of construction manager at risk, 5.2. most suitable cpd technique for sustainable construction based on literature review, 5.2.1. limitations, 5.2.2. recommendations for future research, 6. future trend, 6.1. enhancing innovation through collaborative project delivery, 6.2. open communication and block chain technology, 6.3. multi-party agreement, 6.4. utilizing artificial intelligence in decision support systems, 7. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

Giachino, J.; Cecil, M.; Husselbee, B.; Matthews, C. Alternative Project Delivery: Construction Management at Risk, Design-Build and Public-Private Partnerships. In Proceedings of the Utility Management Conference 2016, San Diego, CA, USA, 24–26 February 2016. [ Google Scholar ]
Shrestha, P.P.; Maharjan, R.; Batista, J.R. Performance of Design-Build and Construction Manager-at-Risk Methods in Water and Wastewater Projects. Pract. Period. Struct. Des. Constr. 2019 , 24 , 04018029. [ Google Scholar ] [ CrossRef ]
Shrestha, P.P.; Batista, J. Lessons Learned in Design-Build and Construction-Manager-at-Risk Water and Wastewater Project. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2020 , 12 , 04520002. [ Google Scholar ] [ CrossRef ]
Xia, B.; Chan, A.P.C. Identification of Selection Criteria for Operational Variations of The Design-Build System: A Delphi Study in China. J. Civ. Eng. Manag. 2012 , 18 , 173–183. [ Google Scholar ] [ CrossRef ]
Shane, J.S.; Bogus, S.M.; Molenaar, K.R. Municipal Water/Wastewater Project Delivery Performance Comparison. J. Manag. Eng. 2013 , 29 , 251–258. [ Google Scholar ] [ CrossRef ]
Sullivan, J.; El Asmar, M.; Chalhoub, J.; Obeid, H. Two Decades of Performance Comparisons for Design-Build, Construction Manager at Risk, and Design-Bid-Build: Quantitative Analysis of the State of Knowledge on Project Cost, Schedule, and Quality. J. Constr. Eng. Manag. 2017 , 143 , 04017009. [ Google Scholar ] [ CrossRef ]
Raouf, A.M.; Al-Ghamdi, S. Effectiveness of Project Delivery Systems in Executing Green Buildings. J. Constr. Eng. Manag. 2019 , 145 , 03119005. [ Google Scholar ] [ CrossRef ]
Francom, T.; El Asmar, M.; Ariaratnam, S.T. Performance Analysis of Construction Manager at Risk on Pipeline Engineering and Construction Projects. J. Manag. Eng. 2016 , 32 , 04016016. [ Google Scholar ] [ CrossRef ]
Gransberg, D.D.; Shane, J.S.; Transportation Research Board. Construction Manager-at-Risk Project Delivery for Highway Programs ; The National Academies Press: Washington, DC, USA, 2010. [ Google Scholar ]
Rahman, M.M.; Kumaraswamy, M.M. Potential for Implementing Relational Contracting and Joint Risk Management. J. Manag. Eng. 2004 , 20 , 178–189. [ Google Scholar ] [ CrossRef ]
Feghaly, J.; El Asmar, M.; Ariaratnam, S.; Bearup, W. Selecting project delivery methods for water treatment plants. Eng. Constr. Archit. Manag. 2019 , 27 , 936–951. [ Google Scholar ] [ CrossRef ]
Park, H.-S.; Lee, D.; Kim, S.; Kim, J.-L. Comparing Project Performance of Design-Build and Design-Bid-Build Methods for Large-sized Public Apartment Housing Projects in Korea. J. Asian Archit. Build. Eng. 2015 , 14 , 323–330. [ Google Scholar ] [ CrossRef ]
Shrestha, P.P.; Batista, J.; Maharajan, R. Risks involved in using alternative project delivery (APD) methods in water and wastewater projects. Procedia Eng. 2016 , 145 , 219–223. [ Google Scholar ] [ CrossRef ]
Hettiaarachchige, N.; Rathnasinghe, A.; Ranadewa, K.; Thurairajah, N. Thurairajah, Lean Integrated Project Delivery for Construction Procurement: The Case of Sri Lanka. Buildings 2022 , 12 , 524. [ Google Scholar ] [ CrossRef ]
Kent, D.C.; Becerik-Gerber, B. Understanding Construction Industry Experience and Attitudes toward Integrated Project Delivery. J. Constr. Eng. Manag. 2010 , 136 , 815–825. [ Google Scholar ] [ CrossRef ]
Franz, B.; Leicht, R.; Molenaar, K.; Messner, J. Impact of Team Integration and Group Cohesion on Project Delivery Performance. J. Constr. Eng. Manag. 2017 , 143 , 04016088. [ Google Scholar ] [ CrossRef ]
Engebø, A.; Klakegg, O.J.; Lohne, J.; Lædre, O. A collaborative project delivery method for design of a high-performance building. Int. J. Manag. Proj. Bus. 2020 , 13 , 1141–1165. [ Google Scholar ] [ CrossRef ]
Ahmed, S.; El-Sayegh, S. Critical Review of the Evolution of Project Delivery Methods in the Construction Industry. Buildings 2020 , 11 , 11. [ Google Scholar ] [ CrossRef ]
Bond-Barnard, T.J.; Fletcher, L.; Steyn, H. Linking trust and collaboration in project teams to project management success. Int. J. Manag. Proj. Bus. 2018 , 11 , 432–457. [ Google Scholar ] [ CrossRef ]
Rodrigues, M.R.; Lindhard, S.M. Lindhard, Benefits and challenges to applying IPD: Experiences from a Norwegian mega-project. Constr. Innov. 2021 , 23 , 287–305. [ Google Scholar ] [ CrossRef ]
Kaminsky, J. The fourth pillar of infrastructure sustainability: Tailoring civil infrastructure to social context. Constr. Manag. Econ. 2015 , 33 , 299–309. [ Google Scholar ] [ CrossRef ]
Al Khalil, M.I. Selecting the appropriate project delivery method using AHP. Int. J. Proj. Manag. 2002 , 20 , 469–474. [ Google Scholar ] [ CrossRef ]
Ibbs, C.W.; Kwak, Y.H.; Ng, T.; Odabasi, A.M. Project Delivery Systems and Project Change: Quantitative Analysis. J. Constr. Eng. Manag. 2003 , 129 , 382–387. [ Google Scholar ] [ CrossRef ]
Jansen, J.; Beck, A. Overcoming the Challenges of Large Diameter Water Project in North Texas via CMAR Delivery Method. In Proceedings of the Pipelines 2020, San Antonio, TX, USA, 9–12 August 2020; Conference Held Virtually. pp. 264–271. [ Google Scholar ] [ CrossRef ]
Bingham, E.; Gibson, G.E.; Asmar, M.E. Measuring User Perceptions of Popular Transportation Project Delivery Methods Using Least Significant Difference Intervals and Multiple Range Tests. J. Constr. Eng. Manag. 2018 , 144 , 04018033. [ Google Scholar ] [ CrossRef ]
Cho, Y.J. A review of construction delivery systems: Focus on the construction management at risk system in the Korean public construction market. KSCE J. Civ. Eng. 2016 , 20 , 530–537. [ Google Scholar ] [ CrossRef ]
Rosayuru, H.D.R.R.; Waidyasekara, K.G.A.S.; Wijewickrama, M.K.C.S. Sustainable BIM based integrated project delivery system for construction industry in Sri Lanka. Int. J. Constr. Manag. 2022 , 22 , 769–783. [ Google Scholar ] [ CrossRef ]
Pishdad-Bozorgi, P.; Beliveau, Y.J. Symbiotic Relationships between Integrated Project Delivery (IPD) and Trust. Int. J. Constr. Educ. Res. 2016 , 12 , 179–192. [ Google Scholar ] [ CrossRef ]
Sherif, M.; Abotaleb, I.; Alqahtani, F.K. Alqahtani, Application of Integrated Project Delivery (IPD) in the Middle East: Implementation and Challenges. Buildings 2022 , 12 , 467. [ Google Scholar ] [ CrossRef ]
Manata, B.; Garcia, A.J.; Mollaoglu, S.; Miller, V.D. The effect of commitment differentiation on integrated project delivery team dynamics: The critical roles of goal alignment, communication behaviors, and decision quality. Int. J. Proj. Manag. 2021 , 39 , 259–269. [ Google Scholar ] [ CrossRef ]
Kraatz, J.A.; Sanchez, A.X.; Hampson, K.D. Hampson, Digital Modeling, Integrated Project Delivery and Industry Transformation: An Australian Case Study. Buildings 2014 , 4 , 453–466. [ Google Scholar ] [ CrossRef ]
Zhang, L.; He, J.; Zhou, S. Sharing Tacit Knowledge for Integrated Project Team Flexibility: Case Study of Integrated Project Delivery. J. Constr. Eng. Manag. 2013 , 139 , 795–804. [ Google Scholar ] [ CrossRef ]
El Asmar, M.; Hanna, A.S.; Loh, W.-Y. Quantifying Performance for the Integrated Project Delivery System as Compared to Established Delivery Systems. J. Constr. Eng. Manag. 2013 , 139 , 04013012. [ Google Scholar ] [ CrossRef ]
Ghassemi, R.; Becerik-Gerber, B. Transitioning to integrated project delivery: Potential barriers and lessons learned. Lean Constr. J. 2011 , 32–52. Available online: https://leanconstruction.org/resources/lean-construction-journal/lcj-back-issues/2011-issue/ (accessed on 11 August 2024).
Mei, T.; Guo, Z.; Li, P.; Fang, K.; Zhong, S. Influence of Integrated Project Delivery Principles on Project Performance in China: An SEM-Based Approach. Sustainability 2022 , 14 , 4381. [ Google Scholar ] [ CrossRef ]
Ilozor, B.D.; Kelly, D.J. Building information modeling and integrated project delivery in the commercial construction industry: A conceptual study. J. Eng. Proj. Prod. Manag. 2012 , 2 , 23–36. [ Google Scholar ] [ CrossRef ]
Zabihi, H.; Habib, F.; Mirsaeedie, L. Sustainability in Building and Construction: Revising Definitions and Concepts. Int. J. Emerg. Sci. 2012 , 2 , 570–578. [ Google Scholar ]
Young, J.W.S. A Framework for the Ultimate Environmental Index—Putting Atmospheric Change Into Context With Sustainability. Environ. Monit. Assess. 1997 , 46 , 135–149. [ Google Scholar ] [ CrossRef ]
Ding, G.K.C. Sustainable construction—The role of environmental assessment tools. J. Environ. Manag. 2008 , 86 , 451–464. [ Google Scholar ] [ CrossRef ] [ PubMed ]
Conte, E. The Era of Sustainability: Promises, Pitfalls and Prospects for Sustainable Buildings and the Built Environment. Sustainability 2018 , 10 , 2092. [ Google Scholar ] [ CrossRef ]
Standardized Method of Life Cycle Costing for Construction Procurement. A Supplement to BS ISO 15686-5. Buildings and Constructed Assets. Service Life Planning. Life Cycle Costing ; BSI British Standards: London, UK, 2008. [ CrossRef ]
Sustainability|Free Full-Text|A Hybrid Multi-Criteria Decision Support System for Selecting the Most Sustainable Structural Material for a Multistory Building Construction. Available online: https://www.mdpi.com/2071-1050/15/4/3128 (accessed on 2 April 2024).
Korkmaz, S.; Riley, D.; Horman, M. Piloting Evaluation Metrics for Sustainable High-Performance Building Project Delivery. J. Constr. Eng. Manag. 2010 , 136 , 877–885. [ Google Scholar ] [ CrossRef ]
Ng, M.S.; Graser, K.; Hall, D.M. Digital fabrication, BIM and early contractor involvement in design in construction projects: A comparative case study. Archit. Eng. Des. Manag. 2021 , 19 , 39–55. [ Google Scholar ] [ CrossRef ]
Moradi, S.; Kähkönen, K.; Sormunen, P. Analytical and Conceptual Perspectives toward Behavioral Elements of Collaborative Delivery Models in Construction Projects. Buildings 2022 , 12 , 316. [ Google Scholar ] [ CrossRef ]
Zupic, I.; Čater, T. Bibliometric Methods in Management and Organization. 2015. Available online: https://journals.sagepub.com/doi/abs/10.1177/1094428114562629 (accessed on 3 April 2024).
Rozas, L.W.; Klein, W.C. The Value and Purpose of the Traditional Qualitative Literature Review. J. Evid.-Based Soc. Work 2010 , 7 , 387–399. [ Google Scholar ] [ CrossRef ] [ PubMed ]
Cobo, M.J.; López-Herrera, A.G.; Herrera-Viedma, E.; Herrera, F. Science mapping software tools: Review, analysis, and cooperative study among tools. J. Am. Soc. Inf. Sci. Technol. 2011 , 62 , 1382–1402. [ Google Scholar ] [ CrossRef ]
Cancino, C.A.; Merigó, J.M.; Coronado, F.C. A bibliometric analysis of leading universities in innovation research. J. Innov. Knowl. 2017 , 2 , 106–124. [ Google Scholar ] [ CrossRef ]
Pedro, L.F.M.G.; Barbosa, C.M.M.d.O.; Santos, C.M.d.N. A critical review of mobile learning integration in formal educational contexts. Int. J. Educ. Technol. High. Educ. 2018 , 15 , 10. [ Google Scholar ] [ CrossRef ]
Wen, S.; Tang, H.; Ying, F.; Wu, G. Exploring the Global Research Trends of Supply Chain Management of Construction Projects Based on a Bibliometric Analysis: Current Status and Future Prospects. Buildings 2023 , 13 , 373. [ Google Scholar ] [ CrossRef ]
Hosseini, M.R.; Martek, I.; Zavadskas, E.K.; Aibinu, A.A.; Arashpour, M.; Chileshe, N. Critical evaluation of off-site construction research: A Scientometric analysis. Autom. Constr. 2018 , 87 , 235–247. [ Google Scholar ] [ CrossRef ]
Toyin, J.O.; Mewomo, M.C. Mewomo, Overview of BIM contributions in the construction phase: Review and bibliometric analysis. J. Inf. Technol. Constr. 2023 , 28 , 500–514. [ Google Scholar ] [ CrossRef ]
Kahvandi, Z.; Saghatforoush, E.; Alinezhad, M.; Noghli, F. Integrated Project Delivery (IPD) Research Trends. J. Eng. 2017 , 7 , 99–114. [ Google Scholar ] [ CrossRef ]
Hale, D.R.; Shrestha, P.P.; Gibson, G.E.; Migliaccio, G.C. Empirical Comparison of Design/Build and Design/Bid/Build Project Delivery Methods. J. Constr. Eng. Manag. 2009 , 135 , 579–587. [ Google Scholar ] [ CrossRef ]
Mollaoglu-Korkmaz, S.; Swarup, L.; Riley, D. Delivering Sustainable, High-Performance Buildings: Influence of Project Delivery Methods on Integration and Project Outcomes. J. Manag. Eng. 2013 , 29 , 71–78. [ Google Scholar ] [ CrossRef ]
Ugwu, O.O.; Haupt, T.C. Key performance indicators and assessment methods for infrastructure sustainability—a South African construction industry perspective. Build. Environ. 2007 , 42 , 665–680. [ Google Scholar ] [ CrossRef ]
Kines, P.; Andersen, L.P.S.; Spangenberg, S.; Mikkelsen, K.L.; Dyreborg, J.; Zohar, D. Improving construction site safety through leader-based verbal safety communication. J. Safety Res. 2010 , 41 , 399–406. [ Google Scholar ] [ CrossRef ] [ PubMed ]
Ballard, G. The Lean Project Delivery System: An Update. 2008. [ Google Scholar ]
Bynum, P.; Issa, R.R.A.; Olbina, S. Building information modeling in support of sustainable design and construction. J. Constr. Eng. Manag. 2013 , 139 , 24–34. [ Google Scholar ] [ CrossRef ]
Choudhry, R.M.; Fang, D.; Lingard, H. Measuring Safety Climate of a Construction Company. J. Constr. Eng. Manag. 2009 , 135 , 890–899. [ Google Scholar ] [ CrossRef ]
Wardani, M.A.E.; Messner, J.I.; Horman, M.J. Comparing procurement methods for Design-Build projects. J. Constr. Eng. Manag. 2006 , 132 , 230–238. [ Google Scholar ] [ CrossRef ]
Liu, J.; Zhao, X.; Yan, P. Risk Paths in International Construction Projects: Case Study from Chinese Contractors. J. Constr. Eng. Manag. 2016 , 142 . [ Google Scholar ] [ CrossRef ]
El-Sayegh, S. Evaluating the effectiveness of project delivery methods. J. Constr. Manag. Econ. 2008 , 23 , 457–465. [ Google Scholar ]
Fang, C.; Marle, F.; Zio, E.; Bocquet, J.-C. Network theory-based analysis of risk interactions in large engineering projects. Reliability Eng. Syst. Safety 2012 , 106 , 1–10. [ Google Scholar ] [ CrossRef ]
Franz, B.; Leicht, R.M. Initiating IPD Concepts on Campus Facilities with a ‘Collaboration Addendum’. In Proceedings of the Construction Research Congress 2012, West Lafayette, IN, USA, 21–23 May 2012; pp. 61–70. [ Google Scholar ] [ CrossRef ]
Kim, H.; Kim, K.; Kim, H. Vision-Based Object-Centric Safety Assessment Using Fuzzy Inference: Monitoring Struck-By Accidents with Moving Objects. J. Comput. Civil Eng. 2016 , 30 . [ Google Scholar ] [ CrossRef ]
Zhou, Y.; Ding, L.Y.; Chen, L.J. Application of 4D visualization technology for safety management in metro construction. Automation Constr. 2013 , 34 , 25–36. [ Google Scholar ] [ CrossRef ]
Wanberg, J.; Harper, C.; Hallowell, M.R.; Rajendran, S. Relationship between Construction Safety and Quality Performance. J. Constr. Eng. Manag. 2013 , 139 . [ Google Scholar ] [ CrossRef ]
Shrestha, P.P.; O’Connor, J.T.; Gibson, G.E. Performance comparison of large Design-Build and Design-Bid-Build highway projects. J. Constr. Eng. Manag. 2012 , 138 , 1–13. [ Google Scholar ] [ CrossRef ]
Torabi, S.A.; Hassini, E. Multi-site production planning integrating procurement and distribution plans in multi-echelon supply chains: An interactive fuzzy goal programming approach. Int. J. Prod. Res. 2009 , 47 , 5475–5499. [ Google Scholar ] [ CrossRef ]
Baradan, S.; Usmen, M. Comparative Injury and Fatality Risk Analysis of Building Trades. J. Constr. Eng. Manag.-ASCE 2006 , 132 . [ Google Scholar ] [ CrossRef ]
Levitt, R.E. CEM Research for the Next 50 Years: Maximizing Economic, Environmental, and Societal Value of the Built Environment1. J. Constr. Eng. Manag. 2007 , 133 , 619–628. [ Google Scholar ] [ CrossRef ]
Araya, F. Modeling the spread of COVID-19 on construction workers: An agent-based approach. Saf. Sci. 2021 , 133 , 105022. [ Google Scholar ] [ CrossRef ]
Zheng, X.; Le, Y.; Chan, A.P.; Hu, Y.; Li, Y. Review of the application of social network analysis (SNA) in construction project management research. Int. J. Proj. Manag. 2016 , 34 , 1214–1225. [ Google Scholar ] [ CrossRef ]
Elghaish, F.; Abrishami, S. A centralised cost management system: Exploiting EVM and ABC within IPD. Eng. Constr. Archit. Manag. 2021 , 28 , 549–569. [ Google Scholar ] [ CrossRef ]
Smith, R.E.; Mossman, A.; Emmitt, S. Lean and integrated project delivery. Lean Constr. J. 2011 , 1–16. [ Google Scholar ]
Bröchner, J.; Badenfelt, U. Changes and change management in construction and IT projects. Autom. Constr. 2011 , 20 , 767–775. [ Google Scholar ] [ CrossRef ]
Monteiro, A.; Mêda, P.; Martins, J.P. Framework for the coordinated application of two different integrated project delivery platforms. Autom. Constr. 2014 , 38 , 87–99. [ Google Scholar ] [ CrossRef ]
Azhar, N.; Kang, Y.; Ahmad, I.U. Factors influencing integrated project delivery in publicly owned construction projects: An information modelling perspective. Procedia Eng. 2014 , 77 , 213–221. [ Google Scholar ] [ CrossRef ]
Mihic, M.; Sertic, J.; Zavrski, I. Integrated Project Delivery as Integration between Solution Development and Solution Implementation. Procedia Soc. Behav. Sci. 2014 , 119 , 557–565. [ Google Scholar ] [ CrossRef ]
Nawi, M.N.M.; Haron, A.T.; Hamid, Z.A.; Kamar, K.A.M.; Baharuddin, Y. Improving integrated practice through building information modeling-integrated project delivery (BIM-IPD) for Malaysian industrialised building system (IBS) Construction Projects. Malays. Constr. Res. J. 2014 , 15 , 29–38. Available online: https://dsgate.uum.edu.my/jspui/handle/123456789/1651 (accessed on 24 April 2024).
Ma, Z.; Zhang, D.; Li, J. A dedicated collaboration platform for Integrated Project Delivery. Autom. Constr. 2018 , 86 , 199–209. [ Google Scholar ] [ CrossRef ]
Yadav, S.; Kanade, G. Application of Revit as Building Information Modeling (BIM) for Integrated Project Delivery (IPD) to Building Construction Project—A Review. Int. Res. J. Eng. Technol. 2018 , 5 , 11–14. [ Google Scholar ]
Salim, M.S.; Mahjoob, A.M.R. Integrated project delivery (IPD) method with BIM to improve the project performance: A case study in the Republic of Iraq. Asian J. Civ. Eng. 2020 , 21 , 947–957. [ Google Scholar ] [ CrossRef ]
Ling, Y.Y.; Lau, B.S.Y. A case study on the management of the development of a large-scale power plant project in East Asia based on design-build arrangement. Int. J. Proj. Manag. 2002 , 20 , 413–423. [ Google Scholar ] [ CrossRef ]
Dalui, P.; Elghaish, F.; Brooks, T.; McIlwaine, S. Integrated Project Delivery with BIM: A Methodical Approach Within the UK Consulting Sector. J. Inf. Technol. Constr. 2021 , 26 , 922–935. [ Google Scholar ] [ CrossRef ]
Pishdad-Bozorgi, P. Case Studies on the Role of Integrated Project Delivery (IPD) Approach on the Establishment and Promotion of Trust. Int. J. Constr. Educ. Res. 2017 , 13 , 102–124. [ Google Scholar ] [ CrossRef ]
Singleton, M.S.; Hamzeh, F.R. Implementing integrated project delivery on department of the navy construction projects: Lean Construction Journal. Lean Constr. J. 2011 , 17–31. [ Google Scholar ]
Tran, D.Q.; Nguyen, L.D.; Faught, A. Examination of communication processes in design-build project delivery in building construction. Eng. Constr. Archit. Manag. 2017 , 24 , 1319–1336. [ Google Scholar ] [ CrossRef ]
Park, J.; Kwak, Y.H. Design-Bid-Build (DBB) vs. Design-Build (DB) in the U.S. public transportation projects: The choice and consequences. Int. J. Proj. Manag. 2017 , 35 , 280–295. [ Google Scholar ] [ CrossRef ]
Wiss, R.A.; Roberts, R.T.; Phraner, S.D. Beyond Design-Build-Operate-Maintain: New Partnership Approach Toward Fixed Guideway Transit Projects. Transp. Res. Rec. J. Transp. Res. Board 2000 , 1704 , 13–18. [ Google Scholar ] [ CrossRef ]
Xia, B.; Chan, A.P. Key competences of design-build clients in China. J. Facil. Manag. 2010 , 8 , 114–129. [ Google Scholar ] [ CrossRef ]
DeBernard, D.M. Beyond Collaboration—The Benefits of Integrated Project Delivery ; AIA Soloso Website: Washington, DC, USA, 2008. [ Google Scholar ]
Chen, Q.; Jin, Z.; Xia, B.; Wu, P.; Skitmore, M. Time and Cost Performance of Design–Build Projects. J. Constr. Eng. Manag. 2016 , 142 , 04015074. [ Google Scholar ] [ CrossRef ]
Xia, B.; Chan, P. Review of the design-build market in the People’s Republic of China. J. Constr. Procure. 2008 , 14 , 108–117. [ Google Scholar ]
Mcwhirt, D.; Ahn, J.; Shane, J.S.; Strong, K.C. Military construction projects: Comparison of project delivery methods. J. Facil. Manag. 2011 , 9 , 157–169. [ Google Scholar ] [ CrossRef ]
Minchin, R.E.; Li, X.; Issa, R.R.; Vargas, G.G. Comparison of Cost and Time Performance of Design-Build and Design-Bid-Build Delivery Systems in Florida. J. Constr. Eng. Manag. 2013 , 139 , 04013007. [ Google Scholar ] [ CrossRef ]
Adamtey, S.; Onsarigo, L. Effective tools for projects delivered by progressive design-build method. In Proceedings of the CSCE Annual Conference 2019, Laval, QC, Canada, 12–15 June 2019; pp. 1–10. [ Google Scholar ]
Adamtey, S.A. A Case Study Performance Analysis of Design-Build and Integrated Project Delivery Methods. Int. J. Constr. Educ. Res. 2021 , 17 , 68–84. [ Google Scholar ] [ CrossRef ]
Gad, G.M.; Adamtey, S.A.; Gransberg, D.D. Gransberg, Trends in Quality Management Approaches to Design–Build Transportation Projects. Transp. Res. Rec. J. Transp. Res. Board. 2015 , 2504 , 87–92. [ Google Scholar ] [ CrossRef ]
Sari, E.M.; Irawan, A.P.; Wibowo, M.A.; Siregar, J.P.; Praja, A.K.A. Project delivery systems: The partnering concept in integrated and non-integrated construction projects. Sustainability 2022 , 15 , 86. [ Google Scholar ] [ CrossRef ]
Chakra, H.A.; Ashi, A. Comparative analysis of design/build and design/bid/build project delivery systems in Lebanon. J. Ind. Eng. Int. 2019 , 15 , 147–152. [ Google Scholar ] [ CrossRef ]
Perkins, R.A. Sources of Changes in Design–Build Contracts for a Governmental Owner. J. Constr. Eng. Manag. 2009 , 135 , 588–593. [ Google Scholar ] [ CrossRef ]
Palaneeswaran, E.; Kumaraswamy, M.M. Contractor Selection for Design/Build Projects. J. Constr. Eng. Manag. 2000 , 126 , 331–339. [ Google Scholar ] [ CrossRef ]
Chan, A.P.C. Evaluation of enhanced design and build system a case study of a hospital project. Constr. Manag. Econ. 2000 , 18 , 863–871. [ Google Scholar ] [ CrossRef ]
Shrestha, P.P.; Davis, B.; Gad, G.M. Investigation of Legal Issues in Construction-Manager-at-Risk Projects: Case Study of Airport Projects. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2020 , 12 , 04520022. [ Google Scholar ] [ CrossRef ]
Marston, S. CMAR Project Delivery Method Generates Team Orientated Project Management with Win/Win Mentality. In Proceedings of the Pipelines 2020, San Antonio, TX, USA, 9–12 August 2020; pp. 167–170. [ Google Scholar ] [ CrossRef ]
Francom, T.; El Asmar, M.; Ariaratnam, S.T. Ariaratnam, Longitudinal Study of Construction Manager at Risk for Pipeline Rehabilitation. J. Pipeline Syst. Eng. Pract. 2017 , 8 , 04017001. [ Google Scholar ] [ CrossRef ]
Peña-Mora, F.; Tamaki, T. Effect of Delivery Systems on Collaborative Negotiations for Large-Scale Infrastructure Projects. J. Manag. Eng. 2001 , 17 , 105–121. [ Google Scholar ] [ CrossRef ]
Mahdi, I.M.; Alreshaid, K. Decision support system for selecting the proper project delivery method using analytical hierarchy process (AHP). Int. J. Proj. Manag. 2005 , 23 , 564–572. [ Google Scholar ] [ CrossRef ]
Randall, T.; Pool, S.; Limke, J.; Bradney, A. CMaR Delivery of Critical Water and Wastewater Pipelines. In Proceedings of the Pipelines 2020, San Antonio, TX, USA, 9–12 August 2020; Conference Held Virtually. pp. 280–289. [ Google Scholar ] [ CrossRef ]
Perrenoud, A.; Reyes, M.; Ghosh, S.; Coetzee, M. Collaborative Risk Management of the Approval Process of Building Envelope Materials. In Proceedings of the AEI 2017, Oklahoma City, OK, USA, 11–13 April 2017; pp. 806–816. [ Google Scholar ] [ CrossRef ]
Parrott, B.C.; Bomba, M.B. Integrated Project Delivery and Building Information Modeling: A New Breed of Contract. 2010. Available online: https://content.aia.org/sites/default/files/2017-03/Integrated%20project%20delivery%20and%20BIM-%20A%20new%20breed%20of%20contract.pdf (accessed on 18 November 2023).
Cheng, R. IPD Case Studies. Report. March 2012. Available online: http://conservancy.umn.edu/handle/11299/201408 (accessed on 1 May 2024).
Lee, H.W.; Anderson, S.M.; Kim, Y.-W.; Ballard, G. Ballard, Advancing Impact of Education, Training, and Professional Experience on Integrated Project Delivery. Pract. Period. Struct. Des. Constr. 2014 , 19 , 8–14. [ Google Scholar ] [ CrossRef ]
Hoseingholi, M.; Jalal, M.P. Jalal, Identification and Analysis of Owner-Induced Problems in Design–Build Project Lifecycle. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2017 , 9 , 04516013. [ Google Scholar ] [ CrossRef ]
Öztaş, A.; Ökmen, Ö. Risk analysis in fixed-price design–build construction projects. Build. Environ. 2004 , 39 , 229–237. [ Google Scholar ] [ CrossRef ]
Lee, D.-E.; Arditi, D. Total Quality Performance of Design/Build Firms Using Quality Function Deployment. J. Constr. Eng. Manag. 2006 , 132 , 49–57. [ Google Scholar ] [ CrossRef ]
Garner, B.; Richardson, K.; Castro-Lacouture, D. Design-Build Project Delivery in Military Construction: Approach to Best Value Procurement. J. Adv. Perform. Inf. Value 2008 , 1 , 35–50. [ Google Scholar ] [ CrossRef ]
Graham, P. Evaluation of Design-Build Practice in Colorado Project IR IM(CX) 025-3(113) ; Colorado Department of Transportation: Denver, CO, USA, 2001. [ Google Scholar ]
Parami Dewi, A.; Too, E.; Trigunarsyah, B. Implementing design build project delivery system in Indonesian road infrastructure projects. In Innovation and Sustainable Construction in Developing Countries (CIB W107 Conference 2011) ; Uwakweh, B.O., Ed.; Construction Publishing House/International Council for Research and Innovation in Building and C: Hanoi, Vietnam, 2011; pp. 108–117. [ Google Scholar ]
Arditi, D.; Lee, D.-E. Assessing the corporate service quality performance of design-build contractors using quality function deployment. Constr. Manag. Econ. 2003 , 21 , 175–185. [ Google Scholar ] [ CrossRef ]
Rao, T. . Is Design-Build Right for Your Next WWW Project? presented at the WEFTEC 2009, Water Environment Federation. January 2009, pp. 6444–6458. Available online: https://www.accesswater.org/publications/proceedings/-297075/is-design-build-right-for-your-next-www-project- (accessed on 3 April 2024).
Touran, A.; Molenaar, K.R.; Gransberg, D.D.; Ghavamifar, K. Decision Support System for Selection of Project Delivery Method in Transit. Transp. Res. Rec. 2009 , 2111 , 148–157. [ Google Scholar ] [ CrossRef ]
Culp, G. Alternative Project Delivery Methods for Water and Wastewater Projects: Do They Save Time and Money? Leadersh. Manag. Eng. 2011 , 11 , 231–240. [ Google Scholar ] [ CrossRef ]
Ling, F.Y.Y.; Poh, B.H.M. Problems encountered by owners of design–build projects in Singapore. Int. J. Proj. Manag. 2008 , 26 , 164–173. [ Google Scholar ] [ CrossRef ]
Pishdad-Bozorgi, P.; de la Garza, J.M. Comparative Analysis of Design-Bid-Build and Design-Build from the Standpoint of Claims. In Proceedings of the Construction Research Congress 2012, West Lafayette, IN, USA, 21–23 May 2012. [ Google Scholar ] [ CrossRef ]
Walewski, J.; Gibson, G.E., Jr.; Jasper, J. Project Delivery Methods and Contracting Approaches Available for Implementation by the Texas Department of Transportation. University of Texas at Austin. Center for Transportation Research. 2001. Available online: https://rosap.ntl.bts.gov/view/dot/14863 (accessed on 3 April 2024).
Alleman, D.; Antoine, A.; Gransberg, D.D.; Molenaar, K.R. Comparison of Qualifications-Based Selection and Best-Value Procurement for Construction Manager–General Contractor Highway Construction. 2017. Available online: https://journals.sagepub.com/doi/abs/10.3141/2630-08 (accessed on 2 April 2024).
Gransberg, N.J.; Gransberg, D.D. Public Project Construction Manager-at-Risk Contracts: Lessons Learned from a Comparison of Commercial and Infrastructure Projects. J. Leg. Aff. Dispute Resolut. Eng. Constr. 2020 , 12 , 04519039. [ Google Scholar ] [ CrossRef ]
Anderson, S.D.; Damnjanovic, I. Selection and Evaluation of Alternative Contracting Methods to Accelerate Project Completion ; The National Academies Press: Washington, DC, USA, 2008; Available online: http://elibrary.pcu.edu.ph:9000/digi/NA02/2008/23075.pdf (accessed on 26 April 2024).
Shrestha, P.P.; Batista, J.; Maharjan, R. Impediments in Using Design-Build or Construction Management-at-Risk Delivery Methods for Water and Wastewater Projects. In Proceedings of the Construction Research Congress 2016, San Juan, PR, USA, 31 May–2 June 2016; pp. 380–387. [ Google Scholar ] [ CrossRef ]
Chateau, L. Environmental acceptability of beneficial use of waste as construction material—State of knowledge, current practices and future developments in Europe and in France. J. Hazard. Mater. 2007 , 139 , 556–562. [ Google Scholar ] [ CrossRef ]
Lam, T.I.; Chan, H.W.E.; Chau, C.K.; Poon, C.S. An Overview of the Development of Green Specifications in the Construction Industry. In Proceedings of the International Conference on Urban Sustainability [ICONUS], 1 January 2008; pp. 295–301. Available online: https://research.polyu.edu.hk/en/publications/an-overview-of-the-development-of-green-specifications-in-the-con (accessed on 2 May 2024).
Tabish, S.Z.S.; Jha, K.N. Success Traits for a Construction Project. J. Constr. Eng. Manag. 2012 , 138 , 1131–1138. [ Google Scholar ] [ CrossRef ]
Niroumand, H.; Zain, M.; Jamil, M. A guideline for assessing of critical parameters on Earth architecture and Earth buildings as a sustainable architecture in various countries. Renew. Sustain. Energy Rev. 2013 , 28 , 130–165. [ Google Scholar ] [ CrossRef ]
Rogulj, K.; Jajac, N. Achieving a Construction Barrier–Free Environment: Decision Support to Policy Selection. J. Manag. Eng. 2018 , 34 , 04018020. [ Google Scholar ] [ CrossRef ]
Sackey, S.; Kim, B.-S. Environmental and Economic Performance of Asphalt Shingle and Clay Tile Roofing Sheets Using Life Cycle Assessment Approach and TOPSIS. J. Constr. Eng. Manag. 2018 , 144 , 04018104. [ Google Scholar ] [ CrossRef ]
Carretero-Ayuso, M.J.; García-Sanz-Calcedo, J.; Rodríguez-Jiménez, C.E. Rodríguez-Jiménez, Characterization and Appraisal of Technical Specifications in Brick Façade Projects in Spain. J. Perform. Constr. Facil. 2018 , 32 , 04018012. [ Google Scholar ] [ CrossRef ]
Golabchi, A.; Guo, X.; Liu, M.; Han, S.; Lee, S.; AbouRizk, S. An integrated ergonomics framework for evaluation and design of construction operations. Autom. Constr. 2018 , 95 , 72–85. [ Google Scholar ] [ CrossRef ]
Jha, K.; Iyer, K. Commitment, coordination, competence and the iron triangle. Int. J. Proj. Manag. 2007 , 25 , 527–540. [ Google Scholar ] [ CrossRef ]
Tabassi, A.A.; Ramli, M.; Roufechaei, K.M.; Tabasi, A.A. Team development and performance in construction design teams: An assessment of a hierarchical model with mediating effect of compensation. Constr. Manag. Econ. 2014 , 32 , 932–949. [ Google Scholar ] [ CrossRef ]
Chen, Y.; Okudan, G.E.; Riley, D.R. Sustainable performance criteria for construction method selection in concrete buildings. Autom. Constr. 2010 , 19 , 235–244. [ Google Scholar ] [ CrossRef ]
Doloi, H.; Sawhney, A.; Iyer, K.; Rentala, S. Analysing factors affecting delays in Indian construction projects. Int. J. Proj. Manag. 2012 , 30 , 479–489. [ Google Scholar ] [ CrossRef ]
Kog, Y.C.; Loh, P.K. Critical Success Factors for Different Components of Construction Projects. J. Constr. Eng. Manag. 2012 , 138 , 520–528. [ Google Scholar ] [ CrossRef ]
Gunduz, M.; Almuajebh, M. Critical success factors for sustainable construction project management. Sustainability 2020 , 12 , 1990. [ Google Scholar ] [ CrossRef ]
Cao, D.; Li, H.; Wang, G.; Luo, X.; Tan, D. Relationship Network Structure and Organizational Competitiveness: Evidence from BIM Implementation Practices in the Construction Industry. J. Manag. Eng. 2018 , 34 , 04018005. [ Google Scholar ] [ CrossRef ]
Clevenger, C.M. Development of a Project Management Certification Plan for a DOT. J. Manag. Eng. 2018 , 34 , 06018002. [ Google Scholar ] [ CrossRef ]
Bygballe, L.E.; Swärd, A. Collaborative Project Delivery Models and the Role of Routines in Institutionalizing Partnering. Proj. Manag. J. 2019 , 50 , 161–176. [ Google Scholar ] [ CrossRef ]
Collins, W.; Parrish, K. The Need for Integrated Project Delivery in the Public Sector. In Proceedings of the Construction Research Congress 2014, Atlanta, GA, USA, 19–21 May 2014; pp. 719–728. [ Google Scholar ] [ CrossRef ]
Turk, Ž.; Klinc, R. Potentials of Blockchain Technology for Construction Management. Procedia Eng. 2017 , 196 , 638–645. [ Google Scholar ] [ CrossRef ]
Elghaish, F.; Abrishami, S.; Hosseini, M.R. Integrated project delivery with blockchain: An automated financial system. Autom. Constr. 2020 , 114 , 103182. [ Google Scholar ] [ CrossRef ]
Fish, A. Integrated Project Delivery: The Obstacles of Implementation. May 2011. Available online: http://hdl.handle.net/2097/8554 (accessed on 3 April 2024).
Pan, Y.; Zhang, L. Roles of artificial intelligence in construction engineering and management: A critical review and future trends. Autom. Constr. 2020 , 122 , 103517. [ Google Scholar ] [ CrossRef ]
Mellit, A.; Kalogirou, S.A. Artificial intelligence techniques for photovoltaic applications: A review. Prog. Energy Combust. Sci. 2008 , 34 , 574–632. [ Google Scholar ] [ CrossRef ]
Smith, C.J.; Wong, A.T.C. Advancements in Artificial Intelligence-Based Decision Support Systems for Improving Construction Project Sustainability: A Systematic Literature Review. Informatics 2022 , 9 , 43. [ Google Scholar ] [ CrossRef ]
Villa, F. Semantically driven meta-modelling: Automating model construction in an environmental decision support system for the assessment of ecosystem services flows. In Information Technologies in Environmental Engineering ; Athanasiadis, I.N., Rizzoli, A.E., Mitkas, P.A., Gómez, J.M., Eds.; Springer: Berlin, Heidelberg, 2009; pp. 23–36. [ Google Scholar ]
Minhas, M.R.; Potdar, V. Decision Support Systems in Construction: A Bibliometric Analysis. Buildings 2020 , 10 , 108. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Paper	Reference	Total Citation TC	TC Per Year	Normalized TC
Kent D.C., 2010, J Constr Eng Manage	(Kent and Becerik-Gerber, 2010) [ ]	300	21.43	7.67
Ugwu O.O., 2007, Build Environ	(Ugwu and Haupt, 2007) [ ]	269	15.82	7.69
Kines P., 2010, J Saf Res	(Kines et al., 2010) [ ]	238	17.00	6.08
Asmar M., 2013, J Constr Eng Manag	(Asmar et al., 2013) [ ]	226	20.55	5.01
Ballard G., 2008, Lean Constr J	(Ballard, 2008) [ ]	221	13.81	6.85
Hale D.R., 2009, J Constr Eng Manag	(Hale et al., 2009) [ ]	211	14.07	6.95
Bynum P., 2013, J Constr Eng Manag	(Bynum et al., 2013) [ ]	185	16.82	4.11
Ibbs C.W., 2003, J Constr Eng Manag	(Ibbs et al., 2003) [ ]	183	8.71	8.58
Choudry R.M., 2009, J Constr Eng Manag	(Choudhry et al., 2009) [ ]	182	12.13	6.00
Mollaoglu-Korkmaz S., 2013, J Manage Eng	(Mollaoglu-Korkmaz et al., 2013) [ ]	152	13.82	3.37
El Wardani M.A., 2006, J Constr Eng Manag	(El Wardani et al., 2006) [ ]	144	8.00	4.65
Ghassemi R., 2011, Lean Constr J	(Ghassemi and Becerik-Gerber, 2011) [ ]	143	11.00	5.54
Liu J., 2016, J Constr Eng Manag	(Liu et al., 2016) [ ]	140	17.50	5.12
El-Sayegh S.M., 2015, J Manag Eng	(El-Sayegh and Mansour, 2015) [ ]	135	15.00	6.59
Fang C., 2012, Reliab Eng Syst Saf	(Fang et al., 2012) [ ]	131	10.92	4.05
Franz B., 2017, J Constr Eng Manag	(Franz et al., 2017) [ ]	126	18.00	5.56
Kim H., 2016, J Comput Civ Eng	(Kim et al., 2016) [ ]	125	15.63	4.57
Ding L.Y., 2013, Autom Constr	(Ding and Zhou, 2013) [ ]	118	10.73	2.62
Wanberg J., 2013, J Constr Eng Manag	(Wanberg et al., 2013) [ ]	116	10.55	2.57
Shrestha, P.P., 2012, J Constr Eng Manag	(Shrestha et al., 2012) [ ]	112	9.33	3.47
Torabi S.A., 2009, Int J Prod Res	(Torabi and Hassini, 2009) [ ]	105	7.00	3.46
Baradan S., 2006, J Constr Eng Manag	(Baradan and Usmen, 2006) [ ]	99	5.50	3.20
Levitt R.E., 2007, J Constr Eng Manag	(Levitt, 2007) [ ]	97	5.71	2.77
Sullivan J., 2017, J Constr Eng Manag	(Sullivan et al., 2017) [ ]	93	13.29	4.11
Araya F., 2021, Saf Sci	(Araya, 2021) [ ]	92	30.67	9.5

Country	Frequency
USA	584
CHINA	167
UK	101
AUSTRALIA	71
SOUTH KOREA	56
CANADA	51
IRAN	39
MALAYSIA	39
INDIA	30
SOUTH AFRICA	22
SPAIN	22
FINLAND	18
FRANCE	17
DENMARK	16
EGYPT	16
SWEDEN	16
INDONESIA	15
NETHERLANDS	14
NEW ZEALAND	14
BRAZIL	13
GERMANY	13
NIGERIA	13
UNITED ARAB ENIRATES	13
JORDAN	12
SAUDI ARABIA	12

Country	TC	Average Article Citations
USA	4933	23.70
CHINA	1106	18.10
UNITED KINGDOM	763	19.10
HONG KONG	703	37.00
AUSTRALIA	494	21.50
SOUTH KOREA	312	16.00
IRAN	198	52.00
SPAIN	191	15.20
SWEDEN	188	21.20
PAKISTAN	182	20.90
FRANCE	164	182.00
UNITED ARAB EMIRATES	163	32.80
MALAYSIA	154	32.60
INDIA	145	15.40
SINGAPORE	130	13.20
CANADA	107	43.30
ITALY	92	7.60
LEBANON	92	18.40
NETHERLANDS	91	18.40
NORWAY	74	18.20

	IPD Advantages
Advantages	% Percentage of Advantages from Ordered List of Publication	Publication List
Collaborative atmosphere and fairness	79	B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] L = [ ] M = [ ] N = [ ] O = [ ] P = [ ] Q = [ ] R = [ ] S = [ ] T = [ ] U = [ ] V = [ ]
Early involvement of stakeholders	63	B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] L = [ ] M = [ ] N = [ ] O U = [ ] V = [ ] W = [ ]
Promoting trust	25	R = [ ] S = [ ] U = [ ] V = [ ] W = [ ] X = [ ]
Reduce schedule time	42	C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] S = [ ] T = [ ]
Reduce waste	42	C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] S = [ ] T = [ ]
Shared cost, risk reward, and responsibilities	75	C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] S = [ ] T = [ ] U = [ ] V = [ ] W = [ ] X = [ ]
Multi-party agreement and noncompetitive bidding	54	C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] N = [ ] Q = [ ] T = [ ] V = [ ]
Integrated decision-making for designs and shared design responsibilities	38	C = [ ] D = [ ] E = [ ] H = [ ] I = [ ] J = [ ] L = [ ] P = [ ] T = [ ]
Open communication and time management	38	D = [ ] E = [ ] F = [ ] O = [ ] R = [ ] S = [ ] T = [ ] U = [ ] V = [ ]
Reduce project duration and liability by fast-tracking design and construction	25	F = [ ] G = [ ] L = [ ] O = [ ] S = V
Shared manpower and changes in SOW, equipment rentage, and change orders	17	A = [ ] F = [ ] G = [ ] Q = [ ]
Information sharing and technological impact	38	A = [ ] D = [ ] G = KLMPRV
Fast problem resolution through an integrated approach	21	B = [ ] C = [ ] D = [ ] E = [ ] S = [ ]
Lowest cost delivery and project cost	33	A = [ ] C = [ ] F = [ ] G = [ ] L = [ ] P = [ ] Q = [ ] S = [ ] T = [ ] U = [ ]
Improved efficiency and reduced errors	29	B = [ ] C = [ ] F = [ ] L = [ ] Q = [ ] S = [ ] T = [ ]
Combined risk pool estimated maximum price (allowable cost)	17	A = [ ] L = [ ] P = [ ] Q = [ ]
Cooperation innovation and coordination	46	CEFLPQRSTUV
Combined labor material cost estimation, budgeting, and profits	25	A = [ ] D = [ ] P = [ ] S = [ ] T = [ ] U = [ ] V = [ ]
Strengthened relationship and self-governance	17	C = [ ] D = [ ] F = [ ]
Fewer change orders, Schedules, and request for information	21	L = [ ] O = [ ] Q = [ ] T = [ ] V = [ ]
Ordered list of publication A = [ ] B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] L = [ ] M = [ ] N = [ ] O = [ ] P = [ ] Q = [ ] R = [ ] S = [ ] T = [ ] U = [ ] V = [ ] W = [ ] X = [ ]

	DB Advantages
Disadvantages	%Percentage of Advantages from Ordered List of Publication	Publication List
Single point of accountability for the design and construction	39	CDIJMOQRT C = [ ] D = [ ] I = [ ] J = [ ] M = [ ] O = [ ] Q = [ ] R = [ ] T = [ ]
Produces time saving schedule	52	CDHJKLMORSTV C = [ ] D = [ ] H = [ ] J = [ ] K = [ ] L = [ ] M = [ ] O = [ ] R = [ ] S = [ ] T = [ ] V = [ ]
Cost effective projects	39	CKLMNOPQSV C = [ ] K = [ ] L = [ ] M = [ ] N = [ ] O = [ ] P = [ ] Q = [ ] S = [ ] V = [ ]
Design build functions as a single Entity	8	DF D = [ ] F = [ ]
Enhances quality and mitigates design errors	21	F = [ ] J = [ ] S = [ ] V = [ ] W = [ ] F = [ ]
Facilitates teamwork between owner and design builder	30	J = [ ] N = [ ] P = [ ] S = [ ] U = [ ] V = [ ] W = [ ]
Insight into constructability of the design build contractor (Early involvement of contractor)	13	H = [ ] I = [ ] T = [ ]
Enhances fast tracking	4	R = [ ]
Good coordination and decision-making	27	C = [ ] D = [ ] E = [ ] M = [ ] O = [ ] Q = [ ]
Clients’ owner credibility	13	A = [ ] C = [ ] G = [ ]
Dispute reduction mitigates disputes	21	B = [ ] H = [ ] I = [ ] J = [ ] Q = [ ]
Ordered list of publication A = [ ] B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] L = [ ] M = [ ] N = [ ] O = [ ] P = [ ] Q = [ ] R = [ ] S = [ ] T = [ ] U = [ ] V = [ ] W = [ ]

	CMAR Advantages
Advantages	Percentage of Advantages from the Ordered List of Publication	Publication List
Early stakeholder involvement	31	H = [ ] I = [ ] L = [ ] M = [ ] O = [ ]
Fast-tracking cost savings and delivery within budget	50	A = [ ] B = [ ] C = [ ] D = [ ] F = [ ] I = [ ] M = [ ] O = [ ]
Reduce project duration by fast-tracking design and construction	6	C = [ ]
Clients have control over the design details and early knowledge of costs	50	B = [ ] C = [ ] D = [ ] H = [ ] I = [ ] K = [ ] M = [ ] P = [ ]
Mitigates against change order	50	A = [ ] C = [ ] E = [ ] H = [ ] I = [ ] K = [ ] M = [ ] P = [ ]
Provides a GMP by considering the risk of price	31	A = [ ] B = [ ] C = [ ] M = [ ] O = [ ]
Reduces design cost and redesigning cost	25	C = [ ] D = [ ] E = [ ] H = [ ]
Facilitates schedule management	75	B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] M = [ ] N = [ ]
Facilitates cost control and transparency	69	C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] M = [ ] N = [ ]
Single point of responsibility for construction and joint team orientation for accountability	44	A = [ ] B = [ ] E = [ ] F = [ ] I = [ ] M = [ ] N = [ ]
Facilitates Collaboration	25	E = [ ] F = [ ] I = [ ] J = [ ]
Ordered list of publication A = [ ] B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] L = [ ] M = [ ] N = [ ] O = [ ] P = [ ]

	IPD Disadvantages
Disadvantages	% Percentage of Disadvantages from Ordered List of Publication	Publication List
Impossibility of being sued internally over disputes and mistrust, alongside complexities in compensation and resource distribution	42	C = [ ] E = [ ] F = [ ] I = [ ] L = [ ]
Skepticism of the added value of IPD and impossibility of owners’ inability to tap into financial reserves from shared risk funds	50	E = [ ] F = [ ] G = [ ] J = [ ] K = [ ] L = [ ]
Difficulty in deciding scope	17	A = [ ] H = [ ]
Difficulty in deciding target cost/Budgeting	25	A = [ ] D = [ ] H = [ ]
Adversarial team relationships and legality issues	50	B = [ ] C = [ ] D = [ ] F = [ ] K = [ ] L = [ ]
Immature insurance policy for IPD and uneasiness to produce a coordinating document	25	A = [ ] J = [ ] K = [ ]
Fabricated drawings in place of engineering drawings because of too early interactions	8	F = [ ]
High initial cost of investment in setting up IPD team and difficulty in replacing a member of IPD team	16	J = [ ] L = [ ]
Inexperience in initiating/developing an IPD team and knowledge level	16	K = [ ] L = [ ]
Low adoption of IPD due to cultural, financial, and technological barriers	33	E = [ ] F = [ ] K = [ ] L = [ ]
High degree of risks amongst teams coming together for IPD and owners responsible for claims, damages, and expenses (liabilities)	25	D = [ ] F = [ ] L = [ ]
Issues with poor collaboration	8	H = [ ]
Non-adaptability to IPD environment	42	E = [ ] G = [ ] J = [ ] K = [ ] L = [ ]
Ordered list of publication A = [ ] B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] L = [ ]

	DB Disadvantages
Disadvantages	Percentage of Disadvantages from Ordered List of Publication	Publication List
Non-competitive selection of team not dependent on best designs of professionals and general contractors	35	B = [ ] C = [ ] D = [ ] E = [ ] G = [ ] I = [ ] J = [ ] K = [ ] L = [ ] M = [ ] O = [ ] P = [ ] Q = [ ] R = [ ] S = [ ]
Deficient checks, balances, and insurance among the designer, general contractor, and owner	30	A = [ ] B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] L = [ ] M = [ ] N = [ ] U = V
Unfair allocation of risk and high startup cost	40	R = [ ] C = [ ] S = [ ]
Architect/Engineer(A/E) not related to clients/owners with no control over the design requirements. A/E has less control or influence over the final design and project requirements	60	C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] S = [ ]
Owner cannot guarantee the quality of the finished project	35	C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] S = [ ]
Difficulty in defining SOW, and alterations in the designs after the contract and during construction with decrease in time	35	C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] M = [ ] N = [ ]
Difficulty in providing track record for design and construction	40	C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] N = [ ]
Discrepancy in quality control and testing intensive of owner’s viewpoint	25	C = [ ] D = [ ] E = [ ] H = [ ] I = [ ] J = [ ] K = [ ] N = [ ]
Delay in design changes, inflexibility, and the absence of a detailed design	35	D = [ ] E = [ ] F = [ ] O = [ ] R = [ ] S = [ ]
Owner/client needs external support to develop SOW/preliminary design of the project	10	E = [ ] F = [ ] L = [ ] O = [ ] S = [ ]
Increased labour costs and tender prices	5	A = [ ] F = [ ] G = [ ] Q = [ ]
Guaranteed maximum price is established with Incomplete designs and work requirement	25	A = [ ] D = [ ] G = [ ] K = [ ] L = [ ] M = [ ] P = [ ] R = [ ]
Responsibility of contractor for omission and changes in design	20	A = [ ] B = [ ] C = [ ] D = [ ] S = [ ]
Ordered list of publication A = [ ] B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] L = [ ] M = [ ] N = [ ] O = [ ] P = [ ] Q = [ ] R = [ ] S = [ ]

	CMAR Disadvantages
Disadvantages	% Percentage of Advantages from Ordered List of Publication	Publication List
Unclear definition and relationship of roles and responsibilities of CM and design professionals	78	A = [ ] B = [ ] C = [ ] D = [ ] G = [ ] H = [ ] I = [ ]
Difficult to enforce GMP, SOW, and construction based on incomplete documents	67	A = [ ] D = [ ] E = [ ] G = [ ] H = [ ] I = [ ]
Not suitable for small projects or hold trade contractors over GMP tradeoffs and prices	56	B = [ ] C = [ ] G = [ ] H = [ ] I = [ ]
Improper education on CMAR methodology, polices, and regulations	56	E = [ ] F = [ ] G = [ ] H = [ ] I = [ ]
Knowledge, conflicts, and communication issues between the designer and the CM	56	B = [ ] E = [ ] F = [ ] G = [ ] H = [ ]
Shift of responsibilities (including money) from owners/clients to CM	44	A = [ ] B = [ ] E = [ ] I = [ ]
Additional cost due to design and construction and design defects	56	A = [ ] C = [ ] D = [ ] G = [ ] H = [ ]
Inability of CMAR to self-perform during preconstruction	11	C = [ ]
Disputes/issues concerning construction quality and the completeness of the design	22	A = [ ] D = [ ]
No information exchange/alignment between the A/E with the CMAR	11	A = [ ]
Ordered list of publication A = [ ] B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ]

Critical Success Factors for Sustainable Construction
Advantages	Percentage of Advantages from Ordered List of Publication %	Publication List
Collaborative atmosphere	47	A = [ ] C = [ ] G = [ ] H = [ ] K = [ ] N = [ ] O = [ ]
Early stakeholder involvement	26	N = [ ] J = [ ] I = [ ]
Reduce design errors	13	N = [ ] O = [ ]
Cost savings and delivery within budget/Client representative	33	ABCEF A = [ ] B = [ ] C = [ ]
Influence of client	13	B = [ ] J = [ ]
Ordered list of publication A = [ ] B = [ ] C = [ ] D = [ ] E = [ ] F = [ ] G = [ ] H = [ ] I = [ ] J = [ ] K = [ ] L = [ ] M = [ ] N = [ ] O = [ ] P = [ ] Q = [ ]

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

Babalola, O.G.; Alam Bhuiyan, M.M.; Hammad, A. Literature Review on Collaborative Project Delivery for Sustainable Construction: Bibliometric Analysis. Sustainability 2024 , 16 , 7707. https://doi.org/10.3390/su16177707

Babalola OG, Alam Bhuiyan MM, Hammad A. Literature Review on Collaborative Project Delivery for Sustainable Construction: Bibliometric Analysis. Sustainability . 2024; 16(17):7707. https://doi.org/10.3390/su16177707

Babalola, Olabode Gafar, Mohammad Masfiqul Alam Bhuiyan, and Ahmed Hammad. 2024. "Literature Review on Collaborative Project Delivery for Sustainable Construction: Bibliometric Analysis" Sustainability 16, no. 17: 7707. https://doi.org/10.3390/su16177707

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Open access
Published: 04 September 2024

Insights into research activities of senior dental students in the Middle East: A multicenter preliminary study

Mohammad S. Alrashdan 1 , 2 ,
Abubaker Qutieshat 3 , 4 ,
Mohamed El-Kishawi 5 ,
Abdulghani Alarabi 6 ,
Lina Khasawneh 7 &
Sausan Al Kawas 1

BMC Medical Education volume 24 , Article number: 967 ( 2024 ) Cite this article

Metrics details

Despite the increasing recognition of the importance of research in undergraduate dental education, limited studies have explored the nature of undergraduate research activities in dental schools in the Middle East region. This study aimed to evaluate the research experience of final year dental students from three dental schools in the Middle East.

A descriptive, cross-sectional study was conducted among final-year dental students from three institutions, namely Jordan University of Science and Technology, University of Sharjah (UAE), and Oman Dental College. Participants were asked about the nature and scope of their research projects, the processes involved in the research, and their perceived benefits of engaging in research.

A total of 369 respondents completed the questionnaire. Cross-sectional studies represented the most common research type (50.4%), with public health (29.3%) and dental education (27.9%) being the predominant domains. More than half of research proposals were developed via discussions with instructors (55.0%), and literature reviews primarily utilized PubMed (70.2%) and Google Scholar (68.5%). Regarding statistical analysis, it was usually carried out with instructor’s assistance (45.2%) or using specialized software (45.5%). The students typically concluded their projects with a manuscript (58.4%), finding the discussion section most challenging to write (42.0%). The research activity was considered highly beneficial, especially in terms of teamwork and communication skills, as well as data interpretation skills, with 74.1% of students reporting a positive impact on their research perspectives.

Conclusions

The research experience was generally positive among surveyed dental students. However, there is a need for more diversity in research domains, especially in qualitative studies, greater focus on guiding students in research activities s, especially in manuscript writing and publication. The outcomes of this study could provide valuable insights for dental schools seeking to improve their undergraduate research activities.

Peer Review reports

Introduction

The importance of research training for undergraduate dental students cannot be overstressed and many reports have thoroughly discussed the necessity of incorporating research components in the dental curricula [ 1 , 2 , 3 , 4 ]. A structured research training is crucial to ensure that dental graduates will adhere to evidence-based practices and policies in their future career and are able to critically appraise the overwhelming amount of dental and relevant medical literature so that only rigorous scientific outcomes are adopted. Furthermore, a sound research background is imperative for dental graduates to overcome some of the reported barriers to scientific evidence uptake. This includes the lack of familiarity or uncertain applicability and the lack of agreement with available evidence [ 5 ]. There is even evidence that engagement in research activities can improve the academic achievements of students [ 6 ]. Importantly, many accreditation bodies around the globe require a distinct research component with clear learning outcomes to be present in the curriculum of the dental schools [ 1 ].

Research projects and courses have become fundamental elements of modern biomedical education worldwide. The integration of research training in biomedical academic programs has evolved over the years, reflecting the growing recognition of research as a cornerstone of evidence-based practice [ 7 ]. Notwithstanding the numerous opportunities presented by the inclusion of research training in biomedical programs, it poses significant challenges such as limited resources, varying levels of student preparedness, and the need for faculty development in research mentorship [ 8 , 9 ]. Addressing these challenges is essential to maximize the benefits of research training and to ensure that all students can engage meaningfully in research activities.

While there are different models for incorporating research training into biomedical programs, including dentistry, almost all models share the common goals of equipping students with basic research skills and techniques, critical thinking training and undertaking research projects either as an elective or a summer training course, or more commonly as a compulsory course required for graduation [ 2 , 4 , 10 ].

Dental colleges in the Middle East region are not an exception and most of these colleges are continuously striving to update their curricula to improve the undergraduate research component and cultivate a research-oriented academic teaching environment. Despite these efforts, there remains a significant gap in our understanding of the nature and scope of student-led research in these institutions, the challenges they face, and the perceived benefits of their research experiences. Furthermore, a common approach in most studies in this domain is to confine data collection to a single center from a single country, which in turn limits the value of the outcomes. Therefore, it is of utmost importance to conduct studies with representative samples and preferably multiple institutions in order to address the existing knowledge gaps, to provide valuable insights that can inform future curricular improvements and to support the development of more effective research training programs in dental education across the region. Accordingly, this study was designed and conducted to elucidate some of these knowledge gaps.

The faculty of dentistry at Jordan University of Science and Technology (JUST) is the biggest in Jordan and adopts a five-year bachelor’s program in dental surgery (BDS). The faculty is home to more than 1600 undergraduate and 75 postgraduate students. The college of dental medicine at the University of Sharjah (UoS) is also the biggest in the UAE, with both undergraduate and postgraduate programs, local and international accreditation and follows a (1 + 5) program structure, whereby students need to finish a foundation year and then qualify for the five-year BDS program. Furthermore, the UoS dental college applies an integrated stream-based curriculum. Finally, Oman Dental College (ODC) is the sole dental school in Oman and represents an independent college that does not belong to a university body.

The aim of this study was to evaluate the research experience of final year dental students from three major dental schools in the Middle East, namely JUST from Jordan, UoS from the UAE, and ODC from Oman. Furthermore, the hypothesis of this study was that research activities conducted at dental schools has no perceived benefit for final year dental students.

The rationale for selecting these three dental schools stems from the diversity in the dental curriculum and program structure as well as the fact that final year BDS students are required to conduct a research project as a prerequisite for graduation in the three schools. Furthermore, the authors from these dental schools have a strong scholarly record and have been collaborating in a variety of academic and research activities.

Materials and methods

The current study is a population-based descriptive cross-sectional observational study. The study was conducted using an online self-administered questionnaire and targeted final-year dental students at three dental schools in the Middle East region: JUST from Jordan, UoS from the UAE, and ODC from Oman. The study took place in the period from January to June 2023.

For inclusion in the study, participants should have been final-year dental students at the three participating schools, have finished their research project and agreed to participate. Exclusion criteria included any students not in their final year, those who have not conducted or finished their research projects and those who refused to participate.

The study was approved by the institutional review board of JUST (Reference: 724–2022), the research ethics committee of the UoS (Reference: REC-22-02-22-3) as well as ODC (Reference: ODC-MA-2022-166). The study adhered to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines [ 11 ]. The checklist is available as a supplementary file.

Sample size determination was based on previous studies with a similar design and was further confirmed with a statistical formula. A close look at the relevant literature reveals that such studies were either targeting a single dental or medical school or multiple schools and the sample size generally ranged from 158 to 360 [ 4 , 8 , 9 , 10 , 12 ]. Furthermore, to confirm the sample size, the following 2-step formula for finite population sample size calculation was used [ 13 ]:

Wherein Z is the confidence level at 95% =1.96, P is the population proportion = 0.5, and E is the margin of error = 0.05. Based on this formula, the resultant initial sample size was 384.

Wherein n is the initial sample size = 384, N is the total population size (total number of final year dental students in the 3 schools) = 443. Based on this formula, the adjusted sample size was 206.

An online, self-administered questionnaire comprising 13 questions was designed to assess the research experience of final year dental students in the participating schools. The questionnaire was initially prepared by the first three authors and was then reviewed and approved by the other authors. The questionnaire was developed following an extensive review of relevant literature to identify the most critical aspects of research projects conducted at the dental or medical schools and the most common challenges experienced by students with regards to research project design, research components, attributes, analysis, interpretation, drafting, writing, and presentation of the final outcomes.

The questionnaire was then pretested for both face and content validity. Face validity was assessed by a pilot study that evaluated clarity, validity, and comprehensiveness in a small cohort of 30 students. Content validity was assessed by the authors, who are all experienced academics with remarkable research profiles and experience in supervising undergraduate and postgraduate research projects. The authors critically evaluated each item and made the necessary changes whenever required. Furthermore, Cronbach’s alpha was used to assess the internal consistency/ reliability of the questionnaire and the correlation between the questionnaire items was found to be 0.79. Thereafter, online invitations along with the questionnaire were sent out to a total of 443 students, 280 from JUST, 96 from UoS and 67 from ODC, which represented the total number of final year students at the three schools. A first reminder was sent 2 weeks later, and a second reminder was sent after another 2 weeks.

In addition to basic demographic details, the questionnaire comprised questions related to the type of study conducted, the scope of the research project, whether the research project was proposed by the students or the instructors or both, the literature review part of the project, the statistical analysis performed, the final presentation of the project, the writing up of the resultant manuscript if applicable, the perceived benefits of the research project and finally suggestions to improve the research component for future students.

The outcomes of the study were the students’ research experience in terms of research design, literature review, data collection, analysis, interpretation and presentation, students’ perceived benefits from research, students’ perspective towards research in their future career and students’ suggestions to improve their research experience.

The exposures were the educational and clinical experience of students, research supervision by mentors and faculty members, and participation in extracurricular activities, while the predictors were the academic performance of students, previous research experience and self-motivation.

The collected responses were entered into a Microsoft Excel spreadsheet and analyzed using SPSS Statistics software, version 20.0 (SPSS Inc., Chicago, IL, USA). Descriptive data were presented as frequencies and percentages. For this study, only descriptive statistics were carried out as the aim was not to compare and contrast the three schools but rather to provide an overview of the research activities at the participating dental schools.

The heatmap generated to represent the answers for question 11 (perceived benefits of the research activity) was created using Python programming language (Python 3.11) and the pandas, seaborn, and matplotlib libraries. The heatmap was customized to highlight the count and percentage of responses in each component, with the highest values shown in red and the lowest values shown in blue.

Potentially eligible participants in this study were all final year dental students at the three dental schools (443 students, 280 from JUST, 96 from UoS and 67 from ODC). All potentially eligible participants were confirmed to be eligible and were invited to participate in the study.

The total number of participants included in the study, i.e. the total number of students who completed the questionnaire and whose responses were analyzed, was 369 (223 from JUST, 80 from UoS and 66 from ODC). The overall response rate was 83.3% (79.6% from JUST, 83.3% from UoS and 98.5% from ODC).

The highest proportion of participants were from JUST ( n = 223, 60.4%), followed by UoS ( n = 80, 21.7%), and then ODC ( n = 66, 17.9%). The majority of the participants were females ( n = 296, 80.4%), while males represented a smaller proportion ( n = 73, 19.6%). It is noteworthy that these proportions reflect the size of the cohorts in each college.

With regards to the type of study, half of final-year dental students in the 3 colleges participated in observational cross-sectional studies (i.e., population-based studies) ( n = 186, 50.4%), while literature review projects were the second most common type ( n = 83, 22.5%), followed by experimental studies ( n = 55, 14.9%). Longitudinal studies randomized controlled trials, and other types of studies (e.g., qualitative studies, case reports) were less common, with ( n = 5, 1.4%), ( n = 10, 2.7%), and ( n = 30, 8.1%) participation rates, respectively. Distribution of study types within each college is shown Fig. 1 .

Distribution in percent of study types within each college. JUST: Jordan University of Science and Technology, UOS: University of Sharjah, ODC: Oman Dental College

The most common scope of research projects among final-year dental students was in public health/health services ( n = 108, 29.3%) followed by dental education/attitudes of students or faculty ( n = 103, 27.9%) (Fig. 2 ). Biomaterials/dental materials ( n = 62, 16.8%) and restorative dentistry ( n = 41, 11.1%) were also popular research areas. Oral diagnostic sciences (oral medicine/oral pathology/oral radiology) ( n = 28, 7.6%), oral surgery ( n = 12, 3.2%) and other research areas ( n = 15, 4.1%) were less common among the participants. Thirty-two students (8.7%) were engaged in more than one research project.

Percentages of the scope of research projects among final-year dental students. JUST: Jordan University of Science and Technology, UOS: University of Sharjah, ODC: Oman Dental College

The majority of research projects were proposed through a discussion and agreement between the students and the instructor (55.0%). Instructors proposed the topic for 36.6% of the research projects, while students proposed the topic for the remaining 8.4% of the projects.

Most dental students (79.1%) performed the literature review for their research projects using internet search engines. Material provided by the instructor was used for the literature review by 15.5% of the students, while 5.4% of the students did not perform a literature review. More than half of the students ( n = 191, 51.7%) used multiple search engines in their literature search. The most popular search engines for literature review among dental students were PubMed (70.2% of cases) and Google Scholar (68.5% of cases). Scopus was used by 12.8% of students, while other search engines were used by 15.6% of students.

The majority of dental students ( n = 276, 74.8%) did not utilize the university library to gain access to the required material for their research. In contrast, 93 students (25.2%) reported using the university library for this purpose.

Dental students performed statistical analysis in their projects primarily by receiving help from the instructor ( n = 167, 45.2%) or using specialized software ( n = 168, 45.5%). A smaller percentage of students ( n = 34, 9.4%) consulted a professional statistician for assistance with statistical analysis. at the end of the research project, 58.4% of students ( n = 215) presented their work in the form of a manuscript or scientific paper. Other methods of presenting the work included PowerPoint presentations ( n = 80, 21.7%) and discussions with the instructor ( n = 74, 19.8%).

For those students who prepared a manuscript at the conclusion of their project, the most difficult part of the writing-up was the discussion section ( n = 155, 42.0%), followed by the methodology section ( n = 120, 32.5%), a finding that was common across the three colleges. Fewer students found the introduction ( n = 13, 3.6%) and conclusion ( n = 10, 2.7%) sections to be challenging. Additionally, 71 students (19.2%) were not sure which part of the manuscript was the most difficult to prepare (Fig. 3 ).

Percentages of the most difficult part reported by dental students during the writing-up of their projects. JUST: Jordan University of Science and Technology, UOS: University of Sharjah, ODC: Oman Dental College

The dental students’ perceived benefits from the research activity were evaluated across seven components, including literature review skills, research design skills, data collection and interpretation, manuscript writing, publication, teamwork and effective communication, and engagement in continuing professional development.

The majority of students found the research activity to be beneficial or highly beneficial in most of the areas, with the highest ratings observed in teamwork and effective communication, where 33.5% rated it as beneficial and 32.7% rated it as highly beneficial. Similarly, in the area of data collection and interpretation, 33.0% rated it as beneficial and 27.5% rated it as highly beneficial. In the areas of literature review skills and research design skills, 28.6% and 34.0% of students rated the research activity as beneficial, while 25.3% and 22.7% rated it as highly beneficial, respectively. Students also perceived the research activity to be helpful for the manuscript writing, with 27.9% rating it as beneficial and 19.2% rating it as highly beneficial.

When it comes to publication, students’ perceptions were more variable, with 22.0% rating it as beneficial and 11.3% rating it as highly beneficial. A notable 29.9% rated it as neutral, and 17.9% reported no benefit. Finally, in terms of engaging in continuing professional development, 26.8% of students rated the research activity as beneficial and 26.2% rated it as highly beneficial (Fig. 4 ).

Heatmap of the dental students’ perceived benefits from the research activity

The research course’s impact on students’ perspectives towards being engaged in research activities or pursuing a research career after graduation was predominantly positive, wherein 274 students (74.1%) reported a positive impact on their research perspectives. However, 79 students (21.5%) felt that the course had no impact on their outlook towards research engagement or a research career. A small percentage of students ( n = 16, 4.4%) indicated that the course had a negative impact on their perspective towards research activities or a research career after graduation.

Finally, when students were asked about their suggestions to improve research activities, they indicated the need for more training and orientation ( n = 127, 34.6%) as well as to allow more time for students to finish their research projects ( n = 87, 23.6%). Participation in competitions and more generous funding were believed to be less important factors to improve students` research experience ( n = 78, 21.2% and n = 63, 17.1%, respectively). Other factors such as external collaborations and engagement in research groups were even less important from the students` perspective (Fig. 5 ).

Precentages of dental students’ suggestions to improve research activities at their colleges

To the best of our knowledge, this report is the first to provide a comprehensive overview of the research experience of dental students from three leading dental colleges in the Middle East region, which is home to more than 50 dental schools according to the latest SCImago Institutions Ranking ® ( https://www.scimagoir.com ). The reasonable sample size and different curricular structure across the participating colleges enhanced the value of our findings not only for dental colleges in the Middle East, but also to any dental college seeking to improve and update its undergraduate research activities. However, it is noteworthy that since the study has included only three dental schools, the generalizability of the current findings would be limited, and the outcomes are preliminary in nature.

Cross-sectional (epidemiological) studies and literature reviews represented the most common types of research among our cohort of students, which can be attributed to the feasibility, shorter time and low cost required to conduct such research projects. On the contrary, longitudinal studies and randomized trials, both known to be time consuming and meticulous, were the least common types. These findings concur with previous reports, which demonstrated that epidemiological studies are popular among undergraduate research projects [ 4 , 10 ]. In a retrospective study, Nalliah et al. also demonstrated a remarkable increase in epidemiological research concurrent with a decline in the clinical research in dental students` projects over a period of 4 years [ 4 ]. However, literature reviews, whether systematic or scoping, were not as common in some dental schools as in our cohort. For instance, a report from Sweden showed that literature reviews accounted for less than 10% of total dental students` projects [ 14 ]. Overall, qualitative research was seldom performed among our cohort, which is in agreement with a general trend in dental research that has been linked to the low level of competence and experience of dental educators to train students in qualitative research, as this requires special training in social research [ 15 , 16 ].

In terms of the research topics, public health research, research in dental education and attitudinal research were the most prevalent among our respondents. In agreement with our results, research in health care appears common in dental students` projects [ 12 ]. In general, these research domains may reflect the underlying interests of the faculty supervisors, who, in our case, were actively engaged in the selection of the research topic for more than 90% of the projects. Other areas of research, such as clinical dentistry and basic dental research are also widely reported [ 4 , 10 , 14 , 17 ].

The selection of a research domain is a critical step in undergraduate research projects, and a systematic approach in identifying research gaps and selecting appropriate research topics is indispensable and should always be given an utmost attention by supervisors [ 18 ].

More than half of the projects in the current report were reasonably selected based on a discussion between the students and the supervisor, whereas 36% were selected by the supervisors. Otuyemi et al. reported that about half of undergraduate research topics in a Nigerian dental school were selected by students themselves, however, a significant proportion of these projects (20%) were subsequently modified by supervisors [ 19 ]. The autonomy in selecting the research topic was discussed in a Swedish report, which suggested that such approach can enhance the learning experience of students, their motivation and creativity [ 20 ]. Flexibility in selecting the research topic as well as the faculty supervisor, whenever feasible, should be offered to students in order to improve their research experience and gain better outcomes [ 12 ].

Pubmed and Google Scholar were the most widely used search engines for performing a literature review. This finding is consistent with recent reviews which classify these two search systems as the most commonly used ones in biomedical research despite some critical limitations [ 21 , 22 ]. It is noteworthy that students should be competent in critical appraisal of available literature to perform the literature review efficiently. Interestingly, only 25% of students used their respective university library`s access to the search engines, which means that most students retrieved only open access publications for their literature reviews, a finding that requires attention from faculty mentors to guide students to utilize the available library services to widen their accessibility to available literature.

Statistical analysis has classically been viewed as a perceived obstacle for undergraduate students to undertake research in general [ 23 , 24 ] and recent literature has highlighted the crucial need of biomedical students to develop necessary competencies in biostatistics during their studies [ 25 ]. One obvious advantage of conducting research in our cohort is that 45.5% of students used a specialized software to analyze their data, which means that they did have at least an overview of how data are processed and analyzed to reach their final results and inferences. Unfortunately, the remaining 54.5% of students were, partially or completely, dependent on the supervisor or a professional statistician for data analysis. It is noteworthy that the research projects were appropriately tailored to the undergraduate level, focusing on fundamental statistical analysis methods. Therefore, consulting a professional statistician for more complex analyses was done only if indicated, which explains the small percentage of students who consulted a professional statistician.

Over half of participating students (58.4%) prepared a manuscript at the end of their research projects and for these students, the discussion section was identified as the most challenging to prepare, followed by the methodology section. These findings can be explained by the students’ lack of knowledge and experience related to conducting and writing-up scientific research. The same was reported by Habib et al. who found dental students’ research knowledge to be less than that of medical students [ 26 ]. The skills of critical thinking and scientific writing are believed to be of paramount importance to biomedical students and several strategies have been proposed to enhance these skills especially for both English and non-English speaking students [ 27 , 28 , 29 ].

Dental students in the current study reported positive attitude towards research and found the research activity to be beneficial in several aspects of their education, with the most significant benefits in the areas of teamwork, effective communication, data collection and interpretation, literature review skills, and research design skills. Similar findings were reported by previous studies with most of participating students reporting a positive impact of their research experience [ 4 , 10 , 12 , 30 ]. Furthermore, 74% of students found that their research experience had a positive impact on their perspectives towards engagement in research in the future. This particular finding may be promising in resolving a general lack of interest in research by dental students, as shown in a previous report from one of the participating colleges in this study (JUST), which demonstrated that only 2% of students may consider a research career in the future [ 31 ].

Notably, only 11.3% of our students perceived their research experience as being highly beneficial with regards to publication. Students` attitudes towards publishing their research appear inconsistent in literature and ranges from highly positive rates in developed countries [ 4 ] to relatively low rates in developing countries [ 8 , 32 , 33 ]. This can be attributed to lack of motivation and poor training in scientific writing skills, a finding that has prompted researchers to propose strategies to tackle such a gap as mentioned in the previous section.

Finally, key suggestions by the students to improve the research experience were the provision of more training and orientation, more time to conduct the research, as well as participation in competitions and more funding opportunities. These findings are generally in agreement with previous studies which demonstrated that dental students perceived these factors as potential barriers to improving their research experience [ 8 , 10 , 17 , 30 , 34 ].

A major limitation of the current study is the inclusion of only three dental schools from the Middle East which my limit the generalizability and validity of the findings. Furthermore, the cross-sectional nature of the study would not allow definitive conclusions to be drawn as students’ perspectives were not evaluated before and after the research project. Potential confounders in the study include the socioeconomic status of the students, the teaching environment, previous research experience, and self-motivation. Moreover, potential sources of bias include variations in the available resources and funding to students’ projects and variations in the quality of supervision provided. Another potential source of bias is the non-response bias whereby students with low academic performance or those who were not motivated might not respond to the questionnaire. This potential source of bias was managed by sending multiple reminders to students and aiming for the highest response rate and largest sample size possible.

In conclusion, the current study evaluated the key aspects of dental students’ research experience at three dental colleges in the Middle East. While there were several perceived benefits, some aspects need further reinforcement and revision including the paucity of qualitative and clinical research, the need for more rigorous supervision from mentors with focus on scientific writing skills and research presentation opportunities. Within the limitations of the current study, these outcomes can help in designing future larger scale studies and provide valuable guidance for dental colleges to foster the research component in their curricula. Further studies with larger and more representative samples are required to validate these findings and to explore other relevant elements in undergraduate dental research activities.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Emrick JJ, Gullard A. Integrating research into dental student training: a global necessity. J Dent Res. 2013;92(12):1053–5.

Article Google Scholar

Ramachandra SS. A comprehensive template for inclusion of research in the undergraduate dental curriculum. Health Professions Educ. 2020;6(2):264–70.

Al Sweleh FS. Integrating scientific research into undergraduate curriculum: a new direction in dental education. J Health Spec. 2016;4(1):42–5.

Nalliah RP, Lee MK, Da Silva JD, Allareddy V. Impact of a research requirement in a dental school curriculum. J Dent Educ. 2014;78(10):1364–71.

Lang ES, Wyer PC, Haynes RB. Knowledge translation: closing the evidence-to-practice gap. Ann Emerg Med. 2007;49(3):355–63.

Fechheimer M, Webber K, Kleiber PB. How well do undergraduate research programs promote engagement and success of students? CBE Life Sci Educ. 2011;10(2):156–63.

Kingsley K, O’Malley S, Stewart T, Howard KM. Research enrichment: evaluation of structured research in the curriculum for dental medicine students as part of the vertical and horizontal integration of biomedical training and discovery. BMC Med Educ. 2008;8:1–10.

Alsaleem SA, Alkhairi MAY, Alzahrani MAA, Alwadai MI, Alqahtani SSA, Alaseri YFY, et al. Challenges and Barriers toward Medical Research among Medical and Dental students at King Khalid University, Abha, Kingdom of Saudi Arabia. Front Public Health. 2021;9:706778.

Soe HHK, Than NN, Lwin H, Htay MNNN, Phyu KL, Abas AL. Knowledge, attitudes, and barriers toward research: the perspectives of undergraduate medical and dental students. J Educ Health Promotion. 2018;7(1):23.

Amir LR, Soekanto SA, Julia V, Wahono NA, Maharani DA. Impact of Undergraduate Research as a compulsory course in the Dentistry Study Program Universitas Indonesia. Dent J (Basel). 2022;10(11).

Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The strengthening the reporting of Observational studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370(9596):1453–7.

Van der Groen TA, Olsen BR, Park SE. Effects of a Research Requirement for Dental students: a retrospective analysis of students’ perspectives across ten years. J Dent Educ. 2018;82(11):1171–7.

Althubaiti A. Sample size determination: a practical guide for health researchers. J Gen Family Med. 2023;24(2):72–8.

Franzén C. The undergraduate degree project–preparing dental students for professional work and postgraduate studies? Eur J Dent Educ. 2014;18(4):207–13.

Edmunds S, Brown G. Doing qualitative research in dentistry and dental education. Eur J Dent Educ. 2012;16(2):110–7.

Moreno X. Research training in dental undergraduate curriculum in Chile. J Oral Res. 2014;3(2):95–9.

Liu H, Gong Z, Ye C, Gan X, Chen S, Li L, et al. The picture of undergraduate dental basic research education: a scoping review. BMC Med Educ. 2022;22(1):569.

Omar A, Elliott E, Sharma S. How to undertake research as a dental undergraduate. BDJ Student. 2021;28(3):17–8.

Otuyemi OD, Olaniyi EA. A 5-year retrospective evaluation of undergraduate dental research projects in a Nigerian University: graduates’ perceptions of their learning experiences. Eur J Dent Educ. 2020;24(2):292–300.

Franzén C, Brown G. Undergraduate degree projects in the Swedish dental schools: a documentary analysis. Eur J Dent Educ. 2013;17(2):122–6.

Thakre SB, Golawar SH, Thakr SS, Gawande AV. Search engines use for effective literature search in biomedical research. 2014.

Gusenbauer M, Haddaway NR. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res Synthesis Methods. 2020;11(2):181–217.

Lorton L, Rethman MP. Statistics: curse of the writing class. J Endod. 1990;16(1):13–8.

Leppink J. Helping medical students in their study of statistics: a flexible approach. J Taibah Univ Med Sci. 2017;12(1):1–7.

Google Scholar

Oster RA, Enders FT. The Importance of Statistical Competencies for Medical Research Learners. J Stat Educ. 2018;26(2):137–42.

Habib SR, AlOtaibi SS, Abdullatif FA, AlAhmad IM. Knowledge and attitude of undergraduate Dental students towards Research. J Ayub Med Coll Abbottabad. 2018;30(3):443–8.

Florek AG, Dellavalle RP. Case reports in medical education: a platform for training medical students, residents, and fellows in scientific writing and critical thinking. J Med Case Rep. 2016;10:86.

Wortman-Wunder E, Wefes I. Scientific writing workshop improves confidence in critical writing skills among trainees in the Biomedical sciences. J Microbiol Biol Educ. 2020;21(1).

Barroga E, Mitoma H. Critical thinking and scientific writing skills of Non-anglophone Medical students: a model of Training Course. J Korean Med Sci. 2019;34(3):e18.

Kyaw Soe HH, Than NN, Lwin H, Nu Htay MNN, Phyu KL, Abas AL. Knowledge, attitudes, and barriers toward research: the perspectives of undergraduate medical and dental students. J Educ Health Promot. 2018;7:23.

Alrashdan MS, Alazzam M, Alkhader M, Phillips C. Career perspectives of senior dental students from different backgrounds at a single Middle Eastern institution. BMC Med Educ. 2018;18(1):283.

Chellaiyan VG, Manoharan A, Jasmine M, Liaquathali F. Medical research: perception and barriers to its practice among medical school students of Chennai. J Educ Health Promot. 2019;8:134.

Jeelani W, Aslam SM, Elahi A. Current trends in undergraduate medical and dental research: a picture from Pakistan. J Ayub Med Coll Abbottabad. 2014;26(2):162–6.

Yu W, Sun Y, Miao M, Li L, Zhang Y, Zhang L, et al. Eleven-year experience implementing a dental undergraduate research programme in a prestigious dental school in China: lessons learned and future prospects. Eur J Dent Educ. 2021;25(2):246–60.

Download references

Acknowledgements

The authors would like to acknowledge final year dental students at the three participating colleges for their time completing the questionnaire.

No funding was received for this study.

Author information

Authors and affiliations.

Department of Oral and Craniofacial Health Sciences, College of Dental Medicine, University of Sharjah, P.O.Box: 27272, Sharjah, UAE

Mohammad S. Alrashdan & Sausan Al Kawas

Department of Oral Medicine and Oral Surgery, Faculty of Dentistry, Jordan University of Science and Technology, Irbid, Jordan

Mohammad S. Alrashdan

Department of Adult Restorative Dentistry, Oman Dental College, Muscat, Sultanate of Oman

Abubaker Qutieshat

Department of Restorative Dentistry, Dundee Dental Hospital & School, University of Dundee, Dundee, UK

Preventive and Restorative Dentistry Department, College of Dental Medicine, University of Sharjah, Sharjah, UAE

Mohamed El-Kishawi

Clinical Sciences Department, College of Dentistry, Ajman University, Ajman, UAE

Abdulghani Alarabi

Department of Prosthodontics, Faculty of Dentistry, University of Science and Technology, Irbid, Jordan

Lina Khasawneh

You can also search for this author in PubMed Google Scholar

Contributions

M.A.: Conceptualization, data curation, project administration; supervision, validation, writing - original draft; writing - review and editing. A.Q: Conceptualization, data curation, project administration; writing - review and editing. M.E: Conceptualization, data curation, project administration; validation, writing - original draft; writing - review and editing. A.A.: data curation, writing - original draft; writing - review and editing. L.K.: Conceptualization, data curation, validation, writing - original draft; writing - review and editing. S.A: Conceptualization, writing - review and editing.

Corresponding author

Correspondence to Mohammad S. Alrashdan .

Ethics declarations

Ethics approval and consent to participate.

The current study was approved by the institutional review board of Jordan University of Science and Technology (Reference: 724–2022), the research ethics committee of the University of Sharjah (Reference: REC-22-02-22-3) and Oman Dental College (Reference: ODC-MA-2022-166).

Informed consent

Agreement to the invitation to fill out the questionnaire was considered as an implied consent to participate.

Consent for publication

not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Alrashdan, M.S., Qutieshat, A., El-Kishawi, M. et al. Insights into research activities of senior dental students in the Middle East: A multicenter preliminary study. BMC Med Educ 24 , 967 (2024). https://doi.org/10.1186/s12909-024-05955-5

Download citation

Received : 12 August 2023

Accepted : 24 August 2024

Published : 04 September 2024

DOI : https://doi.org/10.1186/s12909-024-05955-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Dental research
Dental education
Undergraduate
Literature review
Publication

BMC Medical Education

ISSN: 1472-6920

General enquiries: [email protected]

Open access
Published: 06 September 2024

Association between pregnancy intention and completion of newborn and infant continuum of care in Sub-Saharan Africa: systematic review and meta-analysis

Birye Dessalegn Mekonnen ORCID: orcid.org/0000-0003-3879-1330 1 , 2 ,
Vidanka Vasilevski ORCID: orcid.org/0000-0002-2772-811X 1 , 3 ,
Ayele Geleto Bali ORCID: orcid.org/0000-0001-5139-6568 1 , 3 &
Linda Sweet ORCID: orcid.org/0000-0003-0605-1186 1 , 3

BMC Pediatrics volume 24 , Article number: 567 ( 2024 ) Cite this article

Metrics details

The newborn and infant continuum of care such as essential newborn care, early initiation and exclusive breastfeeding, and immunisation are highly recommended for improving the quality of life and survival of infants. However, newborn and infant mortality remains high across Sub-Saharan African countries. While unintended pregnancies are associated with adverse newborn and infant health outcomes, there is inconclusive evidence on whether pregnancy intention influences newborn and infant continuum of care completion. Therefore, this review aimed to pool findings reported in the literature on the association between pregnancy intention and newborn and infant health care across the continuum of care in Sub-Saharan Africa.

We searched MEDLINE Complete, EMBASE, CINAHL Complete, and Global Health databases for studies potentially eligible for this systematic review and meta-analysis. Two researchers independently screened the identified articles by abstract and title, and then full-text using Covidence. We used the Newcastle–Ottawa Scale to assess the quality of the included studies. The Cochran’s Q test and I 2 were executed to detect and quantify the presence of statistical heterogeneity in the studies. Meta-analysis was done for each outcome when more than one original study reported relevant data, using Stata statistical software version 18.

Eleven studies were included from a total of 235 articles identified by the search. The odds of completing essential newborn care (pooled odds ratio: 3.04, 95% CI: 1.56, 5.90), early initiation of breastfeeding (pooled odds ratio: 1.30, 95% CI: 1.13, 1.52), exclusive breastfeeding (pooled odds ratio: 2.21, 95% CI: 1.68, 2.89), and being fully immunised (pooled odds ratio: 2.73, 95% CI: 1.16, 6.40) were higher among infants born to women with intended pregnancies as compared to women with unintended pregnancies.

Intended pregnancy was positively associated with essential newborn care completion, early initiation and exclusive breastfeeding, and full immunisation of infants in SSA countries. Thus, policy-makers and stakeholders should strengthen the provision of quality family planning services to prevent unintended pregnancy. Furthermore, follow-up of women with unintended pregnancies is needed to increase women’s opportunity to access essential newborn health care services that further reduce the risk of newborn and infant morbidity and mortality.

Systematic review registration

PROSPERO registration number CRD42023409148.

Peer Review reports

Introduction

Improving the quality of life and survival chances of newborns, and children remains an urgent global challenge [ 1 ]. Worldwide, considerable progress has been made to reduce neonatal mortality from 5 million in 1990 to 2.4 million in 2019 [ 2 ], and under-five mortality from 12.5 million in 1990 to 5.3 million in 2018 [ 2 ]. However, substantial differences in child mortality continue to exist across regions where Sub-Saharan Africa (SSA) holds the highest perinatal mortality rates of 34.7 deaths per 1,000 live births [ 3 ], infant mortality rate of 53 deaths per 1,000 live births [ 4 ], and under-five mortality rate of 78 deaths per 1,000 live births [ 2 ]. In addition, about 41% of children in SSA are highly affected by stunting, which possibly contributes to increasing child morbidity and mortality [ 5 ]. Many of these deaths are preventable or curable using interventions that are simple and cost-effective such as adequate nutrition, vaccination, and appropriate newborn and infant care [ 6 , 7 ] as well as by improving access to maternal health care services [ 8 ].

Parental adherence to children’s preventive and curative health care sets children up for better long-term health and considerably reduces the risk for child morbidity and mortality [ 9 , 10 , 11 ]. Similarly, maternal pregnancy intention has a significant effect on parent–child attachment and bonding, with a potential impact on children’s long-term physical, developmental, and psychological health outcomes [ 12 , 13 , 14 ]. Women with unintended pregnancies face more difficulty in establishing a strong attachment with their babies, poor childcare practices, and breastfeeding difficulties [ 12 , 15 , 16 ]. Literature has shown that children born to mothers with unintended pregnancies had less secure mother-to-child attachment during infancy, poor preventive and curative care, and poor behavioural and educational outcomes than women with intended pregnancies [ 12 , 16 , 17 ]. In addition, unintended pregnancy was associated with childhood illness, stunting and underweight, and higher risks of infant and child mortality [ 16 , 18 ]. Furthermore, adverse birth outcomes, such as congenital anomalies, preterm birth, and low birthweight were observed among children born from unintended pregnancies [ 18 , 19 ]. Conversely, intended pregnancy was associated with a reduced risk of stillbirth compared to unintended pregnancies [ 14 ].

Mounting evidence has shown that women who experienced unintended pregnancy had lower likelihoods of timely initiation of breastfeeding, continuing to breastfeed, and exclusive breastfeeding than women with intended pregnancy [ 20 , 21 ], resulting in poorer physical health of their children [ 13 ]. Furthermore, children born from unintended pregnancy were less likely to receive childhood immunisation [ 20 , 22 , 23 ]. These effects could be caused by maternal behaviours during pregnancy, and late initiation, or low use of antenatal care [ 24 , 25 , 26 , 27 ]. Hence, the prevention of unintended pregnancy and effective implementation of the maternity continuum of care are highly recommended to enhance the survival of newborns and infants through promoting preventive health care practices for children [ 28 , 29 ]. A recent study has shown that completing the maternity continuum of care, which includes the use of antenatal care, birthing at health facility, and use of postnatal care can considerably improve the use of essential newborn care practices [ 30 ].

The World Health Organization and United Nations Children’s Fund [ 2 , 31 ] strongly recommended strategies for the newborn and infant continuum of care, such as exclusive and timely initiation of breastfeeding, improving child nutrition, and child vaccination. This is anticipated to achieve the Sustainable Development Goal aimed to decrease the high rates of neonatal deaths to 12 per 1000 live births and under-five deaths to 25 per 1000 live births by 2030 [ 32 , 33 ]. Though strong emphasis has been given to the newborn and infant continuum of care, in SSA, only 56.5% of children were fully vaccinated [ 34 ], 47% of infants were breastfed within one hour of birth and 35% of children received exclusive breastfeeding in the first six months of life [ 35 ]. However, none of these studies have analysed the association between maternal pregnancy intention and completion of the newborn and infant continuum of care including essential newborn care practices, exclusive and timely initiation of breastfeeding, and infant immunisation. Moreover, while unintended pregnancies are associated with adverse newborn and infant health outcomes, there is inconsistent and inconclusive evidence on whether pregnancy intention influences newborn and infant health care across the newborn and infant continuum of care.

Up-to-date evidence with pooled estimates is required to understand whether developing interventions targeting preventing unintended pregnancies with the aim of child survival is required. Therefore, this study aimed to pool findings reported in literature on the association between pregnancy intention and essential newborn and infant health care across the newborn and infant continuum of care in SSA to contribute reliable evidence that would inform newborn and infant health policy and practice.

Registration and reporting

We used the updated Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist [ 36 ] to prepare and report this review (Supplementary File 1). After checking for a lack of other similar existing reviews and protocols, this systematic review was registered on the International Prospective Register of Systematic Reviews (PROSPERO) with registration number CRD42023409148.

Information sources and search strategy

The search terms were prepared based on the following concepts: pregnancy intention, essential newborn care practice, early initiation of breastfeeding, exclusive breastfeeding, full immunisation status, and Sub-Saharan Africa. The search keywords included free text keywords and Medical Subject Headings (MeSH) using Boolean operators, truncation, wildcards, and phrases in various databases. A comprehensive literature search was done in major databases such as MEDLINE Complete (EBSCOhost platform), EMBASE, CINAHL Complete (EBSCOhost platform), and Global Health (EBSCOhost platform) on August 13, 2023. These databases were selected as they index health and medical-related research. Furthermore, bibliographies of reviews and the identified studies were reviewed in Google Scholar for potentially relevant studies. The EBSCOhost MEDLINE was used to develop the initial search strategy, which was then adapted for other databases. Details of search strategies for each database are provided in Supplementary File 2. A scholarly expert librarian reviewed the search plans and provided advice to improve the search.

Eligibility criteria

Inclusion criteria.

We included observational studies that reported essential newborn care practice, or early initiation of breastfeeding, or exclusive breastfeeding, or child immunisation status as an outcome variable and pregnancy intention as an exposure variable. Furthermore, we included peer-reviewed studies published in the English language and conducted in SSA countries. Moreover, studies that reported the odds ratios (OR) with a 95% confidence interval (CI) or have raw data that allowed us to calculate the odds ratios with a 95% confidence interval were included.

Exclusion criteria

We excluded studies that did not report the associations of pregnancy intention with essential newborn care practice, or early initiation of breastfeeding, or exclusive breastfeeding, and full immunisation status. Review articles, qualitative studies, case reports, commentaries, case studies, conference abstracts, case series, and opinion pieces were excluded.

Study selection

We used Covidence systematic review software to screen studies. Two researchers (BDM, VV) independently conducted title and abstract screening after removing duplicates. Similarly, potentially eligible studies for full-text review were retrieved and then systematically screened against the eligibility criteria. The third reviewer (AGB) resolved any conflicts during the screening process. The results of the search in each database, screening, and selection process are summarised in a PRISMA 2020 flow diagram (see Fig. 1 ).

PRISMA flow diagram of the studies screening process

Quality assessment

Two researchers (BDM, AGB) assessed the quality of each primary study using the Newcastle–Ottawa Scale (NOS) for assessing the quality of non-randomised studies [ 37 ], with no discrepancies observed. The NOS tool has selection, comparability, and outcome domains and rate articles with a maximum of ten stars. The selection domain focuses on sample size, representativeness of the target population, response rate, and ascertainment of risk factors with a maximum of five stars. The comparability domain focuses on controlling the confounding factors that potentially influence the outcome of the variable with a maximum of two stars. The outcome domain assesses the outcomes of interest and appropriateness of the statistical tests with a maximum of three stars. The quality of the studies was ranked based on the overall number of stars. Accordingly, studies with a star number of 9 or 10, 7 or 8, and 5 or 6 were deemed to be very-good quality, good quality, and satisfactory quality respectively (Suplementary file 3). Studies with four or less stars were deemed unsatisfactory quality and were to be excluded from the review due to poor quality. The NOS tool has been used in previous similar studies [ 38 , 39 , 40 ].

Data extraction

Two researchers (BDM, VV) independently extracted data in a Microsoft Excel spreadsheet. No discrepancies occurred during data extraction. All relevant data items such as author(s), study setting and design, sample size, study population, data collection methods, response rate, publication year, odds ratio with a 95% CI for the association of pregnancy intention with each of the four outcome variables (essential newborn care, early initiation of breastfeeding, exclusive breastfeeding, and full immunisation status), and the definition of all four outcome variables were extracted from each included study. The pregnancy intention of participants was also extracted for each outcome variable.

Outcome measurement

The association between pregnancy intention and newborn and infant health care across the continuum of care was the outcome of interest in this review. For this research, the newborn and infant health continuum of care includes essential newborn care, early initiation of breastfeeding, exclusive breastfeeding, and full immunisation status. Essential newborn care is defined in this research as the newborn has received all the recommended elements of the services, including safe cord care, initiating skin-to-skin contact, eye care, and delayed baby bathing for at least 24 h after birth [ 41 ]. Early initiation of breastfeeding is defined as the provision of mother’s breast milk to the baby within the first hour of birth [ 42 ]. Exclusive breastfeeding is defined as an infant receiving only breast milk from birth until six months of age without adding water or any other food except for mineral supplements, vitamins, or prescribed medicines [ 42 ]. Full immunisation is defined as a child who has received a single dose of Bacillus Calmette–Guérin (BCG) vaccine, four poliomyelitis vaccines, three doses of pentavalent vaccines (diphtheria, pertussis, tetanus, hepatitis B and Haemophilus influenza type B), three doses of pneumococcal conjugate vaccine (PCV), two doses of Rota vaccine, and one dose of measles vaccine before their first birthday [ 43 ].

Pregnancy intention has been classified into intended and unintended [ 44 ]. For this study, intended pregnancy was when the woman reported that she planned or wanted her last pregnancy at the time of conception; otherwise, a pregnancy is considered an unintended pregnancy. Both unwanted pregnancies (no children are desired at all) and mistimed pregnancies (wanted at some time but occurred sooner than desired) were considered as unintended pregnancies [ 45 ].

Data analysis

The adjusted odds ratios (AOR) of the association between pregnancy intention and each outcome variable were extracted from each primary study as the measure of effect. For the outcome of early initiation of breastfeeding, the crude odds ratios (COR) and confidence intervals were calculated from the raw data. The COR was used for the outcome of early initiation of breastfeeding due to the lack of AOR data in the included studies. Since COR was calculated from all included studies for this outcome, this approach ensured that we did not mix AOR with COR in the same analysis. Logit transformations were made for the individual measure of effect before computing the pooled summary. When two or more studies reported relevant data, a quantitative meta-analysis was done for each outcome. Forest plots and tables are used to graphically display the summary of effect sizes with 95% CI and other results. The random-effects model was employed for the outcomes of essential newborn care, early initiation breastfeeding and child immunisation status. The random-effect model was employed to assume the variations or heterogeneity in the effect estimates across the studies [ 46 , 47 ]. For the outcome of exclusive breastfeeding, the fixed effects model was used as the number of included studies was only two, which makes it inappropriate to perform the random-effects model because the precision in estimating variability in studies is limited [ 46 , 48 ]. The weight of each study in the pooled estimates was calculated using the inverse of the variance. The potential variation between the primary studies was statistically estimated using Cochran’s Q test and quantified by I 2 with significant heterogeneity to be deemed at p -value < 0.1 or I 2 > 50% [ 49 , 50 ]. Since very few studies were included in the meta-analysis of each outcome, publication bias, subgroup analysis, and sensitivity analysis were not conducted [ 48 , 51 ]. In this analysis, early initiation of breastfeeding, while a component of essential newborn care, was analysed separately to better understand how pregnancy intention specifically influences this practice. The studies used for the early initiation of breastfeeding analysis differed from those included in the broader essential newborn care analysis, ensuring that our findings on early initiation are based on a unique set of data. Furthermore, unwanted and mistimed pregnancies were used interchangeably if the original studies had reported one of the two instead of unintended pregnancy. Likewise, wanted and planned pregnancy were used interchangeably if the original studies had reported one of the two instead of intended pregnancy. All statistical analyses were executed using Stata statistical software version 18.

The database searches produced a total of 231 records, with a further 4 articles identified from manual searches of bibliographies of relevant studies. After a comprehensive screening, eleven studies met the inclusion criteria and retained for the review [ 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 ]. The steps and outcomes of the study selection process are shown in the PRISMA flow diagram (see Fig. 1 ).

Studies characteristics

All the studies included in the review used cross-sectional study designs and were published between 2016 and 2022. A total of 18,781 study participants were involved in the original studies. Many studies ( n = 9) were set in Ethiopia [ 52 , 53 , 54 , 56 , 57 , 59 , 60 , 61 , 62 ]. Eight studies collected the data via face-to-face surveys developed from literature reviews [ 52 , 53 , 54 , 57 , 58 , 59 , 60 , 62 ], two studies used the demographic and health survey [ 56 , 61 ], and the remaining one study used data from multiple indicator cluster survey [ 55 ]. Of the included studies in the review, eight studies employed multistage random sampling [ 52 , 53 , 54 , 55 , 56 , 58 , 59 , 61 ], two studies used systematic random sampling [ 60 , 62 ], and one study used simple random sampling to collect data [ 57 ]. Almost all ( n = 10) of the studies were of good quality methodological rigour (7 or 8 stars) on the NOS assessment. The summary of the 11 papers is shown in Table 1 .

Association between pregnancy intention and newborn and infant continuum of care

Three studies examined the association between pregnancy intention and essential newborn care practice [ 52 , 53 , 54 ]. The results of the meta-analysis showed that women whose pregnancy was intended were about three times (AOR: 3.04, 95% CI: 1.56, 5.90) more likely to practice essential newborn care as compared to women with unintended pregnancy (Fig. 2 ).

Pooled odds ratio of the association between essential newborn care and pregnancy intention

Three studies were included to estimate the association between pregnancy intention and early initiation of breastfeeding [ 55 , 56 , 57 ]. Meta-analysis of the three studies showed that women with intended pregnancy had 1.31 times (COR: 1.30, 95% CI: 1.13, 1.52) higher odds of early initiation of breastfeeding than women who had an unintended pregnancy (Fig. 3 ).

Pooled odds ratio of the association between early initiation of breastfeeding and pregnancy intention

Two studies reported the association between pregnancy intention and exclusive breastfeeding [ 58 , 59 ]. Meta-analysis of the two studies showed that women with an intended pregnancy had 2.21 times (AOR: 2.21, 95% CI: 1.68, 2.89) higher odds of exclusive breastfeeding than women who had an unintended pregnancy (Fig. 4 ).

Pooled odds ratio of the association between exclusive breastfeeding and pregnancy intention

Three studies reported the association between pregnancy intention and child immunisation status [ 60 , 61 , 62 ]. Accordingly, the pooled odds of being fully immunised was 2.73 times (AOR: 2.73, 95% CI: 1.16, 6.40) higher in children born from mothers with an intended pregnancy than children born from mothers with an unintended pregnancy (Fig. 5 ).

Pooled odds ratio of the association between child immunisation status and pregnancy intention

Adjusted effect estimates were used to determine the association between pregnancy intention and newborn and infant continuum of care, including essential newborn care, early initiation and exclusive breastfeeding, and full immunisation. However, the studies did not control for the same mix of confounders. Furthermore, the association between pregnancy intention and early initiation of breastfeeding is likely to be affected by several other factors, as the analysis was executed based on an unadjusted effect measure. The results of the present meta-analyses did however suggest that intended pregnancy was associated with greater completion of essential newborn care practices, early initiation and exclusive of breastfeeding, and being fully immunised.

Women with intended pregnancy had higher odds of essential newborn care practice, early initiation and exclusive breastfeeding and fully immunised infants than women with unintended pregnancy. This could be explained by the fact that mothers with intended pregnancy may have greater psychological preparedness for providing care to their newborn infants [ 63 ]. Conversely, women experiencing an unintended pregnancy may not feel equipped to take on a maternal role, leading to poor mother-to-child attachment and low engagement with essential newborn practices [ 13 , 64 ]. Previous studies have indicated that negligence, carelessness, aggression, and maltreatment of children were observed more often in mothers with unintended pregnancies [ 65 , 66 ]. Furthermore, these women may have low health-seeking behaviour for themselves and their infants due to stress associated with unintended pregnancy, and less support from their partners or families [ 24 , 67 , 68 ]. Women who experience an unintended pregnancy may also receive little information and counselling regarding the benefit of appropriate newborn and infant feeding practices and care because of absent or delayed engagement with the continuum of perinatal health services [ 27 , 69 , 70 , 71 , 72 ].

This review implies that preventing unplanned pregnancy may have an important role in improving essential newborn care, early initiation and exclusive breastfeeding, and completion of immunisation of infants in SSA and possibly other low-resource settings. The prevention of unintended pregnancy could be achieved through improving access to quality family planning services [ 73 , 74 ]. Strengthening preconception services and information about the consequences of unintended pregnancy on newborn and infant health outcomes could improve newborn and infant health and survival [ 75 ]. Furthermore, counselling about maternal-child attachment in women with an unplanned pregnancy could contribute to better newborn and infant health outcomes [ 71 ].

Post-conception responses from partners, family and community members, and psychological consequences such as depression and anxiety following an unintended pregnancy possibly influence newborn and infant health care practices among women with unintended pregnancies [ 68 , 76 , 77 ]. Furthermore, women with unintended pregnancies may have several socioeconomic and cultural challenges to visiting health facilities [ 78 , 79 ]. Previous studies in low- and middle-income countries reported that multiple visits to healthcare facilities, which require travel, money, and assistance from others, substantially deter women from using antenatal care, skilled birth attendance, and postnatal care following the occurrence of an unintended pregnancy [ 80 , 81 , 82 ]. Hence, encouraging perinatal service use and preventive and curative care of infants resulting from unintended pregnancies may require policies and practices that support identification of women with unintended pregnancies. Behavioural change approaches regarding unintended pregnancy also need to be considered at individual, family, and community levels [ 83 , 84 , 85 ]. This may reduce the humiliation of women from unsolicited post-conception responses from partner, family and community members following unintended pregnancies [ 86 , 87 ].

This systematic review has the following limitations. In all the included studies, pregnancy intention was examined with other multiple determinants of newborn and infant continuum of care in cross-sectional studies for each outcome; it is thus not possible to establish causal effect relationships with certainty. Although there was significant heterogeneity among the included studies, addressing the sources of heterogeneity was not undertaken due to the limited number of studies included for each outcome. In addition, sensitivity analysis, publication bias and subgroup analyses were not conducted because of the limited number of studies included in each sub-topic. Furthermore, some studies were excluded due inconsistent categorisation of the exposure variable. Moreover, studies published in a language other than English and non-peer-reviewed articles may have been missed as only peer-reviewed articles published in English were included. While grey literature may exist on this topic, we included only peer-reviewed studies to ensure a robust approach and consistency across studies. Finally, the generalizability of this review may be limited because the included studies were not representative of all regions of SSA, with the majority of studies coming from Ethiopia. Nevertheless, this review used adjusted effect measures to quantitatively estimate the association between pregnancy intention on newborn and infant health across the continuum of care in SSA.

Conclusions

This review pinpointed that there were statistically significant positive associations between intended pregnancy and increased completion of essential newborn care practices, early initiation and exclusive breastfeeding and full immunisation of infants in SSA. There is a need for collaboration among stakeholders, policy-makers and health care providers to strengthen the provision of quality family planning services and enhancing preconception care services to prevent unintended pregnancy. Furthermore, strategies that help identify women with unintended pregnancies are needed to increase women’s access to essential maternal and neonatal/infant healthcare services that further reduce the risk of newborn and infant deaths. Moreover, further research with better design is needed to clearly understand the relationship between pregnancy intention and completion of newborn and infant health practices across the continuum of care.

Availability of data and materials

All relevant data is included either in the manuscript or as supplementary files.

Data availability

No datasets were generated or analysed during the current study.

Abbreviations

Cumulative Index to Nursing and Allied Health Literature

Newcastle-Ottawa scale

International Prospective Register of Systematic Reviews

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Sub-Saharan African countries

USAID. Preventing Child and Maternal Deaths: A Framework for Action in a Changing World, 2023–2030: USAID; 2021 [Available from: Available at: https://www.usaid.gov/PreventingChildAndMaternalDeaths/framework . Accesed on April 2023.

WHO. Newborns: improving survival and well-being. 2020. Key facts Available at: https://www.whoint/news-room/fact-sheets/detail/newborns-reducing-mortality . 2021.

Akombi BJ, Renzaho AM. Perinatal mortality in sub-Saharan Africa: a meta-analysis of demographic and health surveys. Ann Glob Health. 2019;85(1):106.

Article PubMed PubMed Central Google Scholar

World-Bank. World Bank. Mortality Rate, Infant (per 1000 Live Births) | Data. Available online: https://data.worldbank.org/indicator/SP.DYN.IMRT.IN . Accessed April 2023. 2018.

Quamme SH, Iversen PO. Prevalence of child stunting in Sub-Saharan Africa and its risk factors. Clin Nutr Open Sci. 2022;42:49–61.

Article Google Scholar

Burstein R, Henry NJ, Collison ML, Marczak LB, Sligar A, Watson S, et al. Mapping 123 million neonatal, infant and child deaths between 2000 and 2017. Nature. 2019;574(7778):353–8.

Article CAS PubMed PubMed Central Google Scholar

De Bernis L, Kinney MV, Stones W, ten Hoope-Bender P, Vivio D, Leisher SH, et al. Stillbirths: ending preventable deaths by 2030. The lancet. 2016;387(10019):703–16.

UNICEF, WHO. Ending preventable newborn and stillbirths by 2030: moving faster towards high-quality universal health coverage in 2020–2025. UNICEF; World Health Organization; 2020.

Yee AZ, Lwin MO, Ho SS. The influence of parental practices on child promotive and preventive food consumption behaviors: a systematic review and meta-analysis. Int J Behav Nutr Phys Act. 2017;14(1):1–14.

Baker S, Morawska A, Mitchell AE. Do Australian children carry out recommended preventive child health behaviours? Insights from an online parent survey. J Paediatr Child Health. 2020;56(6):900–7.

Article PubMed Google Scholar

Jensen SK, Bouhouch RR, Walson JL, Daelmans B, Bahl R, Darmstadt GL, et al. Enhancing the child survival agenda to promote, protect, and support early child development. In: Seminars in perinatology. Elsevier; 2015.

Google Scholar

Singh A, Upadhyay AK, Singh A, Kumar K. The association between unintended births and poor child development in India: evidence from a longitudinal study. Stud Fam Plann. 2017;48(1):55–71.

Pakseresht S, Rasekh P, Leili EK. Physical health and maternal-fetal attachment among women: Planned versus unplanned pregnancy. INternational Journal of Womens Health and Reproduction Sciences. 2018;6(3):335–41.

Hall JA, Barrett G, Copas A, Phiri T, Malata A, Stephenson J. Reassessing pregnancy intention and its relation to maternal, perinatal and neonatal outcomes in a low-income setting: a cohort study. PLoS ONE. 2018;13(10): e0205487.

Sawhill IV, Guyot K. Preventing unplanned pregnancy: lessons from the states. Economic studies at brookings. 2019. p. 1–18.

Chowdhury P, Garg MK, Sk MIK. Does mothers’ pregnancy intention affect their children’s preventive and curative care in India? Evidence from a longitudinal survey. BMJ Open. 2021;11(4): e042615.

Article PubMed Central Google Scholar

Foster DG, Biggs MA, Raifman S, Gipson J, Kimport K, Rocca CH. Comparison of health, development, maternal bonding, and poverty among children born after denial of abortion vs after pregnancies subsequent to an abortion. JAMA Pediatr. 2018;172(11):1053–60.

Johnson EL, Burke AE, Wang A, Pennell PB. Unintended pregnancy, prenatal care, newborn outcomes, and breastfeeding in women with epilepsy. Neurology. 2018;91(11):e1031–9.

Hall JA, Benton L, Copas A, Stephenson J. Pregnancy intention and pregnancy outcome: systematic review and meta-analysis. Matern Child Health J. 2017;21:670–704.

Chatterjee E, Sennott C. Fertility intentions and child health in India: Women’s use of health services, breastfeeding, and official birth documentation following an unwanted birth. PLoS ONE. 2021;16(11): e0259311.

Lindberg L, Maddow-Zimet I, Kost K, Lincoln A. Pregnancy intentions and maternal and child health: an analysis of longitudinal data in Oklahoma. Matern Child Health J. 2015;19:1087–96.

Singh A, Singh A, Thapa S. Adverse consequences of unintended pregnancy for maternal and child health in Nepal. Asia Pac J Public Health. 2015;27(2):NP1481–91.

Sharrow D, Hug L, You D, Alkema L, Black R, Cousens S, et al. Global, regional, and national trends in under-5 mortality between 1990 and 2019 with scenario-based projections until 2030: a systematic analysis by the UN Inter-agency Group for Child Mortality Estimation. Lancet Glob Health. 2022;10(2):e195–206.

Kost K, Lindberg L. Pregnancy intentions, maternal behaviors, and infant health: investigating relationships with new measures and propensity score analysis. Demography. 2015;52(1):83–111.

Abajobir AA, Maravilla JC, Alati R, Najman JM. A systematic review and meta-analysis of the association between unintended pregnancy and perinatal depression. J Affect Disord. 2016;192:56–63.

Tolossa T, Turi E, Fetensa G, Fekadu G, Kebede F. Association between pregnancy intention and late initiation of antenatal care among pregnant women in Ethiopia: a systematic review and meta-analysis. Syst Rev. 2020;9:1–10.

Ranatunga IDJC, Jayaratne K. Proportion of unplanned pregnancies, their determinants and health outcomes of women delivering at a teaching hospital in Sri Lanka. BMC Pregnancy Childbirth. 2020;20(1):1–15.

Owili PO, Muga MA, Chou Y-J, Hsu Y-HE, Huang N, Chien L-Y. Associations in the continuum of care for maternal, newborn and child health: a population-based study of 12 sub-Saharan Africa countries. BMC Public Health. 2016;16(1):1–15.

Phway P, Kyaw AT, Mon AS, Mya KS. Continuum of care of mothers and immunization status of their children: A secondary analysis of 2015–2016 Myanmar Demographic and Health Survey. Public Health in Practice. 2022;4: 100335.

Zelka MA, Yalew AW, Debelew GT. Effects of continuity of maternal health services on immediate newborn care practices, Northwestern Ethiopia: multilevel and propensity score matching (PSM) modeling. Heliyon. 2022;8(12): e12020.

UNICEF D. Improving maternal, infant and young child nutrition expands opportunities for every child to reach his or her full potential. United Nations Children’s Fund 2023 [Available from: Available from: https://data.unicef.org/topic/nutrition/child-nutrition/ .

UN. Department of Economic Social Affairs. Transforming our world: the 2030 Agenda for Sustainable Development. United Nations General Assembly. Cited 2023. Available at: https://sdgs.un.org/2030agenda ; 2015.

UNICEF D. Analytics Section. Progress for every child in the SDG era: are we on track to achieve the SDGs for children? UNICEF Division of Data. Research and Policy: New York Cited 2023 Available from: https://www.uniceforg/media/56516/file . 2018.

Bobo FT, Asante A, Woldie M, Dawson A, Hayen A. Child vaccination in sub-Saharan Africa: Increasing coverage addresses inequalities. Vaccine. 2022;40(1):141–50.

Pretorius CE, Asare H, Genuneit J, Kruger HS, Ricci C. Impact of breastfeeding on mortality in sub-Saharan Africa: a systematic review, meta-analysis, and cost-evaluation. Eur J Pediatr. 2020;179:1213–25.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg. 2021;88: 105906.

Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. 2000. p. 1–12.

Biset G, Woday A, Mihret S, Tsihay M. Full immunization coverage and associated factors among children age 12–23 months in Ethiopia: systematic review and meta-analysis of observational studies. Hum Vaccin Immunother. 2021;17(7):2326–35.

Awoh AB, Plugge E. Immunisation coverage in rural–urban migrant children in low and middle-income countries (LMICs): a systematic review and meta-analysis. J Epidemiol Community Health. 2016;70(3):305–11.

Alamneh Y, Adane F, Yirga T, Desta M. Essential newborn care utilization and associated factors in Ethiopia: a systematic review and meta-analysis. BMC Pregnancy Childbirth. 2020;20:1–9.

Organization WH. WHO recommendations on maternal and newborn care for a positive postnatal experience: World Health Organization; 2022. Available at: https://www.who.int/publications/i/item/9789240045989 . Accessed Apr 2023.

Wojcieszek AM, Bonet M, Portela A, Althabe F, Bahl R, Chowdhary N, et al. WHO recommendations on maternal and newborn care for a positive postnatal experience: strengthening the maternal and newborn care continuum. BMJ Glob Health. 2023;8(Suppl 2): e010992.

WHO. WHO recommendations for routine immunization-summary tables. WHO Geneva, Switzerland; 2014. Available at: https://www.who.int/teams/immunization-vaccines-andbiologicals/policies/who-recommendations-forroutine-immunization---summary-tables . Accessed Apr 2023.

Stanford JB, Hobbs R, Jameson P, DeWitt MJ, Fischer RC. Defining dimensions of pregnancy intendedness. Matern Child Health J. 2000;4:183–9.

Article CAS PubMed Google Scholar

Santelli J, Rochat R, Hatfield-Timajchy K, Gilbert BC, Curtis K, Cabral R, et al. The measurement and meaning of unintended pregnancy. Perspect Sex Reprod Health. 2003;35(2):94–101.

Tufanaru C, Munn Z, Stephenson M, Aromataris E. Fixed or random effects meta-analysis? Common methodological issues in systematic reviews of effectiveness. JBI Evidence Implementation. 2015;13(3):196–207.

Tufanaru C, Munn Z, Aromataris E, Campbell J, Hopp L. Chapter 3: Systematic reviews of effectiveness. In: Aromataris E, Munn Z, editors. JBI manual for evidence synthesis. JBI; 2020. [cited August 2023]. Available from: https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-04 .

Cooper H. Research synthesis and meta-analysis: A step-by-step approach. 5th ed. Thousand Oaks (CA): Sage Publications; 2015. p. 1–364.

Huedo-Medina TB, Sánchez-Meca J, Marín-Martínez F, Botella J. Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychol Methods. 2006;11(2):193.

Higgins JP, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.

Chandler J, Cumpston M, Li T, Page MJ, Welch V. Cochrane handbook for systematic reviews of interventions. Hoboken: Wiley; 2019.

Habte A, Lukas K, Tamirat T. The level of Community-Based Essential Newborn Care utilization and associated factors among rural women in Southern Ethiopia, 2020: Based on the updated Community-Based Essential Newborn Care guideline. SAGE open medicine. 2022;10:20503121211067690.

Alemu A, Eshete A. Newborn care practices and associated factors among lactating mothers at home in the rural districts of Gedeo Zone, southern Ethiopia. Pediatric Health Med Ther. 2020;11:47–54.

Chichiabellu TY, Mekonnen B, Astawesegn FH, Demissie BW, Anjulo AA. Essential newborn care practices and associated factors among home delivered mothers in Damot pulasa Woreda, southern Ethiopia. Reprod Health. 2018;15(1):1–11.

Apanga PA, Kumbeni MT. Prevalence and predictors of timely initiation of breastfeeding in Ghana: an analysis of 2017–2018 multiple indicator cluster survey. Int Breastfeed J. 2021;16:1–8.

Gedefaw G, Goedert MH, Abebe E, Demis A. Effect of cesarean section on initiation of breast feeding: Findings from 2016 Ethiopian Demographic and Health Survey. PLoS ONE. 2020;15(12): e0244229.

Gebremeskel SG, Gebru TT, Gebrehiwot BG, Meles HN, Tafere BB, Gebreslassie GW, et al. Early initiation of breastfeeding and associated factors among mothers of aged less than 12 months children in rural eastern zone, Tigray, Ethiopia: cross-sectional study. BMC Res Notes. 2019;12(1):1–6.

Duarte Lopes E, Monteiro AMRL, Varela DOBFC, Trigueiros DELR, Monteiro Spencer Maia I, de Jesus Xavier Soares J, et al. The prevalence of exclusive breastfeeding and its associated factors in Cape Verde. BMC Nutr. 2022;8(1):1–7.

Mamo K, Dengia T, Abubeker A, Girmaye E. Assessment of exclusive breastfeeding practice and associated factors among mothers in west Shoa zone, Oromia. Ethiopia Obstet Gynecol Int. 2020;2020:3965873.

PubMed Google Scholar

Mebrate M, Workicho A, Alemu S, Gelan E. Vaccination Status and Its Determinants Among Children Aged 12 to 23 Months in Mettu and Sinana Districts, Oromia Region, Ethiopia: A Comparative Cross Sectional Study. Pediatric Health Med Ther. 2022;13:335–48.

Fenta SM, Fenta HM. Individual and community-level determinants of childhood vaccination in Ethiopia. Archives of Public Health. 2021;79(1):1–11.

Gualu T, Dilie A. Vaccination coverage and associated factors among children aged 12–23 months in debre markos town, Amhara regional state, Ethiopia. Adv Pub Health. 2017;2017(1):5352847.

Tsegaw HZ, Cherkos EA, Badi MB, Mihret MS. Intended pregnancy as a predictor of good knowledge on birth preparedness and complication readiness: the case of northern Ethiopia pregnant mothers. Int J Reprod Med. 2019;2019:9653526.

Abajobir AA, Kisely S, Najman JM. A systematic review of unintended pregnancy in cross-cultural settings: Does it have adverse consequences for children? Ethiopian Journal of Health Development. 2017;31(3):138–54.

Guterman K. Unintended pregnancy as a predictor of child maltreatment. Child Abuse Negl. 2015;48:160–9.

Isumi A, Fujiwara T. Synergistic effects of unintended pregnancy and young motherhood on shaking and smothering of infants among caregivers in Nagoya City. Japan Frontiers in public health. 2017;5:245.

Faisal-Cury A, Menezes PR, Quayle J, Matijasevich A. Unplanned pregnancy and risk of maternal depression: secondary data analysis from a prospective pregnancy cohort. Psychol Health Med. 2017;22(1):65–74.

Barton K, Redshaw M, Quigley MA, Carson C. Unplanned pregnancy and subsequent psychological distress in partnered women: a cross-sectional study of the role of relationship quality and wider social support. BMC Pregnancy Childbirth. 2017;17:1–9.

Ochako R, Gichuhi W. Pregnancy wantedness, frequency and timing of antenatal care visit among women of childbearing age in Kenya. Reprod Health. 2016;13:1–8.

Amo-Adjei J, Anamaale TD. Effects of planned, mistimed and unwanted pregnancies on the use of prenatal health services in sub-Saharan Africa: a multicountry analysis of demographic and health survey data. Tropical Med Int Health. 2016;21(12):1552–61.

Ekrami F, Mohammad-Alizadeh Charandabi S, Babapour Kheiroddin J, Mirghafourvand M. Effect of counseling on maternal-fetal attachment in women with unplanned pregnancy: a randomized controlled trial. J Reprod Infant Psychol. 2020;38(2):151–65.

Khan MN, Harris ML, Shifti DM, Laar AS, Loxton D. Effects of unintended pregnancy on maternal healthcare services utilization in low-and lower-middle-income countries: systematic review and meta-analysis. Int J Public Health. 2019;64:743–54.

Norton M, Chandra-Mouli V, Lane C. Interventions for preventing unintended, rapid repeat pregnancy among adolescents: a review of the evidence and lessons from high-quality evaluations. Global Health: Science and Practice. 2017;5(4):547–70.

Rees H, Pillay Y, Mullick S, Chersich M. Strengthening implant provision and acceptance in South Africa with the ‘Any woman, any place, any time’approach: An essential step towards reducing unintended pregnancies. S Afr Med J. 2017;107(11):939–44.

Goodfellow A, Frank J, McAteer J, Rankin J. Improving preconception health and care: a situation analysis. BMC Health Serv Res. 2017;17:1–8.

McNamara J, Risi A, Bird AL, Townsend ML, Herbert JS. The role of pregnancy acceptability in maternal mental health and bonding during pregnancy. BMC Pregnancy Childbirth. 2022;22(1):1–10.

Yong MQY, Yeo Y, Shorey S. Factors affecting unintended pregnancy resolution from the perspectives of pregnant women and people: A systematic review of qualitative evidence. Midwifery. 2023;127: 103866.

Hajizadeh M, Nghiem S. Does unwanted pregnancy lead to adverse health and healthcare utilization for mother and child? Evidence from low-and middle-income countries. Int J Public Health. 2020;65:457–68.

Klann EM, Wong YJ. A pregnancy decision-making model: psychological, relational, and cultural factors affecting unintended pregnancy. Psychol Women Q. 2020;44(2):170–86.

Guliani H, Sepehri A, Serieux J. Determinants of prenatal care use: evidence from 32 low-income countries across Asia, Sub-Saharan Africa and Latin America. Health Policy Plan. 2014;29(5):589–602.

Khan MN, Harris ML, Loxton D. Assessing the effect of pregnancy intention at conception on the continuum of care in maternal healthcare services use in Bangladesh: Evidence from a nationally representative cross-sectional survey. PLoS ONE. 2020;15(11): e0242729.

Rahman MM, Rahman MM, Tareque MI, Ferdos J, Jesmin SS. Maternal pregnancy intention and professional antenatal care utilization in Bangladesh: a nationwide population-based survey. PLoS ONE. 2016;11(6): e0157760.

Uthaphan P, Mueaithaisong A, Phanphoon A, Karsapone S, Sopajorn S, Neadpuckdee R, et al. Using an Integrated Training Course with Family, Community, and Buddhism in Solving Unintended Pregnancies among Vulnerable Thai Youth. High Educ Stud. 2022;12(4):137–42.

Moseson H, Mahanaimy M, Dehlendorf C, Gerdts C. “… Society is, at the end of the day, still going to stigmatize you no matter which way”: A qualitative study of the impact of stigma on social support during unintended pregnancy in early adulthood. PLoS ONE. 2019;14(5): e0217308.

Feld H, Barnhart S, Wiggins AT, Ashford K. Social support reduces the risk of unintended pregnancy in a low-income population. Public Health Nurs. 2021;38(5):801–9.

Crowley JL, High AC, Thomas LJ. Desired, expected, and received support: How support gaps impact affect improvement and perceived stigma in the context of unintended pregnancy. Health Commun. 2019;34(12):1441–53.

Smith W, Turan JM, White K, Stringer KL, Helova A, Simpson T, et al. Social norms and stigma regarding unintended pregnancy and pregnancy decisions: a qualitative study of young women in Alabama. Perspect Sex Reprod Health. 2016;48(2):73–81.

Download references

Acknowledgements

The authors would like to thank Olivia Larobina (Scholarly Services Librarian at Deakin University) for her assistance in developing the systematic search strategy. We also would like to acknowledge Deakin University for providing PhD scholarship for the principal investigator of this research.

The first author is supported by Deakin University with a PhD research scholarship.

Author information

Authors and affiliations.

School of Nursing and Midwifery, Centre for Quality and Patient Safety Research in the Institute for Health Transformation, Deakin University Geelong, Victoria, Australia

Birye Dessalegn Mekonnen, Vidanka Vasilevski, Ayele Geleto Bali & Linda Sweet

Amhara Public Health Institute, Bahir Dar, Ethiopia

Birye Dessalegn Mekonnen

Western Health Partnership, Victoria, Australia

Vidanka Vasilevski, Ayele Geleto Bali & Linda Sweet

You can also search for this author in PubMed Google Scholar

Contributions

All authors developed the conception of the review and study design. BDM conducted the literature review, study screening, data extraction, quality assessment, data analysis, interpretation of results, and drafting of the manuscript. VV, AGB and LS contributed to study screening, quality assessment, data extraction, assist data analysis, and critically reviewed the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Birye Dessalegn Mekonnen .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Reprints and permissions

About this article

Cite this article.

Mekonnen, B.D., Vasilevski, V., Bali, A.G. et al. Association between pregnancy intention and completion of newborn and infant continuum of care in Sub-Saharan Africa: systematic review and meta-analysis. BMC Pediatr 24 , 567 (2024). https://doi.org/10.1186/s12887-024-05036-y

Download citation

Received : 18 December 2023

Accepted : 27 August 2024

Published : 06 September 2024

DOI : https://doi.org/10.1186/s12887-024-05036-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Intended pregnancy
Unintended pregnancy
Essential newborn care
Breastfeeding
Immunisation
Sub-Saharan Africa
Systematic review

BMC Pediatrics

ISSN: 1471-2431

General enquiries: [email protected]

Search Menu
Sign in through your institution
Chemical Biology and Nucleic Acid Chemistry
Computational Biology
Critical Reviews and Perspectives
Data Resources and Analyses
Gene Regulation, Chromatin and Epigenetics
Genome Integrity, Repair and Replication
Nucleic Acid Enzymes
RNA and RNA-protein complexes
Synthetic Biology and Bioengineering
Molecular and Structural Biology
Advance Articles
Breakthrough Articles
Molecular Biology Database Collection
Special Collections
Scope and Criteria for Consideration
Author Guidelines
Data Deposition Policy
Database Issue Guidelines
Web Server Issue Guidelines
Submission Site
About Nucleic Acids Research
Editors & Editorial Board
Information of Referees
Self-Archiving Policy
Dispatch Dates
Advertising and Corporate Services
Journals Career Network
Journals on Oxford Academic
Books on Oxford Academic

Article Contents

Introduction, comprehensive review, purposes of microarray meta-analysis, databases and software, meta-analysis for de gene detection, open questions, conclusion and discussion, acknowledgements.

< Previous

Comprehensive literature review and statistical considerations for microarray meta-analysis

Article contents
Figures & tables
Supplementary Data

George C. Tseng, Debashis Ghosh, Eleanor Feingold, Comprehensive literature review and statistical considerations for microarray meta-analysis, Nucleic Acids Research , Volume 40, Issue 9, 1 May 2012, Pages 3785–3799, https://doi.org/10.1093/nar/gkr1265

Permissions Icon Permissions

With the rapid advances of various high-throughput technologies, generation of ‘-omics’ data is commonplace in almost every biomedical field. Effective data management and analytical approaches are essential to fully decipher the biological knowledge contained in the tremendous amount of experimental data. Meta-analysis, a set of statistical tools for combining multiple studies of a related hypothesis, has become popular in genomic research. Here, we perform a systematic search from PubMed and manual collection to obtain 620 genomic meta-analysis papers, of which 333 microarray meta-analysis papers are summarized as the basis of this paper and the other 249 GWAS meta-analysis papers are discussed in the next companion paper. The review in the present paper focuses on various biological purposes of microarray meta-analysis, databases and software and related statistical procedures. Statistical considerations of such an analysis are further scrutinized and illustrated by a case study. Finally, several open questions are listed and discussed.

With the rapid advances in biological high-throughput technology, generation of various kinds of genomic data is commonplace in almost every biomedical field. Effective data management and analytical approaches are essential to fully decipher the biological knowledge contained in the tremendous amount of experimental data. In the past decade, the accumulation of transcriptomic data mainly from microarray experiments was particularly significant, and resulted in several large public data depositories (such as Gene Expression Omnibus and ArrayExpress). Similarly, genome-wide association studies (GWAS) are another example: thousands of GWAS have been performed world-wide and results and/or raw data for many are publicly available (see companion review paper for GWAS meta-analysis). It is common that multiple transcriptomic studies or GWAS are available for the same or related disease condition and each study has relatively small sample size with limited statistical power. Combining information from these studies to increase sensitivity and validate conclusions is a natural step. Such genomic information integration is akin to the classical meta-analysis in statistics where results of multiple studies of a similar research hypothesis are combined for a conclusive finding.

A major distinction in the genome-wide setting compared with the classical one is that we are typically analyzing data on thousands of genes. We term genomic information integration in which we combine results from multiple transcriptomic studies or GWAS as ‘horizontal genomic meta-analysis’ ( Figure 1 A). Figure 1 B demonstrates another type of multi-dimensional integrative analysis that combines multiple sources of -omics information on a given cohort of patients. The multi-dimensional -omics data usually include, but are not limited to, transcriptome profile, genotypes, DNA copy number variation, methylation, microRNA, proteome and phenome. Examples of publicly available databases that include this type of information include the Cancer Genome Atlas (TCGA; cancergenome.nih.gov) and the Therapeutically Applicable Research to Generate Effective Treatments (TARGET; target.cancer.gov). Integration of this type of data is called ‘vertical genomic integrative analysis’. In this article, we will focus on horizontal genomic meta-analysis through extensive search of PubMed database and manual literature referencing. Of the 582 papers related to genomic meta-analysis, we will concentrate on 333 microarray meta-analysis papers in this article. The other 249 GWAS meta-analysis papers are discussed in the companion paper. The goal of this article is 3-fold. First, we aim to provide a summary of the methodologies used in the microarray meta-analysis papers. In this light, the article can be viewed as a ‘meta’–meta-analysis paper. The second goal of the article is to provide a critique of the methodologies used in the literature. Finally, we outline some further issues in the field that need more attention.

Types of information integration of genomic studies. ( A ) Horizontal genomic meta-analysis that combines different sample cohorts for the same molecular event. ( B ) Vertical genomic integrative analysis that combines different molecular events usually in the same sample cohort.

The article is structured as follows. ‘Comprehensive review’ section summarizes details of the comprehensive literature review. In ‘Purposes of Microarray Meta-Analysis’ and ‘Databases and Software’ sections, we discuss various purposes of microarray meta-analysis and related software and database resources. In ‘Meta-Analysis for DE Gene Detection’ section, we discuss statistical considerations behind meta-analysis for differentially expressed (DE) gene detection, an analysis commonly encountered in microarray meta-analysis. ‘Open questions’ section describes a list of open questions and further discussions. ‘Conclusion and discussion’ section provides final conclusions.

Papers under review came from two sources: PubMed search and manual collection. 745 papers were obtained from searching the PubMed database by keywords on 29 December 2010 (see legend of Figure 2 ), and 102 papers were identified from cross-referencing accumulated in our research activities. After removing duplicates and irrelevant papers, a total of 620 distinct papers were formally reviewed and summarized. Among them, 22 papers belong to the vertical genomic integrative analysis category and 598 papers were horizontal genomic meta-analysis. Of the 598 papers, 333 papers were related to microarray meta-analysis, 256 papers were in the GWAS meta-analysis category and 9 papers were meta-analysis of other categories (e.g. copy-number variation or genome-wide linkage scan). The flow diagram is shown in Figure 2 .

Flow chart of paper collection and categorization. Papers were collected from PubMed search and manual collection. After removing duplicates and irrelevant papers, 620 papers were formally reviewed. Commands used in PubMed search: a (“meta-analysis”[Title/Abstract]) AND ((“microarray”[Title/Abstract]) OR (“expression profiles”[Title/Abstract]) OR (“expression profile”[Title/Abstract]) OR (“gene expression”[Title/Abstract]) OR (“Affymetrix”[Title/Abstract]) OR (“Illumina”[Title/Abstract])); b (“meta-analysis”[Title/Abstract]) AND (“genome-wide association”[Title/Abstract]); c (“meta-analysis”[Title/Abstract]) AND ((“CGH”[Title/Abstract]) OR (“CNV”[Title/Abstract]) OR (“copy number”[Title/Abstract])); d (“meta-analysis”[Title/Abstract]) AND ((“miRNAs”[Title/Abstract]) OR (“miRNA”[Title/Abstract]) OR (“microRNAs”[Title/Abstract])).

Figure 3 illustrates a summary of our microarray meta-analysis review. Detailed information of the paper list and categorization to generate Figure 3 is available in the Supplementary Data . Of the 333 microarray meta-analysis papers, 7 (2%) were descriptive review without quantitative information integration, 42 (13%) were meta-analysis on one or several targeted genes (not at genome-wide scale) and the remaining 284 (85%) represented genome-wide meta-analysis on a global basis ( Figure 3 A). In Figure 3 B, the 333 papers were categorized into review papers (11 papers; 3%), biological applications (201 papers; 60%), novel methodologies (83 papers; 25%) and database/software (38 papers; 12%). For different purposes of meta-analysis shown in Figure 3 C, the majority of papers targeted on DE gene or pathway detection (218 papers; 66%). Other purposes include ‘network or co-expression analysis’ (32 papers; 10%), ‘classification analysis’ (25 papers; 8%), ‘reproducibility or bias analysis’ (19 papers; 6%) and ‘others’ (34 papers; 10%). We will further survey these various meta-analysis purposes later in ‘Purposes of microarray meta-analysis’ section. Since two-thirds (218 papers; 66%) of the microarray meta-analysis papers were related to DE gene or pathway detection which conceptually were extensions from traditional meta-analysis, we scrutinized this category and summarized four types of statistical methodologies used ( Figure 3 D). Of the 191 papers that could be clearly categorized, 81 papers (42%) used meta-analysis methods that combine P -values from individual studies, while 41 papers (22%) combined effect sizes, 18 papers (9%) combined ranks and 51 papers (27%) directly merged data after proper normalization. ‘Types of meta-analysis methods’ section will go over these four types of statistical methodologies in more detail.

Summary of microarray meta-analysis review. ( A ) Types of information integration; ( B ) Types of paper; ( C ) Purposes of meta-analysis; and ( D ) Types of statistical methods for DE gene detection.

When the term ‘microarray meta-analysis’ is used, it usually means meta-analysis for DE gene (or marker) detection. Although two-thirds of identified publications ( Figure 3 C) were of this type, microarray studies have also been combined for many other biological purposes, as described below.

DE gene detection (218 papers)

DE gene detection is a commonly used downstream analysis in microarray that identifies genes differentially expressed across two or more conditions with statistical significance and/or biological significance (e.g. fold change). In the simple case that we are looking at one gene, this type of analysis is usually performed using a two-sample t -test or a Wilcoxon rank-sum test. However, when this analysis is performed genome-wide, a major issue becomes the fact that there can be many spurious associations that are expected by chance. To counteract this problem, some type of multiple comparisons adjustment is usually done; a popular one is to use the q -value ( 1 ). The task is usually a first step to identify gene targets for understanding genetic mechanisms under a disease or for guiding the search of treatment targets. From Figure 3 C, detection of DE genes covers two-thirds of papers (218 papers) in the microarray meta-analysis literature. Most existing methods or applications are for two-class comparison (e.g. identify DE genes comparing cases versus controls). Other types of outcome variables (e.g. multi-class, continuous, censored survival or time series) have also been considered in microarray meta-analysis ( 2 ). Details of these methods will be further described in ‘Types of meta-analysis Methods’ section.

Pathway analysis

Pathway analysis (a.k.a. gene set analysis) is a statistical tool to infer correlation of differential expression evidence in the data with pathway knowledge from established databases ( 3 , 4 ). The idea behind pathway analysis is to determine if there is enrichment in the detected DE genes based on an a priori defined biological category. Such a category might come from one or multiple databases such as Gene Ontology (GO; www.geneontology.org ), the Kyoto Encyclopedia of Genes and Genomes (KEGG; http://www.genome.jp/kegg/ ), Biocarta Pathways ( http://www.biocarta.com/ ) and the comprehensive Molecular Signatures Database (MSigDB; http://www.broadinstitute.org/gsea/msigdb/ ). For the majority of recent microarray meta-analysis applications, pathway analysis has been a standard follow-up to identify pathways associated with detected DE genes [e.g. ( 5 ) and many others]. The result provides more insightful biological interpretation and it has been reported that pathway analysis results are usually more consistent and reproducible across studies than DE gene detection ( 6 ). Shen and Tseng ( 7 ) developed a systematic framework of Meta-Analysis for Pathway Enrichment (MAPE) by combining information at gene level, at pathway level and a hybrid of the two.

Network and co-expression analysis (32 papers)

Co-expression analysis and network analysis of microarray data are used to investigate potential transcriptional co-regulation and gene interactions. Network analyses typically work with the gene–gene co-expression matrix, which represents the correlation between each pair of genes in the study. A crucial assumption is that the magnitude of the co-expression between any pair of genes is associated with a greater likelihood that the two genes interact. Thus, networks of interactions between genes are inferred from the co-expression matrix. Many papers have extended this analysis to the meta-analysis scenario. Of the 32 papers identified, some directly merge multiple studies to construct a network as if from a single study ( 8–15 ). Others combine pairwise gene interaction evidence across studies by vote counting ( 16–18 ) or Fisher's ( 19 , 20 ) method, similar to meta-analysis for DE gene detection. Segal et al . ( 21 ) was probably the first large-scale microarray meta-analysis for network or co-expression analysis. They developed a ‘module map’ by combining 1975 arrays in 26 cancer studies to characterize expression behavior of 2849 modules collected from various sources (e.g. Gene Ontology, KEGG pathways and gene expression clusters). Wang et al . ( 22 ) formulated a regularized approach to combine multiple time-course microarray studies for inferring gene regulatory networks. Zhou et al . ( 23 ) proposed a 2nd-order correlation analysis to construct network and functional annotation by combining 39 yeast data sets. Huttenhower et al . ( 24 ) used a scalable Bayesian framework to combine studies for pairwise meta-correlation and predicted functional relationship. Wang et al . ( 25 ) developed a semi-parametric meta-analysis approach for combining co-expression relationships from multiple expression profile data sets to evaluate similarity and dissimilarity of gene network across species. Steele et al . ( 26 ) proposed a weighted meta-analysis Bayesian network based on combining statistical confidences attached to network edges and a consensus Bayesian network to identify consistent network features across all studies.

Inter-study prediction analysis (25 papers)

Prediction analysis (a.k.a. classification analysis or supervised machine learning) is probably the most commonly applied microarray analysis that leads to clinical utility. In this type of analysis, the goal is to construct an improved discrimination between two or more study populations with accuracy beyond existing criteria in clinical practice ( 27 ). There now exists an extensive literature on classification methods for gene expression data; we refer the reader to Perez-Diaz et al . ( 28 ) for a recent review. In a single microarray study analysis, cross-validation has been routinely used by splitting the entire cohort into training and testing groups, constructing a prediction rule in the training group and finally validating in the test group. To demonstrate validity of microarray signatures or prediction models in other studies, two major strategies for developing prognostic signatures have been pursued. The first approach focuses on validity of biomarkers in external data. The prognostic signatures (a small number of genes) generated from training data are usually subsequently developed from a more traditional platform such as qRT–PCR. Reasons for failure of external validation in this regard have been widely surveyed and discussed in the literature ( 27 , 29–35 ). The second type of external validation focuses on inter-study prediction (i.e. construct a prediction model in one study and use the model to make predictions in another study). Although external validation of a gene expression-based prediction model has been shown valid in some publications ( 36 , 37 ), it has been found to be difficult in general. The failure of direct inter-study prediction is mainly due to discrepancy of probe design and experimental protocols across array platforms, plus possible heterogeneous patient cohorts across studies. Some reports avoided the major cross-platform obstacle by directly merging studies of the same platform (usually Affymetrix) to construct a prediction signature ( 38–42 ) and conventional cross-validation can be performed. Others developed sophisticated normalization techniques to solve or alleviate such a problem, including cross-platform normalization (XPN) ( 43 ), distance-weighted discrimination (DWD) ( 44 ), ratio-adjusted gene-wise normalization (rGN) ( 45 ) and module-based prediction (MBP) ( 46 ). In these approaches, data are normalized across studies so the prediction model can be applied across studies ( 47–50 ). Rank-based robust approaches have also been used ( 41 , 51 ).

Reproducibility and bias analysis (19 papers)

Evaluating reproducibility and bias across microarray studies was an important topic, especially when array technology and experimental protocols were in an early developmental stage. Simple Pearson correlation and Venn diagrams have been widely used ( 52–55 ). Other sophisticated statistical measures have been proposed to quantify similarity of any two microarray studies, including integrative correlation coefficient ( 56 ), similarities of ordered gene lists (SOGL) ( 57 , 58 ), BayesGen ( 59 ) and co-inertia analysis (CIA) ( 60 ).

Others (34 papers)

Additional purposes of microarray meta-analysis include: (i) discover or validate disease subtypes ( 61–65 ); (ii) predict unknown gene functions ( 66 , 67 ) or transcriptional regulations ( 13 ); (iii) dimension reduction ( 68 ); (iv) gene clustering ( 69 ). Targeted gene detections other than classical DE gene analysis have also been pursued. For example, phase-coupled models ( 70 ) or Bayesian approaches ( 71 ) have been used to combine multiple studies to detect periodic or cell cycle-related genes. Sequence information and gene expression have been combined for cyclic gene detection ( 72 ). Others have also combined large-scale microarray studies to identify house-keeping genes (defined as genes having consistent expression across various cellular or environmental changes) ( 73–75 ) or conversely highly variable genes ( 76 , 77 ).

Many web databases are available for public storage and meta-analysis of microarray data sets. Gene Expression Omnibus (GEO) from NCBI and ArrayExpress from EBI are probably the two largest public repositories. On 3 April 2011, GEO contained 22 170 data series and 546 633 samples. Several other databases are housed in specific universities or groups, including Stanford Microarray Database (SMD), caArray at NCI, UPenn RAD Database, UNC Microarray Database, Yale Microarray Database, MUSC Database and UPSC-BASE. These websites are considered primary databases, where the main purpose is to provide downloadable and searchable microarray data sets. Other secondary databases import data sets from primary data archives, preprocess the data, perform in-depth analyses and deliver it through convenient interfaces for fast query, data mining and information integration. GEO Profiles and Gene Expression Atlas ( 78 ) are two secondary databases that accompany GEO and ArrayExpress. Other secondary databases include Genevestigator ( 79 ), ArrayTrack ( 80 ), Gemma, NextBio ( 81 ), LOLA ( 82 ), L2L ( 83 ), A-MADMAN ( 84 ), PrognoScan ( 85 ), MiMiR ( 86 ), Microarray retriever ( 87 ), TranscriptomeBrowser ( 88 ), M 2 DB ( 89 ), MAMA ( 90 ) and GeneSigDB ( 91 ). These tools contain various types of gene signature, regulatory network and differential expression information available for fast query, retrieval and evaluation.

In addition to the general-purpose microarray databases listed above, many databases are specialized to particular disease or species, including aging databases [AGEMAP ( 92 ) and Gene Age Nexus ( 93 )], Pancreatic Expression database ( 94 ), COXPRESdb for gene networks in mammals ( 95 ), CYCLONET for cell cycle regulation ( 96 ), HCNet for heart and calcium functional network ( 14 ), and general cancer databases [Oncomine ( 97 ) and Cancer Genome Workbench (CGWB) ( 98 )]. Of these, Oncomine has been used and cited widely in cancer research particularly when only a few targeted genes are scrutinized. While the statistical methods in these databases are relatively simple, a major advantage of these is the ease of use for biological scientists who are generating microarray data sets.

Despite the availability of many web databases and many microarray meta-analysis methods (to be discussed in detail in the ‘Types of meta-analysis methods’ section), there exist surprisingly few user-friendly software packages for microarray meta-analysis implementation, in terms of their documentation and workflow. Compared with popular microarray packages (e.g. SAM, LIMMA or BRB array tool), existing meta-analysis packages are relatively primitive and difficult to use. In the R and Bioconductor environment, GeneMeta (implements fixed and random effects model; http://www.bioconductor.org/packages/release/bioc/html/GeneMeta.html ; version 1.24.20), metaMA (implements random effects model and Stouffer's method; http://cran.r-project.org/web/packages/metaMA/ ; version 2.1), metaArray (implements meta-analysis of probability of expression, POE; http://www.bioconductor.org/packages/release/bioc/html/metaArray.html ; version 1.28.20) ( 99 ), OrderedList (compares ordered gene lists; http://www.bioconductor.org/packages/release/bioc/html/OrderedList.html ; version 1.24.20) ( 100 ), SequentialMA (for determining sensitivity and judge whether more samples are needed to assure firm conclusion) ( 101 ), RankProd (implement rank product method; http://www.bioconductor.org/packages/release/bioc/html/RankProd.html ; version 2.24.20) ( 102 ) and RankAggreg (implements various rank aggregation methods; http://cran.r-project.org/web/packages/RankAggreg/ ; version 0.4-2) ( 103 ) are available. GODiff ( 104 ) ( http://fishgenome.org/bioinfo/godiff/index.htm version 1.2) allows investigation of functional differentiation across studies using Gene Ontology annotation. Integrative Array Analyzer ( 105 ) ( http://zhoulab.usc.edu/iArrayAnalyzer.htm ; version 1.1.13) provides data mining and visualization tools to combine studies for simple co-expression analysis and differential expression analysis. For visualization, UCSC Genome Browser ( 106 ) and Genome Graphs provide flexible tools to compare and explore multiple genomic studies. Other commercial packages, including JMP Genomics from SAS ( http://www.jmp.com/software/genomics/index.shtml ; version 5.1) and Partek Genomic Suite ( http://www.partek.com/software ), also provide similar or more advanced visualization and graphical tools but with less statistical information integration capabilities.

In addition to scarcity of software packages in the field, quality of software packages should be enhanced. The concept of ‘literate programming’ ( 107 ) (e.g. the ‘sweave’ package in R) has been developed for reproducible research and should be promoted in future software development. For example, all packages available in Bioconductor now meet this requirement. Such a programing practice allows users to easily understand program design and rationale in the source code and to reproduce the results by other researchers.

Ramasamy et al . ( 108 ) outlined a seven-step practical guidelines for conducting microarray meta-analysis: ‘(1) Identify suitable microarray studies; (2) Extract the data from studies; (3) Prepare the individual datasets; (4) Annotate the individual datasets; (5) Resolve the many-to-many relationship between probes and genes; (6) Combine the study-specific estimates; (7) Analyze, present, and interpret results’. In the section below, we will focus on steps 6 and 7 for DE gene detection of microarray meta-analysis. We will discuss four major types of statistical meta-analysis methods in the ‘Types of meta-analysis methods’ section. In the ‘Statistical considerations behind the methods’ and ‘A case study’ sections, related statistical considerations and a case study are discussed to illustrate the issue of choosing a suitable method.

Types of meta-analysis methods

As shown in Figure 3 C, microarray meta-analysis for DE gene detection is a commonly encountered application. In this sub-section, we will discuss four categories of methods to combine information for DE gene detection: combine P -values, combine effect sizes, combine ranks and directly merge after normalization. In addition to these major categories, sophisticated latent variable approaches have also been developed.

Combining P -values (81 papers)

Combining P -values from multiple studies for information integration has a long history in statistical science. It has two major advantages (e.g. compared with another popular category of combining effects sizes below), including its simplicity and extensibility to different kinds of outcome variables. When the outcome variable is not binary (e.g. multi-class, continuous or censored survival), effects sizes may not be well defined, while association P -values can still be calculated. Below, we briefly introduce five P -value combination methods and use the examples in the ‘A case study’ section for illustration later. A major advantage of the P -value-based approaches is that they allow for standardization of the associations from genomic studies to a common scale.

Results of the case study

PT: primary tumor Met: metastasis .	Types of hypothesis setting .	Total number of detected DE genes (FDR = 1%) .	PTTG1 .	FOLR3 .	TPM2 .	BRAF .
Study analysis
Lapointe (62 PT, 9 Met)	–	364	= 1.6E-3; = 1.5E-2; FC = 2.75	= 0.65; = 0.80; FC = 0.92	= 9.4E-7; = 9.3E-5; FC = 0.36	= 2.9E-4; = 5E-3; FC = 1.65
Tomlins (30 PT, 19 Met)	–	598	= 4.7E-7; = 3.4E-5; FC = 1.42	= 1E-20; = 0; FC = 0.58	= 0.92; = 0.95; FC = 0.99	= 3.4E-3; = 1.9E-2; FC = 0.81
Varambally (7 PT, 6 Met)	–	587	= 1.7E-4; = 3E-3; FC = 8.49	= 0.96; = 0.97; FC = 1.02	= 1E-20; = 0; FC = 0.04	= 1.4E-2; = 4.8E-2; FC = 0.58
Yu (65 PT, 25 Met)	–	1073	= 4.7E-7; = 8.1E-6; FC = 3.34	= 0.43; = 0.56; FC = 1.13	= 1E-20; = 0; FC = 0.16	= 8.5E-6; = 9E-5; FC = 2.3
Meta-analysis
Fisher	HS	2287	= 0; = 0	= 0; = 0	= 0; = 0	= 4E-10; = 3E-9
Stouffer	HS	1472	= 0; = 0	= 1.1E-5; = 4.9E-3	= 0; = 0	= 0.36; = 0.97
minP	HS	1740	= 4E-20 ( = 4E-19)	= 4E-20 ( = 4E-19)	= 4E-20 ( = 4E-19)	= 1E-5 ( = 9E-5)
AW	HS	2312	= 0 ( = 0) (1,1,1,1)	= 0 ( = 0) (0,1,0,0)	= 0 ( = 0) (1,0,1,1)	= 0 ( = 0) (1,1,1,1)
RankSum
Up	HS	672	= 0 ( = 0)	= 0.93 ( = 1)	= 1 ( = 1)	= 2E-6 ( = 4E 5)
Down	HS	626	= 1 ( = 1)	= 0.06 ( = 0.23)	= 0 ( = 0)	= 0.99 ( = 1)
RankProd
Up	HS	490	= 0 ( = 0)	= 0.84 ( = 1)	= 1 ( = 1)	= 0 ( = 0)
Down	HS	462	= 1 ( = 1)	= 0.02 ( = 0.02)	= 0 ( = 0)	= 0.99 ( = 1)
Vote counting
S ≥ 3, = .01	HS or HS	453	Yes	No	Yes	Yes
S ≥ 3, = .05	HS or HS	1021	Yes	No	Yes	Yes
S = 4, = .01	HS or HS	80	Yes	No	No	Yes
S = 4, = .05	HS or HS	217	Yes	No	No	Yes
Random effects model	HS	350	= 2E-14 ( = 1E-11)	= 0.33 ( = .56)	= 0.002 ( = 0.02)	= 0.89 ( = 0.95)
maxP	HS	549	= 2E-19 ( = 2E-16)	= 0.79 ( = 0.86)	= 0.05 ( = 0.13)	= 2E-8 ( = 1E-6)

PT: primary tumor Met: metastasis .	Types of hypothesis setting .	Total number of detected DE genes (FDR = 1%) .	PTTG1 .	FOLR3 .	TPM2 .	BRAF .
Study analysis
Lapointe (62 PT, 9 Met)	–	364	= 1.6E-3; = 1.5E-2; FC = 2.75	= 0.65; = 0.80; FC = 0.92	= 9.4E-7; = 9.3E-5; FC = 0.36	= 2.9E-4; = 5E-3; FC = 1.65
Tomlins (30 PT, 19 Met)	–	598	= 4.7E-7; = 3.4E-5; FC = 1.42	= 1E-20; = 0; FC = 0.58	= 0.92; = 0.95; FC = 0.99	= 3.4E-3; = 1.9E-2; FC = 0.81
Varambally (7 PT, 6 Met)	–	587	= 1.7E-4; = 3E-3; FC = 8.49	= 0.96; = 0.97; FC = 1.02	= 1E-20; = 0; FC = 0.04	= 1.4E-2; = 4.8E-2; FC = 0.58
Yu (65 PT, 25 Met)	–	1073	= 4.7E-7; = 8.1E-6; FC = 3.34	= 0.43; = 0.56; FC = 1.13	= 1E-20; = 0; FC = 0.16	= 8.5E-6; = 9E-5; FC = 2.3
Meta-analysis
Fisher	HS	2287	= 0; = 0	= 0; = 0	= 0; = 0	= 4E-10; = 3E-9
Stouffer	HS	1472	= 0; = 0	= 1.1E-5; = 4.9E-3	= 0; = 0	= 0.36; = 0.97
minP	HS	1740	= 4E-20 ( = 4E-19)	= 4E-20 ( = 4E-19)	= 4E-20 ( = 4E-19)	= 1E-5 ( = 9E-5)
AW	HS	2312	= 0 ( = 0) (1,1,1,1)	= 0 ( = 0) (0,1,0,0)	= 0 ( = 0) (1,0,1,1)	= 0 ( = 0) (1,1,1,1)
RankSum
Up	HS	672	= 0 ( = 0)	= 0.93 ( = 1)	= 1 ( = 1)	= 2E-6 ( = 4E 5)
Down	HS	626	= 1 ( = 1)	= 0.06 ( = 0.23)	= 0 ( = 0)	= 0.99 ( = 1)
RankProd
Up	HS	490	= 0 ( = 0)	= 0.84 ( = 1)	= 1 ( = 1)	= 0 ( = 0)
Down	HS	462	= 1 ( = 1)	= 0.02 ( = 0.02)	= 0 ( = 0)	= 0.99 ( = 1)
Vote counting
S ≥ 3, = .01	HS or HS	453	Yes	No	Yes	Yes
S ≥ 3, = .05	HS or HS	1021	Yes	No	Yes	Yes
S = 4, = .01	HS or HS	80	Yes	No	No	Yes
S = 4, = .05	HS or HS	217	Yes	No	No	Yes
Random effects model	HS	350	= 2E-14 ( = 1E-11)	= 0.33 ( = .56)	= 0.002 ( = 0.02)	= 0.89 ( = 0.95)
maxP	HS	549	= 2E-19 ( = 2E-16)	= 0.79 ( = 0.86)	= 0.05 ( = 0.13)	= 2E-8 ( = 1E-6)

Results of DE gene detection from individual study analysis and meta-analysis (using nine different methods) are listed. Four representative genes are scrutinized for the P -value and q -value results.

Despite availability of powerful statistical tools described above, many biological applications we surveyed chose to apply naïve Venn diagram (used in 21 papers in our survey) or vote counting methods (used in 24 papers) for convenience. Venn diagram is a useful visualization tool, when combining few (usually 2–4) studies, to demonstrate the intersection and union distribution of DE gene lists detected by each individual study under a fixed threshold (e.g. FDR = 5%). The naïve diagram, however, does not perform real information integration but only displays a consistency summary. When many studies are combined, naïve vote counting is often chosen by biologists instead. For each gene, the method simply counts the number of studies with P -values under a given threshold (e.g. P < 0.05). In the statistical literature, it is well known that vote counting is statistically inefficient ( 113 , 114 ). On the other hand, vote counting is useful when raw data and complete P -value information of all genes are unavailable while only a list of DE genes under certain P -value threshold is available. This happened frequently in many early microarray studies, in which DE gene lists were summarized in supplemental tables of publications but raw data were not uploaded to public domain. Due to the significant loss of information and efficiency, the vote counting method should be avoided whenever possible in the applications.

Combining effect sizes (41 papers)

Many meta-analysis methods have been based on the assumption that the standardized effect sizes are combinable across studies. Fixed and random effects models (FEM & REM) are the two most popular approaches in this category. In FEM, the estimated effect size in each study is assumed to come from an underlying true effect size plus measurement error (that may come from experimental or population sampling error). In REM, each study further contains a random effect that can incorporate unknown cross-study heterogeneities in the model. Choi et al . ( 115 ) was among the first to apply these models to microarray meta-analysis. In a given application, a Q -statistic was used to determine the need for a random effects model and the underlying effect size was estimated under FEM or REM. Bayesian meta-analysis was also developed with Markov Chain Monte Carlo (MCMC) simulation to estimate the underlying effect size. Others have also developed different variations of effect size models ( 116–118 ).

Combining ranks (18 papers)

One apparent downside of methods combining P -values or effect sizes is that the results can often be dominated by outliers. This can be a significant problem when thousands of genes are analyzed simultaneously in the noisy nature of microarray experiments. Methods combining robust rank statistics are used to alleviate this problem. Instead of P -values or effect sizes, the ranks of DE evidence are calculated for each gene in each study. The product, mean ( 119 ) or equivalently sum ( 120 ) of ranks from all studies is then calculated as the test statistic. Permutation analysis can be performed to assess the statistical significance and to control FDR. Hong et al . ( 102 ) proposed a more advanced RankProd algorithm that calculates the product of the ranks of fold change in each inter-group pair of samples. In a follow-up comparative study, they showed its better performance as compared to Fisher's method and the random effects model ( 121 ). DeConde ( 122 ) applied various ‘rank aggregation’ methods, which were developed for the meta-search problem for combining top-k lists in the computer science literature. The methods effectively aggregate the rankings of, say the top 100 most upregulated or downregulated genes in each study.

Directly merging the raw data (51 papers)

Despite the concern of heterogeneity across studies, many microarray meta-analysis applications chose to normalize across studies and directly merge data sets for DE gene detection. This approach is often called ‘mega-analysis’, especially in GWAS meta-analysis. In microarray meta-analysis, such applications usually restrict selection of studies from the same or similar array platform, e.g. a single Affymetrix U133 or multiple Affymetrix platforms ( 38 , 123 ). The collection of only Affymetrix arrays allows pre-processing by model-based robust multi-array (RMA) normalization ( 124 ) on the CEL files of all samples simultaneously. Others have developed advanced normalization techniques to eliminate cross-study discrepancy and allow direct merge of studies [e.g. XPN ( 43 ), DWD ( 44 ) and rGN ( 45 )]. Although direct merging can be attractive in applications for its convenience, cautions have to be taken that normalizations do not guarantee to remove all cross-study discrepancies. In fact, Goldstein et al . ( 125 ) demonstrated that RMA does not remove batch effects even when two studies are from the same lab and same Affymetrix platform but performed at different time.

Latent variable approaches

There are more sophisticated approaches in place that attempt to model the pre-processed microarray data sets using latent variable-based models and attendant inference using either expectation–maximization routines or Markov Chain Monte Carlo algorithms. For example, the probability of expression (POE) was a latent variable used in several papers that was not observable in the data but could be inferred from other observed variables. Papers of this category include metaArray ( 99 ) which employs two types of inferential strategies, frequentist and Bayesian (see the ‘Statistical considerations behind the methods’ section) for modeling data from multiple platforms, and XDE ( 126 ), which fits a joint parametric Bayesian model for multi-study meta-analysis. In particular, the latter paper shows some compelling simulation evidence for a joint modeling strategy using these latent variable models. For more specialized settings, Conlon et al . ( 127 ) and Fan et al . ( 71 ) have presented Bayesian modeling approaches for combining data from multiple microarray studies. While the hierarchical models used in these papers are statistically more sophisticated than the methods described in the previous section, they offer the potential of pooling information across genes to sharpen inferences about which genes are differentially expressed. However, due to their complexity, they have not been used much in practice. One notable exception is Shen et al . ( 128 ), which applied a precursor of the metaArray algorithm to identification of gene expression signatures for aggressive breast cancer.

Statistical considerations behind the methods

Null and alternative hypothesis assumptions behind the methods.

Although the concept of combining studies for meta-analysis is seemingly straightforward, the targeted biomarker characteristics implicitly reflected by different statistical hypothesis settings behind the methods can be varied. Following the convention of Birnbaum ( 129 ), Li and Tseng ( 111 ) presented two major hypothesis settings behind microarray meta-analysis methods described in the ‘Types of meta-analysis methods’ section. Suppose K studies are combined and θ k is the effect size of study k . The first hypothesis setting (HS A ) detects candidate genes differentially expressed in ‘all’ studies ( H 0 : θ 1 = θ k = 0 for one or more k versus H a : θ k ≠ 0, 1 ≤ k ≤ K ) whereas, HS B identifies markers differentially expressed in ‘partial’ (one or more) studies ( H 0 : θ 1 = … = θ k = 0 versus H a : θ k ≠ 0 for one or more k ). For example, Fisher's method takes sum of log-transformed P -values as the statistics. If, for a given gene, a study has very significant P -value (e.g. P = 1E-20) but all other studies do not have significant P -values (e.g. the FOLR3 gene in the ‘A case study’ section), the Fisher's method still concludes a large Fisher's score and declares this gene as a DE gene. As a result, Fisher's method pursues the second hypothesis setting, HS B . Similarly, Stouffer, minP, maxP, AW, as well as rank sum and RankProd, all adopt similar hypothesis setting HS B . On the other hand, the maxP method takes the maximum P -value as the statistics. It requires that P -values from all studies are small and thus it pursues the first hypothesis setting, HS A . The random effects model has the same hypothesis setting that all studies have the same overall effect size while each study may contain an additional random effect component. One might somewhat relax HS A to detect genes differentially expressed in ‘majority’ of studies (denoted as HS A− ). The vote counting method follows this relaxed hypothesis setting. The hypothesis setting of each method is presented in Table 1 .

Frequentist versus Bayesian inference

Implicit in the discussion about inference has been the use of a frequentist framework. In particular, we assume that there is a test statistic, larger values which indicate stronger evidence against the null hypothesis. However, one could also perform Bayesian hypothesis testing using these hypotheses. This is done by consideration of posterior probabilities of the specific hypotheses (e.g. P( θ 1 = … = θ k = 0|data) versus P( θ k ≠ 0 ∀ k |data)). Computation of these posterior probabilities requires the use of a likelihood for the parameters of interest along with prior probabilities of the specific hypotheses being tested. The prior probabilities are typically selected based on the relative costs of a type I error (rejecting the null hypothesis when it is true) versus a type II error (accepting the null hypothesis when it is false). The larger the relative cost, the larger the prior probability for the null hypothesis should be. Bayesian hypothesis testing procedures are amenable with the latent variable models for meta-analysis described in the ‘Databases and software’ section. In the literature, another advantage of Bayesian approach is the use of Bayes factor that does not require a prior probability of the two hypotheses and can work as an alternative of classical hypothesis testing.

Consistent up or downregulation

Comparing the first three categories of meta-analysis methods in the ‘Types of meta-analysis methods’ section, combining effects sizes (e.g. random or fixed effects model) automatically identifies genes that have consistent up or downregulation in all studies. This may not be the case for methods combining P -values or ranks if the P -values and ranks are obtained from two-sided hypothesis testing. In this case, up- and down-regulation are treated as equally strong evidence and a gene may be detected from the meta-analysis with strong up-regulation evidence in one study but strong down-regulation evidence in another study, which leads to confusing conclusions. Theoretically, the discordance may reflect underlying biological truth due to population heterogeneity but it may as well be a result of technical artifacts such as gene annotation mistakes or cross-hybridization. Distinguishing the two is often a difficult, if not impossible, task. A convenient solution to avoid detecting genes with such discordances is by combining P -values or ranks from one-sided tests. For example, a modified Stouffer's method can apply a z-transformation that automatically utilizes one-sided tests and splits up- and downregulation evidences into positive and negative z-scores, respectively. Owen ( 130 ) applied a similar Pearson one-sided test adjustment for Fisher's method and the modification can be extended to minP, maxP and other methods. Note that the consistent up- or downregulation issue only exists in two-class comparison in DE gene detection and does not apply to other types of response variables (e.g. multi-class, continuous or survival).

A case study

To illustrate some properties of the methods described in the ‘Types of meta-analysis methods’ section, we performed a simple case study. The motivation of this small case study was to help understand how the algorithm of each method works and to explain pros and cons of each method. The result provides general insight for selecting an adequate method in applications. This case study is, however, neither comprehensive nor conclusive enough as a comparative study to judge performance of the methods. In this case study, four prostate cancer expression profiles (Lapointe, Tomlins, Varambally and Yu) containing metastasis versus primary tumor samples were combined for meta-analysis. After gene matching by official gene symbols, pre-processing and filtering, 4260 genes were analyzed in the meta-analysis. We used the R package ‘siggenes’ to perform DE gene analysis in each study. ‘siggenes’ allows implementation of the Significance Analysis of Microarray (SAM) method and the Empirical Bayes Analyses of Microarrays (EBAM) method. For simplicity, we applied the popular SAM method with B = 500 permutation. According to Phipson and Smyth ( 131 ), the P -values from permutation analysis should never be zero but the ‘siggenes’ package does occasionally generate zero P -values. If P = 0 is obtained for a certain gene in an individual study, we set it to P = 1E-20 to avoid failure of logarithmic or inverse normal transformation in the Fisher's and Stouffer's methods. After P -values are generated, Benjamini–Hochberg procedure is applied to calculate q -values and correct for multiple comparison (‘p.adjust’ function in R is used). The random effects model was implemented using the ‘GeneMeta’ package in R. RankSum and RankProd methods were performed in the R package ‘RankProd’. In the ‘RankProd’ package, the RankSum and RankProd methods could only be implemented with up- and downregulation analysis separately. Theoretically, it is easy to modify the algorithm to analyze up- and downregulation simultaneously. For the vote counting method, the method determines a DE gene if it has P -values smaller than a threshold P in greater or equal to S studies among the four studies combined. In Table 1 , we list results for P = 0.01 or 0.05 and S = 3 or 4. Table 1 shows results of four single-study analyses and nine meta-analysis methods in four selected genes.

The first example gene, ‘PTTG1’, was up-regulated in the metastatic group with strong statistical significance in all four studies ( P = 1.9E-5, 1E-20, 2E-5 and 1E-20). As expected, all nine meta-analysis methods concluded very strong statistical significance even after multiple comparison correction. As a comparison, the second selected gene ‘FOLR3’ was down-regulated with strong statistical significance in the Tomlins study ( P = 1E-20; fold change FC = 0.58) but was not statistically significant in the other three studies ( P = 0.65, 0.96 and 0.43). Such sporadic high statistical significance in a subset of studies might be a result of unknown experimental artifacts (e.g. non-specific probe design that causes cross-hybridization in the cDNA array design) but might instead be a biological truth in the specific cohort. Fisher, minP, AW, RankSum and RankProd all obtained strong to moderate statistical significance after meta-analysis for this gene (see FOLR3 column in Table 1 ). This reflected the underlying HS B hypothesis setting of these methods to detect a DE gene if the gene is differentially expressed in one or more studies (see ‘Statistical considerations behind the methods’ section). On the other hand, vote counting, the random effects model and maxP required a gene to be differentially expressed in all or ‘majority’ of the studies (i.e. hypothesis setting HS A ) and thus did not generate significant q -values. The third gene, ‘TPM2’, was differentially expressed in three studies ( P = 9.4E-7, 1E-20 and 1E-20 in Lapointe, Varambally and Yu) but not differentially expressed in Tomlins ( P = 0.92). Among the nine methods, it was detected by seven methods, excepting only maxP ( q = 0.13) and vote counting (S = 4). This result shows that methods to detect genes differentially expressed in ‘all’ studies might be too stringent and could ignore an important marker gene when many studies are combined. It was interesting that, in the random effects model, although it is aimed at HS A , the random effects assumption provided robustness so that TPM2 was statistically significant ( q = 0.02). The fourth example gene, ‘BRAF’, was differentially expressed in all four studies but was surprisingly down-regulated in two studies but up-regulated in the other two studies. Among the nine methods, Fisher, minP, AW, vote counting and maxP detected BRAF as a DE gene because the methods combined two-sided P -values without distinguishing DE direction. RankSum and RankProd, although considered DE directions in the algorithm, still determined BRAF as an upregulated DE gene. Stouffer and random effects model were two methods that considered DE directions in the algorithm and generated non-significance q -values. Whether detecting a discordant gene like BRAF is favorable or not depends on the inferential goals of the experiment. It can be the case that BRAF is an important marker and the discordance is generated from an unknown meaningful confounding variable (e.g. race; say, BRAF is up-regulated in black but down-regulated in white). It is equally possible that the discordance comes from unknown technical artifacts.

Below, we further scrutinize the biological functions of the four genes using the NCBI database. PTTG1 has been related to DNA repair, cell division and mitosis cell cycle and has been correlated with tumor aggressiveness in multiple tumors. The strong statistical significance in all four studies is biologically verified. On the contrary, there is no direct evidence of cancer association found for FOLR3. The strong DE statistical significance in the Tomlins study might indeed be an artifact. For TPM2, a recent paper has identified a novel splice variant of TPM2 related to prostate cancer cell lines ( 132 ). The high statistical significance in three out of four studies might be strong enough evidence for its association with metastasis. The fourth gene, ‘BRAF’, plays a role in regulating the MAP kinase/ERKs signaling pathway, has been associated to multiple cancers and is in the KEGG prostate cancer pathway (05215). Indeed, the confusing discordant direction of fold changes might be the result of unknown confounding factors such as age or race. Further investigation of demographic or experimental information for the four studies might help elucidate the mystery. We also note that interpretation of detected DE genes also depend on other genes due to gene dependency.

Despite the popularity of microarray meta-analysis, many issues remain unresolved that can hamper the effectiveness of its application. In this section, we discuss a few open questions and related problems.

Quality assessment and inclusion/exclusion criteria

To date, the decision to include or exclude microarray studies in a meta-analysis has been mostly ad hoc and subjective in the literature. Researchers usually apply arbitrary criteria, such as number of samples or array platforms (e.g. ( 112 , 133 , 134 ) and many others), to make the decision. Inclusion of a low quality or outlying study into the meta-analysis, however, can greatly reduce the statistical power or even result in a false conclusion. As a first step, keyword searching in primary data repositories can provide a useful initial screening to identify studies to combine. Some biological terminology systems (e.g. Unified Medical Language System, UMLS) may help provide a refined and unbiased selection for more homogeneous studies. Ramaswamy et al . ( 108 ) has suggested to apply the integrative correlation technique by Parmagiani et al . ( 56 ) to select ‘reproducible’ genes for meta-analysis. This approach potentially can be extended for objective inclusion/exclusion decisions. In general, a data-driven quantitative evaluation for inclusion/exclusion criteria is still an open question in the field. This is tied to the classical question of between-study variation. In the case of a single gene, the issue of between-study variation has been carefully studied; a review of available methods can be found in ( 135 ). How to adapt this to the genomic, high-dimensional data setting is still an open question. This issue is also discussed in the companion paper for GWAS meta-analysis, under the terminology of ‘heterogeneity’.

Practical guidelines from large-scale comparative study and simulation

Among the papers we have surveyed, only two papers performed systematic comparative analysis on microarray meta-analysis methods: Hong et al . ( 121 ) and Campain and Yang ( 136 ). Although the two studies provided insightful conclusions, the number of methods compared (three and five methods, respectively) and the number of real examples examined (two and three examples, respectively with each example combining only 2–5 microarray studies) were very limited. Some key conclusions from the two papers were even contradictory. A large-scale comparative study and simulation study with adequate evaluation measures will help provide insights and practical guidelines for choosing the ‘best’ meta-analysis method(s) in practice.

Combining studies with censored information

As mentioned in ‘Types of meta-analysis methods’ section, vote counting has a natural advantage to combine information from studies with censored P -value information (i.e. raw data are not accessible but only a top ranked DE gene list under certain P -value threshold is available), though it suffers greatly from low statistical power. Although many grant agencies and journals now enforce data sharing policies, many old studies or new studies funded by private foundations are still not openly accessible. Studies with censored information can be an obstacle for meta-analysis. Researchers are forced to either drop studies with censored information or use inefficient vote counting methods in the meta-analysis. In the literature, Bushman and Wang ( 137 ) have transformed P -values to pseudo effect sizes to combine vote counting and effect size combination methods. Extension of other existing methods, such as Fisher, Stouffer and maxP, to analyze such censored P -value data in partial studies will provide a more powerful solution to this practical problem.

Meta-analysis to guide and design future studies

In modern evidence-based medicine, meta-analysis is often used (or required) to combine existing evidence in the literature when planning for a new study. Similarly, genomic meta-analysis should be used more frequently to narrow down gene targets or scope of study when designing new studies (e.g. targeted sequencing).

Meta-analysis on a pathway basis

While the work of authors such as Shen et al . ( 37 ) and Shen and Tseng ( 7 ) has led the way in the area of combining information from multiple studies at the pathway level, there are several issues that remain to be addressed. Adjusting for inference due to pathway dependence remains an important open problem, as the dependence in pathway data might render many of the statistical methods available for multiple testing (e.g. q -values/false discovery rate control) invalid.

Development of user-friendly software

In our review, only a few microarray meta-analysis methods are developed with R packages. When we tested the packages, most of them either did not have clear manuals or had functions that were not easy to apply (especially compared with mature and popular microarray packages such as SAM, PAM, LIMMA, BRB Array Tool or GSEA). Convenient R packages or packages in a programmable environment will allow researchers to test and compare methods and motivate further methodological development. Software with friendly graphical user interfaces (GUI) will further assist biologists in daily applications.

Adjust for potential confounding variables

Heterogeneities caused by demographic, clinical and technical variables often exist within and across studies. Failure to consider these variables in the statistical models and meta-analysis can result in reduced statistical power or false positives. In a microarray meta-analysis, these systematic variabilities should be considered and incorporated in the analysis whenever possible. Leek and Storey ( 138 ) proposed surrogate variable analysis (SVA) to further account for unmeasured and unmodeled factors in a genome-wide expression analysis. The result has shown improved sensitivity and accuracy. Similar techniques can be extended to microarray meta-analysis.

In this article, we performed a comprehensive review of microarray meta-analysis and discussed the related statistical issues. Although many methods have been proposed and used in published applications, the detailed meta-analysis workflow and the hypothesis behind the analysis needs more attention. Selection of a suitable method depends on the type of analysis desired (various purposes described in ‘Purposes of microarray meta-analysis’ section) and the hypothesis setting behind each method (‘Statistical considerations behind the methods’ section). In our review, we noticed that easy to use software packages are rare in the field. We have also addressed several important open questions (‘Open questions’ section), including developing a quantitative inclusion/exclusion evaluation, performing comparative study for a practical guideline and adjusting for confounding variables. As many high-throughput experimental technologies are rapidly developed and widely applied nowadays, data management and effective integrative analysis will become more and more essential to fully utilize the rich information contained in the tremendous amount of data. The analytical techniques and concepts may also extend to information integration of other types of genomic data.

One limitation of this review article is the restricted scope of literature search by PubMed. We have attempted to include 102 manually collected references. The inclusion, however, cannot be exhaustive. For example, many related approaches are termed ‘integrative analysis’ in the literature and thus cannot be included in the review. This is especially true in categories other than DE gene analysis (e.g. pathway analysis, prediction analysis or network analysis). We attempted to include ‘integrative analysis’ in the keyword search but failed because it generated thousands of publications with most of them irrelevant to the purpose of this article.

National Institutes of Health (NIH) (R01MH077159 and RC2HL101715, to G.C.T.); (R01HD38979 and R01DE14899, to E.F. and F.B.); NIH (R01GM72007, to D.B.); Huck Institute for Life Sciences (to D.B.). Funding for open access charge: University of Pittsburgh.

Conflict of interest statement . None declared.

The authors thank C. Song, X. Wang and G. Liao for collecting and printing papers.

Google Scholar

Google Preview

Supplementary data

Month:	Total Views:
November 2016	2
December 2016	14
January 2017	4
February 2017	50
March 2017	39
April 2017	30
May 2017	34
June 2017	16
July 2017	37
August 2017	24
September 2017	38
October 2017	27
November 2017	39
December 2017	78
January 2018	57
February 2018	90
March 2018	118
April 2018	55
May 2018	83
June 2018	91
July 2018	54
August 2018	50
September 2018	80
October 2018	75
November 2018	89
December 2018	76
January 2019	102
February 2019	80
March 2019	182
April 2019	126
May 2019	117
June 2019	107
July 2019	100
August 2019	100
September 2019	120
October 2019	149
November 2019	162
December 2019	114
January 2020	123
February 2020	111
March 2020	94
April 2020	67
May 2020	90
June 2020	75
July 2020	104
August 2020	81
September 2020	123
October 2020	100
November 2020	127
December 2020	77
January 2021	117
February 2021	92
March 2021	136
April 2021	117
May 2021	119
June 2021	135
July 2021	74
August 2021	102
September 2021	77
October 2021	76
November 2021	114
December 2021	69
January 2022	132
February 2022	94
March 2022	99
April 2022	128
May 2022	104
June 2022	111
July 2022	128
August 2022	80
September 2022	97
October 2022	72
November 2022	77
December 2022	92
January 2023	101
February 2023	75
March 2023	73
April 2023	72
May 2023	59
June 2023	57
July 2023	62
August 2023	57
September 2023	60
October 2023	109
November 2023	62
December 2023	80
January 2024	79
February 2024	116
March 2024	181
April 2024	98
May 2024	94
June 2024	89
July 2024	72
August 2024	70
September 2024	8

Email alerts

Citing articles via.

Editorial Board

Affiliations

Online ISSN 1362-4962
Print ISSN 0305-1048
About Oxford Academic
Publish journals with us
University press partners
What we publish
New features
Open access
Institutional account management
Rights and permissions
Get help with access
Accessibility
Advertising
Media enquiries
Oxford University Press
Oxford Languages
University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

Cookie settings
Cookie policy
Privacy policy
Legal notice

This Feature Is Available To Subscribers Only

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

A geographical analysis of social enterprises: the case of Ireland

Social Enterprise Journal

ISSN : 1750-8614

Article publication date: 29 April 2024

Issue publication date: 4 July 2024

This study aims to conduct a geographical analysis of the distribution and type of activities developed by social enterprises in rural and urban areas of Ireland.

Design/methodology/approach

The study analyses data of more than 4,000 social enterprises against a six-tier rural/urban typology, using descriptive statistics and non-parametric tests to test six hypotheses.

The study shows a geographical rural–urban pattern in the distribution of social enterprises in Ireland, with a positive association between the remoteness of an area and the ratio of social enterprises, and a lack of capital-city effect related to the density of social enterprises. The analysis also shows a statistically significant geographical rural–urban pattern for the types of activities developed by social enterprises. The authors observe a positive association between the remoteness of the areas and the presence of social enterprises operating in the community and local development sector whereas the association is not significant for social enterprises developing welfare services.

Research limitations/implications

The paper shows the potential of using recently developed rural–urban typologies and tools such as geographical information systems for conducting geographical research on social enterprises. The findings also have implications for informing spatially sensitive policymaking on social enterprises.

Originality/value

The merging of a large national data set of social enterprises with geographical tools and data at subregional level contributes to the methodological advancement of the field of social enterprises, providing tools and frameworks for a nuanced and spatially sensitive analysis of these organisations.

Rural social enterprises
Urban social enterprises
Quantitative research
Social economy organisations

Olmedo, L. , O. Shaughnessy, M. and Holloway, P. (2024), "A geographical analysis of social enterprises: the case of Ireland", Social Enterprise Journal , Vol. 20 No. 4, pp. 499-521. https://doi.org/10.1108/SEJ-09-2023-0105

Emerald Publishing Limited

Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode

1. Introduction

Social and solidarity economy organisations, and especially social enterprises, have recently been brought to the fore by international institutions including the European Commission, the organisation for economic co-operation and development (OECD) and the United Nations ( European Commission, 2021 ; OECD, 2022 ; United Nations, 2023 ). These institutions acknowledge the contribution and potential of social enterprises to address complex challenges such as climate change, ageing population and lack of access to employment for vulnerable groups; namely, due to the combination of social and/or environmental aims with an economic activity and democratic decision-making which characterise social enterprises ( Galera and Borzaga, 2009 ; Defourny and Nyssens, 2017 ).

Social enterprises in Ireland have been traditionally considered relevant actors providing goods and services to disadvantaged communities and enabling work integration of vulnerable groups ( O’Hara and O’Shaughnessy, 2021 ). In 2019, the Irish Government launched the first National Social Enterprise Policy for Ireland, representing a milestone for the recognition and institutionalisation of social enterprises in the country ( Olmedo et al. , 2021 ). This policy establishes an official definition of social enterprises as follows:

An enterprise whose objective is to achieve a social, societal or environmental impact, rather than maximising profit for its owners or shareholders. It pursues its objectives by trading on an ongoing basis through the provision of goods and/or services, and by reinvesting surpluses into achieving social objectives. It is governed in a fully accountable and transparent manner and is independent of the public sector. If dissolved, it should transfer its assets to another organisation with a similar mission ( Government of Ireland, 2019 , p. 8).

This policy recognises, in line with previous research reports on Irish social enterprises ( Hynes, 2016 ; European Commission, 2020 ), the contribution of Irish social enterprises to deliver a wide range of goods and services, as well as supporting the attainment of government policy goals in areas such as labour market activation but also in health care, climate action, social cohesion and rural development.

Despite common features shared across social enterprises, previous research has highlighted differences between social enterprises operating in rural and urban areas in terms of their community focus, leadership style and funding sources ( Smith and McColl, 2016 ; Barraket et al. , 2019 ). The geographical context where social enterprises operate has been acknowledged as a relevant factor for explaining the work of these organisations ( Steiner and Teasdale, 2019 ; Olmedo et al. , 2023 ) and their contribution to urban and rural development ( Angelidou and Mora, 2019 ; Olmedo and O’Shaughnessy, 2022 ). Geographically sensitive research on social enterprises has been developed mainly at the local level ( Mazzei, 2017 ; Jammulamadaka and Chakraborty, 2018 ; Pinch and Sunley, 2016 ), with some research also conducted at the regional level ( Buckingham et al. , 2011 ; Woo and Jung, 2023 ); however, less is known about the differences in the distribution and the type of activities that social enterprises develop in different rural–urban areas of a country. Therefore, the aim of this paper is to explore the distribution and type of activities developed by social enterprises in different rural and urban areas in Ireland.

To achieve this, we use a six-tier rural–urban typology developed by the Irish Central Statistics Office ( CSO, 2019 ) combined with data on 4,335 social enterprises collected in Ireland. Using geographical information systems (GIS) we georeferenced social enterprises and tested six hypotheses. The spatially sensitive and quantitative empirical data analysis provided by this study adds knowledge to previous calls for geographical research on social enterprises ( Munoz, 2010 ) and provides relevant evidence for the development of spatially sensitive policies for social enterprises ( Mazzei and Roy, 2017 ).

The rest of the paper is structured as follows, Section 2 presents a literature review on previous geographical research on social enterprises. Section 3 outlines the research framework and the hypotheses of this study. Section 4 explains the methodology used in the research. Section 5 presents the findings of this study, with a subsection presenting descriptive statistics and another presenting the analysis of the hypotheses’ tested. Section 6 discusses the findings and Section 7 outlines the conclusions and limitations of this study, ending with some proposals for further research.

2. Literature review – geographical research on social enterprises

The publication in 2010 of a seminal call “Towards a geographical research agenda for social enterprise” ( Munoz, 2010 ) meant a significant milestone for the development of a geographically sensitive perspective towards the study of social enterprises. Within this body of research [ 1 ], some authors have adopted a micro-geographical perspective to study social enterprises as spaces of well-being ( Munoz et al. , 2015 ). For example, Farmer et al. (2020) used GIS to link specific sites within a social enterprise to the well-being experienced by the employees of three Australian Work Integration Social Enterprises. Their findings show how the social enterprises studied acted as “socially-supportive workplaces which focus on deploying, developing and supporting talents and not simply allocating people to one job in one location for all time” ( Farmer et al. , 2020 , p. 9).

Another stream of studies has focused on the local geography of social enterprises ( Jammulamadaka and Chakraborty, 2018 ). Some of these studies have a specific urban focus. For example, Pinch and Sunley (2016) investigated whether social enterprises in four major UK cities benefited from urban agglomeration effects, concluding that agglomeration enables greater demand for social enterprises goods and services and better access to institutional support, funding, knowledge and networks. Similarly, Mazzei (2017) stressed the influence of “place” on the incentives and opportunities for two social enterprises operating within English cities.

Previous research has also taken a geographical perspective to study social enterprises in rural areas. Drawing from social network theory, Richter (2019) showed how social enterprises operating in rural Austria and Poland act as embedded intermediaries between their localities and supra-regional networks. In studies conducted in rural Scotland, Steiner and Steinerowska-Streb (2012) and Steiner and Teasdale (2019) stated that rural areas are a fertile ground for social enterprises due to characteristics associated to rurality, such as reduced market competitors and high levels of social capital. Moreover, these studies further explain how rural social enterprises use advantages of the rural context, such as the skills and knowledge of retired people who moved to rural localities, to develop social entrepreneurial activities. In a study conducted in rural Scotland exploring social enterprises in addressing social isolation and loneliness, Kelly et al. (2019) concluded that despite these organisations offer more flexible solutions than statutory services, relying on social enterprises as solutions to these challenges is not realistic. This was posited to features associated with the rural context of the study, such as remoteness, small labour markets and depopulation.

This echoes research on social enterprises in rural Ireland conducted by O’Shaughnessy and O’Hara (2016) , who stated that geographic isolation and limited job creation associated to the rural context challenges the development of social enterprises. More recently, Olmedo et al. (2023) showed how social enterprises in three Irish rural localities, through a process of “placial substantive hybridity”, harness and (re)valorise untapped local resources and complement these with extra-local resources to foster social innovation and contribute to an integrated development of their localities.

Geographical research has also been conducted comparing social enterprises operating in rural and urban localities. Smith and McColl (2016) explored the influence of the context in four social enterprises based in Scottish urban and rural communities. The authors found that rural social enterprises show a great linkage between the geographical characteristics of where they are based, their community identity and ownership and type of business developed. Contrarily in the urban social enterprises they studied, it was a social need rather than a geographical aspect which drove the organisations’ aim. In a study conducted in Australia, Barraket et al. (2019) compared 11 locally oriented urban and rural social enterprises resourcefulness strategies. The authors showed the great relevance of community networks within rural based social enterprises to access financial and physical assets; however, those social enterprises based in urban areas were more inclined to leverage public funding related to welfare objectives and resources from corporates.

Despite the plethora of research investigating social enterprises at the urban and rural levels, few studies have researched social enterprises through a regional perspective. In this regard, Buckingham et al. (2011) attempted to unmask the “enigmatic regional geography of social enterprises in the UK” using statistical data from different surveys related to social enterprises conducted between 2005 and 2009. The authors concluded that interregional variations (north–south and east–west) were relatively small and without statistical significance; except for high levels of social enterprise activity in London due to its dynamic and innovative business environment and the effect that headquarters location of national social enterprises (mainly in London) might have in the data. More recently, Woo and Jung (2023) have explored the regional determinants of the emergence of social enterprises in South Korea. Combining longitudinal data sets (2012–2019) from the Korea Social Enterprise Promotion Agency and Korea Statistics and using an entrepreneurial ecosystems perspective, the authors concluded that the emergence of social enterprises is especially significant in regions experiencing government or market failure and in regions with greater incidences of start-ups, human capital and financial resources.

At the national (country) and international level, research on social enterprises has been mainly conducted from an institutional perspective, influenced by the seminal work of Kerlin (2013) and the international comparative social enterprise models project ( Defourny and Nyssens, 2017 ; Defourny et al. , 2020 ), with scarce studies adopting a geographical perspective. A notable exception can be found in a study conducted by Douglas et al. (2018) exploring social enterprises in Fiji, in which the geography of the country, a small remote island in the Pacific Ocean, is considered (together with its history, social, economic, political and cultural institutions) a determinant factor shaping social enterprises in the country.

In summary (see Table 1 ), the review of the literature shows how geographical research on social enterprises has been conducted at various levels, from micro-organisational to national level; however, to-date this research has predominantly focused on the influence of local geographical elements in shaping the work of social enterprises. Within the local level, urban and rural localities have been subject to research and some differences have been identified in the ways rural–urban social enterprises operate. Regarding the methodologies used by studies, most geographical research on social enterprises have used qualitative methods, with some exceptions in studies that take a regional perspective. In these instances, studies have predominantly used existing survey data and registers of social enterprises ( Woo and Jung, 2023 ). In terms of theoretical perspectives, some studies are based on economic geography theories such as agglomeration and cluster theory (e.g. Pinch and Sunley, 2016 ; Jammulamadaka and Chakraborty, 2018 ) and concepts such as “place” borrowed from human geography (e.g. Mazzei, 2017 ; Olmedo et al. , 2023 ). However, generally the studies reviewed rather use theories from disciplines such as sociology, e.g. social network theory, and business/entrepreneurship, e.g. entrepreneurial ecosystems, complementing these with spatially sensitive elements such as the use of methodological tools such as GIS in their analysis ( Farmer et al. , 2020 ), the multi-scalar analysis of networks ( Richter, 2019 ) or a spatial rural–urban comparison of the cases studied ( Barraket et al. , 2019 ).

Despite the significant progress of geographical research on social enterprises in recent years, studies have focused on how geographical elements of the context influence the features and work of social enterprises, rather than exploring the basic and critical question (for research and policy) of how social enterprises are geographically distributed, and why. According to Buckingham et al. (2011 , p. 90), it “seems likely that the most significant geographical differences in the distribution of social enterprises are to be found at the sub-regional level […] and there is clearly a need for further, more fine-grained investigation”, see also Steiner et al. (2019) . This study aims to fill this gap for the case of Ireland by exploring the distribution and type of activities developed by social enterprises in rural and urban areas. To do so, we draw from a combination of increasingly complex thinking about rural–urban spatial heterogeneity, the advancement of methodological tools for rural–urban spatial classification at sub-regional level and from statistical information gathered on Irish social enterprises.

3. Research framework and hypothesis

3.1 territorial, rural–urban and classifications.

This paper is based on a geographical perspective towards the study of social enterprises in Ireland, and more specifically on the analysis of social enterprises in rural and urban areas. The definition of what constitutes a rural and urban area has been subject to extensive debate (see, for example, Mantino et al. , 2023 ; Eurostat, 2021 ). Within Europe there is no definitive agreement between Member States of what is considered as a rural/urban area; for example, in Ireland, rural areas are defined in terms of settlements with a population of less than 1,500 persons ( CSO, 2019 ), whereas in Spain rural areas are considered as those municipalities with less than 5,000 inhabitants but also those with less than 30,000 inhabitants and a density lower than 100 inhabitants/km 2 ( Government of Spain, 2007 ). These definitions classify rural–urban areas mainly in terms of population densities.

population density;

the percentage of the population of a region living in rural communities; and

the presence of large urban centres in such regions.

According to these criteria, NUTS 3 [ 2 ] regions are classified into Predominantly Rural; Intermediate and Predominantly Urban ( OECD, 2006 ) [ 3 ]. This methodology has been revised by Eurostat (2021) incorporating finer-grain data at Local Administrative Units Level 2 (LAU2) and grid cells of 1 km 2 to categorised territories into cities, towns, semi-dense areas and rural areas. Eurostat (2021) has also included a further subclassification based on population density and size. Towns and semi-dense areas were sub-divided into dense towns, semi-dense towns and suburban or peri-urban areas. Rural areas were also sub-divided into villages, dispersed rural areas and mostly inhabited rural areas. This finer analysis allows for a more precise analysis of the rural–urban continuum overcoming an abrupt differentiation between urban and rural areas but approaching it rather as a continuum that acknowledges the heterogeneity of rural and urban areas.

Besides the classification of rural–urban areas based on population density and size, classifications based on the functions and relations between areas have also been developed ( Mantino et al. , 2023 ). These classifications tend to incorporate indicators related to economic factors, for example, the economic growth/decline, the degree of productive activities (agriculture, forestry, manufacturing and construction) and consumption activities (tourism, recreation, housing and services) ( Copus et al. , 2011 ). Environmental indicators, for example, related to ecosystems functions (climate regulation, water supply and regulation, soil retention and formation, biodiversity) are also incorporated to classify rural–urban areas based on their (multi)functionality ( Mantino et al. , 2023 ). A key aspect of the relationship between rural–urban areas includes the mobility of workers and the access to services. In this regard indicators of proximity related, for example, to the time needed to access to services and infrastructures have also been considered in the classification of rural–urban areas ( Eurostat, 2021 ).

These functional classifications are usually interlinked with the abovementioned rural–urban classifications based on population density creating increasingly nuanced typologies through the multiple criteria that reflects the complexity of relationships between urban and rural areas ( Perpiñá Castillo et al. , 2022 ). In this line, the Central Statistics Office of Ireland (CSO) developed in 2019 a six-tier rural–urban typology ( CSO, 2019 ). This typology was developed using the place of work as a measure of distance to services and amenities, combined with population density from Census 2016. The typology is applied to small area population (SAP), and includes the following six categories: cities, satellite urban towns, independent urban towns, rural areas with high urban influence, rural areas with moderate rural influence and highly rural/remote rural areas (see Table 2 ).

3.2 Hypothesis development

Our study uses the typology developed by the CSO to conduct a geographical analysis of social enterprises in Ireland. Based on this framework, and some of the characteristics of social enterprises presented in the literature review of this paper, six hypotheses have been developed.

Previous studies have suggested that social enterprises are influenced by their geographical context with differences in the spread of social enterprises in rural and urban areas ( Buckingham et al. , 2011 ; CEIS and Social Value Lab, 2023 ). Some studies stress that rural areas represent a fertile ground for social enterprises ( Steiner and Steinerowska-Streb, 2012 ) and that social enterprises tend to emerge and develop in regions experiencing government or market failure ( Woo and Jung, 2023 ). However, the studies of Buckingham et al. (2011) and Pinch and Sunley (2016) suggest a capital-city effect attraction for social enterprises due to its dynamic and innovative business environment, the presence of headquarters location of national social enterprises, greater demand for social enterprises goods and services and better access to institutional support, funding, knowledge and networks, therefore, more supportive social entrepreneurial ecosystems (see also Diaz Gonzalez and Dentchev, 2020 ).

States that the presence of social enterprises is significantly associated with the type of rural–urban areas.

States that the presence of social enterprises is positively associated with areas with lower population density and greater distance to services and amenities (remoteness).

States that the presence of social enterprises within the capital city (Dublin) is significantly higher compared to the national average and to other rural and urban areas of Ireland.

Previous research has also pointed towards the influence of the geographical context in the activities developed by social enterprises ( Mazzei, 2017 ; Smith and McColl, 2016 ). Looking at rural–urban differences and the sector of activities of social enterprises, research has highlighted the key role of social enterprises in community and local development in (remote) rural areas ( van Twuijver et al. , 2020 ; Olmedo et al. , 2023 ) and in providing services related to welfare objectives in urban centres ( Barraket et al. , 2019 ).

States that there is a significant relationship between the sectors of activities in which social enterprises operate and the type of rural–urban areas in which they are based.

States that there is a positive association between areas with lower population density and greater distance to services and amenities (remoteness) and the presence of social enterprises in the sector of community and local development.

States that there is a negative association between areas with lower population density and greater distance to services and amenities (remoteness) and the presence of social enterprises operating in sectors related to welfare objectives.

4. Methodology

Nationwide data on Irish social enterprises were obtained from a social enterprise baseline data collection exercise conducted in 2022. This baseline data collection exercise followed a bottom-up methodology in which a population of social enterprises for Ireland was built from social enterprises lists provided by 36 intermediary organisations and public institutions delivering social enterprise programmes [ 4 ]. The population of social enterprises included 4,335 organisations, geographical-location information was gathered for 4,234 social enterprises and data about their sector of activity was gathered for 4,329 organisations.

Location information of social enterprises was georeferenced using organisation’s Eircodes (postal code/zip code equivalent for Ireland), thus allowing for a precise geolocation. The Eircode was either provided by the social enterprises or when not available the address of the organisation was introduced on the website “Eircode finder” to obtain the Eircode. Geographical coordinates for each Eircode were obtained using ArcGIS Online. Once the geographical coordinates were obtained each social enterprise was mapped using the software QGIS [ 5 ].

Data related to the CSO rural–urban typology containing information about the type of area (six categories) and population [ 6 ] was obtained from the Ordnance Survey Ireland – Open Data Portal [ 7 ]. The rural–urban typology developed by the CSO (2019) used in this study was applied to small area levels. Small areas are the lowest level of geography for the compilation of statistics by the CSO in line with data protection guidelines and typically contain between 80 and 120 dwellings ( CSO, 2019 ). A shapefile with small areas ungeneralised – National Statistical Boundaries was used, this contains a subdivision of the territory of the Republic of Ireland into 18,641 small areas. Information of small areas was vectorised and mapped using QGIS. Information about the six rural–urban categories was joined to each small area within QGIS and a choropleth map was created to differentiate between the types of rural–urban areas. Colours from light green (rural areas with high urban influence) to dark green (highly rural/remote areas) were used for rural areas, whereas dark red was used for cities, light red for satellite urban towns and pink for independent urban towns (see Figure 1 ).

The statistical analysis of this study includes three variables: type of rural–urban area, presence of social enterprises and sector of activities of social enterprises. As the aforementioned six-tier typology combines population density with distance to services and amenities, the categories have been ordered according to their level of remoteness, creating a dummy ordinal variable in which cities are converted into 1 (less remote) and highly rural/remote areas into 6 (most remote). The presence of social enterprises was calculated by the ratio of social enterprises divided by 10,000 inhabitants, following international guidelines from previous social enterprises census/baseline studies (see, for example, CEIS and Social Value Lab, 2023 ). The activities of social enterprises were codified following sectoral categories from the Scottish social enterprise census. This decision was made given the similarities between the countries (Scotland and Ireland) and the long experience of Scotland in constructing this census.

Statistical analysis for this study was conducted using the software R, version 4.2.2, within RStudio. We conducted a descriptive analysis of the variables before undertaking bivariate analysis of the variables to test our hypotheses. Due to the (partially categorical) nature of our data, we used non-parametric statistical tests such as Kruskal–Wallis H test, including post hoc Dunn’s test, chi-square test and Jonckheere–Terpstra test to investigate our hypotheses. The specific tests used for testing each hypothesis are explained in the following section.

5.1 Descriptive statistics

Social enterprises are distributed across rural and urban areas of Ireland (see Figure 2 ). In terms of total number, social enterprises are often concentrated in counties with the most populated Irish cities, such as Dublin (17.9% of total social enterprises) and Cork (10.5%) (see Figure 3 ). However, when considering the ratio of social enterprises by population (social enterprises/10,000 inhabitants), higher ratios of social enterprises are found, namely, in the north and northwest of the country (see Figure 4 ) and in counties with a high density of rural areas, such as Leitrim (26.2 social enterprises per 10,000 inhabitants), Donegal (18.5), Monaghan (17.3) and Mayo (16.5).

The descriptive analysis of social enterprises in relation to the rural–urban typology (see Table 3 ), shows that rural areas present a higher ratio of social enterprises (10.8 social enterprises per 10,000 inhabitants) than urban areas (8.0). However, the ratios show important differences when analysing the rural and urban subcategories, with highly rural/remote areas having a ratio of 21.0 social enterprises per 10,000 inhabitants against the 5.9 social enterprises per 10,000 inhabitants of rural areas with high urban influence. Within urban areas, independent urban towns have a higher concentration of social enterprises (12.9), than cities (6.7) and satellite urban towns (4.9).

The descriptive statistical analysis of the sector of activities of Irish social enterprises also shows some differences between rural–urban areas (see Table 4 ). For example, over 20% of social enterprises within each type of rural areas focus on community infrastructure and local development, whereas only 7.9% of social enterprises in cities operate within this sector. On the other hand, approximately 20% of social enterprises in cities and satellite urban towns develop activities related to health, youth services and social care, whereas in rural areas less than 10% of social enterprises operate within this sector. Social enterprises in sectors such as training and work integration, and information and support services are more prominent in cities, approximately 10% of city-based social enterprises operate in these sectors, whereas these sectors represent less than 5% of the total social enterprises based in Irish rural areas.

5.2 Hypothesis testing

Based on previous literature we developed six hypotheses to be tested related to the distribution and sectors of activities of social enterprises in rural and urban areas in Ireland (see Appendix for the results of the statistical test conducted).

H1 stated that the presence of social enterprises (measured by the ratio of social enterprises per 10,000 inhabitants) is significantly associated with the type of rural–urban areas (operationalised following the six-tier typology developed by the Irish CSO). To analyse this hypothesis a Kruskal–Wallis H test, a non-parametric version of ANOVA suitable for assessing the differences among three or more groups of a categorical/ordinal variable (rural–urban typology) related to a non-normally distributed continuous variable (social enterprise ratio), was conducted ( Vargha and Delaney, 1998 ). The results from this test show a statistically significant relationship between the variables ( p < 0.01), supporting H1 . As the rural–urban areas typology is formed by six categories, a post hoc Dunn test (adjusted with Bonferroni) ( Dinno, 2022 ) was conducted to compare the relationship between each of the pair categories. The results from this test show a significant relationship between all categories except for “cities and satellite urban towns” and “cities and rural areas with high urban”.

H2 refers to the positive association between the presence (ratio) of social enterprises and areas with lower population density and greater distance to services and amenities (remoteness). The six rural–urban categories have been ordered into a dummy variable from 1 to 6 according to their degree of “remoteness”. To test the (positive) directional association between the ratio of social enterprises and the rural–urban areas according to their degree of “remoteness” a Jonckheere–Terpstra test, a non-parametric test similar to Kruskal–Wallis H test, but preferred when the groups are assumed to be arranged in order (ascendent or descendent), was conducted ( Ali et al. , 2015 ). The results show a significant positive association ( p < 0.01) between the remoteness of the rural–urban areas studied and the presence (ratio) of social enterprises, supporting H2 .

H3 refers to the significantly higher presence (ratio) of social enterprises within the capital city (Dublin) compared to the national average and to other rural–urban areas of Ireland. To test this hypothesis, first, we calculated the ratio of social enterprises for the specific SAPs belonging to the category “cities” within County Dublin which accounts for 6.2 social enterprises per 10,000 inhabitants. Although social enterprises based in the city of Dublin represent 16.4% of total Irish social enterprises, the ratio of social enterprises in the city of Dublin (6.2) is below the national average (9.0) and lower than in other urban areas, including other Irish cities of more than 50,000 inhabitants (8.3) and independent urban towns (12.9). The ratio of social enterprises in Dublin city is also lower than in rural areas with moderate urban influence (9.9) and highly rural/remote areas (21.0).

Alternatively, the ratio of social enterprises in Dublin city is higher than in satellite urban towns (4.9) and rural areas with high rural influence (5.9). To analyse the statistical significance between the ratios of Dublin city and the categories with lower ratios we used Welch’s two-sample t -test, suitable for comparing means of groups with unequal variances ( Lu and Yuan, 2010 ). The results show no statistically significant difference between these means ( p > 0.05), thus H3 was not supported.

H4 refers to the significant relationship between the sectors of activities in which social enterprises operate and the type of rural–urban areas in which they are based. Due to the categorical nature of both variables, a Pearson chi-square test (test of independence) was conducted ( Franke et al. , 2012 ). The results show a statistical significance relationship between the variables ( p < 0.01), supporting H4 .

H5 refers to a positive association between areas with lower population density and greater distance to services and amenities (remoteness) and the presence of social enterprises in the community and local development sector and; H6 refers to a negative association between areas with lower population density and greater distance to services and amenities (remoteness) and the presence of social enterprises operating in sectors associated with welfare objectives such as “childcare” and “health, youth services and social care”. We followed the procedure explained in H2 of using a dummy variable to order the rural–urban categories according to their remoteness. Social enterprises within the category “community infrastructure and local development” were used to test H5 . Data of social enterprises from two categories, i.e. “childcare”, and “health, youth services and social care”, were used to test H6 .

To test the directional association between the ratio of social enterprises in community and local development ( H5 ) and in welfare services ( H6 ) with the rural–urban areas according to their degree of “remoteness” a Jonckheere–Terpstra test ( Ali et al. , 2015 ) was conducted. The results show a statistically significant relationship ( p < 0.05) for the variables of H5 , supporting this hypothesis. However, results for H6 were not statistically significant ( p > 0.05), thus this hypothesis was not supported.

In summary, our statistical analysis shows support for four of our six hypotheses (see Table 5 ). The hypothesis supported by our statistical analysis show a geographical rural–urban pattern in the distribution of social enterprises in Ireland ( H1 ) with a positive statistically significant association between the remoteness of the area and the ratio of social enterprises ( H2 ). However, our analysis suggests that there is not a capital effect that attracts a higher ratio of social enterprises to Dublin city ( H3 ). The statistical analysis also shows a geographical rural–urban pattern between the types of activities developed by social enterprises and the type of areas where they are based ( H4 ), with a positive association between the degree of remoteness of the area where social enterprises are based and the ratio of social enterprises in the community and local development sector ( H5 ). However, our analysis does not support a negative association between the degree of remoteness of the areas and the ratio of social enterprises in activities related to welfare services such as childcare and health, youth services and social care ( H6 ).

6. Discussion

The aim of this paper is to explore the distribution and type of activities developed by social enterprises in different rural and urban areas in Ireland. The results from our analysis show distinctive rural–urban patterns in the distribution of these organisations. Our research advances previous regional analysis of social enterprises ( Buckingham et al. , 2011 ) through the provision of fine-grained statistical data at subregional level and with a focus on heterogeneous rural and urban areas instead of following regional/county administrative divisions. The use of the six-tier rural–urban typology and the geo-localisation of social enterprises provides detailed evidence which can be used as a base by regional development actors and public authorities to develop targeted measures for social enterprises in geographically diverse areas ( Mazzei and Roy, 2017 ; Steiner and Teasdale, 2019 ).

Our results show the positive association between the presence of social enterprises and the degree of remoteness (low density of population and low access to services and amenities). These results align with previous studies that suggested rural areas and regions characterised by state and market failure as fertile grounds for social enterprises. ( Steiner and Steinerowska-Streb, 2012 ; Woo and Jung, 2023 ). Our study does not support the hypothesis that the capital city, in this case Dublin, with its greater entrepreneurial and innovation ecosystem acts as a significant area of social enterprises development – at least relative to its population. This result contradicts the analysis of Buckingham et al. (2011) which stressed the greater presence of social enterprises in London compared to other UK regions due to its capital effect.

Our results show the relevance of social enterprises in “lagged behind areas” and their aim to respond to unsatisfied needs, especially of marginalised people and territories ( Olmedo et al. , 2023 ). The great presence of social enterprises in these remote territories has meant the development of a wide range of services and community infrastructure which otherwise would have not been provided to the local population ( Aiken et al. , 2016 ; van Twuijver et al. , 2020 ). However, the presence of social enterprises cannot be automatically related to a greater capacity of these areas to overcome their challenges. Previous studies on rural social enterprises have shown their great potential to contribute to a socially inclusive and territorial integrated development when cooperating with other development actors including for-profit businesses and public institutions; however, these previous studies also show the incapacity of rural social enterprises to change, by themselves, structural-exogenous forces affecting marginalised territories ( Bock, 2016 ; Olmedo and O’Shaughnessy, 2022 ).

Our analysis of social enterprises by sectors of activities in different geographical areas does not show a relationship between social enterprises operating in urban areas and their greater focus on welfare objectives, contrary to the findings of Barraket et al. (2019) . It is important to note than in Ireland (community) childcares represent an important number of social enterprises (over 25%) and these are spread across the whole territory without a clear distinctive geographical pattern. Descriptive statistics by sectors of activity show that social enterprises focusing on activities related to health, youth services and social care represent over 10% in urban areas and only approximately 5% in rural areas which would be more in line with the results of Barraket et al. (2019) in Australia and Smith and McColl (2016) in Scotland when comparing urban and rural social enterprises.

Our results also show a significant focus of social enterprises on remote and rural areas in community and local development activities. This aligns with previous research on rural social enterprises that stress the relevance of community social entrepreneurship in rural territories ( Peredo and Chrisman, 2006 ) and the important role of rural (community-based) social enterprises in local development ( O’Shaughnessy et al. , 2011 ; Steiner and Teasdale, 2019 ; van Twuijver et al. , 2020 ). The significant developmental role of social enterprises in rural areas aligns with a key feature of rural social enterprises, which is their tendency to merge social, economic and/or environmental aims, contributing to an integrated territorial development ( Olmedo et al. , 2023 ). However, this significant focus of social enterprises in rural areas on community and local development activities often implies the development of basic infrastructure and services that are usually provided by public administrations in urban areas ( Bock, 2016 ). Thus social enterprises can, in this instance, be interpreted as a substitute arising from the absence and/or retrenchment of the state and public services ( Roy and Grant, 2019 ); this, in turn, can create an overburden to the citizens of these areas and increase the disparities between those better equipped and vulnerable social groups and territories ( Bock, 2016 ).

7. Conclusions, limitations and further research

This paper explored the distribution and sectors of activity of social enterprises in Ireland against a six-tier rural–urban typology that combines population density and access to services and amenities, adding a timely contribution to the body of geographical research on social enterprises. We suggest that the combination of national data of social enterprises with geographical tools and data at subregional level contributes to the methodological advancement of the field of social enterprises, through the provision of tools and frameworks for a nuanced and spatially sensitive analysis of these organisations. Moreover, this study contributes to testing, through a quantitative analysis, hypotheses developed from the findings of previous geographical research on social enterprises.

Our findings show geographical patterns in the distributions of social enterprises, such as their greater presence in highly rural/remote areas and the lack of a capital city effect in terms of density of social enterprises. Our analysis also shows a geographical rural–urban differentiation in terms of sectors of activity, with social enterprises in the community and local development sector being especially relevant in rural areas. Against this evidence, we conclude that social enterprise policies should incorporate territorially sensitive and place-based measures that account for the diversity of rural and urban areas. To this end, the alignment of social enterprises and rural development policies is a key aspect for harnessing the potential of these organisations in rural areas. However, we also conclude that there is great scope for the development of social enterprises in specific sectors in rural and remote areas, such as the creative industry, sustainable agri-food and the circular economy. The development of social enterprises within these sectors is linked to fostering a more socially and territorially inclusive society, but also to wider aspects related to the twin (digital and green) transitions.

This study is not absent of limitations. Social enterprises are context-specific, and the rural–urban typology used in this study was created by the Irish CSO with specific criteria. This makes international comparability difficult and any generalization of the results from this study to other contexts/countries should be taken with caution. Interestingly the Scottish Social Enterprise Census (latest version is of 2021) also follows a six-tier rural–urban typology, showing an important presence of social enterprises in remote rural areas; however, the use of different indicators for developing the Scottish rural–urban typology does not allow for a rigorous comparison with the data shown in this study. Recently developed methodologies such as the Global Human Settlement Layer by the Joint Research Centre of the European Commission ( Dijkstra et al. , 2021 ), which harmonise indicators for urban and rural areas to support consistent international comparisons across countries represent an interesting avenue for further research that compares geographical patterns of social enterprises in different countries. In this regard, the increasing amount of geolocation information and geographically sensitive data collection on social enterprises, and more generally on social economy organisations, can also represent an important advancement for future research.

A final suggestion for further research relates to the combination of geographical and institutional frameworks for the (quantitative) study of spatial patterns in social enterprises that can inform place-based social enterprise policies. This study can be further developed by isolating specific clusters of social enterprises at regional level and exploring their impact on the development of their areas and the critical factors supporting and/or hindering this impact.

Map rural–urban typology for the Republic of Ireland

Map of social enterprises by rural–urban typology

Map total number of social enterprises by county

Map ratio of social enterprises by county

Summary of literature on geographical research on social enterprises

Geographical analytical level	Relevant findings	Examples of articles
Micro	Social enterprises and spaces of well-being	(2015); (2020)

Urban	Agglomeration in cities enables greater demand and better access to institutional support, funding, knowledge and networks for social enterprises Characteristics of place influence in incentives and opportunities for social enterprises	;
Rural	Social enterprises as embedded intermediaries between their localities and supra-regional networks Social enterprises harness and (re)valorise untapped local resources and complement these with extra-local resources for integrated development of localities Rural areas are a fertile ground for social enterprises due to some characteristics associated to rurality	; ; ; (2023)
Urban–rural	Rural social enterprises more attached to geographical needs and community networks; urban social enterprises more focus on social needs and welfare objectives	; (2019)
Regional	Low interregional variations (UK) in distribution of social enterprises, except for capital Emergence of social enterprises related to regions experiencing government or market failure	(2011);
National	Geographical location of Fiji influence in shaping social enterprises	(2018)

Authors’ own creation

Type	Definition

Cities	Towns/settlements with populations greater than 50,000
Satellite urban towns	Towns/settlements with populations between 1,500 and 49,999, where 20% or more of the usually resident used population’s workplace address is in “Cities”
Independent urban towns	Towns/settlements with populations between 1,500 and 49,999, where less than 20% of the usually resident employed population’s workplace address is in “Cities”

Rural areas with high urban influence	Rural areas (themselves defined as having an area type with a population less than 1,500 persons, as per census 2016) are allocated to one of three sub-categories, based on their dependence on urban areas Again, employment location is the defining variable. The allocation is based on a weighted percentage of resident used adults of a rural small area who work in the three standard categories of urban area (for simplicity the methodology uses main, secondary and minor urban area). The percentages working in each urban area were weighted through the use of multipliers. The multipliers allowed for the increasing urbanisation for different sized urban areas. For example, the percentage of rural people working in a main urban area had double the impact of the same percentage working in a minor urban area. The weighting acknowledges the impact that a large urban centre has on its surrounding area The adopted weights for: Main urban areas is 2 Satellite urban communities is 1.5 Independent urban communities is 1 The weighted percentages is divided into tertials to assign one of the three rural breakdowns
Rural areas with moderate urban influence
Highly rural/remote areas

Area/Typology	Social enterprises	Population	Ratio (SE/10,000 inhabitants)
Highly rural/remote areas	865	20.4	412,457	8.8	21.0	10.8 (total rural)
Rural areas with moderate urban influence	580	13.7	587,041	12.5	9.9
Rural areas with high urban influence	447	10.6	754,794	16.1	5.9
Independent urban towns	991	23.4	770,329	16.4	12.9	8.0 (total urban)
Satellite urban towns	293	6.9	597,355	12.8	4.9
Cities	1,058	25.0	1,567,945	33.4	6.7
Total	4,234	100	4,689,921	100	9.0

Type of area	Childcare (%)	Community infrastructure and local development (%)	Health, youth services and social care (%)	Heritage, festivals, arts and creative industry (%)	Sport and leisure (%)	Training and work integration (%)	Information, support and financial services (%)	Housing (%)	Food, agriculture, catering (%)	Environment, circular economy and renewable energy (%)	Retailing (%)	Transport (%)	Manufacturing (%)	Other (%)
Highly rural/remote areas	28.7	23.8	8.6	15.7	5.3	3.5	3.6	2.9	3.1	2.3	1.4	0.2	0.2	0.6
Rural areas with moderate urban influence	32.2	21.9	9.0	10.5	9.5	3.6	2.4	2.2	3.1	2.4	1.2	0.9	0.2	0.9
Rural areas with high urban influence	23.7	22.4	9.4	10.1	13.4	4.7	5.6	3.4	3.6	2.2	0.4	0.0	0.2	1.1
Independent urban towns	23.8	14.6	14.5	12.7	8.9	5.8	7.1	5.2	2.2	1.3	1.8	0.7	0.4	1.1
Satellite urban towns	23.5	16.4	20.1	8.5	7.5	6.1	4.1	4.4	1.4	2.4	2.7	1.7	0.3	0.7
Cities	28.9	7.9	18.9	5.6	4.3	9.5	9.8	7.0	2.2	4.0	0.6	0.2	0.4	0.7
All Ireland	26.7	16.4	13.7	10.7	7.6	6.1	5.8	4.5	2.7	2.6	1.2	0.5	0.3	0.9

Hypothesis	Decision
: the presence of social enterprises is significantly associated with the type of rural–urban areas	Supported
: the presence of social enterprises is positively associated to areas with lower population density and greater distance to services and amenities (remoteness).	Supported
: the presence of social enterprises within the capital city (Dublin) is significantly higher compared to the national average and to other rural and urban areas of Ireland	Not supported
: there is a significant relationship between the sectors of activities in which social enterprises operate and the type of rural–urban areas in which they are based	Supported
: there is a positive association between areas with lower population density and greater distance to services and amenities (remoteness) and the presence of social enterprises in community and local development	Supported
: there is a negative association between areas with lower population density and greater distance to services and amenities (remoteness) and the presence of social enterprises operating in sectors related to welfare objectives such as childcare, health and social care	Not supported

		df	-value	Decision
SEs ratio – rural/urban area	309.17	5	2.2e **	Supported

Comparison (pairwise)	Z	P. unadj	P. adj (Bonferroni)
Highly rural/remote areas – Rural areas with moderate urban influence	6.432	1.26E-10	1.89E-09**
Highly rural/remote areas – Rural areas with high urban influence	9.694	3.21E-22	4.81E-21**
Highly rural/remote areas – Independent urban towns	3.866	0.000111	0.0017**
Highly rural/remote areas – Satellite urban towns	12.304	8.65E-35	1.308E-33**
Highly rural/remote areas – Cities	−14.341	1.21E-46	1.81E-45**
Rural areas with moderate urban influence – Rural areas with high urban influence	−3.256	0.001129	0.0169*
Rural areas with moderate urban influence – Independent urban towns	3.007	0.002637	0.0396*
Rural areas with moderate urban influence – Satellite urban towns	6.11	9.98E-10	1.50E-08**
Rural areas with moderate urban influence – Cities	−6.657	2.79E-11	4.19E-10**
Rural areas with high urban influence – Independent urban towns	6.491	8.55E-11	1.28E-09**
Rural areas with high urban influence – Satellite urban towns	2.979	0.002889	0.0433*
Rural areas with high urban influence – Cities	−2.772	0.005563	0.0834
Independent urban towns – Satellite urban towns	9.378	6.74E-21	1.01E-19**
Cities – Independent urban towns	−11.05	2.19E-28	3.28E-27**
Cities – Satellite urban towns	0.879	0.379665	1

	Alternative hypothesis	JT	-value	Decision
Positive association area remoteness and ratio social enterprises	Increasing	73161607	0.001**	Supported

	t-test (Welch Two Sample t-test)
Dublin City – satellite urban towns	1.6129	5,163.3	0.1068	(−0.22, 2.24)	Not supported
Dublin City – rural areas with high urban influence	1.1337	6,491.1	0.2569	(−0.46, 1.75)	Not supported

		df	-value	Decision
Association between sector of activity SEs and rural–urban typology	445.99	70	2.2e **	Supported

and	Alternative hypothesis	JT	-value	Decision
: Positive association rural–urban remoteness and ratio social enterprises in community local development	Increasing	13	0.02778*	Supported
Negative association rural–urban remoteness and ratio social enterprises in welfare services	Decreasing	3	0.06806	Not supported

IMAGES

Types of Reviews
(PDF) Comprehensive literature review and statistical considerations
Review: Statistical Analysis with R: Beginner’s Guide by John M. Quick
A BRIEF REVIEW OF THE STATISTICAL LITERATURE
Summary table for literature review on statistical analysis used for
Statistical analysis chart of published years of included literature

VIDEO

#literature review #research #article
Literature Review
Statistical pattern recognition
Why Many Scientific Findings Don’t Hold Up Under Scrutiny: Emily Kaplan
Literature Review
Meta-analysis (Pt. 1)

COMMENTS

A practical guide to data analysis in general literature reviews
The general literature review is a synthesis and analysis of published research on a relevant clinical issue, and is a common format for academic theses at the bachelor's and master's levels in nursing, physiotherapy, occupational therapy, public health and other related fields. ... Quantitative studies tend to contain a lot of statistical ...
A practical guide to data analysis in general literature reviews
The general literature review is a synthesis and analysis of published research on a rel-evant clinical issue, and is a common format for academic ... Quantitative studies tend to contain a lot of statistical data, and not all of it is relevant in a general literature review, especially at the undergraduate level. Tables that
Introduction to systematic review and meta-analysis
A systematic review collects all possible studies related to a given topic and design, and reviews and analyzes their results [1]. During the systematic review process, the quality of studies is evaluated, and a statistical meta-analysis of the study results is conducted on the basis of their quality. A meta-analysis is a valid, objective, and ...
Conducting systematic literature reviews and bibliometric analyses
However, a literature review that only offers an arbitrary selection of evidence is often not fully representative of the state of existing knowledge, and the selection of some studies over others ultimately leads to what is known in statistical analysis as a sample selection bias - a type of bias caused by choosing a non-random sample of ...
Meta‐analysis and traditional systematic literature reviews—What, why
Meta-analysis statistical assesses the robustness of findings in an area and identifies and resolves conflicting findings in past research to provide more ... is a web-based software that manages the entire literature review process and meta-analysis. The meta-analyst uploads all studies to RevMan library, where they can be managed and ...
Systematic Reviews and Meta-Analysis: A Guide for Beginners
Meta-analysis is a statistical tool that provides pooled estimates of effect from the data extracted from individual studies in the systematic review. The graphical output of meta-analysis is a forest plot which provides information on individual studies and the pooled effect. Systematic reviews of literature can be undertaken for all types of ...
Comprehensive literature review and statistical considerations for
The goal of this article is 3-fold. First, we aim to provide a summary of the methodologies used in the microarray meta-analysis papers. In this light, the article can be viewed as a 'meta'-meta-analysis paper. The second goal of the article is to provide a critique of the methodologies used in the literature.
Literature review as a research methodology: An overview and guidelines
This is why the literature review as a research method is more relevant than ever. Traditional literature reviews often lack thoroughness and rigor and are conducted ad hoc, rather than following a specific methodology. ... such as the meta-analysis, are used to integrate the results of the included studies. A meta-analysis is a statistical ...
How to Do a Systematic Review: A Best Practice Guide for Conducting and
The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information.
PDF Undertaking a literature review: a step'by-step approacii
literature review gathers information about a particular subject from many sources. It is well written and contains few if any personal biases. It should contain a clear search ... quantitative findings and conducting statistical analysis in order to integrate those findings and enhance understanding.
Literature Review Research
Literature Review - Research Guides - University of Delaware
Introduction to Research Statistical Analysis: An Overview of the
Introduction. Statistical analysis is necessary for any research project seeking to make quantitative conclusions. The following is a primer for research-based statistical analysis. It is intended to be a high-level overview of appropriate statistical testing, while not diving too deep into any specific methodology.
Systematic Review
Systematic Review | Definition, Example & Guide
5. Writing your literature review
The purpose of statistical analysis is usually to generalize from a the small number of people in a study's sample to a larger population of people. ... A literature review is an iterative process, one that stops, starts, and loops back on itself multiple times before completion. As research is a practice behavior of social workers, you ...
Writing a literature review
A formal literature review is an evidence-based, in-depth analysis of a subject. There are many reasons for writing one and these will influence the length and style of your review, but in essence a literature review is a critical appraisal of the current collective knowledge on a subject. Rather than just being an exhaustive list of all that ...
How to Write a Literature Review
Examples of literature reviews. Step 1 - Search for relevant literature. Step 2 - Evaluate and select sources. Step 3 - Identify themes, debates, and gaps. Step 4 - Outline your literature review's structure. Step 5 - Write your literature review.
The Literature Review, Part 1: What to Include
The literature review also helps to support your research problem and rationalize why your study is necessary by identifying gaps in the literature and the methodological weaknesses of previous studies. Below is what to include in your literature review. Include recent, peer-reviewed studies and articles.
LSBU Library: Literature Reviews: Developing a Literature Review
This ensures a comprehensive coverage of the literature. Avoid listing sources without analysis. Use tables, bulk citations, and footnotes to manage references efficiently and make your review more readable. Writing a literature review is an ongoing process. Start writing early and revise as you read more.
Chapter 9 Methods for Literature Reviews
Chapter 9 Methods for Literature Reviews
Literature Review on Collaborative Project Delivery for Sustainable
This paper aims to conduct a bibliometric analysis and traditional literature review concerning collaborative project delivery (CPD) methods, with an emphasis on design-build (DB), construction management at risk (CMAR), and integrated project delivery (PD) Methods. This article seeks to identify the most influential publications, reveal the advantages and disadvantages of CPD, and determine ...
A practical guide to data analysis in general literature reviews
This article is a practical guide to conducting data analysis. in general literature reviews. The general literature review. is a synthesis and analysis of published research on a rel-. evant ...
Insights into research activities of senior dental students in the
In addition to basic demographic details, the questionnaire comprised questions related to the type of study conducted, the scope of the research project, whether the research project was proposed by the students or the instructors or both, the literature review part of the project, the statistical analysis performed, the final presentation of ...
Guidance on Conducting a Systematic Literature Review
Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...
Association between pregnancy intention and completion of newborn and
Registration and reporting. We used the updated Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist [] to prepare and report this review (Supplementary File 1).After checking for a lack of other similar existing reviews and protocols, this systematic review was registered on the International Prospective Register of Systematic Reviews (PROSPERO) with ...
Comprehensive literature review and statistical considerations for
Pathway analysis (a.k.a. gene set analysis) is a statistical tool to infer correlation of differential expression evidence in the data with pathway knowledge from established databases (3, 4). The idea behind pathway analysis is to determine if there is enrichment in the detected DE genes based on an a priori defined biological category.
Is Clot Composition Associated With Cause of Stroke? A Systematic
Our study was performed according to the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses guidelines. 18 We searched MEDLINE (PubMed), Embase, and the Cochrane Library to identify studies between January 1, 2000 and March 20, 2024 that reported clot histology in adult patients who underwent MT for large vessel occlusion‐AIS. . Review and meta‐analysis articles on the ...
A geographical analysis of social enterprises: the case of Ireland
The statistical analysis also shows a geographical rural-urban pattern between the types of activities developed by social enterprises and the type of areas where they are based ... The main source for selecting the papers for the literature review was a search on Scopus (conducted in early 2023), with the search string: TITLE-ABSTRACT ...
Full article: Extremely low-frequency electromagnetic fields from
This review adopted a literature search period as defined by Draborg et ... Use of statistical techniques (such as matching or statistical adjustment) to account for confounding. ... C., Tirelli, E., Geuzaine, C., & Bruyère, O. (2022). Exposure to magnetic fields and childhood leukaemia: a systematic review and meta-analysis of case-control ...
Democracy and Foreign Direct Investment in BRICS-TM ...
A literature review indicates a preference for general country-based time series analyses over new-generation tests, with classical panel data analyses commonly employed for the selected country group. In summary, an examination of the literature reveals that studies on this issue predominantly rely on first- and second-generation linear panel ...

literature review statistical analysis

Acknowledgements

Corresponding author

Report an issue or find answers to frequently asked questions

Democracy and Foreign Direct Investment in BRICS-TM Countries for Sustainable Development

Cite this article

Graphical Abstract

Introduction

Theoretical Frame and Literature Review

The Status of the Literature on BRICS-TM Countries and Democracy and Foreign Direct Investment

Overview of the Literature, Identified Gaps, and Requirements for New Research

Research Method and Econometric Analysis

Data Set and Model

Econometric Method

Descriptive Statistics and Graphical Analysis of Variables

Cross-section Dependence Test

Panel Unit Root Test

Homogeneity Test of Cointegration Coefficients

Panel Cointegration Test with Structural Break

Kónya’s Causality Test

Data Availability

Abbreviations

Acknowledgements

Author information

Corresponding author

Ethics declarations

Conflict of Interest

Permission to reproduce material from other sources

Additional information

Practice Points/Highlights

Supplementary Information

About this article

Share this article

JEL Classification

IMAGES

VIDEO

COMMENTS

Save citation to file

Add to My Bibliography

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Similar articles

LinkOut - more resources

Other Literature Sources

Miscellaneous

Research Methods

Literature Review

5 5. Writing your literature review

5.1 Reading the results section

Read beyond the abstract

Quantitative results

Statistical significance

Confidence intervals

Qualitative results

Key Takeaways

5.2 Organizing information

Organizing empirical articles using a summary table

Organizing facts using a topical outline

Putting the pieces together: Building a concept map

Revising your working question

5.3 Writing your literature review

The problem statement

The structure of a literature review

Organize your review by argument

Use signposts

Editing your literature review

Share This Book

Have a language expert improve your writing

How to Write a Literature Review | Guide, Examples, & Templates

Instantly correct all language mistakes in your text

Table of contents

Receive feedback on language, structure, and formatting

Make a list of keywords

Search for relevant sources

Take notes and cite your sources

Don't submit your assignments before you do this

Chronological

Methodological

Theoretical

Cite this Scribbr article

Is this article helpful?

Shona McCombes

The Literature Review, Part 1: What to Include

Free Help Session: Chapters 1 and 2

Register Here

Literature Reviews

Developing a Literature Review

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

9.1. Introduction

9.2. Overview of the Literature Review Process and Steps

9.3. Types of Review Articles and Brief Illustrations

9.3.1. Narrative Reviews

9.3.2. Descriptive or Mapping Reviews

9.3.3. Scoping Reviews

9.3.4. Forms of Aggregative Reviews

9.3.5. Realist Reviews

9.3.6. Critical Reviews

9.4. Summary

9.5. Concluding Remarks

In this Page

Related information

Recent Activity

Information

Initiatives

Article Menu

JSmol Viewer

1. Introduction

Share and Cite

Article Metrics

Insights into research activities of senior dental students in the Middle East: A multicenter preliminary study

Conclusions

Introduction

Materials and methods

Data availability

Acknowledgements

Author information

Contributions

Corresponding author