Main

The 2015 Sustainable Development Goals (SDGs) have motivated the development community to focus on improving the livelihoods and climate resilience of small-scale farms. To focus research, programmatic efforts and donor funding towards supporting small-scale farmers, SDG 2.3 aims to "double the agricultural productivity and incomes of small-scale food producers [by 2030]"1. Critical to achieving SDG 2.3 is to sustain farmers’ production and incomes through climate shocks and stressors. As climate projections indicate that drier regions will become drier and that droughts are likely to increase2, small-scale farmers in already water scarce regions will require additional support3.

Tackling water scarcity in small-scale farming systems remains a top priority for development organizations, researchers and donors4,5. As precipitation patterns change with climate change, the livelihoods of small-scale farmers who lack access to water and technologies will become even more marginalized6,7. Small-scale farmers represent >80% of the world’s farms8 and are major contributors to the food system9,10,11; farms of <5 ha produce nearly 50% of the global food supply11. Finding options for this large and diverse set of farmers to adapt to increasing water scarcity is a growing central tenet to alleviate poverty and to ensure a resilient food system4,5.

A key question for donors is which interventions have proven impacts. For instance, improved water access and water-use efficiency have been shown to increase yields and farmer incomes but interventions that are successful in one region do not necessarily work elsewhere. Many interventions (such as water harvesting, soil improvement strategies, drought-resistant crops and livestock breed selection) are heterogeneous and their impacts can vary across physical, social and political dimensions12. This creates a challenge for donors seeking to focus their investments around the goals of increasing food production, improving livelihoods and reducing climate vulnerability for small-scale farmers. Despite the large volume of evidence produced every year in peer-reviewed journals, grey literature and monitoring and evaluation reports, it is difficult to track which interventions have adequate research support. For instance, in water scarce regions it remains unknown if general research trends have kept up to date with policy trends that have shifted from the focus of the Green Revolution era on improving yields to modern development initiatives (for example, SDGs) that include livelihoods, commitment to reducing environmental impacts and gender. To date, the evidence base addressing interventions to improve the livelihoods of small-scale farmers in water scarce regions is limited. Structured evidence syntheses such as systematic reviews and meta-analyses do exist13,14,15,16,17,18,19,20, as do more traditional literature reviews21,22, but all tend to focus on subtopics such as conservation agriculture, irrigation or policy frameworks.

This scoping review collates many different types of interventions aimed at small-scale farmers in water scarce regions across low- and middle-income countries (LMICs) to understand the breadth of evidence for available on-farm interventions. Our effort seeks to help donors identify future research funding, to focus the efforts of researchers towards filling knowledge gaps and to build a repository of studies on a broad swath of interventions that may improve the yields and incomes of small-scale farmers.

We focus our evidence synthesis on water scarce, small-scale farms across LMICs to assess whether research effort is being conducted in the locations that need it most. Our research identifies countries that have limited research for adaptation solutions, yet high numbers of small-scale farmers in water scarce regions. On the basis of the available evidence, we systematically map research effort onto intervention types to identify which interventions lack research (Box 1). We also assess whether studies examined environmental trade-offs or gender effects to ensure that there is evidence for integrated solutions that reflect the relationship between SDG 2.3 and other development goals. Our scoping review provides a rapid way to identify the breadth of evidence represented in the literature to assist the adaptation of small-scale farmers to water scarcity.

Results

Research needs versus research effort

We found that 76.7% of small-scale farms and 72.4% of small-scale farm area across all LMICs were probably located in water scarce regions. We also found that existing technological penetration of basic irrigation infrastructure for these farms was low. For example, we estimated that fewer than 37.2% of small-scale farms in water scarce regions across all LMICs were irrigated, compared to 43.2% of non-small-scale farms (Fig. 1 and Supplementary Fig. 1). The largest disparities in irrigation coverage between non-small-scale farms and small-scale farms were in Latin America and the Caribbean, South Asia and Sub-Saharan Africa. Yet, in East Asia and the Pacific and in the Middle East and North Africa regions, small-scale farms had greater irrigation coverage than non-small-scale farms.

Fig. 1: Irrigation coverage for small-scale farms in water scarce regions.
figure 1

The percentage of small-scale and non-small-scale farms under irrigation (as opposed to rainfed) by region and across all LMICs. The left plot shows the relationships in water scarce regions, while the right plot shows the relationships in non-water scarce regions.

Full size image

We also found a large variation across countries of the number of studies of interventions for farmers living in water scarce regions (Fig. 2). When comparing where research on interventions was conducted with where high numbers of water scarce, small-scale farms were located, we observed considerable gaps for countries where few studies met our criteria, such as Nigeria, Mali, Uganda, Chad and Ivory Coast (Fig. 3 shows that these countries had 12, 10, 7, 1 and 0 studies, respectively). There were also hotspots of water scarce, small-scale farms with more moderate research effort, which included India, Ethiopia, Kenya, Tanzania and Ghana (Fig. 3 shows that these countries had 74, 54, 48, 33 and 28 studies, respectively). These differences in the amount of studies are quite large when considering that the included studies ranged from 1962 to 2019.

Fig. 2: Number of studies per country included in literature synthesis.
figure 2

A map showing the number of studies that measured the impact of an intervention on the incomes and/or yields of small-scale farmers in water scarce regions. Countries in grey indicate that no studies met our inclusion criteria. High-income countries were only included if they tested an intervention relevant to small-scale farmers in water scarce regions of LMICs.

Full size image
Fig. 3: Number of small-scale farms in water scarce regions compared to research coverage.
figure 3

Each point represents the number of small-scale farms in a country that are in water scarce regions. The size of the points corresponds to the number of studies that measured the impact of an intervention on the incomes and/or yields of small-scale farmers in water scarce regions. The colours indicate the economic grouping of the countries as low, lower-middle or upper-middle income.

Full size image

A promising finding was that 55% of studies tested interventions in controlled field trials and 18% were modelling studies both of which focus on the causal relationships between the interventions and yield and/or incomes. In addition, 28% of studies were from household surveys to determine if interventions designed for water scarce, small-scale farms worked in farmers’ local contexts according to farmers’ responses on yield and livelihood outcomes. Studies in high-income countries were excluded unless they were relevant to LMICs (for example, studies that used experiments and modelling to mimic water scarce conditions) and those accounted for 2% of the studies we analysed. We found eight meta-analyses and six systematic reviews in our literature search. Five of the meta-analyses found that seed priming, soil management and water management contributed to increased yields in water scarce contexts; the three meta-analyses that examined conservation agriculture showed mixed results in its effects on yield for water scarce contexts (Table 1).

Table 1 Results from the eight meta-analyses included in our evidence synthesis
Full size table

Research focus on yields versus livelihoods

We found that most articles assessed outcomes associated with yield (91% of articles) as compared to livelihoods (21% of articles examined incomes and/or expenditures) (Fig. 4). Despite shifts from Green Revolution rhetoric in the 1950s–1970s to language used in the Millennium Development Goals that launched in 2000 and the subsequent 2015 SDGs, research continues to focus on yields compared to livelihoods (Supplementary Fig. 2).

Fig. 4: Number of studies per outcome by intervention.
figure 4

The number of studies classified by intervention type, outcome and methodology. The left plot classifies studies that examined the effects of interventions on farmer incomes or expenditures, while the right plot classifies studies examining the effects of interventions on yields. Different colours correspond with different methodological approaches (for example, experimental plots, observational household or plot surveys, statistical modelling, meta-analysis or systematic review). Interventions are grouped into eight broad classes (see Supplementary Table 1 for full definitions). Water refers to direct water interventions (for example, irrigation or water harvesting). Soil includes direct soil-based interventions (for example, fertilizer or liming). Crop and pest include crop species and crop variety-based interventions and pest management as these can consist of overlapping interventions. Tillage includes all variants of tillage. Cropping system includes crop rotation, intercropping, fallow, monocropping and so on. Livestock refers to any intervention directly related to livestock (for example, species feed or vaccinations). Natural resource protection includes protection of farm natural resources (for example, via erosion control or buffer strips). Digital technology refers to any digital intervention (for example, weather advisories or precision agriculture).

Full size image

We checked this research bias towards yields compared to livelihoods by assessing meta-analyses that synthesized the literature on interventions for small-scale farmers in water scarce regions. We found that all eight meta-analyses focused on yield (Table 1). This shows that not only is the focus on yield prevalent in the primary literature but it is also the sole focus of existing research syntheses that quantifies the relationships between interventions and outcome for small-scale farmers in water scarce regions to date.

In addition to outcomes, the coverage and evidence for different kinds of interventions is also diverse. Farm-level interventions represent a toolbox of actions that farmers can take to tackle water scarcity directly, which include drip irrigation, soil improvement strategies, integrated pest management, crop rotation and so on (Fig. 4). While we found higher coverage for interventions around irrigation, tillage, soil amendments, cropping systems, crop varieties and pest management (145–210 articles), we found little work on key interventions of interest, including livestock interventions (43 articles), solutions that protect natural resources at the farm scale, such as buffer strips and contouring (15 articles) and digital technology interventions (three articles) (see Supplementary Table 1 for full definitions of interventions). Changes in the number of studies increased for all interventions over time but the relative share of studies per intervention remained stagnant since the early 1990s (Supplementary Fig. 3).

Research on environmental impacts and effects on gender

We found relatively high coverage of research that examined environmental impacts of the intervention assessed (68.9% of interventions and 50.4% of articles; Fig. 5a). Most of these studies assessed win–wins between farmers and the environment in the context of water scarcity, such as improving water efficiency, increasing water availability (for example, through rainwater harvesting) and managing soil for greater water retention. A smaller share of research (7% of interventions and 6.3% of articles; Fig. 5b) addressed the negative environmental effects of interventions, such as greenhouse gas emissions, biodiversity loss and land conversion/degradation.

Fig. 5: Number of studies per cross-cutting theme over time.
figure 5

The upper plot shows the cumulative number of studies that included analysis on environmental impacts and/or gender dimensions of the interventions assessed. The lower plot disaggregates the environmental impacts into eight subcategories.

Full size image

We found few studies that considered gender aspects of the effects of interventions on yields and incomes. Only 9.6% of interventions (and 9.6% of articles; Fig. 5a) either assessed gender differences in the outcomes measured or framed their research with a gender lens, with most of these studies examining if the intervention could reduce the time burden many female farmers face. This limited number of studies was uncovered even though we used broad inclusion criteria to include studies not only if they directly addressed gender outcomes but also if they were framed in terms of gender issues.

Discussion

This scoping review quantified the global irrigation gap of small-scale farms and found that water scarce regions needed equitable irrigation infrastructure the most. While our results generally found that small-scale farmers face unequal access to one critical resource in adapting to water scarcity across LMICs, local political, economic and environmental realities may determine irrigation access. These realities may explain the variation in irrigation coverage between small-scale and non-small-scale farmers across regions. Hence, donors and researchers should focus on water scarce regions with the largest disparities in access to critical resources (for example, irrigation or other) and the lowest amount of evidence for the efficacy of on-farm interventions on yields and livelihoods. A limitation to these findings is that our research highlights only one type of disparity that small-scale farmers face (irrigation coverage) but future studies can build on our methods to examine other types of marginalization (for example, poor soil quality, distance to markets, climate exposure, land use, land tenure/governance and so on) to help prioritize research funding towards evidence-backed interventions for disadvantaged small-scale farmers.

We found that the shift in funding priorities for small-scale farmers from improving yields during the Green Revolution to including livelihoods, environmental trade-offs and gender impacts in the SDGs was not always reflected in the research. Yields have remained the main focus in this research literature since the 1960s (Supplementary Fig. 2). Yet, yields only provide a partial farm productivity measure based on output per unit of land cultivated. They cannot proxy for productivity measures that incorporate more production factors (for example, total factor productivity), for economic measures (for example, farm profitability or farm household welfare measures based on income, consumption or asset holdings) or for dynamic measures that address the probability that the welfare of a farm household falls below some threshold in a given year (for example, vulnerability or resilience). All of these outcomes can be difficult to measure and even yields, which are relatively straightforward to measure, are subject to bias when self-reported23. One cannot conclude whether an intervention has been successful for small-scale farmers without incorporating economic measures that properly account for self-provisioned inputs such as family labour and capture aspects of farmers’ livelihoods24.

Critical environmental trade-offs of interventions need to be tested to ensure that funded projects do not have deleterious effects on local natural resources and, in turn, on small-scale farmers’ future livelihoods. While clear environmental and productivity win–wins were prevalent in the literature, such as water-use efficiency25,26,27, it was concerning that so few studies jointly addressed cost-cutting solutions to both environmental degradation and productivity, which include, sustainable intensification, minimizing biodiversity loss and solutions across the food–energy–water nexus28,29,30. Future research should help identify where and when win–win interventions exist or when environmental trade-offs are inevitable.

Similarly, there was a large research gap on gender dimensions of interventions. Donors should focus funding research that assesses interventions specifically designed to improve women’s outcomes, for example by addressing their large time burden. In addition, existing interventions need to be tested to ensure avoidance of unintended consequences on women. We were concerned to find that gender themes were so rarely addressed, even though women comprise 43% of the agricultural labour force, account for two-thirds of livestock keepers in LMICs and are often responsible for very large shares of agricultural tasks (for example, 90% of hand weeding across Sub-Saharan Africa), while consistently farming plots that are smaller than male farmers’ plots31. Studies examining the gender effects of on-farm interventions in water scarce regions should focus on gender-specific interventions, such as reducing unpaid labour requirements of women32, to free up women’s time towards the most productive activities. Improving the gender balance of female extension agents or agricultural researchers—for instance, only 24% of African agricultural researchers are female31—may help to better align research priorities towards the needs of female small-scale farmers.

Given the importance of livestock for adaptation in water scarce conditions, climate solutions that protect farm-level natural resources, and the benefits of climate services and digital technology for in-season recommendations and responses, the lack of evidence we found suggests a need for more research addressing these three types of interventions. While the dearth of research in digital technology is probably due to its relatively recent emergence, the low interest in livestock and interventions that protect natural resources have been persistently low throughout the last several decades (Supplementary Fig. 3). The lack of research on livestock interventions to improve productivity or incomes in water scarce regions was particularly concerning considering an estimated 23% of farmers own cattle33. Our findings compliment the notion that pastoral systems in LMICs have tended to receive less investment from the international research community and other institutions, compared to livestock systems in high-income countries and cropping systems more broadly34. While interest in livestock systems has, however, been growing in recent years, driven by increasing concerns about food security, land competition, the vulnerability of remote communities and the environment as well as by a raising awareness regarding the contribution of livestock systems to all these aspects35,36, we did not see relative increases in the number of studies examining these systems over time (Supplementary Fig. 3). Climate resilience strategies for livestock keepers require increased institutional support through policies that address the issues of land tenure, fragmentation and degradation35. The evidence gap we found pertaining to natural resource protection may reflect the focus of this particular literature on different outcome metrics, such as contouring to prevent soil erosion or using buffer strips to improve biodiversity instead of the focused outcomes of improving yields and livelihoods of the SDG 2.3. However, as ecosystem services are an important dimension of a farm’s resilience to climate stressors and shocks37, natural resource protecting interventions require more evidence for their effects on farmer livelihoods in water scarce regions.

Digital solutions for water scarce, small-scale farms is an emerging field but funders should prioritize research grants to test the livelihood impacts of these solutions. There are more applications for small-scale farmers generally38 but there is limited evidence to apply these interventions even outside of water scarce regions. For example, a meta-analysis carried out in 2019 only found five studies with nine observations that examined digital solutions that improved small-scale farmers’ yields (researchers found a 0–8% increase) and the odds of adopting a recommended agricultural input (researchers found a 13–31% increase)38. While the goal of our systematic map was not to identify studies suitable for a meta-analysis, we found seven studies that examined digital solutions for farmers in water scarce regions. These studies focused on providing farmers with tools to improve the water-use efficiency of their irrigation systems39, climate information to farmers40 and to aid decisions in nutrient, water and weed management to reduce production risks41. Despite the promise of digital solutions in water scarce environments, we suggest that there needs to be better proof of digital technologies that are tested in low-bandwidth settings since many small-scale farmers live outside of 3G and higher mobile coverage areas42. Research in these geographies, metrics and outcomes will bring us closer to meeting the needs of water scarce, small-scale farms on the ground.

To achieve the SDG 2.3 goal of bolstering small-scale farmers’ yield and livelihoods, a greater research focus is needed in water scarce regions of LMICS. Future research needs to test the effects of on-farm interventions not only on yields but also on more outcomes that are relevant for farmer livelihoods, such as farm profitability, farmer income and resilience. Trade-offs between interventions and the environment need to continue to become part of research designs and donor requirements to ensure that unintended environmental impacts can be avoided, especially in regions with limited water resources and for populations that are systematically disadvantaged from accessing critical on-farm infrastructure. Similarly, gender dimensions of interventions require a tremendous amount of support in research funding since these remain under-researched despite being an SDG cross-cutting theme. We suggest that funders incorporate these outcomes into their impact assessment frameworks for research grants. For specific policy recommendations see Box 2.

Methods

Spatial analysis

To contextualize the disparity small-scale farmers face in accessing critical resources, we quantified the number of small-scale farms with irrigation in water scarce regions in LMICs. We created a spatial layer of small-scale farms and overlaid it with available water scarcity and irrigation spatial layers. Our analysis provided estimates aggregated to the country level. While this was a coarse method that relied on spatial overlays, it enabled us to take a high-level view on where water scarce small-scale farmers live and how many of these farms have access to irrigation.

To create the small-scale farms layer, we implemented the SDG 2.3 definition of small-scale farms through spatial proxies at 10 km2 resolution. SDG 2.3 defines small-scale farms as the smallest 40% of farms in a country and farms with the lowest 40% of agricultural revenue in a country43. We used an available farm size map at 10 km2 resolution with a global spatial extent42. It was created by using a crowd-sourced field size map44 to downscale the national farm size distributions of the World Census of Agriculture8. Each grid cell contains the most common farm size using World Census of Agriculture categories: 0–1 ha, 1–2 ha, 2–5 ha, 5–10, ha, 10–20 ha, 20-50 ha, 50–100 ha, 100–200 ha, 200–500 ha, 500–1,000 ha and farms >1,000 ha. We computed the smallest 40% of farms in a country on the basis of this map. Since there are no comparable spatial data on agricultural revenue, we used a proxy for agricultural revenue. We used a downscaled gross domestic product (GDP) per capita dataset available at 110 km2 resolution45, which we subsetted to cropland area46. We computed the lowest 40% of GDP per capita per country using this data. We aggregated the farm size map to the same spatial resolution as the GDP per capita map and overlaid these two layers to find pixels that were both the smallest 40% of farms in each country and in agricultural areas with the lowest 40% of GDP per capita in each country. We estimated the number of farms in each pixel by dividing the agricultural area by the predominant farm size of that pixel. This approach allowed us to identify the poorest agricultural areas with the smallest farms in each country.

We created the water scarcity layer by combining two different measures of water scarcity at 10 km2 resolution. Each grid cell needed to be either green (rainwater stored in the soil) or blue (fresh surface water and groundwater) water scarce. We defined green water scarcity according to the IPCC definitions of a semi-arid region, which is an area with <250 mm of rainfall in a year47. We used the Global Precipitation Climatology Centre rainfall dataset (full v.2018)48 to calculate the median rainfall per grid cell from 1996 to 2005. We relied on Mekonnen and Hoekstra’s blue water scarcity dataset to identify grid cells that had on average at least one month of blue water scarcity a year from 1996 to 200549. Through using subnational and intra-annual green and blue water scarcity data, we implemented a more detailed view on water scarcity than commonly reported country-level water scarcity metrics reported at annual intervals.

To determine the irrigation area for small-scale farms compared to non-small-scale farms, we used the global rainfed and irrigated croplands (GRIPC) layer, which is a global irrigation map at 500 m2 resolution50. The irrigation map was created through remote sensing and calibrated to nationally reported irrigation coverage statistics to represent circa 2005 values. This irrigation map provided the percentage of agricultural area in a grid cell covered by irrigated cultivation, rice paddy or rainfed cultivation. For our analysis, we only used the irrigated and rainfed categories. We excluded paddy cropland (and the corresponding farming populations) from our analysis because GRIPC does not further classify paddy fields as irrigated or rainfed in their pixel level dataset but they do offer that irrigated paddy accounts for 66 Mha of global cropland and non-irrigated paddy accounts for 63 Mha of global cropland in their manuscript; put another way, this near 50/50 split should not have a major effect on our estimates of the share of water scarce areas that are irrigated but future studies would benefit from better spatial data that splits irrigated and non-irrigated paddy. To provide the full range of values, our scoping review did not include paddy and found that 37.22% of crop area farmed by small-scale farmers in water scarce regions is irrigated (versus 43.19% farmed by not small-scale farmers). If we assume all paddy fields are rainfed, then 26.48% of crop area farmed by small-scale farmers in water scarce regions is irrigated (versus 37.24% farmed by not small-scale farmers). If we assume all paddy fields are irrigated, then 55.07% of crop area farmed by small-scale farmers in water scarce regions is irrigated (versus 52.82% farmed by not small-scale farmers).

We overlaid the small-scale farm layer, the water scarcity layer and the irrigation layer to calculate the number of farms with and without irrigation. We repeated this analysis for non-small-scale farms and non-water scarce regions. Results were aggregated to the country level and summed across all LMICs and per world region.

Evidence synthesis

We used a ‘systematic map’ assisted by machine learning and natural language processing (NLP) to perform our evidence review. Systematic maps (also referred to as ‘evidence maps’ and ‘gap maps’) are an emerging type of systematic review that attempt to identify patterns of research, to identify gaps in a field and future priorities for research51. A systematic map is not like traditional expert-based or narrative reviews in that it attempts to capture all of the research on a given topic and reduce the authors’ biases51. Systematic maps can capture the full-breadth of interventions relevant to a population, such as our scoping review that asks: What is the spectrum of farm-level interventions that have been tested to increase small-scale farmers’ incomes, yields and productivity in water scarce regions?

Our systematic map method had six steps: (1) forming the research question; (2) querying academic and grey literature databases for relevant studies; (3) screening titles and abstracts to determine if a study should be included in our synthesis; (4) screening the full text of studies that passed step 3 to determine if a study should be included in our synthesis; (5) extracting relevant data from each included study; and (6) summarizing and reporting the results. The protocol for this scoping review was registered on the Open Science Framework (https://osf.io/c6n4k/) before study selection, which can also be accessed in our Supplementary Information.

The guiding question for this systematic map was: ‘What spectrum of farm-level interventions to alleviate water scarcity has been tested to increase small-scale farmers’ incomes, yields and productivity?’

An exhaustive search strategy was developed and applied to 26 academic and grey literature databases to identify all available research pertaining to on-farm interventions that have been tested to increase small-scale farmers’ incomes and yields in water scarce regions in low- and middle-income countries. Search terms included variations of the key concepts in the research question: small-scale farmers, water scarcity, and income, yield and productivity. Searches were performed in the following bibliographic databases: CAB Abstracts and Global Health (access via CAB Direct), Web of Science Core Collection (access via Web of Science), Scopus (access via Elsevier), Agricola (access via EBSCOhost), EconLit (access via EBSCOhost) and ProQuest Dissertations & Theses Global (access via ProQuest). A search of grey literature sources (20 specialist organizations and online databases) was also conducted. Full search strategies for each database, including grey literature, can be accessed in their entirety in our protocol available in our Supplementary Information or at https://osf.io/c6n4k/. Search results were de-duplicated to remove redundant citations identified from multiple sources, resulting in 18,365 unique publication records.

We screened the 18,365 titles and abstracts to include or exclude from our scoping review. Two independent reviewers assessed each title and abstract. If there was disagreement between reviewers on whether the study was to be included, a third independent reviewer decided. Articles needed to meet the following eligibility criteria for inclusion in our systematic map. (1) Studies needed to address small-scale farms explicitly. For inclusion in the evidence synthesis, we defined a small-scale farm to meet two of four dimensions: land size, labour input (especially of family members), market orientation and economic size. (2) Studies examining on-farm production management techniques or technologies explicitly addressing water scarcity, drought adaptation or water efficiency adaptation. For the systematic map, we erred on the side of inclusivity and used a general definition of water scarcity to include a broad range of studies and interventions across a spectrum of agricultural regions as well as the concept of water stress. We considered water scarcity as when there is not enough water to be used by a farmer for agricultural purposes, which includes blue and green water. We considered water stress to be an additional subset of water scarcity where certain farmers are economically disadvantaged due to poor access to water resources. (3) Studies examining the effect of an on-farm intervention on yields or incomes. While interventions that improve small-scale farmers welfare can range across plot-level technologies, farm-level management, collective action, government infrastructure projects and bi/multilateral trade agreements, we only focused on-farm-level interventions to represent a toolbox of actions that farmers can take to tackle water scarcity directly. (4) Studies including either a control case for comparison (temporal or spatial) for identifying counterfactual outcomes in the intervention’s absence (for example, by using randomized control trials, pre–post designs, random block designs, modelling and so on) or studies comparing alternative interventions. We included reviews if they were systematic reviews or meta-analyses. The most common reason an article was excluded was because it characterized a farming practice, rather than measuring the effects of the farming practice on yields or livelihoods. Other common reasons that studies were excluded were because the interventions were not explicitly linked to water scarcity, water savings or improving adaptation to drought. Several studies were not in English, which was a limitation of our research.

To assist the time-consuming task of sorting the 18,365 titles and abstracts that we identified in the academic and grey literature databases, we used a machine learning approach. Using NLP and machine learning for this stage of systematic reviews is an emerging method52. While studies have used Naive Bayes and Support Vector Machines (SVM) models52,53 the Google Development Team released the Bidirectional Encoder Representations from Transformers (BERT) model on Tensorhub in 2019 and it has outperformed other NLP models in a variety of tasks54. BERT is a deep learning language representation model that is context aware, in which the word in context of the sentence and the sentence in the context of the paragraph are embedded in the structure of the model. The BERT comes pretrained, which speeds up processing time since the end user only needs to fine-tune the model. Our scoping review tests the BERT model against other classification models to assist a systematic review.

Our team manually classified 1,500 titles and abstracts to include or exclude from our scoping review. We split the manually classified titles and abstracts into training and test data to build and validate several machine learning classifiers. We tested the accuracy of Naive Bayes and SVM, where we used a bag-of-words model with term-frequency times inverse document-frequency (TFIDF) to construct the features of the model. We conducted cross-validated grid searches to identify optimal sets of hyperparameters, which included removing stop words (common words in the English language) and stemming (converting the word to the root word); all hyperparameters assessed can be found in the supplemental code provided. We compared these scores to a BERT model that we fine-tuned to minimize binary cross entropy loss in a classification layer. We used the multilanguage base version of BERT (12-layer, 768-hidden, 12-heads, 110 M parameters), which was trained on the top 102 languages with the largest Wikipedias. Our supplemental code details all parameterization (https://github.com/vinnyricciardi/Ricciardi_etal_2020_ceres). For all models, we calculated accuracy, precision, recall and F1 scores through k-fold cross validation (k = 10) (Supplementary Table 3). The goal was for the model to perform better than the level of agreement our team achieved during the manual classification of titles and abstracts. During the manual classification, two random reviewers (out of a team of ten reviewers) classified each title and abstract. If there was disagreement, a third reviewer broke the tie. Reviewers agreed on the classification 82% of the time. Each of the NLP models performed better than 82%. BERT was the best-performing model with an 88% accuracy (Supplementary Table 3). We applied this final model to the 18,365 titles and abstracts to classify each study to be included or excluded from our scoping review. The model included 1,423 studies in our scoping review for full-text review, of which we were able to find and download 1,355 texts.

In the next phase of our assessment, we downloaded the 1,355 full-texts and manually checked if they were to be included or excluded from our scoping review. In the fourth stage of our assessment, for each of the 560 articles included, we manually extracted the location each study was conducted, the type of method, the type of farming system (crop, livestock or mixed), the intervention assessed (see Supplementary Table 1 for full list), the outcome assessed (yield or income) and whether the study measured a cross-cutting theme (gender effects or environmental impacts). This final stage consisted of one reviewer extracting the information from each text, with communication between reviewers to ensure consistency. Supplementary Fig. 4 shows the number of publications included and excluded at each stage. Our final analysis consisted of cross-tabulations and descriptive statistics.

Through our evidence synthesis, we identified eight meta-analyses. We extracted the intervention and outcome assessed, the location(s) that the meta-analysis included, also the number of studies the meta-analysis included. We recorded the main finding of the meta-analysis to understand the impact of the intervention on the outcome (for example, did water harvesting increase, decrease or have no effects on yields of water scarce, small-scale farms). To assess the quality of each meta-analysis, we relied on Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT)55, which provides a score sheet of 16 questions to appraise the rigour, transparency and limitations of the systematic review or meta-analysis. For each question, the meta-analysis can receive a red score for the lowest score, an amber score, a green score or a gold score for the highest rating. We converted these scores from one to four, averaged the scores and presented them in Table 1. Our Supplementary Information provides our scoring for the CEESAT score sheet.

Reporting Summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.