Article | Open | Published:

Translating large-scale climate variability into crop production forecast in Europe

Scientific Reportsvolume 9, Article number: 1277 (2019) | Download Citation


Studies show that climate variability drives interannual changes in meteorological variables in Europe, which directly or indirectly impacts crop production. However, there is no climate-based decision model that uses indices of atmospheric oscillation to predict agricultural production risks in Europe on multiple time-scales during the growing season. We used Fast-and-Frugal trees to predict sugar beet production, applying five large-scale indices of atmospheric oscillation: El Niño Southern Oscillation, North Atlantic Oscillation, Scandinavian Pattern, East Atlantic Pattern, and East Atlantic/West Russian pattern. We found that Fast-and-Frugal trees predicted high/low sugar beet production events in 77% of the investigated regions, corresponding to 81% of total European sugar beet production. For nearly half of these regions, high/low production could be predicted six or five months before the start of the sugar beet harvesting season, which represents approximately 44% of the mean annual sugar beet produced in all investigated areas. Providing early warning of crop production shortages/excess allows decision makers to prepare in advance. Therefore, the use of the indices of climate variability to forecast crop production is a promising tool to strengthen European agricultural climate resilience.


By 2050, the global demand for agricultural goods is expected to grow sharply, driven by the projected demands from an expanding world population, dietary shifts, and increasing biofuel consumption1,2,3. At the same time, there are several major obstacles to boosting crop yields, including a decrease in the area of arable land per person4, and variability in global climate. Creating a resilient agricultural system requires the incorporation of preparedness measures against weather-related events that can trigger disruptive risks such as droughts.

The use of climate information with long-lead times, such as the seasonal predictions of the El Niño Southern Oscillation (ENSO), has allowed farmers to anticipate risks and to improve their management in several parts of the world5,6,7,8,9,10. ENSO influences global agriculture in several ways, including through changes in hydro-meteorological extremes11,12,13,14,15,16 and climate extremes17,18,19,20, which directly or indirectly impact crop yield, production and prices7,21,22,23. However, ENSO only slightly modulates the European climate24,25, where the interannual anomalies in common atmospheric variables such as temperature and precipitation are driven mostly by other atmospheric oscillations11. For instance, the North Atlantic Oscillation (NAO), the East Atlantic/West Russian pattern (EA/WR) and the East Atlantic Pattern (EA) are known to be related with precipitation patterns in Europe, especially in the Iberian Peninsula11,26,27,28,29,30,31, and the Scandinavian Pattern (SCA) influences rainfall in north-eastern Europe32.

Whilst several studies have found connections between these indices of climate variability and common atmospheric variables, few have addressed the role of large-scale atmospheric oscillations on the variability of agricultural production, especially at the pan-European level. Initial studies33,34 investigating the relationship between modes of climate variability and winter wheat anomalies concluded that NAO and EA patterns are strong indicators of yearly wheat deviations, and other studies found significant connection between major European crops and indices of large-scale climate variability35,36,37,38,39.

Agricultural producers make decisions regularly throughout the year, including tactical ones (actions to be taken within weeks or months) and strategic ones (actions to be taken within future seasons or years)6,40. Introducing climate forecasting into producer management depends on the availability of relevant information during the decision-making process, which requires an understanding of the relationship between European climate variability and crop production at several lead times (LD)6,41,42,43. To the best of our knowledge, there is no climate-based decision model that uses indices of atmospheric oscillation to predict agricultural production risks in Europe at different lead times.

In this paper, we develop such a model for multiple time-scales by exploring the relationship between large-scale indices of climate variability and anomalies in sugar beet production. Therefore, we aim at identifying those regions where a robust model can be established based on the indices of atmospheric oscillation investigated. For this, we applied a supervised Machine Learning decision tree-based algorithm44, using predictors (in this case the ENSO, NAO, SCA, EA and EA/WR) recorded within the growing season to establish a prediction between high and low values of the predictands (sugar beet production). Based on the accuracy and predictive skill of the model, we also discuss how this information potentially improves the management of the agricultural sector by combining the findings with a seasonal forecasting system of crop production.


General performance of the FFT models

In this section, we analyse the performance of the Fast-and-Frugal Trees (FFT) in predicting high/low sugar beet production events. The best performing indices of climate variability for each NUTS2 (Nomenclature of territorial units for statistics) region are shown in the supplementary material (Table S1.1–S1.6 and Fig. S2).

In total, the cross-validated FFT models distinguished between sugar beet high/low production events in 160 out of 207 NUTS2 regions, covering nearly 77% of investigated areas (Fig. 1G); 81% of the mean annual sugar beet production is harvested in these regions (Fig. 2). An overview of the mean sugar beet production in all NUTS2 regions investigated is available in Fig. S3. In some locations, the FFT models have skill to predict high/low production for more than one LD, and already at LD6 (March) before the start of the sugar beet harvesting season.

Figure 1
Figure 1

Regions where the FFT models have predictive skill (Area Under the Curve index or AUC > 0.7): (A) six months (March) to (F) one month (August) before the beginning of the harvesting season. In (G) the maps were overlaid in descending order from longest to shortest lead time. Regions without predictive skill (AUC < 0.7) are shown in grey. Field significance of the results was assessed using the binomial distribution and found to be highly significant (P < 0.001). Lead times that were found to be significant only due to bootstrapping (P < 0.1) are indicated with an asterisk.

Figure 2
Figure 2

Standard deviation of the mean sugar beet production in NUTS2 regions where the FFT models have predictive skill (AUC > 0.7) in all lead times.

Predicting sugar beet high/low production events

In March and April, six (LD6) and five (LD5) months before sugar beet harvesting in Europe respectively, we found that the FFT models have predictive skill in a total of 79 out of 207 NUTS2 regions, with an Area Under the Curve (AUC) index ranging from 0.70 to 1.00 (Supplementary Table S1.1 and S1.2). Western and eastern Europe have the highest number of NUTS2 regions with AUC > 0.7. In 56% and 55% of the NUTS2 regions located in western and eastern Europe respectively, significant predictions (P < 0. 1) are already observed in these lead times. For LD6 and LD5, the overall balanced accuracy (BACC), which represents the skill of the FFT in correctly predicting high/low production events, is on average, 79% for both lead times. Approximately 44% of the mean annual sugar beet is produced in these 79 NUTS2 regions. The Hit Rate (HR), thus the probability of a predicting a true low sugar beet production event in these regions is, on average, 74% and 84% at LD6 and LD5, respectively. The spatial distribution of the HR, False Alarm Rate (FAR) and Positive Predictive Value (PPV) are shown in Fig. 3. The HR values are especially high (above 90%) in central Europe, particularly over large areas in Germany. On average, the probability of predicting a false low sugar beet production event (FAR) is 17% and 26% at LD6 and LD5, respectively. The probability of a true low production (hereafter referred to as PPV) over all low production events predictions represents the trade-off between HR and FAR. PPV values are, on average, 90% and 83% at LD6 and LD5, respectively. In Fig. 4, the results are presented for predicting high sugar beet production events. We found that the probability of predicting a true high sugar beet production event in these regions (defined as Correct Rejection Rate or CR) is, on average, 83% and 74% at LD6 and LD5 respectively (Fig. 4A,B). The probability of predicting a false high sugar beet production event (defined as Miss Rate or MS) is, on average, 26% and 16% at LD6 and LD5, respectively. The average Negative Predictive Value (NPV), which represents the probability of a true high production (CR) over all high production events predictions, is 74% and 81% at LD6 and LD5 respectively. In the supplementary material Fig. S2, we display the respective large-scale indices of climate variability that were used by the FFT models as a predictor of high/low production events; results are shown for areas with significant predictive skill (AUC > 0.7 and P < 0. 1). In March and April, during the early growing stages of sowing and emergence, sugar beet is sensitive to water deficits and frost, and winter and early spring weather conditions in Europe are strongly associated with EAWR, NAO and EA11,25,35,45,46. Relationships between sugar beet production and mean EAWR, EA and NAO from January to March, and mean EA and NAO from February to April were found in this study, and also by others when assessing the relationship between these atmospheric oscillations and other crop types, productivity and vegetation dynamics35,47,48,49. In addition, we observed that in most of the regions, multiple indices of atmospheric oscillations were used simultaneously as a predictor of sugar beet production instead of a single index (Figs S2, S4). We observed that winter and early spring NAO influences summer crop production in large areas in Europe, especially in Germany (LD5), as previously found by others35,50, where positive NAO in January and February drives more intense precipitation in northern and north western Europe and the opposite in the south of the continent11,25, which might affect the early growing season. However, we neither find a north-south dipole impact of the NAO at these lead times, as often is observed for rainfall, nor a large influence of NAO in southern Europe, as found by others48,51. Other important predictors of sugar beet high/low production events are EA and EA/WR patterns, especially in regions in western and eastern Europe. Since precipitation and temperature variability in Europe are more strongly modulated by winter and spring oscillation regimes, the relevance of winter and spring atmospheric oscillations may be twofold: (a) winters and early spring modes of climate variability provide soil moisture for crop development in summer, as also suggested by others50,52,53; (b) winter and spring weather conditions affect sowing and early growing stage, as found by previous research35,54.

Figure 3
Figure 3

Performance metrics for predicting low production sugar beet events for areas with AUC > 0.7 at six lead times (A to F). Hit Rate (HR) is the probability of low production occurrences that were correctly predicted; Positive Predictive Value (PPV) index represents the probability of FFT to detect true low production sugar beet events over all low production (including False Alarms); False Alarm Rate (FAR) is the probability of a false low production occurrence. Regions without predictive skill (AUC < 0.7) are shown in grey. The FAR is cut off at <40% because in more than 90% of the NUTS2 regions the results are below this threshold.

Figure 4
Figure 4

Display of standard classification statistics for predicting high production sugar beet events for areas with AUC > 0.7 at six lead times (A to F). Correct Rejection Rate (CR) shows the probability of high production occurrences that were correctly predicted; Negative Predictive Value (NPV) index represents the probability of the FFT detecting a true high production sugar beet event over all high production (including Misses); Miss Rate (MS) is the probability of a false high production occurrence. Regions without predictive skill (AUC < 0.7) are shown in grey. The MS is a cut off at <40% because in more than 90% of the NUTS2 regions the results are below this threshold.

In May and June, four and three months before the start of the sugar beet harvesting season, the FFT models have predictive skill in a total of 92 out of 207 NUTS2 regions, as displayed in Fig. 1C, D. Within the investigated areas, approximately 47% of the mean annual sugar beet is produced in these 92 NUTS2 regions. Comparing all lead times investigated, LD3 was found to be the one with the most regions with predictive skill (74 out of 207). In May and June, western and northern Europe have the highest number of NUTS2 regions with AUC > 0.7. In 54% and 67% of the NUTS2 located in western and northern Europe respectively, significant predictions (P < 0. 1) are found in these lead times. In addition to LD6 and LD5, the FFT models extend the predictive skill to some western and southern European regions and the United Kingdom. The AUC index ranges from 0.70 to 1.00 (Supplementary Table S1.3 and S1.4). On average, the mean BACC is 78% (LD4) and 83% (LD3). In these areas, HR, PPV and FAR are, on average, 78%, 86% and 22% (LD4) and 89%, 82% and 27% (LD3) respectively. The CR and NPV are, on average, 78% and 74% (LD4), and 72% and 88% (LD3) respectively. The mean MS at LD4 and LD3 is 22% and 11% respectively (Fig. 4C,D). In June, sugar beet root has reached its vegetative growth, and beet leaves are subjected to both heat and water stress55. FFT models primarily used EA/WR and ENSO averages from spring and late spring (Fig. S2 LD3) as the main predictors of sugar beet production in western and eastern Europe. These indices have also been found to have links with maize grain and wheat yield in other studies7,35. Spring/late spring weather in Europe are associated with a range of atmospheric oscillations including EA/WR and EA56, and less influenced by NAO11. The EA/WR pattern influences mainly western and eastern Europe, where a positive (negative) EA/WR phase is related to high temperature anomalies35 and drier (wetter) conditions. ENSO was previously found to have significant associations with changes in precipitation in spring, especially over western and northern Europe11,57. However, in Spain, where we would expect a larger spatial link between SCA, EA and ENSO patterns on precipitation11 and consequently on crop production35, the effect of these climate patterns may be weaker than elsewhere, probably due to common irrigation practices in some areas, especially in southern and north western Spain58,59,60.

In July and August, the FFT models predict sugar beet high/low production events in 62 of the 207 NUTS2 regions (Fig. 1E,F), with an AUC index ranging from 0.70 to 1.00 (Supplementary Table S1.5 and S1.6). In July and August, western and eastern Europe have the highest number of NUTS2 regions with AUC > 0.7. In 35% and 41% of the NUTS2 located in western and eastern Europe respectively, significant predictions (P < 0. 1) are found at LD2 and LD1. In addition to models at LD6 to LD3, the FFT models extend the predictive skill to several regions in Europe, especially in large areas of Spain and Finland. On average, the mean BACC for LD2 and LD1 is 80%. Within the investigated areas, approximately 22% of the mean annual sugar beet is produced in these 62 NUTS2 regions. The average HR, FAR and PPV is 81%, 22% and 82% for LD2, and 83%, 22% and 84% for LD1 (Fig. 3E,F). We found that the CR, MS and NPV is, on average, 78%, 19% and 83% for LD2, and 78%, 16% and 81% for LD1. (Fig. 4E,F). In July and August, when the sugar beet has reached its late development stage, reduced water availability has a small impact on the yield. EA/WR from May to July in addition to EA and NAO averages in June to August are most used as sugar beet production predictors in large areas of Europe. In summer, the EA/WR pattern was found in previous research35 to have strong links with positive temperature anomalies and below-average rainfall in western and south eastern Europe, which is beneficial for harvesting. However, our results also suggest that multiple indices of atmospheric oscillation show links with sugar beet production in scattered areas in Europe, especially at LD2 (Fig. S2). In addition, previous investigations found that the precipitation, temperature, vegetation dynamics and maize yield in Europe have links with these atmospheric oscillations in summer11,35,47. Large-scale atmospheric oscillation indices may have less predictive skill for production in early summer than winter and spring for two reasons: (a) due to the higher importance of regional-to-local atmospheric phenomena during summer, also highlighted by others35. This means that sugar beet production estimates might be more efficiently captured by the regional crop model forced by observed surface climate variables locally; and (b) in July and August, when the sugar beet has reached its late development stage, reduced water availability has a small impact on the crop61.

Discussion and Conclusion

In this section we discuss how our findings could be used to support agricultural management and decision-making processes in Europe, followed by recommendations, limitations and conclusions of the study.

Currently, the Joint Research Centre (JRC) MARS crop yield forecasting system (M-CYFS) forecasts national sugar beet yields for the current growing season based on a scenario technique and trend analysis62, which substantially differs from the method applied in this study, which also targets production. The M-CYFS is a decision support system with the purpose of providing evidence-based information on the status of annual crops in the EU and neighbouring countries by monitoring crop growth and forecasting crop yields along the season5,63,64. It uses agro-meteorological indicators derived from observed meteorological data as well as crop growth models and remote sensing information, which are applied together to build a statistical yield forecast using best-fit criterion to explain a cause-effect relationship with historical yield statistics at national level. Sugar beet forecasts early in the season are purely based on observed trends, but could potentially gain predictive skill if indices of climate variability, such the ones investigated in this study, are integrated in M-CYFS based on the outcomes of this study. In general, agro-meteorological and remote sensing indicators start to demonstrate a certain reliability for the regression forecasts (r-squared value > 0.5 if de-trended data are used) from the end of June or beginning of July depending on the country. Furthermore, since large-scale indices of climate variability can be predicted with higher lead times than weather variables and related crop growth variables, the M-CYFS could further extend its lead-time if predictions of the large-scale indices of climate variability were used from dynamic climate models35.

Anomalies in temperature and precipitation driven by climate variability do not always explain crop production, and regions without predictive skill, may have been masked by a number of local agro-management activities. For instance, irrigation practices, which are observed in Spain and Bulgaria, may have lowered the negative impacts of unfavourable weather conditions and the skill of the FFT models in detecting high and low production. Overall, the decision-model performed more consistently in eastern and western Europe. These regions also have low shares of irrigated areas, which may have influenced the model performance58. Additionally, in some regions, limitations in data availability on crop production were encountered during this investigation (described in the methods section), and this may have affected the extraction of the high/low sugar beet production indicator per NUTS2 region. For instance, we disaggregated national crop production statistics data in several NUTS2 regions (available in supplementary S7), and our downscaling method adds uncertainty to the model predictions. Therefore, the FFT models may not be performing satisfactorily in some regions for three reasons: (a) presence of irrigation practices, which may inhibit the impacts of unfavourable weather conditions; (b) lack of underlying physical relationship between sugar beet production and the investigated indices of climate variability; and (c) due to limitations in the sugar beet production downscaling method.

In Europe, the growing period of sugar beet is normally between 140 and 160 days61, mostly in the northern half of Europe, where the climate is more suited to growing beet65. Sugar beet is planted in early spring and harvested in late September, before the cold season starts. In March, beet producers make strategic decisions regarding the amount of sowing for the year and planning tactical actions for the germination and early plant development stages, which comprise the most sensitive periods of the crop61. Due to strict regulations of the EU sugar market, the 2007 Sugar Reform limited total EU production to 14.7 million tonnes of raw sugar until the marketing year 2016/201765. If climatic conditions indicate a probable “high production” scenario at this early stage, the EU could better plan “out-of-quota” measures, such as: exporting the excess of sugar beet production to the EU’s annual World Trade Organisation quota, which is limited to 1.374 million tonnes; disposing excess on the EU market for industrial purposes; or counting against the following year’s sugar “quota”65. In addition, each year the EU market must decide by March 16th for a first “preventive” withdrawal to allow producers to reduce their beet sowings. However, the quota management ended as of 30 September 2017. On the other hand, if climatic conditions show signs of shortage in production, tactical measures can be taken to increase supplies as follows: (1) better preparation or further investment in responsive irrigation schemes as sugar beet is particularly sensitive to water deficits in early spring55,66; (2) taking measures to prevent freeze damage to crops such as active methods (e.g. adding heat and covering crops) and passive methods (e.g. proper scheduling of planting within the safe freeze-free period) as night frost in spring can damage sugar beet and delay seed germination67,68; and (3) before planting, producers could decide to reduce their financial losses by purchasing appropriate crop insurance products against deviations from their long-term yields. Prior information about the spatial configuration of risk would support insurance companies to better allocate resources to comply with the EU solvency requirements, which demands that insurers have adequate reserves for 99.5% of potential loss events69.

By June, sugar beet root has reached its vegetative growth, and its leaves are subjected to both heat and water stress55. Temperatures greater than 30 °C greatly decrease sugar yields61, and water stress is considered the major limitation to crop productivity and yield stability66. In central and western Europe, drought stress can reduce sugar beet yields by 10–30% compared to the long-term average70,71. Beet water requirements are moderate, and if low production is predicted, adequate water should be available to allow the sugar beet to develop a good root system for extracting water from the soil. Predictions of low or high production, mainly needed on a regional scale for industries and policy makers72, supports the sugar industry in adapting normative plans for optimizing processing campaigns73, such as factory operations concerning delivery schedules and storage capacity, transport logistics and export sales. Sugar beet production forecasts could also be a useful aid for marketing operations, where prices fluctuate based on the supply and demand of the product72.

In August, when the sugar beet has reached its late development stage, reduced water availability has a small impact on the yield61. Forecasts of low production could support sugar beet producers to better prepare against cold and wet days, which often lead to deterioration of harvesting conditions and increase the probability of fungal infections, increasing the risk of late harvesting in autumn74. Moreover, forecasts of high production may indicate favourable conditions to the harvesting period, when certain levels of soil moisture and rain-free days are preferred74.

Our study did not aim to produce quantitative prognostic information about crop production; instead, we focused on identifying those regions where a robust model can be established based on the indices of atmospheric oscillation investigated, and used as an early warning indicator for crop impacts. This is a primary step towards the adoption and use of climate-related forecasts in agricultural decision-making: if there were no climate variability influence on crop production, it is unlikely that agricultural stakeholders and markets would benefit from long-lead time climate information. In addition, the observed indices of climate variability assessed in this study can be forecast with varying levels of skill and lead times. Skilful predictions of NAO have been extended to more than a year ahead35,75. EA summer and autumn anomalies can be properly hindcast with a lead time of 1 to 2 months25,76, while ENSO forecasting is more developed, and most prediction systems have some skill for predicting events up to 14 months lead times77. Given that the seasonal predictability of large-scale climate variability is generally higher than that of surface weather variables in Europe35, empirical seasonal risk outlooks could potentially be developed based on predicted values of the indices of climate variability78.

The current study is a statistical analysis of the effect of climate variability on sugar beet production, and resulted in the selection of related predictors for each region in Europe out of a total of five indices of climate variability. Future work could benefit from using different methods to classify the different “low/high production”, and examining time-lags between the indices of climate variability on the agricultural impact indicators. Moreover, some of the significant results may had occurred by random chance (LD4 and LD1 Fig. 1), and results may be interpreted with caution. Further insights into relationship between climate variability and crop production can be obtained by applying compensatory models since a more complex model can reveal important features that are not being captured by a simple model. In addition, this study can benefit from testing different decision algorithms, since the ifan algorithm, which was adopted in this study, assumes independence between predictors. Furthermore, crop production databases, such as the one used in this study, are also known to face limitations, such as reporting errors. Testing the proposed method on other crop production databases and crop types could provide further insight in the strengths and limitations of the approach. Lastly, climate can only partly explain sugar beet production. Other important aspects such as changes in farm-level management, economy, agronomy and quality of the land were not included in this study.


We use FFT79 to predict impacts on agricultural production applying five large-scale indices of climate variability: ENSO, NAO, SCA, EA and EA/WR. The FFT models identify which indices of climate variability are capable of classifying production in given years into high or low production classes. From a database of historical sugar beet production of the European statistical office (EUROSTAT), we derived a yearly agricultural production indicator from 1975–2013, namely high/low production based on observed anomalies. For the same period, we obtained 3-month average values for the indices of climate variability from January to March, February to April, March to May, April to June, May to July and June to August, further referred to as LD6, LD5, LD4, LD3, LD2 and LD1, respectively. An overview of the methodological framework is displayed in Fig. 5. The methods and datasets are described in detail in the following subsections.

Figure 5
Figure 5

Flowchart representing the methodological framework applied in this study, handled in three steps: (1) collection of two main datasets; (2) extraction of five climate indicators at six lead times, and a discrete variable based on sugar beet climatological anomalies; and (3) example of an FFT output model containing standard classification statistics for a specific NUTS2 region and LD3.

Indices of climate variability

In this study, we represent climate variability using a 3-month average of the ENSO, NAO, EA, EA/WR and SCA indices from the National Oceanic and Atmospheric Administration Climate Prediction Center ( We extracted 3-month average values from 1975–2013, corresponding to the months of January to March, February to April, March to May, April to June, May to July and June to August. These values represent six different LD before the start of the sugar beet harvesting season (September) in Europe80. For instance indices of climate variability at LD6 represent a 3-month average between January to March, therefore six months before the start of the sugar beet harvesting season (September).

We used the standardized Southern Oscillation Index (SOI) from 1951–2016, calculated from observed sea level pressure differences between Tahiti and Darwin (Australia), as a continuous measure of ENSO strength81. The time series of the four Northern Hemisphere teleconnection patterns from 1950–2016 represent monthly mean standardized 500-mb height anomalies at 20°N–90°N82. The NAO is the main mode of low-frequency variability over the North Atlantic, and consists of a north-south dipole of anomaly in surface pressure between Greenland and the central latitudes of the North Atlantic between 35°N and 40°N82. The EA pattern is the second prominent mode of low-frequency variability over the North Atlantic, consisting of a north-south dipole of anomaly centers that extends across the entire region from the East to the West. The EA/WR represents four main anomalies, centred over Europe and northern China, central North Atlantic and north of the Caspian Sea, while the SCA shows anomalies mainly in Scandinavia and western Russia82. The Euro-Atlantic region is mainly dominated by these four Northern Hemisphere teleconnections11,35.

Index of agricultural impact

We obtained annual historical records of sugar beet production (in 1000 ton of fresh weight) from 1975 to 2013 for 232 NUTS2 from EUROSTAT. We examined sugar beet records for three major reasons: the European Union (EU) is the world’s leading producer of sugar beet83; sugar beet production is wide-spread within the EU territory; and generally, this crop is not extensively irrigated, thus having a strong dependency on rainfall58. We performed the analysis on historical records of sugar beet production instead of sugar beet yield (tons/hectare) due to the large unavailability of datasets of the latter in the vast majority of NUTS2 regions.

Years without sugar beet production records (MN2) at the NUTS2 level were filled according to the following method. First, we compared sugar beet production data registered at the national level (PN) by EUROSTAT (if available) with the sum registered at the NUTS2 level (PN2) for a given country (c) and year (t).

$${{{\rm{M}}}_{{\rm{N}}2,{\rm{t}},{\rm{c}}}={P}_{N,t,c}-\sum }^{}{{\rm{P}}}_{\mathrm{N2},t,c}$$

If there was a positive difference between both datasets \(({{\rm{P}}}_{{\rm{N}},{\rm{t}},{\rm{c}}} > {\sum }^{}{{\rm{P}}}_{\mathrm{N2},t,c})\) and more than one NUTS2 region without a record, the missing production record at the NUTS2 (MPN2) region was filled proportionally with respect to the sum of its sugar beet harvesting area (HAN2):

$${{\rm{M}}}_{{\rm{PN}}2,{\rm{t}},{\rm{c}}}={{\rm{M}}}_{N2,t,c}/\frac{100}{{\sum }^{}H{A}_{N2}}$$

We assumed a positive and direct relationship between harvesting area and production: the larger the harvesting area, the higher the production, even though other factors, such as agricultural management, could affect production. Estimates of sugar beet harvesting areas are obtained from MIRCA 2000 and described in Sack et al. 201080 (available in Fig. S5). In case of missing records in the NUTS2 region but no difference in the overall production \(({{\rm{P}}}_{{\rm{N}},{\rm{t}},{\rm{c}}}={\sum }^{}{{\rm{P}}}_{{\rm{N}}2,{\rm{t}}})\), we assigned MPN2,t,c = 0. Last, in the case of a negative difference \(({{\rm{P}}}_{{\rm{N}},{\rm{t}},{\rm{c}}} < {\sum }^{}{{\rm{P}}}_{{\rm{N}}2,{\rm{t}}})\), the missing sugar beet production remained unaltered.

For each NUTS2 region, we calculated sugar beet production anomalies (observed value minus the multiyear mean) after removal of the linear trend (if the p-value is less than or equal to 0.1). We classified below zero anomalies as “low production”, and above zero anomalies as “high production”, creating a discrete agricultural impact indicator. Only NUTS2 regions with time series longer than 20 years were further investigated (NUTS2 areas that fit the criteria, and their respective summary of statistics are displayed in supplementary material Fig. S5 and Table S6, consecutively). In total, we were able to fit a FFT model in 207 NUTS2 regions.

Statistical approach: Fast-and-Frugal Decision Tree (FFT)

In this study, we used FFT to predict sugar beet production as a function of indices of climate variability. In heuristic decision-making, FFT are simple decision trees for classifying cases (e.g. sugar beet production) into one of two classes (e.g. low production vs. high production) based on a particular predictor, or cue. FFT models establish simple rules for making decisions based on fast-and-frugal heuristics approach84,85, and offer simple and transparent search rules for practical decision problems79 as a competitive alternative for more complex Machine Learning and regression methods86.

As displayed in Fig. 5 step 3, the structure of an FFT determines the exact number, sequence and threshold of predictors that are applied to reach a final classification84,87. The FFTs algorithm applied in this study is limited to maximum five cues79. Consequently, a five-cue decision tree is based on the best five performing indices of climate variability. However, FFTs can be based on 1 to 5 cues since it uses non-compensatory decision rules, which apply a limited subset of all predictors for establishing a binary classification86. Non-compensatory algorithms ignore information, once a decision is completed, and therefore, no additional predictors can change such decision79. This aspect is often perceived to have both practical and statistical advantages over compensatory algorithms, such as regression models86. First, because by only using a partial subset of predictors, FFTs are relatively simple, and can perform better in predicting new data, thus they tend to avoid overfitting44. Second, FFT algorithms uses search rules that specify where to look for information and when to end search, which can guide decision makers in gathering information and assist in supporting decision tasks44,85.

The FFT algorithm is designed to learn from and make predictions on data. FFTs are fitted to a training dataset, which is used for learning the model, and deriving its parameter. In summary, the FFT’s algorithm is constructed as instructed: a) select predictors; b) determine a decision threshold for each predictor; c) determine the order of predictors; and d) determine the exit for each predictor86. By definition, FFTs must have either a negative or a positive exit (or both in the case of the final node of a decision tree)79.

The accuracy of the FFT is measured by the BACC Index, which is calculated based on the amount of correct decisions (Table 1) obtained from the ifan decision algorithm, which is described in detail in previous research79. In summary, the ifan decision algorithm tests several different thresholds of the investigated indices of climate variability to find one that maximizes the predictor’s accuracy. Consequently, once the set of multiple FFTs has been created, ifan selects the decision tree with the highest balanced accuracy.

Table 1 Definition of Standard Classification Statistics.

In order to avoid overfitting, we cross-validated the FFTs using the train-test split method. For this, we partitioned 70% of all data for training the FFT models, and the other 30% for testing the models. We chose this validation method to assess the FFT potential success (if applied in a practical case), and its hindcast skill in predicting past sugar beet production events. In addition, this method takes less computation power than other cross-validation techniques such as k-fold. A more detailed explanation of the train-test split method is available in the Supplementary Material S8. For each FFT, we assessed the skill of the model to predict classes of “low production” and “high production” using the AUC index88. AUC measures how well the FFT can distinguish binary classes (low/high)89, displayed in supplementary material Fig. S9. The AUC index was calculated using the trapezoidal rule, and values can vary between 0 and 1, where a perfect prediction has an AUC = 1.0, and predictions that are randomly drawn are presumed to provide an AUC = 0.590. We tested the statistical significance of the AUC by bootstrapping the values of the high and low sugar beet production events at the NUTS2 level using 1000 iterations (more details in the supplementary material 10). Field significance of the results was assessed using the binomial distribution91.

For NUTS2 regions with an AUC > 0.7, we display standard classification statistics such as HR, FAR, CR, MS, PPV and NPV. Their definition and formula are given in Table 1. We obtained FFT for each particular lead time and NUTS2 region through the following steps:

  1. 1.

    Calculating the pruning parameter of the model, meaning that we assessed the ideal size of decision trees by cross-validating the FFT models using train-test split method (see supplementary material S8);

  2. 2.

    Selecting the pruning parameter and decision tree that maximizes the BACC index of the tested model;

  3. 3.

    Calculating the AUC index for each best performing decision tree;

  4. 4.

    Assessing the statistical significance of the AUC obtained in step 3 by bootstrapping and calculating field significance (see supplementary material S10);

  5. 5.

    Analysing the performance of the significant FFT model by calculating standard classification statistics (Table 1).

After following such steps, we built a new FFT after recombining the two samples (training and testing) and adopting the pruning parameter that was found to maximize the BACC index. Hence, this procedure enables us to find the set of predictors which are most important for each NUTS2 region and lead time (Fig. 5 step 3). Since we aim at identifying regions where a robust model can be established based on the indices of atmospheric oscillation, we focus our results in regions with AUC > 0.7.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.


  1. 1.

    Godfray, H. C. J. et al. Food security: The challenge of feeding 9 billion people. Science 327, 812–818 (2010).

  2. 2.

    Pingali, P. Westernization of Asian diets and the transformation of food systems: Implications for research and policy. Food Policy 32, 281–298 (2007).

  3. 3.

    Ray, D. K., Mueller, N. D., West, P. C. & Foley, J. A. Yield Trends Are Insufficient to Double Global Crop Production by 2050. PLoS One 8 (2013).

  4. 4.

    FAO. Achieving sustainable gains in agriculture. at, FAO- South Am. (2000).

  5. 5.

    Bussay, A., van der Velde, M., Fumagalli, D. & Seguini, L. Improving operational maize yield forecasting in Hungary. Agric. Syst. 141, 94–106 (2015).

  6. 6.

    Haigh, T. et al. Climate Risk Management Mapping the decision points and climate information use of agricultural producers across the U. S. Corn Belt. Clim. Risk Manag. 7, 20–30 (2015).

  7. 7.

    Iizumi, T. et al. Impacts of El Niño Southern Oscillation on the global yields of major crops. Nat. Commun. 5, 3712 (2014).

  8. 8.

    Meinke, H. & Stone, R. C. Seasonal and inter-annual climate forecasting: the new tool for increasing preparedness to climate variability and change in agricultural planning and operations. Clim. Change 70, 221–253 (2005).

  9. 9.

    Motha, R. P. & Baier, W. In Increasing Climate Variability and Change. 137–164 (Springer, 2005).

  10. 10.

    Nnaji, A. O. Forecasting seasonal rainfall for agricultural decision-making in northern Nigeria. Agric. For. Meteorol. 107, 193–205 (2001).

  11. 11.

    Casanueva, A., Rodríguez-Puebla, C., Frías, M. D. & González-Reviriego, N. Variability of extreme precipitation over Europe and its relationships with teleconnection patterns. Hydrol. Earth Syst. Sci. 18, 709–725 (2014).

  12. 12.

    Sun, X., Renard, B., Thyer, M., Westra, S. & Lang, M. A global analysis of the asymmetric effect of ENSO on extreme precipitation. J. Hydrol. 530, 51–65 (2015).

  13. 13.

    UNMGCY. Youth Science Policy Interface Publication – Special Edition: Disaster Risk Reduction: A Road of Opportunities. at (2017).

  14. 14.

    Veldkamp, T. I. E., Eisner, S., Wada, Y., Aerts, J. C. J. H. & Ward, P. J. Sensitivity of water scarcity events to ENSO-driven climate variability at the global scale. Hydrol. Earth Syst. Sci. 19, 4081–4098 (2015).

  15. 15.

    Ward, P. J., Beets, W., Bouwer, L. M., Aerts, J. C. J. H. & Renssen, H. Sensitivity of river discharge to ENSO. Geophys. Res. Lett. 37, 1–6 (2010).

  16. 16.

    Ward, P. J. et al. Strong influence of El Nino Southern Oscillation on flood risk around the world. Proc. Natl. Acad. Sci. 111, 15659–15664 (2014).

  17. 17.

    Barlow, M., Nigam, S. & Berbery, E. H. ENSO, Pacific decadal variability, and U.S. summertime precipitation, drought, and stream flow. J. Clim. 14, 2105–2128 (2001).

  18. 18.

    Dilley, M. & Heyman, B. N. ENSO and disaster: droughts, floods and El Niño/Southern Oscillation warm events. Disasters 19, 181–93 (1995).

  19. 19.

    Donat, M. G. et al. Changes in extreme temperature and precipitation in the Arab region: Long-term trends and variability related to ENSO and NAO. Int. J. Climatol. 34, 581–592 (2014).

  20. 20.

    Trenberth, K. E. & Fasullo, J. T. Climate extremes andclimate change: The Russian heat wave and other climate extremes of 2010. J. Geophys. Res. Atmos. 117, 1–12 (2012).

  21. 21.

    Ferreyra, R. A. et al. A linked-modeling framework to estimate maize production risk associated with ENSO-related climate variability in Argentina. Agric. For. Meteorol. 107, 177–192 (2001).

  22. 22.

    Ray, D. K., Gerber, J. S., MacDonald, G. K. & West, P. C. Climate variation explains a third of global crop yield variability. Nat. Commun. 6, 5989 (2015).

  23. 23.

    Rowhani, P., Lobell, D. B., Linderman, M. & Ramankutty, N. Climate variability and crop production in Tanzania. Agric. For. Meteorol. 151, 449–460 (2011).

  24. 24.

    Brönnimann, S. Impact of El Niño-Southern Oscillation on European climate. Rev. Geophys. 45 (2007).

  25. 25.

    Guimarães Nobre, G., Jongman, B., Aerts, J. & Ward, P. J. The role of climate variability in extreme floods in Europe. Environ. Res. Lett. 12, 84012 (2017).

  26. 26.

    Lopez-Bustins, J. A., Martin-Vide, J. & Sanchez-Lorenzo, A. Iberia winter rainfall trends based upon changes in teleconnection and circulation patterns. Glob. Planet. Change 63, 171–176 (2008).

  27. 27.

    Mariotti, A. Euro-Mediterranean rainfall and ENSO—a seasonally varying relationship. Geophys. Res. Lett. 29, 54–59 (2002).

  28. 28.

    Markovic, D. & Koch, M. Long-term variations and temporal scaling of hydroclimatic time series with focus on the German part of the Elbe River Basin. Hydrol. Process. 28, 2202–2211 (2014).

  29. 29.

    Rios-Cornejo, D., Penas, A., Alvarez-Esteban, R. & del Rio, S. Links between teleconnection patterns and precipitation in Spain. Atmos. Res. 156, 14–28 (2015).

  30. 30.

    Rodó, X., Baert, E. & Comin, F. A. Climate Dynamics Variations in seasonal rainfall in Southern Europe during the present century: relationships with the North Atlantic Oscillation and the. Clim. Dyn. 13, 275–284 (1997).

  31. 31.

    Struglia, M. V., Mariotti, A. & Filograsso, A. River discharge into the Mediterranean sea: Climatology and aspects of the observed variability. J. Clim. 17, 4740–4751 (2004).

  32. 32.

    Bueh, C. & Nakamura, H. Scandinavian pattern and its climatic impact. Q. J. R. Meteorol. Soc. 133, 2117–2131 (2007).

  33. 33.

    Cantelaube, P., Terres, J. & Doblas-Reyes, F. J. Influence of climate variability on European agriculture- analysis of winter wheat production. Clim. Res. 27, 135–144 (2004).

  34. 34.

    Kettlewell, P. S., Stephenson, D. B., Atkinson, M. D. & Hollins, P. D. Summer rainfall and wheat grain quality: relationships with the North Atlantic Oscillation. Weather 58, 155–164 (2003).

  35. 35.

    Ceglar, A., Turco, M., Toreti, A. & Doblas-Reyes, F. J. Linking crop yield anomalies to large-scale atmospheric circulation in Europe. Agric. For. Meteorol. 240–241, 35–45 (2017).

  36. 36.

    Gouveia, C. & Trigo, R. M. In geoENV VI–Geostatistics for Environmental Applications. 335–345 (Springer, 2008).

  37. 37.

    Lorenzo, M. N., Taboada, J. J., Lorenzo, J. F. & Ramos, A. M. Influence of climate on grape production and wine quality in the Rías Baixas, north-western Spain. Reg. Environ. Chang. 13, 887–896 (2013).

  38. 38.

    Marta, A. D., Grifoni, D. & Mancini, M. The influence of climate on durum wheat quality in Tuscany, Central Italy. Int. J. Biometeorol. 55, 87–96 (2011).

  39. 39.

    Gimeno, L. et al. Identification of empirical relationships between indices of ENSO and NAO and agricultural yields in Spain. Clim. Res. 21, 165–172 (2002).

  40. 40.

    Hollinger, S. E. Incorporating Weather and Climate data into Integrated Crop Management Systems. Clim. Agric. Drought Misc. Pap. 1 (1991).

  41. 41.

    Calanca, P., Bolius, D., Weigel, A. P. & Liniger, M. A. Application of long-range weather forecasts to agricultural decision problems inEurope. J. Agric. Sci. 149, 15–22 (2011).

  42. 42.

    Easterling, W. E. & Mjelde, J. W. The importance of seasonal climate prediction lead time in agricultural decision making. Agric. For. Meteorol. 40, 37–50 (1987).

  43. 43.

    Mase, A. S. & Prokopy, L. S. Unrealized potential: A review of perceptions and use of weather and climate information in agricultural decision making. Weather. Clim. Soc. 6, 47–61 (2014).

  44. 44.

    Phillips, N. D., Woike, J. K. & Gaissmaier, W. FFTrees: A toolbox to create, visualize, and evaluate fast-and-frugal decision trees. Judgm. Decis. Mak. 12, 344–368 (2017).

  45. 45.

    Hurrell, J. W. & Deser, C. North Atlantic climate variability: The role of the North Atlantic Oscillation. J. Mar. Syst. 79, 231–244 (2010).

  46. 46.

    Ionita, M. The impact of the East Atlantic/Western Russia pattern on the hydroclimatology of Europe from mid-winter to late spring. Climate 2, 296–309 (2014).

  47. 47.

    Gouveia, C., Trigo, R. M., DaCamara, C. C., Libonati, R. & Pereira, J. M. C. The North Atlantic Oscillation and European vegetation dynamics. Int. J. Climatol. 28, 1835–1847 (2008).

  48. 48.

    Kim, M. & Mccarl, B. A. The Agricultural Value Of Information On The North Atlantic Oscillation: Yield And Economic Effects. Clim. Change 71, 117–139 (2005).

  49. 49.

    Heino, M. et al. Two-thirds of global cropland area impacted by climate oscillations. Nat. Commun. 9 (2018).

  50. 50.

    Gonsamo, A. & Chen, J. M. Winter teleconnections can predict the ensuing summer European crop productivity: Fig. 1. Proc. Natl. Acad. Sci. (2015).

  51. 51.

    Res, C. et al. Identification of empirical relationships between indices of ENSO and NAO and agricultural yields in Spain. Clim. Res. 21, 165–172 (2002).

  52. 52.

    Kettlewell, P. S., Easey, J., Stephenson, D. B. & Poulton, P. R. Soil moisture mediates association between the winter North Atlantic Oscillation and summer growth in the Park Grass Experiment. Proc. R. Soc. B Biol. Sci., (2006).

  53. 53.

    Wang, G., Dolman, A. J. & Alessandri, A. A summer climate regime over Europe modulated by the North Atlantic Oscillation. Hydrol. Earth Syst. Sci. (2011).

  54. 54.

    Petkeviciene, B. & others. The effects of climate factors on sugar beet early sowing timing. Agron. Res 7, 436–443 (2009).

  55. 55.

    Clarke, N., Hetschkun, H., Jones, C., Boswell, E. & Marfaing, H. In Interacting stresses on plants in a changing climate 511–524 (Springer, 1993).

  56. 56.

    Gao, T., Yu, Jyi & Paek, H. Impacts of four northern-hemisphere teleconnection patterns on atmospheric circulations over Eurasia and the Pacific. Theor. Appl. Climatol. 129, 815–831 (2017).

  57. 57.

    Shaman, J. The seasonal effects of ENSO on European precipitation: Observational analysis. J. Clim. 27, 6423–6438 (2014).

  58. 58.

    EUROSTAT. Agri-environmental indicator - irrigation. at,,_2010_(%25_of_total_area_of_each_crop).png. (2010).

  59. 59.

    Wriedt, G., der Velde, M., Aloe, A. & Bouraoui, F. Water requirements for irrigation in the European Union. EUR-Scientific Tech. Res. Reports, EUR 23453 (2008).

  60. 60.

    Wriedt, G., Van der Velde, M., Aloe, A. & Bouraoui, F. Estimating irrigation water requirements in Europe. J. Hydrol, (2009).

  61. 61.

    FAO. Crop Water Information: Sugarbeet. at (2015).

  62. 62.

    Baruth, B., Royer, A., Klisch, A. & Genovese, G. The use of remote sensing within the MARS crop yield monitoring system of the European Commission. Proc. ISPRS 27, 935–940 (2008).

  63. 63.

    van der Velde, M. et al. Use and relevance of European Union crop monitoring and yield forecasts. Agric. Syst (2018).

  64. 64.

    Lecerf, R., Ceglar, A., López-Lozano, R., Van Der Velde, M. & Baruth, B. Assessing the information in crop model and meteorological indicators to forecast crop yield overEurope. Agric. Syst (2018).

  65. 65.

    European Comission. EU sugar policy. at, (2017).

  66. 66.

    Romano, A. et al. Morpho-physiological responses of sugar beet (Beta vulgaris L.) genotypes to drought stress. Acta Physiol. Plant. 35, 853–865 (2012).

  67. 67.

    Pidgeon, J. D. et al. Climatic impact on the productivity of sugar beet inEurope, 1961-1995. Agric. For. Meteorol. 109, 27–37 (2001).

  68. 68.

    Snyder, R. L. & de Melo-Abreu, J. P. Frost protection: fundamentals, practice and economics. (Food and agriculture organization of the United Nations, 2005).

  69. 69.

    European Parliament and Council. Directive on the taking-up and pursuit of the business of insurance and reinsurance (Solvency II), Directive2009/138/EC (2009).

  70. 70.

    Ober, E. The search for drought tolerance insugar beet. Br. sugar beet Rev. 69, 40–43 (2001).

  71. 71.

    Van Swaaij, A., Heijbroek, W. & Basting, J. L. Testing and improving seed vigour in sugar beet. Int. sugar J. 103, 467–472 (2001).

  72. 72.

    Vandendriessche, J. Crop Models and Decision Support Systems for Yield Forecasting and Management of the Sugar Beet Crop. Eur. J. Agron. 4, 269–279 (1995).

  73. 73.

    Kenter, C., Hoffmann, C. M. & Märländer, B. Effects of weather variables on sugar beet yield development (Beta vulgaris L.). Eur. J. Agron. 24, 62–69 (2006).

  74. 74.

    JRC MARS Bulletin. Crop monitoring in Europe, November 2016: Arrival of the first frosts. 24 (2016).

  75. 75.

    Dunstone, N. et al. Skilful predictions of the winter North Atlantic Oscillation one year ahead. Nat. Geosci. 9, 809–814 (2016).

  76. 76.

    Iglesias, I., Lorenzo, M. N. & Taboada, J. J. Seasonal predictability of the east atlantic pattern from sea surface temperatures. PLoS One 9, 1–8 (2014).

  77. 77.

    Gonzalez, P. L. M. & Goddard, L. Long-lead ENSO predictability from CMIP5 decadal hindcasts. Clim. Dyn. 46, 3127–3147 (2016).

  78. 78.

    Ossó, A., Sutton, R., Shaffrey, L. & Dong, B. Observational evidence of European summer weather patterns predictable from spring. Proc. Natl. Acad. Sci. 201713146 (2017).

  79. 79.

    Phillips, N. D., Neth, H., Woike, J. K. & Gaissmaier, W. FFTrees: A toolbox to create, visualize, and evaluate fast-and-frugal decision trees. Judgm. Decis. Mak. 12, 344–368 (2017).

  80. 80.

    Sacks, W. J., Deryng, D. & Foley, J. A. Crop planting dates: an analysis of global patterns. Glob. Ecol. Biogeogr. 19, 607–620 (2010).

  81. 81.

    NOAA. The Southern Oscillation Index. at, (2005).

  82. 82.

    Barnston, A. G. & Livezey, R. E. Classification, Seasonality and Persistence of Low-Frequency Atmospheric Circulation Patterns. Mon. Weather Rev. 115, 1083–1126 (1987).

  83. 83.

    EUROSTAT. Agricultural production - crops. at, (2016).

  84. 84.

    Gigerenzer, G., Czerlinski, J. & Martignon, L. How good are fast and frugal heuristics? Decis. Sci. Technol. 81–103 (1999).

  85. 85.

    Raab, M. & Gigerenzer, G. The power of simplicity: A fast-and-frugal heuristics approach to performance science. Front. Psychol. (2015).

  86. 86.

    Nobre, G. G. et al. Financing agricultural drought risk through ex-ante cash transfers. Sci. Total Environ (2018).

  87. 87.

    Gigerenzer, G. & Todd, P. M. In Simple heuristics that make us smart 3–34 (Oxford University Press (1999).

  88. 88.

    Metz, C. E. Basic principles of ROC analysis. Semin. Nucl. Med. 8, 283–298 (1978).

  89. 89.

    Zweig, H., Mark & Campbell, G. Receiver-Operating Clinical Medicine (ROC) Plots: A Fundamental Evaluation Tool 39, 561–577 (1993).

  90. 90.

    Hamill, T. M. & Juras, J. Measuring forecast skill: is it real skill or is it the varying climatology? Q. J. R. Meteorol. Soc. 132, 2905–2923 (2006).

  91. 91.

    Livezey, R. E. & Chen, W. Y. Statistical Field Significance and its Determination by Monte Carlo Techniques. Monthly Weather Review 111, 46–59 (1983).

Download references


The research leading to this article is funded by the Horizon 2020 Framework programme through the project IMPREX (grant agreement no. 641811). J.C.J.H.A. and P.J.W. received additional support from the Netherlands Organisation for Scientific Research (NWO) in the form of VICI grant 453.140.006 and VIDI grant 016.161.324 respectively. We thank the Joint Research Centre for supplying data on sugar beet production from the EUROSTAT database.

Author information


  1. Institute for Environmental Studies (IVM), Vrije Universiteit Amsterdam, De Boelelaan 1087, 1081 HV, Amsterdam, The Netherlands

    • Gabriela Guimarães Nobre
    • , Jeroen C. J. H. Aerts
    •  & Philip J. Ward
  2. FutureWater, Cartagena, Spain

    • Johannes E. Hunink
  3. Directorate Sustainable Resources, European Commission, Joint Research Centre, Ispra, Italy

    • Bettina Baruth


  1. Search for Gabriela Guimarães Nobre in:

  2. Search for Johannes E. Hunink in:

  3. Search for Bettina Baruth in:

  4. Search for Jeroen C. J. H. Aerts in:

  5. Search for Philip J. Ward in:


G.G.N., J.E.H. and P.J.W. contributed to the design of the study. G.G.N. and B.B. collected data. G.G.N. performed statistical analysis. P.J.W. and J.C.J.H.A. provided suggestions and supervised statistical analysis. G.G.N. drafted the manuscript. All authors critically revised the manuscript and approved the final version.

Competing Interests

The authors declare no competing interests.

Corresponding author

Correspondence to Gabriela Guimarães Nobre.

Supplementary information

About this article

Publication history






By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.