Human mobility continues to increase in terms of volumes and reach, producing growing global connectivity. This connectivity hampers efforts to eliminate infectious diseases such as malaria through reintroductions of pathogens, and thus accounting for it becomes important in designing global, continental, regional, and national strategies. Recent works have shown that census-derived migration data provides a good proxy for internal connectivity, in terms of relative strengths of movement between administrative units, across temporal scales. To support global malaria eradication strategy efforts, here we describe the construction of an open access archive of estimated internal migration flows in endemic countries built through pooling of census microdata. These connectivity datasets, described here along with the approaches and methods used to create and validate them, are available both through the WorldPop website and the WorldPop Dataverse Repository.
Machine-accessible metadata file describing the reported data (ISA-tab format)
Background & Summary
According to the International Organization for Migration1 and The World Bank2, without accounting for seasonal and temporary migrants, more than 1 billion people are currently living outside their places of origin, with about 740 million of them classified as internal migrants. Additionally, in 2014 around 67 million passengers travelled on international and domestic flights every week3 and hundreds of millions are estimated to commute daily by public transport and private vehicles4. Human mobility is expected to continue rising in volume and reach, producing increasing global connectivity that has a range of impacts, including rising numbers of invasive species, the spread of drug resistance, and disease pandemics. In this context, quantifying human mobility across multiple temporal and spatial scales, becomes crucial for quantifying its effects on society5,
In public health, the role of human mobility in the spread of infectious diseases is exemplified by the presence of HIV/AIDS in areas outside where it first emerged at the beginning of the twentieth century20,
Local malaria elimination and global malaria eradication are rising up the international agenda34,
As detailed in Tatem7, sources of human mobility data potentially useful for modelling pathogen movements include: air and sea travel data records (including open access modelled versions of them); census migration data; travel history and displacement surveys; GPS tracking data and volunteered geographic information (with the latter including geolocated social media data), and even satellite night-time light data. In particular, patient travel history data, containing detailed demographic information and travel motivations, are traditionally used to understand malaria parasite importation patterns48,
However, because of difficulties in sharing and accessing CDRs (mostly due to commercial and privacy concerns)7,56,57, alternative datasets are required in order to quantify and map internal connectivity across continental scales. To this end, using CDRs, Wesolowski et al.56 and Ruktanonchai et al.58 demonstrated that widely-available and easy-to-obtain census-derived internal migration flow data can serve as reliable proxies for the relative strength of within-country human connectivity across multiple temporal scales.
Within the framework of the WorldPop Project (www.worldpop.org), and following the approaches described in Henry et al.59 and Garcia et al.60 (Fig. 1), internal census-based migration microdata available through the online IPUMS-International (IPUMSI) database61, along with a number of other ancillary datasets, were assembled and processed to produce an open access archive of estimated 5-year (2005–2010) internal human migration flows for every Plasmodium falciparum and Plasmodium vivax (hereafter simply referred as Pf and Pv, respectively) endemic country62,63 (Supplementary Table 1).
Estimating internal migration flows between administrative units
Following Garcia et al.60 a gravity model-based approach was used to estimate the total number of people migrating from one administrative unit to any other administrative unit, between 2005 and 2010, within each malaria endemic country located in Africa, Asia, Latin America and the Caribbean62,63 (Supplementary Table 1).
The simplest gravity-type spatial interaction model, proposed by Zipf64, considers the total population in a location of origin i and in a location of destination j (henceforth simply indicated as i and j), and the distance between the two locations to predict the migration flow (MIGij) between them. Thus, migration flows between administrative units can be estimated using the following function: where and represent the populations in the location of origin i and of destination j, respectively, and represents the distance between i and j; with α, β, and γ being parameters, used to indicate the magnitude of the effect for each covariate, that are typically estimated in the statistical modelling framework.
In this study, following the notation from Henry et al.59 and Garcia et al.60, the basic gravity-type spatial interaction in equation (1) was extended in order to include additional geographical and socioeconomic factors described in detail in the Data collection and preparation subsection below. Since the census-based migration microdata extracted from the IPUMSI database61 represent only a sample of the total census, a logistic regression was used to model the proportion of people migrating between administrative units65. In particular, the logistic regression was used to model the proportion of people residing in j in the census year who were in i ‘n’ years prior to the census. Thus, the proportion of migrants in j in the census year that were previously residing in i was estimated using the following logistic regression function: where ; with MIGij and TOTj representing the number of people residing in j in the census year that were in i ‘n’ years prior to the census and the total population residing in j in the census year, respectively.
Initially, a separate vector β=(β0, β1, …, βn) of coefficients was used in the linear predictor of the gravity model for each country (including malaria non-endemic countries), in Africa, Asia, Latin America and the Caribbean, for which migration data were available in the IPUMSI database61 (hereafter referred as IPUMSI countries; Table 1).
However, since the main aim of this study was to estimate internal human migration flows for malaria endemic countries for which migration data are not available, ultimately, models where the linear predictors were common across all countries located in the same continent were constructed (under the assumption of homogeneity of the process along the space). To investigate possible nonlinear relationships, models where linear predictors were replaced by additive predictors, using a Generalized Additive Modelling (GAM) framework66, were also explored.
GAM is a type of regression that, while preserving the functionality of using linear terms, allows covariates to have different and possibly opposite effects on the response variable by incorporating regression coefficients with smooth nonlinear form (Fig. 2).
Thus, all possible combinations of covariates (listed in Table 2 and Supplementary Table 2) were tested in a logistic regression model and then only the linear predictors of all continuous covariates of the best predictive logistic regression model were also modelled using a GAM.
For each continent, the overall combinations of covariates and model types were explored using a multi-step approach to identify the model with the greatest predictive power in countries for which migration data were not available. The best model was then selected using a leave-one-out cross-validation approach67 in which the observed proportion of migrants in j previously residing in i for all countries except one were used for fitting models, that were subsequently used to predict the proportion of migrants in j previously residing in i in the withheld country. The correlation coefficient (R2) was selected to measure the variance explained after verifying homoscedasticity and testing overdispersion using a chi-squared test. This process was then repeated through iteratively withholding one country at the time. For each model, the R2 values for all withheld countries were averaged and used to rank each models according to their predictive power averaged across all withheld countries (Fig. 3). The overall best predictive model for each continent (Supplementary Table 3) was then used to predict the proportion of migrants residing in j who were previously residing in i for every malaria endemic country located in the corresponding continent (refer to Supplementary Table 4a,b and c for summary statistics of each best predictive model for Africa, Asia, and Latin America and the Caribbean, respectively).
Finally, in order to estimate the total number of people that migrated from i to j between 2005 and 2010 (Figs 4,5,6), for each country the predicted proportion of migrants residing in j was multiplied by the 2010 total population in j; with the latter calculated using either the corresponding WorldPop68,
Data collection and preparation
In most of the countries available through the online IPUMSI database, internal migration variables were recorded by asking respondents either their administrative unit of residence 15, 5, or 1 prior to the census, or their previous residence and the number of years they are residing in the current locality. Considering that 5-year was the temporal interval available for most of the countries in the IPUMSI database and the fact that it has been demonstrated that both 1- and 5-year census-based internal migration data generally align well with shorter-term population movements in terms of relative strength of connections56,58, the 5-year migration data were used in this study. This maximised the amount of data that could be used to fit the gravity models subsequently used for predicting internal migration flows for every malaria endemic country. Thus, for each country listed in Table 1, harmonized, census-based 5-year internal migration data were extracted from the most recent census microdata available through the IPUMSI database61, downloaded locally, and eventually uploaded into a PostgreSQL database using a Microsoft Visual Studio 2010 user interface. The IPUMSI data stored in the PostgreSQL database were subsequently queried, using SQL, to quantify the number of people that migrated from each subnational administrative unit i to every other subnational administrative unit j during the 5-year timespan. These numbers were then matched to the corresponding country administrative unit spatial dataset, extracted from either the Global Administrative Areas (GADM)75 or the Global Administrative Unit Layers (GAUL)76 database, in a GIS environment. This was done by manually adding a unique ‘ID’ to each spatial unit corresponding to the one in the PostgreSQL database (hereafter referred as ‘IPUMSID’). In some cases, depending on the country, either the spatial detail of the IPUMSI migration data had to be reduced to match the lower spatial detail of the corresponding administrative unit dataset or spatially contiguous units in the administrative unit dataset had to be merged together to match the lower spatial detail of the IPUMSI migration data. In some other cases, ‘IPUMSIDs’ had to be edited or spatially contiguous units in the administrative unit dataset had to be merged together to match the reorganisation of the administrative units during the 5 years prior to the census. Finally, before calculating the migration flows between administrative units, another SQL query was used to classify each person in the census sample as either an internal migrant (1) or not (0). Examples of SQL queries used to perform the tasks described above are included in the WorldPop-InternalMigration-v1 code74 briefly described in the Code availability subsection below.
Response variable and covariates
For each country, the response variable, or the proportion of migrants residing in j in the census year that were residing in i 5 years prior to the census, was obtained by dividing the number of migrants residing in j in the census year that were residing in i 5 years prior the census by the total population residing in j in the census year; with both numbers based only on the information contained in IPUMSI census samples.
The administrative units spatially matching the IPUMSI migration microdata were used to calculate the distance between each pair of administrative units, their area, total population, and proportion of urban population. These main covariates (Table 2), along with other covariates derived from them (Supplementary Table 2), represent the pull and push migration factors, known to influence internal migration59,60,77, that were used to extend the basic gravity model proposed by Zipf64.
Other factors, including environmental factors59,60, and country-specific factors, such as literacy and percentage of male population59 or infrastructure and transportation78, were not used because (i) the factors listed in the previous paragraph alone proved to be able to explain most of the variance in the gravity models of Garcia et al.59, and (ii) only globally available datasets were explored in order to consistently model internal migration across all countries.
Calculating response variable and covariates
For each country, the total population in each administrative unit was calculated using the corresponding WorldPop79 (Data Citation 1: Harvard Dataverse http://dx.doi.org/10.7910/DVN/PUGPVR) or GPWv4 (ref. 80) population count raster dataset adjusted to match UNPD estimates for 201072. The GPWv4 datasets were resampled to the spatial resolution of the WorlpPop datasets and used only for countries for which the WorldPop datasets were not available (Supplementary Table 1).
The area of each unit was calculated using each country vector administrative unit dataset projected to the most appropriate country-specific projected coordinate system, in order to minimize areal distortion, and ultimately reprojected to GCS WGS84.
The proportion of people in urbanized areas in each unit was calculated using the MODIS 500 m Global Urban Extent raster dataset81,82. The latter was converted to vector polygons, using the ArcGIS ‘Raster to Polygon’ tool83, and intersected with the reprojected country vector administrative unit dataset using the ArcGIS ‘Intersect’ tool83. Then, both the intersect output (containing polygons representing the total urban area within each unit uniquely identified by its ‘IPUMSID’) and the country vector administrative unit dataset were rasterized, at the resolution of the corresponding raster population dataset (i.e., 3 arc seconds 3 arc equals to approximately 100 m at the equator), and co-registered with it.
The two raster outputs, along with the population count raster dataset, were then input to the ArcGIS ‘Zonal Statistics as Table’ tool83 to generate two tables containing the total population and urban population in each unit (with the rasterized administrative units and thus their ‘IPUMSIDs’ used to define the zones). Subsequently, both tables were joined to the attribute table of the vector administrative unit dataset, using the ‘IPUMSID’ field to perform the join operation, and the proportion of urban population in each unit was calculated simply dividing its urban population by its total population.
The geodesic distance between each pair of administrative units, with the latter represented by their centroids, was calculated using the ArcGIS ‘Generate Near Table (Analysis)’ tool83. The ‘IN_FID’ and ‘NEAR_FID’ fields (identifying the administrative unit of origin and destination, respectively) in the output ‘distance’ table were then used for joining twice the ‘centroid attribute’ table using the centroid ‘ID’ field to perform the join operation. Since the ‘centroid attribute’ table contains the attributes of each administrative unit represented by the corresponding centroid, the join operation allowed to generate a ‘distance’ table containing all pairs of origin and destination administrative units along with their ‘IPUMSIDs’ and attributes including the unit’s area, total population, and proportion of urban population. Origin and destination ‘IPUMSID’ fields were then renamed ‘NODEI’ and ‘NODEJ’, respectively.
A ‘contiguity’ table containing information about spatial contiguity of administrative units (defined based on polygons sharing an edge) was generated using the ArcGIS ‘Generate Spatial Weights Matrix’ tool83 and subsequently joined with the ‘distance’ table to obtain a new table containing all main covariates, listed in Table 2, calculated at the unit level. This join operation (based on both the ‘NODEI’ and ‘NODEJ’ field the in the ‘distance’ table and the corresponding ‘IPUMSID’ and ‘NID’ field in the ‘contiguity’ table) was performed through two different R scripts depending on whether the country is an IPUMSI or a non-IPUMSI countries. In particular, the R script for the IPUMSI countries added to the new table a ‘MIGIJ’ field containing the number of people that migrated from each ‘NODEI’ to each other ‘NODEJ’ according to the IPUMSI migration microdata and calculated the response variable.
Finally, on a continent basis, all IPUMSI country tables were merged together and input to an R73 script that generated the additional covariates listed in Supplementary Table 2, identified the best predictive model for each continent, as described in the previous section, and was used to estimate the 5-year (2005–2010) internal human migration flows for every malaria endemic country using the best predictive model selected for the corresponding continent.
All operations described above, excluding the reprojection of the vector administrative unit datasets and the calculation of their surface areas, for all IPUMSI and non-IPUMSI countries, were performed using the WorldPop-InternalMigration-v1 code74 briefly described in the Code availability subsection below.
The WorldPop-InternalMigration-v1 code74, used to produce the open access archive of estimated 5-year (2005–2010) internal human migration flows described in this article, is publicly available through Figshare. It consists of 1) a Microsoft Visual Studio 2010 user interface allowing users to upload the IPUMSI census microdata to a PostgreSQL database; 2) example SQL queries that were used to match the spatial detail of the IPUMSI migration data to spatial detail of the corresponding administrative unit dataset and to identify internal migrants within the IPUMSI census samples 3) an ArcToolbox geoprocessing tool82 that assigns a unique ID to each administrative unit and calculates the corresponding total population and proportion of urban population; 4) a Python84/ArcPy83 script that creates two tables, one containing spatial contiguity information between each pair of administrative units (‘contiguity.csv’) and another one containing the ISO country code, the continent in which the country is located, the distance between each pair of administrative units, their total population, proportion of urban population, surface area, and the geographic coordinates (GCS WGS84) of their centroid (‘distance.csv’); 5) two R73 scripts, one for the IPUMSI countries used to query the IPUMSI migration microdata loaded in the PostgreSQL database, calculate the response variable, and join the query result with the two output tables of the python script, and another one for the non-IPUMSI countries used just to join together the two output tables of the python script; and 6) an R73 script that performs the model selection and estimates the 5-year (2005–2010) internal human migration flows between subnational administrative units.
All available sets of code are named progressively and must be run sequentially according to the order in which they are presented above. They are also internally documented in order to both briefly explain their purpose and, when required, guide the user through their customization.
All datasets described in this article, referring to all Pf and Pv endemic countries listed in Supplementary Table 1, are publicly and freely available both through the WorldPop Dataverse Repository (Data Citation 2: Harvard Dataverse http://dx.doi.org/10.7910/DVN/RUWQQK) and the WorldPop website (http://www.worldpop.org.uk/data/data_sources/). However, it is important to note that while the datasets stored in the Dataverse Repository represent the datasets produced at the time of writing, and will be preserved in their published form, the datasets stored on the WorldPop website may be updated as more recent IPUMSI migration data for the countries listed in Table 1, become available. Similarly, the datasets stored on the WorldPop website may be updated as IPUMSI census-based migration microdata become available for additional malaria endemic and non-endemic countries located in Africa, Asia, Latin America and the Caribbean. Indeed, the availability of migration data for additional countries may enable further improvements of the predictive power of the gravity models used to estimate the internal migration flows. For each county, the corresponding internal migration dataset, along with a point dataset showing the nodes of the migration network, (Table 3) can be obtained by downloading the corresponding zipped archive associated with the continent in which the country of interest is located.
Goodness of fit and error p-value
All countries available in the IPUMSI database were used to assess the accuracy of the predicted proportion of migrants in j in the census year that were previously residing in i 5 years prior to the census by comparing them with the corresponding observed values from the IPUMSI migration microdata. For each country, the goodness of fit (R2) between predicted and observed values and the corresponding error P-value, representing the average probability that predicted migration values lay outside the distribution of the observed values, are reported in Table 4 below. Both metrics were derived using (i) the observed IPUMSI migration flows from each i to any other j and (ii) the predicted IPUMSI-based migration flows calculated by multiplying the predicted proportion of migrants residing in j in the census year by the IPUMSI-based total number of people residing in j in the census year.
The estimated internal human migration flows between subnational administrative units can be used to support a range of applications from planning interventions, to measuring progress, designing strategies, and predicting response variables that are intrinsically dependent on migration flows and internal connectivity.
Ongoing work involves the integration of these datasets with malaria prevalence raster datasets85,
There are a number of limitations, caveats, and assumptions inherent in the approach that should be considered when using the datasets outlined here. For consistency, internal migration flows were estimated using a fixed set of pull and push factors common to all countries and thus only a limited number of covariates were used to fit the gravity-type spatial interaction models and to create predictions. For this reason, as is a trade-off in the production of generalizable models, the model fit varied between countries and for some of them, such as Malawi, China, Cambodia, India, and Venezuela (Table 4), poor fits could be improved by considering additional, locally-specific migration drivers that could help to increase the percentage of variance explained60,78. Other limitations are the fact that migration models were fitted using only a small sample (ranging between 0.07 and 10%) of the full census for each country, and that in each sample a small number of households were swapped across administrative units. Moreover, the spatial detail at which migration is captured and summarized varies by country. Because of this, for some countries, the modelled role of some of the pull and push factors, may not have been captured at the spatial level at which they influence migration as recorded in the census. It is also important to consider that the underlying migration data are based only on permanent movements captured by the census and other types of migrations, such as seasonal movements and forced displacements, may be not captured by the model88,
The two main assumptions behind the approach presented here are that for each country (i) the census samples are considered to be representative at the administrative unit level at which migration was recorded and (ii) the percentage of people migrating between administrative units is considered to be constant over time. Regarding the second assumption, it is important to highlight that the use of census data from many years ago for some countries may have generated inaccurate estimates for the period considered in this study (i.e., 2005–2010), for example because of major changes in the countries’ socio-economic conditions from the time period covered by the census (e.g., the rapid economic development and urbanization that has occurred in China during the last two decades91,92). Similarly, in some other countries, either the presence of conflicts93 or the occurrence of natural disasters88,89 during the specific time period covered by the census may have produced fluctuations in the number of internal migrants and consequently biased results for the period considered in this study.
Finally, the estimated internal flows represent modelling outputs generated using ancillary covariate datasets, and thus, to avoid circularity they should not be used to make predictions or explore relationships with any of these ancillary datasets. It is also important to note that these ancillary datasets are modelling outputs in themselves and thus they have a degree of uncertainty that will carry over into the migration estimates.
How to cite this article: Sorichetta, A. et al. Mapping internal connectivity through human migration in malaria endemic countries. Sci. Data 3:160066 doi: 10.1038/sdata.2016.66 (2016).
Sorichetta, A Harvard Dataverse http://dx.doi.org/10.7910/DVN/RUWQQK (2016)
The authors wishes to acknowledge the statistical offices that provided the underlying data making this research possible: Central Bureau of Census and Population Studies, Cameroon; Ghana Statistical Services; National Statistics Directorate, Guinea; National Statistical Office, Malawi; National Directorate of Statistics and Informatics, Mali; National Agency of Statistics and Demography, Senegal; Statistics South Africa; Bureau of Statistics, Uganda; Central Statistics Office, Zambia; Central Agency for Public Mobilization and Statistics, Egypt; Department of Statistics, Morocco; National Statistical Service, Armenia; National Statistical Committee, Kyrgyzstan; Ministry of Statistics and Programme Implementation, India; BPS Statistics Indonesia; National Statistical Office, Thailand; National Institute of Statistics, Cambodia; National Bureau of Statistics, China; Department of Statistics, Malaysia; National Statistics Office, Philippines; General Statistics Office, Vietnam; National Statistical Office, Mongolia; Bureau of Statistics, Fiji; National Institute of Statistics and Censuses, Argentina; National Institute of Statistics, Bolivia; Institute of Geography and Statistics, Brazil; National Administrative Department of Statistics, Colombia; National Institute of Statistics and Censuses, Costa Rica; National Statistics Office, Dominican Republic; National Institute of Statistics and Censuses, Ecuador; Department of Statistics and Censuses, El Salvador; Institute of Statistics and Informatics, Haiti; National Institute of Statistics, Geography, and Informatics, Mexico; National Institute of Information Development, Nicaragua; National Institute of Statistics and Informatics, Peru; National Institute of Statistics, Venezuela; Office of National Statistics, Cuba; Statistical Institute, Jamaica; National Institute of Statistics, Uruguay. A.S. is supported by funding from the Bill & Melinda Gates Foundation (OPP1106427, 1032350). A.J.T. is supported by funding from NIH/NIAID (U19AI089674), the Bill & Melinda Gates Foundation (OPP1106427, 1032350, 1134076), the Clinton Health Access Initiative, National Institutes of Health, and a Wellcome Trust Sustaining Health Grant (106866/Z/15/Z). This work forms part of the outputs of WorldPop (www.worldpop.org) and the Flowminder Foundation (www.flowminder.org). The funders had no role in study design, data collection and analysis, decision to publish, and preparation of the manuscript.
About this article
Scientific Data (2017)