Completeness of open access FluNet influenza surveillance data for Pan-America in 2005–2019

For several decades, the World Health Organization has collected, maintained, and distributed invaluable country-specific disease surveillance data that allow experts to develop new analytical tools for disease tracking and forecasting. To capture the extent of available data within these sources, we proposed a completeness metric based on the effective time series length. Using FluNet records for 29 Pan-American countries from 2005 to 2019, we explored whether completeness was associated with health expenditure indicators adjusting for surveillance system heterogeneity. We observed steady improvements in completeness by 4.2–6.3% annually, especially after the A(H1N1)-2009 pandemic, when 24 countries reached > 95% completeness. Doubling in decadal health expenditure per capita was associated with ~ 7% increase in overall completeness. The proposed metric could navigate experts in assessing open access data quality and quantity for conducting credible statistical analyses, estimating disease trends, and developing outbreak forecasting systems.


Data and methods
FluNet weekly records. We abstracted FluNet weekly records on 24-27 April 2020 for 29 Pan American countries from Week 1 (03 January) 2005 through Week 52 (29 December) 2019. Due to the absence of a public application programming interface (API) and the challenges of the website's dynamic AJAX interface 24 , we acquired public data with a custom scraper built using RSelenium 25 . We downloaded each country's records individually and used a scripted pipeline to standardize and merge country-specific datasets. Codes are available in the Supplementary Materials Appendix.
FluNet reports time series data of influenza confirmed cases representing a complex array of data streams. For each country download, we extracted time series data for 6 available testing outcomes: specimens collected, specimens processed (tests), total positives, total negatives, influenza A positives, and influenza B positives. We also extracted time series data for 8 influenza subtypes: A(H1), A(H1N1)pdm09, A(H3), A(H5), A(Unsubtyped), B(Yamataga), B(Victoria), and B(Undetermined). The overall data compilation included 14 variables for 29 countries covering 782 weeks.
Surveillance systems attributes. We compiled FluNet case definitions, surveillance strategies, reporting quota and timeliness, NICs, and reporting facilities using several WHO reports 6,[26][27][28] to assess surveillance attributes associated with completeness (Table 1). Influenza case definitions are not fully standardized: they have subtle but important differences in their evaluation setting and diagnostic criteria. Case definitions for each country include severe acute respiratory illness (SARI), influenza-like-illness (ILI), pneumonia, influenza cases (Influenza), acute respiratory infection (ARI), and deaths (Mortality) (Supplementary Table S1). Mortality was defined as deaths from influenza unless otherwise specified. Six FluNet countries (Chile, Cuba, Dominica, Dominican Republic, El Salvador, and Honduras) have fully adopted the WHO case definitions. We marked countries utilizing non-WHO definitions.
Surveillance strategies include national, sentinel, and universal, which offer different population coverage for each influenza case definition (Supplementary Table S2). NICs are nationally recognized institutions approved by the WHO and responsible for reporting influenza surveillance records to FluNet. Influenza reporting facilities include SARI hospitals, ILI centers, PCR testing facilities, and influenza (IF) testing laboratories, and their number vary by country. Each facility processes tests at different volumes and speeds, resulting in differing reporting quotas and timeframes. Reporting quotas describe the fraction of cases that are reported to FluNet from incountry surveillance systems. They include all cases or a specific number of cases based on each country's health objectives. Reporting timeframe is the difference between when disease cultures are laboratory confirmed and when they are reported by surveillance facilities to a national or global database. Though FluNet publishes weekly records, reporting timeframes range from daily to monthly across countries. We found no resources that compile these surveillance system characteristics to allow for clear side-by-side comparison across continental countries.
Economic and health expenditure indicators. We  www.nature.com/scientificreports/ Gross National Income (GNI) per capita (GNIPC), domestic general government health expenditure per capita (DHEPC), and out-of-pocket health expenditure as a percentage of current health expenditure (OOPHE%) 29 . GNIPC was reported in purchasing power parity (PPP) constant 2011 international US dollars (USD). DHEPC was reported in current international USD. OOPHE% is reported as the percentage of current government expenditure (Supplementary Table S3 Completeness metric. We measured completeness based on an effective time series length (ETSL), or the extent of time series data that can be used in data analysis. Time series length greatly influenced how and what the completeness metric described. We used an annual ETSL to calculate completeness from Week 1 to Week 52/53 to compare completeness between outcomes and across countries. We used these values to calculate the overall completeness, or the mean across all years from 2005 to 2019. We also calculated the average completeness using the full time series ETSL for the entire 782-week study period. Finally, we calculated the corrected average values and corrected overall completeness by including only years when selected outcomes or subtypes were reported. This prevented the deflating of the completeness metric by including all study years irrespective of whether a country's surveillance records were present or not. We calculated the annual completeness, C i,j,k , as a fraction of the time series length for which reliable data are available to the overall length of the considered time series, or the number of full weeks between the start and end of the time period, multiplied by 100: where C i,j,k is completeness for i-outcome (i = 1-14), j-country (j = 1-29), k-year (k = 1-15); n i,j,k -the number of time units (weeks) in the time series when records are available (e.g. weeks with reported counts ≥ 0) for i-outcome, j-country, k-year; L 1 -the number of full weeks (52 or 53) for k-year (Table 2). We calculated the annual completeness by using the total length in weeks for each year.
We calculated the completeness for each outcome and country for the full time series using the total 782-week length covering the study period from Week 1 (03 January) 2005 to Week 52 (29 December) 2019, as: where C i,j is completeness for i-outcome (i = 1-14), j-country (j = 1-29); n i,j -the number of time units (weeks) in the time series when records are available (e.g. weeks with reported counts ≥ 0) for i-outcome, j-country; L i , 2 -the C i,j,k = n i,j,k L 1 * 100%  www.nature.com/scientificreports/ number of full weeks (782) or the full study period. To draw comparisons across countries, we calculated the overall completeness as the average completeness across all 14 outcomes for each country. To more accurately reflect completeness for specific influenza outcomes, we corrected the average estimates to only include the time period when reporting is meaningful. For example, Jamaica, Paraguay, and Mexico were the first Pan American countries to report the new influenza subtype A(H1N1)pdm09 in 2008. All 29 countries have continued reporting this subtype as of 2019. Thus, we estimated the average estimates for A(H1N1)pdm09 to account for the start of pandemic strain reporting in 2008 for all countries. Specimens were also first reported in 2008 by Paraguay with Canada and the United States continuing reporting through 2019; completeness for specimens was similarly estimated for a 626-week time period from Week 1 2008 to Week 52 2019. For A(H5), we estimated average completeness from Week 1 2008 to Week 52 2016 (470 weeks). For B(Yamataga) and B(Victoria), we estimated average completeness for a 678-week time period from Week 1 2007 to Week 52 2019. All analyses of average estimates were performed using values of L 3 as shown in Table 2.
Completeness analysis. We examined trends in annual completeness for all 14 influenza variables and produced heatmaps illustrating the country ranking with respect to completeness. To further examine trends and associations with national economic indicators (GNIPC, DHEPC, and OOPHE%), we selected annual completeness estimates for tests, positives, A(H1N1)pdm09, and overall. We applied loess smoothers with a span of 0.5 to illustrate trends across all years and countries. We transformed the GNIPC and DHEPC values using the natural logarithm function to minimize the effect of skewed distributions in regression models. For each influenza variable, we estimated the change in annual completeness associated with time and national economic indicators using a mixed effects regression model (Model 1): where C i,j,k is completeness for i-outcome (i = 1-4), j-country (j = 1-29), k-year; E j,k -one of three national indicators for j-country and k-year; β 1 -fixed effects for the annual trend, α j -random effects for individual countries. The length of the time series used in each regression varied according to the length of available records for the economic or health expenditure indicator in each country.
We expanded the model to adjust for surveillance systems' attributes (Model 2): where S j,m -matrix of the national surveillance system attributes as defined in Table 1. Attributes included in the analysis were: case definition type, including ARI, ILI, Influenza, Pneumonia, Mortality, and SARI as the reference category; adherence to WHO definition as a binary variable; surveillance strategy, including Sentinel, National, and Universal as the reference category; reporting quota, including categories for reported quota, varying or unknown quota, and reporting all cases as the reference category; reporting timeframe, as weekly, not reported (NR), and daily reporting as the reference; number of NICs, and the natural log of reporting facilities number (we applied the transformation given the skewed distribution of reporting facilities as fewer countries have many facilities).
To estimate the effect size (ES) from regression model results, we calculated the decadal change between 2005 and 2015 for each economic and health expenditure indicator. For Venezuela, we estimated decadal changes in GNIPC from 2005 through 2014 due to limited data availability. Using the coefficients from Models 1 and 2, we estimated ES and its 95% confidence intervals (CI) associated with time and health expenditure indicators in the completeness of tests, positives, A(H1N1)pdm09, and overall completeness. For GNIPC and DHEPC, the effect size was associated with a doubling in these predictors; the 95% confidence interval was estimated as: ES = ln(2) * (β 2 + 1.96se(β 2 )) . For OOPHE%, ES was associated with a 10% increase in expenditures; 95%CI was estimated as: ES = 10 * (β 2 + 1.96se(β 2 )).
We used Akaike Information Criterion (AIC) to assess model performance. Data management, statistical analyses, and maps were conducted using Stata/SE 15.1 and R versions 1.1.419 and 4.0.0.

Results
Annual completeness, trends, and spot-checking. The annual completeness values for each influenza outcome in each country and year of the 15-year study period are compiled in Supplementary Table S4. Figures 1 and 2 show the completeness values for influenza outcomes along with the trend line for average completeness across all 29 countries presented as a heatmap and a line-plot, respectively. In each country, completeness for tests, positives, influenza A positives, and influenza B positives are almost identical (completeness for positives shown in Fig. 1a). In 2005, these 4 outcomes are nearly 100% complete for nine countries (Argentina, Brazil, Chile, Colombia, Dominican Republic, Mexico, Paraguay, Peru, and the United States). Since the 2009 pandemic, 19 countries reached > 95% completeness with further improvements by 2019 when 24 countries reached > 95% completeness. This annual progression in influenza outcome surveillance illustrates the maturity of FluNet over time.
Jamaica, Paraguay, and Mexico were the first countries with available records starting in 2008 for both influenza A(H1N1)pdm09 and A(H5) (Fig. 1b,c). For A(H1N1)pdm09, completeness across the region reached ~ 70% by 2010 and slowly grew to ~ 80% by 2019. Some countries, including Uruguay, Ecuador and Nicaragua, show a reduction in completeness for A(H1N1)pdm09 after 2014. Some countries such as Suriname, Haiti, and Barbados did not offer surveillance data for A(H1N1)pdm09 until 2015. For influenza A(H5), completeness across countries almost reached 50% in 2010, then declined to 0% by 2016 and has not been reported by any country www.nature.com/scientificreports/ www.nature.com/scientificreports/ since. A similar trajectory was seen for specimens though this testing outcome continues to be reported by the United States and Canada as of 2019 (Fig. 1d). The completeness of influenza A(H1) for the region grew from 43 to 70% between 2009 and 2011 and decreased gradually from 75% in 2014 to 56% as of 2019 (Fig. 2a). The completeness of influenza A(Unsubtyped) increased briefly in 2008 just prior to the A(H1N1)pdm09 pandemic. In 2009, the average completeness across countries increased from 46 to 75% completeness by 2011; it began to decline in 2014, surging back to ~ 70% completeness in 2017 but gradually declining again to 68% completeness as of 2019 (Fig. 2b).
Influenza B(Yamataga) and B(Victoria) subtypes have showed increased trends in completeness since 2012 (Fig. 2c,d). Both subtypes increased in completeness from 2007 to 2010 however the average completeness across countries did not exceed 50% for either subtype. After a brief decline from 2010 to 2011, the average completeness grew for both subtypes from 32% in 2012 to 75% in 2014. While declining from 2014 to 2016, completeness for these influenza B subtypes continuously increased thereafter and reached an average of ~ 67% across the region as of 2019.
In addition to temporal trends, we detected anomalies in annual completeness for influenza subtypes. For example, Fig. 1a showed that the United States, despite having 100% completeness in all other years, had only 63.5% completeness in 2006. During this year, we found that records for total positives are missing from Week 21 (22 May) to Week 49 (10 December). Yet, the United States Centers for Disease Control and Prevention (CDC) reported influenza records during these missing weeks 30 . Though cases are near zero, the national surveillance system does collect data on influenza positives that were not reported in FluNet.
Missing reports for influenza positives in 2018 from Week 30 (23 July) to Week 38 (23 September) in Peru provided another example of using annual completeness for spot-checking data quality (Fig. 1a). Reports from the Pan American Health Organization (PAHO) suggest that Peru reported a surge in influenza A(H1N1)pdm09 and SARI activity during these weeks [31][32][33] . Despite increased case counts, influenza activity dipped below the alert threshold with pneumonia cases increasing for infants < 5 years of age. Like in the United States, PAHO reported case information on influenza A positives, influenza B positives, and subtypes including A(H1N1)pdm09 and A(Unsubtyped) during these weeks 33 though no data is reported within FluNet.
We also used annual completeness to identify patterns across variables and study years for specific countries.  Table S5). Though surveillance data was reported for 2005-2007, completeness dropped to 0% for all influenza variables for 2008-2010 during the known rise of the A(H1N1) pdm09 pandemic. While 100% completeness was achieved for nearly all influenza outcomes and subtypes in 2011, completeness again dropped to 0% in 2012 and 2013. Since 2014, however, Venezuela maintained > 75% completeness for 5 of 6 influenza outcomes and 7 of 8 influenza subtypes. This fluctuation in annual completeness suggested that attributes related to the surveillance system or factors influencing surveillance performance such as economic stability and health expenditure could be influencing completeness.

Overall completeness ranking and economic and health expenditure indicators. The overall
and average completeness values for six influenza outcomes for each country are compiled in Table 3 (all outcomes are reported in Supplementary Table S6). A map of average overall completeness is shown in Fig. 3. The overall completeness was > 80% for 4 countries (Argentina, Brazil, Mexico, United States) and > 70% for an additional 6 countries (Chile, Peru, El Salvador, Paraguay, Honduras, Panama). Completeness for tests, positives, influenza A positives, and influenza B positives were nearly identical for all countries except Brazil, Peru, Honduras, Jamaica, Costa Rica, Venezuela, and Ecuador where test completeness was slightly less than positives completeness. Eight countries (Argentina, Brazil, Mexico, United States, Chile, Peru, Colombia, and the Dominican Republic) had > 90% completeness for tests and total, influenza A, and influenza B positives. The average completeness values for influenza A(H1N1)pdm09 and A(H5) across all countries were 66% and 25%, respectively. Seven countries (Suriname, Haiti, Belize, Barbados, St. Lucia, Dominica, and St. Vincent & Grenadines) had < 40% completeness for all influenza outcomes, subtypes, and overall completeness. Figure 4 demonstrates that high and low overall completeness was likely in countries of mid-range DHEPC and OOPHE%. Given the similarity in completeness for multiple outcomes, we selected tests, positives, A(H1N1)pdm09, and overall completeness for further regression analyses.
Completeness, economic and health expenditure indicators, and surveillance system attributes. Trends in country-specific completeness for tests, positives, A(H1N1)pdm09, and overall and GNIPC, DHEPC, and OOPHE% are shown in Figs. 5 and 6, respectively. The decadal change of economic indicators for each country and averaged across countries (with standard deviation estimates) is provided in Supplementary  Table S7. GNIPC and DHEPC almost doubled (2.02 ± 0.67 vs 2.19 ± 0.77, respectively) between 2005 and 2015; and OOPHE% declined by 16 ± 18%. Table 4 shows the annual change in completeness for four outcome variables estimated from unadjusted (Model 1) and adjusted (Model 2) mixed effects models. All variables exhibited a strong positive trend with an annual increase in completeness from 4.2 to 6.3% on average. Improvements in completeness were most prominent for influenza A(H1N1)pdm09 showing a strong positive relationship with GNIPC. DHEPC was the strongest predictor of completeness across all influenza outcome variables. With the doubling in DHEPC achieved on average in the region, all four variables exhibited an improvement in completeness up to 9.4%. A projected 10% change in OOPHE% had no association with changes in completeness.
The adjusted models also indicated that the high numbers of NICS and reporting facilities were consistently associated with an increase in completeness. For every additional NIC, the completeness for tests, positives, and overall completeness increased by ~ 18-24% irrespective of the economic or expenditure indicator assessed. www.nature.com/scientificreports/ Similarly, every additional NIC was associated with a 15-20% increase in influenza A(H1N1)pdm09 completeness. A doubling in the number of in-country reporting facilities was associated with a 1.85-5.00% increase in completeness across all influenza outcomes and economic indicators. While some countries use multiple surveillance systems or followed numerous reporting quotas, timeframes, and case definitions, on average there was no difference in completeness values across surveillance system attributes.

Discussion
In this study, we demonstrated the progress made by FluNet over the last decade towards achieving high overall completeness in publicly available influenza surveillance records. The proposed metric shows that influenza surveillance reporting has improved especially after the A(H1N1) 2009 pandemic, highlighting the effort by national systems and the WHO. This metric further demonstrated the substantial increase in completeness after the 2013 WHO guidelines for influenza surveillance 5 . These improvements were practically identical for tests,   www.nature.com/scientificreports/ total positives, influenza A positives, and influenza B positives indicating the systematic approach taken in their reporting. As of 2019, 24 of 29 Pan-American countries were operating at > 95% of completeness for these key outcomes, though efforts are still needed to ensure consistent surveillance for other indicators. National reporting infrastructures continued to increase in their richness and heterogeneity, which indicates wider in-country surveillance coverage 14 . Annual completeness estimates continue to improve over time by ~ 5% annually and the rates of improvement are similar among countries with different national surveillance systems attributes. Yet, countries with higher numbers of NICs, more reporting facilities, and greater health expenditures showed the best performance. The proposed completeness metric provides essential information on data availability and suitability for statistical analysis and modeling and can improve the utility of existing data for all users. Our metric is based on the effective time series length and can be computed for any pre-specified time periods. The metric ultimately reflects the fraction of weeks when surveillance reports are missing. We define 'missing' as a week for which case counts are undefined or not reported. This is not to be mistaken with zero reported observations, or the absence of counts for a specific case definition. To further the utility of the completeness www.nature.com/scientificreports/ metric, we recommend supplementing publicly disseminated reports with metadata on why data are missing, what missing data means, and how much missing data are reported. Although the metric is simple to implement and interpret, the metric could not take into consideration patterns of missingness. Missing records could be distributed randomly or systematically throughout the study period when the data displays structural missingness with records lost in chunks or more frequently during specific times of the year. However, low completeness estimates call for attention to further investigate the pattern of missingness. This was demonstrated above in examples with the United States, Peru, and Venezuela where we identified periods of missing weeks and years using our metric. By knowing the temporal distribution of missing records some correction can be made during the analysis stage, for example by using completeness as an additional variable to reduce the weight of years with incomplete records. Closer inspection into patterns of missingness can also be used for verifying, inspecting, or updating public records within a data source prior to analysis.
The metric provides greater clarity on how a country's surveillance system quality changes over time and helps identify anomalies in surveillance reporting. If an unusual drop in completeness is noted yet the national surveillance records exist for the time period in question, such discrepancies can be curated in a timely manner. Furthermore, such discrepancy calls into question whether external data users could help data curating organizations with checking the fidelity and accuracy of the data they use and their assumptions for handling missing data.
By examining annual completeness estimates, users could determine the study interval with reliable information. In the extracted records from 2005 to 2019, each country-outcome-specific time series had varying numbers and patterns of weeks with missing information. While a longer historical reference period indicates greater  www.nature.com/scientificreports/ statistical power in estimating disease trends, some outcomes are collected over limited time frames. The length of these time series is influenced by the country of interest, outcome of interest, and date of data extraction. For example, positives of A(H1N1)pdm09 showed consistently high completeness across most countries from 2009 to 2019. Yet, for some countries, records were available for a fraction of that period. Based on our findings, we encourage data users to clearly specify the start and end date of their time series, the completeness of that time series, and the date of data extraction. In recent years, the WHO has taken numerous efforts to evaluate the economic burden associated with seasonal influenza, especially in lower-and middle-income countries [34][35][36] . These efforts, including a published Manual for estimating the economic burden of seasonal influenza in 2016, aim to demonstrate the value of population surveillance by calculating the direct medical, direct non-medical, and indirect costs of influenza illness 34,35 . Studies applying methodologies outlined in this manual confirm that increased country income and health expenditure is associated with increased immunization policies, vaccination coverage, health infrastructure, and surveillance coverage 35,36 . The WHO 2016 manual further recognizes that data validity and completeness can influence assessments of economic burdens related to national income, domestic health expenditure, and out-of-pocket health expenditure 34 .
Our study faced several challenges related to data availability and accessibility. First, the FluNet data portal requires country time series to be downloaded individually and merged using a scripted pipeline. This process is both inefficient and prone to human error during data alignment and compilation. While our data extraction and merging code overcomes this challenge, we encourage FluNet curators to allow for multi-country data downloads and improve data accessibility. www.nature.com/scientificreports/ Next, we examined only the most recent records starting in 2005. Our initial intention was to use all available data from 1995 to 2019. We completed two attempts to extract records: first on 15 December 2019 and second on 26 April 2020 to retrieve data for the final weeks of 2019. Between extraction dates, however, available FluNet data changed dramatically: data originally available from Week 1 1995 to Week 52 2004 were reported as missing at the time of second extraction. No justification is provided regarding this change. Thus, we encourage FluNet curators to provide information on when, by how much, and why retrospective records are modified to ensure accuracy and validity of analyses performed with the open source records.
Finally, we recognize that examined national economic indicators, such as DHEPC and OOPHE% describe all national health expenditures and the fraction of health expenditures dedicated to influenza may vary dramatically each year. During the A(H1N1)pdm09 pandemic, health expenditures may have been quite high for influenza vaccinations and testing. Moving forward, the onset of the coronavirus disease 2019 (COVID-19) pandemic may have decreased this fraction of health expenditures for influenza monitoring much lower. A better tracking of influenza-related expenditures at the national and global levels is needed to confirm or refute our findings.
In prior works, we have developed tools and explored the seasonality of influenza and other infections. In a study of pandemic seasonal epidemics in Wisconsin from 1967 to 2004, we found that seasonal peak timing varied greatly and while viral evolution played an important role, the variability of seasonality estimates was also influenced by the data granularity 23 . The estimates (and their confidence intervals) of seasonal peak timing and intensity could be in part influenced by data aggregation and completeness of surveillance data and thus affect our understanding of deviations in influenza seasonality 21 . Our mini-review of mathematical modeling www.nature.com/scientificreports/ techniques for influenza transmission highlighted that variations in theories governing seasonal dynamics could also be attributed to data availability limitations 22 . Examining disease trends and seasonal epidemic signatures allows for greater understanding of influenza transmission and developing preparedness strategies at the local, national, and regional levels 2,[18][19][20]36 . The proposed metric of completeness is essential in estimating the statistical power to detect disease trends and temporal changes by providing the effective length of disease surveillance time series data. Further work is needed to understand how completeness influences the reliability of modeling results. The use of the proposed metric will also allow for better assessment of the quality of historical data for tracking disease trends. The continuously updated surveillance records and the ensemble of disease outcomes allows for adaptive modeling to create realtime forecasts and detection of local events with high spatiotemporal granularity 2 .

Conclusion
This study provided the first attempt at quantifying the percentage of available data usable for closer examination of disease trends or seasonality analyses. As more surveillance data becomes available for public use, indicators such as completeness should be applied to ensure quality, accuracy, and reliability of trend estimations. The proposed metric of completeness is vital to any secondary time series data analysis where the data user did not curate the data source. This metric can also be estimated and reported by external data users to ensure reliability and improve understanding of data structure. This next step in data sharing can help external data users detect outbreak signatures more accurately and reliably as well as improve health policies, programming, and recommendations. Combined with access to already developed WHO M&E indicators, the completeness metric for publicly disseminated data will strengthen disease surveillance systems.
Received: 14 September 2020; Accepted: 16 December 2020 Table 4. Estimated annual change in completeness of four influenza outcome variables and three economic and health expenditure indicators in 29 Pan American countries over the 782-week time series starting from Week 1 (03 January) 2005 to Week 52 (29 December) 2019. The effect size estimation was calculated for a doubling in per capita indicator estimates and a 10% increase in out-of-pocket health expenditure. LCI and UCI refer to the lower and upper bounds of the 95% confidence interval. a Models were adjusted for case definition type, adherence to WHO definition, surveillance strategy, reporting quota, reporting timeframe, number of NICs and reporting facilities.