Hourly rainfall data from rain gauge networks and weather radar up to 2020 across the Hawaiian Islands

With increasing needs for understanding historic climatic events and assessing changes in extreme weather to support natural hazard planning and infrastructure design, it is vital to have an accurate long-term hourly rainfall dataset. In Hawaiʻi, annual, monthly, and daily gauge data have been well-compiled and are accessible. Here, we compiled hourly rainfall data from both gauges and radars. We arranged the metadata from various data sources, acquired data, and applied quality control to each gauge dataset. In addition, we compiled and provided hourly radar rainfall, and filtered out areas with low confidence (larger error). This paper provides (1) a summary of available hourly data from various observation networks, (2) 293-gauge rainfall data from their installation date to the end of 2020, and (3) a 5-year 0.005° by 0.005° (~250 × 250 m2) gridded radar rainfall dataset between 2016 and 2020 across the Hawaiian Islands.

www.nature.com/scientificdata www.nature.com/scientificdata/ to any study: (1) the stations distribute among different networks and data stewards, and some of them cannot be accessed online directly; and (2) a quality-controlled dataset of hourly rainfall does not exist in most the gauge networks. The challenges and drawbacks of applying NEXRAD Level III rainfall data to hydrological studies include: (1) the non-rectangular HRAP grid of the rainfall dataset cannot be used directly to most of hydrological models; (2) the finest spatial resolution of the rainfall product is still considered coarse at one dimension (>1 km); (3) the confidence of radar and its products' performance in Hawaiʻi are unknown; and (4) the parameters applied to the rain rate calculation are unclear. All of these hinder studies of radar rainfall in Hawaiʻi.
The goal of this paper is to describe the availability of hourly rainfall data in Hawaiʻi and provide: (1) rainfall data of 293 gauges from their installation date (the earliest, 1962) to 2020; and (2) a 0.005° by 0.005° ( × m 250 250 2 ) gridded radar rainfall dataset between 2016 and 2020 under a level of confidence. For hourly gauge rainfall, we acquired all the data, applied quality controls (QCs), and compiled the metadata. For radar rainfall, we produced the rainfall product by converting raw radar reflectivity into hourly rainfall using the Lidar Radar Open Software Environment (LROSE) 13 . The paper is organized as follows: Methods-rainfall networks, step-by-step method for gauge rainfall QC, and the process to derive radar rainfall; Data records-an overview of the data files and their formats; and Technical Validations-comparing hourly radar rainfall with hourly gauge rainfall and things to be concerned when using the data.

Methods
Rain gauge networks. To compile a comprehensive Hawaiʻi-wide hourly rainfall dataset, we first identified and acquired data from all rain gauge networks. Based on the daily rain gauge networks between 1990 and 2014 in Longman et al. 9 , we extended the work to hourly rainfall data within all listed rain gauge networks and to the end of 2020. Additional networks and data repositories were explored and supplemented through this effort. The hourly rainfall data were assembled through several national and international online data repositories and networks (Table 1). For Hawaiʻi-only networks, data typically can only be obtained from project principal investigators (PIs) or data managers. Detailed information of nine out of 12 networks can be found in Longman et al. 9 under the section, Data Records, with one network, Ua-Hydro Net, updated from Hydro Net. Besides the updated Ua-Hydro Net, three additional networks were identified and added to the data compilation in this study:

Ua-Hydro Net
The Ua-Hydro Net ("ua" means "rain" in Hawaiian language) includes Ua Net and Hydro Net, maintained by Pacific Region Headquarters of National Weather Service (NWS). The gauges in Hydro Net (see Longman et al. 9 for more detail) were gradually discontinued or converted to Ua Net beginning in mid-2015. Moreover, most of Ua Net data within the latest seven days can be retrieved from the Hydrometeorological Automated Data System (HADS).

Cooperative Observer Program Version 2
The Cooperative Observer Program version 2 (COOPV2) has hourly gauge rainfall from 1940 to the present. These stations, nearly all of which were part of Hourly Precipitation Data (HPD) version 1 (a.k.a., DSI-3240), were gradually upgraded from paper punch tape data recording systems to a modern electronic www.nature.com/scientificdata www.nature.com/scientificdata/

Hydrometeorological Automated Data System
The Hydrometeorological Automated Data System (HADS) is a real-time data acquisition and data distribution system operated by the NWS. The HADS system acquires raw observations from Geostationary Operational Environmental Satellites (GOES) Data Collection Platforms (DCPs). Most of the data in Hawaiʻi acquired and processed by HADS come from the DCPs owned and/or operated by the NWS, Western Regional Climate Center (WRCC), Water Resources Division of the U.S. Geological Survey (USGS), and numerous local agencies and state departments of natural resources.

National Centers for Environmental Information Integrated Surface Database
The NCEI Integrated Surface Database (ISD) is a global database that consists of hourly and synoptic surface observations compiled from numerous sources into a certain format and includes various meteorological parameters. Additionally, the ISD Lite, an advanced product of ISD, is a subset of the full ISD containing eight common surface parameters (air temperature, dew point temperature, sea level pressure, wind direction and wind speed, total cloud cover, hourly rainfall, and six-hour accumulated rainfall) in a fixed-width format free of duplicate values. Currently, there are 14 active NCEI ISD Lite stations updated daily in the NCEI ISD lite database for Hawaiʻi. Data can be obtained through the NCEI Data Access app (https://www.ncei.noaa.gov/products/land-based-station/integrated-surface-database).

Radar networks.
To derive radar quantitative precipitation estimates (QPEs), we use the raw radar reflectivity from NEXRAD LEVEL II dataset 11  Rain gauge data. We applied Quality Assessment and Quality Control (QAQC) to ensure a level of data homogeneity. Raw rainfall data acquired from various sources are subject to different formats and a wide variety of potential errors. Different networks have their own data format, and errors can occur during the data management chain of collecting, processing, transferring, and storing 16 . The World Meteorological Organization (WMO) developed a set of procedures for rescuing and quality controlling data 17 . Before conducting any scientific analysis, rainfall data must undergo a strict set of QAQC procedures to minimize the proportion of errors in the data. The data provided here were screened through an automated and a manual QC process ( Fig. 1). We summarized the WMO QC steps and the adapted QC steps in Table 2. In addition, we compiled metadata for all gauges including their coordinates, record period, sources, and flags for the gauges with potentially unreliable data. We updated the latest coordinates in the metadata if the gauge was relocated yet retained the same gauge ID to be consistent with the source agencies. Prior to conducting QC, a set of pre-processing protocols are applied to 293 gauges (Fig. 2). 4. Accumulated rainfall is changed to rain rate. 5. Hourly rainfall is aggregated by rain rate. 6. All rainfall values are converted to metric units (e.g., millimeters). 7. All stations were plotted using Geographic Information System (GIS) software to verify land-based locations. When two (or more) stations were co-located between NCEI IDS, COOPV2, and Ua-Hydro Net, we checked to see if they are the same stations included in multiple networks based on the Historical Observing Metadata Repository from NCEI (https://www.ncdc.noaa.gov/homr/). Once the station is confirmed, rainfall records of different periods were combined into one file. When periods of records overlapped, the records of the Ua-Hydro Net were prioritized to be kept in the file.
There are three main processes in the QC procedure-automated QC, manual QC ( Fig. 1), and consistency with neighbor gauges. Once a common format of records is established, the following set of automated QC is applied. During these two processes, "flag codes" were given to the values detected by the QC criteria. A description of the automated QC process is given below, and flag codes are listed in Table 3. 1. Identify the beginning of the records. Delete the missing values (NA) at the beginning of the records. 2. Trace precipitation (<0.254 mm) 18 or negative rainfall are set to 0 and flagged as A1. 3. Threshold test, flagged as A2.2: • Rainfall values above 150 mm/hr 17 are flagged. 4. Variability test, flagged as A2.3: • If an hour before and after includes no missing values and the inspecting rainfall value varies more than eight times of standard deviation of gauge records (modified from WMO QAQC procedures 17 ), the rainfall value is flagged. • If an hour before or/and after the inspecting value includes missing value(s), the missing value(s) are assumed to be 0. If the inspecting rainfall value varies more than four times of standard deviation of gauge records, the rainfall value is flagged.
After each file goes through the automated QC (steps 1 to 5), a manual QC (steps 6 to 7) was conducted to investigate and determine if the flagged rainfall values should be accepted, corrected, or deleted. A detailed description of the manual flag codes is given in Table 4. 6. Manual inspection of threshold flag A2.2: • If the rainfall value occurs in a very extreme event, according to the verified information from NWS and the climatology of gauge, we referred the rainfall amount to the surrounding gauges and the entire records of the gauge to confirm that the rainfall value could possibly go over the defined threshold of 150 mm/hr. www.nature.com/scientificdata www.nature.com/scientificdata/ 7. Manual inspection of variability flag, A2.3: • If the rainfall value is a spike (--˄--) in the time series • and there is no rainfall event (i.e., no rain recorded within six hours before or after the flagged rainfall value hour), the rainfall value is accepted if it is under the 75th percentile of the entire dataset.
For rainfall values above the 75th percentile, the values are set to 0 and flagged as M2. The threshold here was rationalized by the characteristics of locally convective rainfall-the variance during the no rainfall event should be lower than the variance of a larger and longer rainfall event (i.e., rain recorded within six hours). • and a rainfall event is detected (i.e., rain recorded within six hours before or after the flagged rainfall value hour), the rainfall value is accepted if it is under the 95th percentile of the entire dataset.
For rainfall values above the 95th percentile, the values are set to NA and flagged as M1. The 95th percentile was selected based on the fact that the 95th percentile value of an hourly rainfall dataset is large. Therefore, as large rainfall variations are expected when a rain event is observed, it is logical to compare the flagged value to a large value from the entire dataset.
• If the rainfall value is a dip (-v--) and is 0 • and rainfall records preceding and following the 0-rainfall value are flagged 2.3, the zero-rainfall value is set as NA and flagged as M1. • and rainfall records preceding or following the zero-rainfall value are not flagged 2.3, the zero-rainfall value is accepted and flagged as M0.
• For the rainfall values that are not a spike (--˄--) or dip (-v--) • and the value is lower than the 90th percentile value of the entire dataset, the value is accepted and flagged as M0. • and the value is higher than the 90th percentile value of the entire dataset: • values within a rainfall event (rain observed six hours before or after the flagged rainfall value) are accepted and flagged as M0. • values that are not in a rainfall event (no rain observed six hours before or after the flagged rainfall value) are compared with satellite data, Tropical Rainfall Measuring Mission (TRMM) 19 , which provides 3-hour rainfall data for Hawaiʻi: • if rain is observed with satellite data, the value is accepted and flagged as M0.
• if no rain is observed with satellite data, the value is set as NA and flagged as M1.
After flagging on hourly rainfall data of each gauge, we inspected the consistency of the gauge record and applied the correlation coefficient to the rainfall of each gauge and its nearest three (neighbor) gauges within a 6-km radius to examine any underreporting issues, based on the aggregated rainfall at a daily scale. We used three different flags to mark gauges including "Not Applicable", "Warning", and "Pass" (Table 5). Not Applicable was assigned when the nearest three gauges were outside of 6-km radius or when three neighbor gauges had no overlapping data periods for comparison. Warning was assigned when: (1) there were two neighbor gauges not applicable (missing due to the distance threshold or no overlapping data period), and the only remaining gauge had a correlation coefficient (R) less than 0.5; and (2) there were at most one not applicable neighbor gauge and at least two remaining gauges had R smaller than 0.5. Gauges that were not flagged as Not Applicable or Warning were considered passing the QC of neighbor gauge data comparison and marked as Pass. We included this flag under the column, CORR_FLAG in our metadata.
Radar rainfall. We derived radar QPE by using the open sources software, Lidar Radar Open Software Environment (LROSE) 13 . LROSE is built on the legacy left by TITAN 20 and developed by the National Center for Atmospheric Research (NCAR) to handle weather radar and lidar data. LROSE has very wide applications including beam blockage identification, converting radar variables into rain rate, and storm tracking, and is continuously updated. We use LROSE-cyclone, which was released in 2020 21 , to derive the QPE from the raw NEXRAD LEVEL II radar reflectivity.
There are three basic steps to obtain the radar QPE: (1) estimate precipitation rate aloft within the radar volume; (2) estimate the applicable rate at the surface; and (3) converse precipitation rate to precipitation depth over a certain period. Using LROSE, we estimated the precipitation rate at each radar grid by running the particle identification (PID) along with the nearest radiosonde soundings in time and space to estimate the 0 °C isotherm. The PID is derived by modified NCAR's fuzzy-logic-based PID algorithm that can identify the particle  www.nature.com/scientificdata www.nature.com/scientificdata/ types at each radar grid. After identifying particle types, we calculated the radar based QPE at the surface and applied the modified NCAR Hybrid method 22 with localized parameters 23 (Table 6) to calculate the precipitation rate at each grid point.
Prior to deriving the applicable precipitation rate at the surface, beam blockage was estimated at the lower elevation radar angles with the SRTM elevation data. The beam blockage algorithm considered standard atmospheric propagation effects and the convolution of the beam pattern with terrain features 22 . The algorithm produced a blocked fraction for each elevation angle. The beam blockage is crucial to derive QPE in Hawaiʻi because of the mountainous terrain. If it is blocked, then the remaining unblocked angle would be used for    www.nature.com/scientificdata www.nature.com/scientificdata/ the precipitation estimation. The precipitation rate in polar coordinates at the surface then inherits the lowest elevation angles with the following criteria: (1) the signal to noise ratio (SNR) <5 dBZ; (2) beam blockage < = 25%; (3) PID does not show clutter, insects or second trip; and (4) the precipitation rate is not missing at the grid. The surface precipitation rate is then converted into Cartesian coordinates. The QPE was estimated by computing precipitation depth from the rate at the original time interval and summing the precipitation depth over time for accumulation periods (in this study, hourly), and was then converted into Cartesian grids. Finally, the precipitation rates from individual radars (if multiple radars were included) were integrated by taking the maximum value.
Validation of radar rainfall. For this study we selected 18 severe rainfall events (  [24][25][26] . The hourly radar rainfall was validated with the quality-controlled gauge rainfall at each gauge location except for the gauges with correlation coefficients flagged as "Warning". We defined the validation period of each event as low or no rainfall before and after the reported event time. For each event, the rain gauges for validating hourly radar rainfall must contain more than 80% of hourly rainfall records and the number of gauges can be slightly (<10) different. For each event, we applied four error matrices, including bias (BIAS), root mean square errors (RMSE), and their normalized values (nBIAS and nRMSE). BIAS indicates if radar rainfall is overall under-or over-estimated compared to the gauge rainfall, while RMSE quantifies the errors accumulated during the study periods and penalizes the larger error. To ensure that BIAS and RMSE are comparable among gauges and can be interpolated for creating masks of radar rainfall, the relative error matrices, nBIAS and nRMSE, were derived by dividing the range between event maximum and minimum rainfall of each gauge. In addition, we calculated the correlation coefficient (R) by combining all events. BIAS and RMSE indicate the accuracy at the gauges. nBIAS and nRMSE denote the spatial accuracy. The R value shows how well each gauge aligned with the radar estimate.
Mask of radar rainfall. To provide the gridded rainfall data with a level of confidence, we removed (masked out) areas with high uncertainties and bias by considering the high beam blockage areas, nBIAS, and nRMSE. Beam blockage is indicated by the higher beam blockage fraction or higher angle of the radar beam, which cause higher uncertainties and bias in reflectivity measurements. The beam blockage height is set for the lowest level that has the beam blockage fraction < = 25%. In addition, the nBIAS and nRMSE of radar rainfall against gauge rainfall were used to examine the confidence of radar measurement. Specifically, we examined the nBIAS and nRMSE of radar rainfall for the previously mentioned 18 weather events to ensure that the radar rainfall's bias and error are not due to the type of event selected in this process. We filtered out the outliers of event nBIAS and nRMSE when Z-score >1 within the events at each gauge location. We then interpolated event-averaged nBIAS and nRMSE at each gauge location with inverse distance weight. Next, we applied thresholds: 5% for the interpolated nRMSE (Fig. 3a); 3.5% for the interpolated nBIAS (Fig. 3b); and 2.0 km for the beam blockage height (Fig. 3c). Lastly, we combined the masks to create a final mask (Fig. 3d). The mask is then applied to the radar rainfall to remove it from areas with higher uncertainties/errors. The remaining, unmasked, radar rainfall data are more reliable and have a higher level of confidence.

Not applicable
The flag is not applicable on the gauge because there is not enough data to compare between its data and its neighbor gauges' data.

Warning
The gauge needs to be used with caution because it has different daily rainfall pattern with its neighbor gauges.

Pass
The gauge is reliable because its data is consistent with its neighbor gauges' data.  Table 6. Parameters and thresholds used in modified NCAR Hybrid method in Lidar Radar Open Software Environment (LROSE), where ZH a and ZH b are the coefficients of rain rate reflectivity relationship, dBZ is radar reflectivity, Kdp is specific differential phase, and Zdr is differential reflectivity. We applied the reflectivity-rain rate relationship, = . www.nature.com/scientificdata www.nature.com/scientificdata/ Data records Gauge rainfall. This dataset includes one metadata CSV file describing gauge data and location information (Table 8) and 293 comma-separated value (CSV) files of hourly gauge rainfall 27 . The metadata has been compiled in metadata_pub_V20220401.csv. The filename of each gauge is {FILENAME}.csv (FILENAME in metadata). Each hourly gauge rainfall csv file includes five columns: DateTime; RF_raw (raw hourly rainfall in millimeters); RF_mm (quality-controlled hourly rainfall in millimeters); flag_code_auto (the automated QC flag code); and flag_code_manual (the manual QC flag code). Note that the value, NA, means no measurement or uncertain rainfall amount. The metadata includes station information such as station name, coordinates, start and end date, data sources, etc. Table 8 shows the definition and unit of each column name in the metadata. The COMPLETENESS is provided so that users can gain a quick understanding of the percentage of how many hours have rainfall value compared to the entire observation period of the gauge. The hourly rainfall data starts from the earliest station record of hourly rainfall data and ends in the end of 2020 (i.e., December 31, 2020). One can refer to the SOURCES in the metadata to access rainfall data beyond 2020. The metadata and data records of this study dataset can be downloaded from the figshare electronic repository 27 . It is in the plan that gauge metadata and rainfall data will be updated to the Hawaiʻi Climate Data Portal (https://www.hawaii.edu/climate-data-portal/) prepared by Mclean et al. 28 in the future.
Radar rainfall. The masked hourly radar rainfall (2016-2020) is also provided on the same repository in figshare with the Network Common Data Form (NetCDF) 27 . The filename is "Hourly_radarRF_{YYYYMM}_ masked.nc" (e.g., Hourly_radarRF_201601_masked.nc for masked radar rainfall in January 2016). The masked hourly radar rainfall is stacked into 60 NetCDF monthly files with three dimensions-time (UTC), latitude (degree), and longitude (degree). Each grid is by 0.005 degree (~ 250 m). The radar rainfall values on the grid with no radar, missing hours, and masked areas were filled as NaN with the same spatial dimension. There are 465 hours missing (260 hours in 2016, 25 hours in 2017, 12 hours in 2018, 137 hours in 2019, 31 hours in 2020) over the total 43,848 hours, and the data is 98.94% complete.

Technical Validations
All hourly gauge rainfall data provided in this study are quality controlled and the radar rainfall was validated for 18 rainfall events. Out of a total of 293 stations, 215 stations remained active after December 31, 2020. The earliest data can be traced back to 1962 at Honolulu International Airport (911820-22521). Two new stations were established in 2020 at Pāpaʻikou well (PPWH1) and Waimea tank (WMTH1). The completeness ranges from 2.29% to 100% with a mean value of 87.04% (Fig. 2). The neighbor gauge comparison allows us to identify gauges with potentially unreliable data 29 . This QC process resulted in 225 Pass, 56 Not Applicable, and 12 Warning in our collective dataset. It successfully flagged the gauges that have inconsistent records with neighbor records to alert potential long period errors. For example, the gauge, 911905-22524, has many trace rainfall and potentially incorrect zero values (e.g., zero values at the station, while high rainfall values at other neighbor gauges) in the raw data. For these reasons, we recommend excluding the gauges flagged as Warning, or being extra cautious when using the data of these gauges.  www.nature.com/scientificdata www.nature.com/scientificdata/ When validating the mean BIAS, RMSE, nBIAS, nRMSE, and correlation coefficient over all gauges are 0 32 − . mm, 0.61 mm, 1 30% − . , . 5 23%, and 0.60, respectively, before applying the mask. The hourly radar rainfall has better agreement with the hourly gauge rainfall on Oʻahu and Molokaʻi (Fig. 4). The radar rainfall often underestimates the rainfall on Kauaʻi, Maui, and Hawaiʻi compared to gauge rainfall (Fig. 4). When validating the remaining radar data against 153 gauges, the final masked-out radar rainfall has an increased confidence with mean BIAS, RMSE, nBIAS, nRMSE, and CORR as 0 10 − . mm, 0.43 mm, 0 14% − . , . 4 43%, and 0.67, respectively. The masked hourly radar rainfall has more confidence and reliability than the former unmasked data, thus, we provide the masked radar rainfall here.

SOURCES
The source(s) of downloading data