Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

A new climate data record of upper-tropospheric humidity from microwave observations

Abstract

We generated a new Climate Data Record (CDR) of Upper Tropospheric Humidity (UTH) based on observations from the microwave sounders Special Sensor Microwave Temperature - 2 (SSMT-2), Advanced Microwave Sounding Unit - B (AMSU-B) and Microwave Humidity Sounder (MHS). The data record covers the time period between 1994 and 2017 and provides monthly mean 183.31 ± 1 GHz brightness temperatures and derived UTH along with estimates of measurement uncertainty on a 1° × 1° latitude-longitude grid covering the tropical region (30° S to 30° N). For the UTH retrieval we introduce a new definition of UTH. Forgoing the use of the humidity Jacobian as a weighting function, it is easier to apply than the traditional definition without compromising the retrieval accuracy. The same definition can be used to derive UTH from infrared observations, allowing for a more synergistic use of infrared and microwave UTH in the future. The new UTH CDR is validated against an existing UTH data record.

Measurement(s) humidity
Technology Type(s) Radiometry (Microwave)
Factor Type(s) time period • geographic location
Sample Characteristic - Environment troposphere • climate system

Machine-accessible metadata file describing the reported data: https://doi.org/10.6084/m9.figshare.12465830

Background & Summary

In the framework of the Horizon 2020 project Fidelity and Uncertainty in Climate Data Records from Earth Observations (FIDUCEO) a new Level 3 Climate Data Record (CDR) of Upper Tropospheric Humidity (UTH) has been generated based on observations from the microwave humidity sounders Special Sensor Microwave Temperature - 2 (SSMT-2), the Advanced Microwave Sounding Unit - B (AMSU-B) and the Microwave Humidity Sounder (MHS). It covers the time period between 1994 and 2017 and provides monthly mean data on a 1° × 1° latitude-longitude grid covering the tropical region between 30° S and 30° N. UTH is derived from 183.31 ± 1 GHz brightness temperatures provided by the FIDUCEO Microwave Fundamental Climate Data Record (FCDR) Version 4.11,2.

UTH is an important climate variable, because it has a significant impact on the Earth’s radiation budget3 and the associated water vapour feedback amplifies the climate system’s response to increases in other greenhouse gasses such as carbon dioxide4. Virtually all climate models show a water vapour feedback that is consistent with an approximately constant upper tropospheric relative humidity5. To evaluate climate model simulations, long-term observational data records of UTH are needed6.

Compared to existing UTH products, there are two major new aspects of the FIDUCEO UTH CDR: (1) A revised definition of UTH is used and (2) estimates of observational uncertainties are provided.

The first aspect is an attempt to solve a problem specifically related to the quantity UTH: Besides the microwave (MW) measurements used for this CDR, UTH can also be derived from observations in the infrared (IR) spectral region. However, comparing UTH derived from MW and IR measurements is hardly possible due to the traditional definition of UTH. UTH is traditionally defined as a weighted mean of the RH profile, where the weights are given by the humidity Jacobian. The humidity Jacobian peaks in the upper troposphere for both MW and IR water vapour channels. However, the exact shape of the Jacobian varies between channels, complicating a synergistic use of MW and IR UTH. We introduce a new UTH definition that is solely based on the vertical distribution of water vapour and forgoes the Jacobian. We show that this definition can be used to retrieve UTH from measurements by both microwave sounders (SSMT-2, AMSU-B and MHS) and the High-resolution Infrared Radiation Sounder 2 (HIRS/2) without loss in retrieval accuracy compared to the traditional approach. As HIRS/2 data is available in the time period from 1979 to 2016 it could be used at a later time to expand our UTH data record to earlier years. The new definition has the additional advantage that UTH can be calculated directly from given atmospheric profiles of humidity and temperature without a detour via radiative transfer simulations.

Detailed uncertainty information in CDRs derived from satellite-based Earth observations are needed to support the application of the data in climate research7,8. Providing such information on CDR level (level 2 or 3) has mainly been constrained by the availability of uncertainty information in the underlying FCDRs (level 1) in the past. Within the FIDUCEO project four new versions of such FCDRs were created, among them the FIDUCEO Microwave FCDR used as input for our UTH CDR. The FCDR includes information on observational uncertainty on pixel level, which is the result of rigorous uncertainty analyses based on metrological principles9. These uncertainties are propagated to the spatially and temporally averaged quantities in the UTH CDR. Depending on the spatial and temporal correlation behaviour of the underlying error sources, uncertainties are divided into three different classes, enabling the user to propagate them to spatial and temporal averages of the data.

The FIDUCEO UTH CDR is validated against an exisiting microwave UTH data record provided by the Satellite Application Facility on Climate Monitoring (CM-SAF). Differences in monthly tropical mean UTH do not exceed 2% RH and can be attributed to differences in the underlying FCDR and in the CDR processing in approximately equal parts.

The structure of this paper is as follows: The Methods chapter introduces the satellite instruments, the UTH retrieval method and the new definition of UTH. Furthermore, a detailed description of the CDR processing is provided. This is followed by the Data Records chapter, which includes a description of the CDR data file format as well as the satellite missions and time periods covered. The subsequent chapter Technical Validation consists of an evaluation of the UTH retrieval performance, the comparison of our CDR with the CM-SAF UTH CDR and a description of uncertainties not estimated in the CDR.

Methods

Instruments

The FIDUCEO UTH CDR combines measurements from the instruments Special Sensor Microwave Temperature - 2 (SSMT-2), the Advanced Microwave Sounding Unit - B (AMSU-B) and its successor, the Microwave Humidity Sounder (MHS). All three of them are passive microwave radiometers operating on polar-orbiting satellites. They have a similar design with two surface channels and three sounding channels located around the 183.31 GHz water vapour absorption line at 183.31 ± 1 GHz, 183.31 ± 3 GHz and 183.31 ± 7 GHz (183.31 + 7 for MHS), respectively. Typically the 183.31 ± 1 GHz channel is used to derive UTH since the signal reaching the instrument in this channel originates from the upper troposphere10. The channel has a total bandwidth of 1000 MHz for all three instruments. It is called channel H3 for MHS, channel 18 for AMSU-B and channel 2 for SSMT-2. For simplicity we will hereafter refer to the 183.31 ± 1 GHz channel of all instruments as the MW UTH channel.

Table 1 provides an overview of the scanning properties of MHS, AMSU-B and SSMT-2. All three are cross-track scanning instruments. For MHS11 each scan line consists of 90 Earth views, 45 on each side of the sub-satellite point. Each scan covers about 50° on both sides of the sub-satellite point, resulting in a swath width of 2180 km. The viewing angles range from 0.55° (with respect to the nadir view) to 48.95° in steps of 1.1°. With an antenna beamwidth of 1.1° the ground footprint at the innermost scan position has a diameter of approximately 16 km. The scanning geometry of AMSU-B11 is almost identical to that of MHS, only the viewing angles differ slightly from those of MHS. SSMT-212 scans the Earth in only 28 views and has a larger beamwidth of 3.0°, resulting in a larger nadir footprint diameter of approximately 48 km.

Table 1 Basic scanning properties of SSMT-2, AMSU-B and MHS. Note that the numbers given for SSMT-2 are only valid for channel 2 since the beamwidth changes between the instrument channels.
Table 2 UTH scaling parameters determined from the ECMWF data set for all viewing angles of AMSU-B (aA, bA) and MHS (aM, bM).
Table 3 Satellite missions and time periods covered by the FIDUCEO UTH CDR.

For the development of a new UTH definition we simulated MW UTH channel brightness temperature (Tb) with a radiative transfer model. Furthermore, since the aim is to find a common definition for MW and IR instruments, we additionally simulated Tb for an IR instrument, the High-Resolution Infrared Radiation Sounder (HIRS)13. HIRS is a cross-track scanning infrared radiometer with 20 frequency channels covering a spectral range from 0.69 to 15 μm. Channel 12 is designed to observe water vapour in the upper troposphere. For HIRS/2, the earliest version of the instrument, channel 12 is centred at 6.7 μm and has a spectral bandwidth of approximately 0.45 μm (2998 GHz). We will refer to it as the IR UTH channel in the following. When the instrument was upgraded to HIRS/3 with the launch of the satellite NOAA15 in 1998, the spectral response function of the channel changed and its centre wavelength moved to 6.5 μm14. Since only the 6.7 μm channel of the HIRS/2 instrument probes a similar altitude region of the atmosphere as the MW UTH channel, only the HIRS/2 instrument is considered here. HIRS scans through 56 Earth views, with instrument viewing angles ranging from 0.9° to 49.5° (from nadir), resulting in a total swath width of approximately 2240 km. With a field of view of 1.4° the footprint at the Earth’s surface has a size of 20.4 km at the innermost scan position15.

UTH retrieval

To retrieve UTH from a measured brightness temperature Tb we make use of a linear relationship between the Tb and the logarithm of UTH, which was derived for 6.3 μm Tbs16 and has also been successfully applied to microwave (183.31 ± 1 GHz) Tbs10:

$${\rm{ln}}({\rm{UTH}})=a+b{T}_{b}.$$
(1)

The scaling parameters a and b are typically determined by a linear regression (Fig. 1) using a training data set of atmospheric temperature and humidity profiles. On the one hand, the Tb measured by the satellite instrument is simulated for every training profile using a radiative transfer model. On the other hand, UTH is calculated for each training atmosphere as an average of the vertical profile of relative humidity (RH) in a certain atmospheric layer in the upper-troposphere. The exact position of this layer depends on the chosen definition of UTH. A major requirement for a UTH definition is to fulfil Eq. (1). Note that all quantities in Eq. (1) depend on the viewing angle of the instrument.

Fig. 1
figure1

Linear regression to determine the UTH scaling parameters for two different satellite viewing angles. Logarithm of UTH versus AMSU-B 183.31 ± 1 GHz Tb (a) and HIRS/2 6.7 μm (b) for the nadir view (violet dots) and the most off-nadir view (green dots). Each dot corresponds to one training atmosphere in the 137-level ECMWF data set. The new UTH definition was used to calculate UTH for the training atmospheres. The linear fits to the data are indicated by solid lines.

Fig. 2
figure2

Optimised IWV thresholds for the new UTH definition. Most suitable IWV thresholds for the upper boundary (a) and lower boundary (b) of the UTH layer for AMSU-B (dark blue), MHS (light blue) and HIRS (red) at all instrument viewing angles (0° corresponds to the nadir view). For the final definition the mean of the thresholds determined for HIRS and the MW instruments (orange dashed line) is used.

Fig. 3
figure3

Schematic illustration of the FIDUCEO Microwave UTH CDR processing chain. The processing starts from the FIDUCEO Microwave FCDR and is subdivided into three main parts: pre-screening of pixels (blue), transformation of Tb to UTH (green) and gridding and temporal averaging (red). The processing chain consists of three main branches corresponding to the three core CDR variables Tbfull, Tbfiltered and UTH.

Fig. 4
figure4

Performance of the new UTH retrieval compared to the traditional retrieval. UTH retrieved from AMSU-B nadir 183.31 ± 1 GHz brightness temperatures versus true UTH of all ECMWF training atmospheres for the retrieval with the new UTH definition (a) and the retrieval with the traditional UTH definition based on the VMR Jacobian (b).

Fig. 5
figure5

Retrieval statistics. Retrieval bias (a,c) and standard deviation (b,d) as a function of UTHtrue in absolute units (a,b) and relative units (c,d) for the retrieval with the new definition (solid lines) and with the traditional definition based on the VMR Jacobian (dashed lines). Numbers in the upper right of each panel denote overall biases and standard deviations calculated from bins with UTH ≤ 80% RH.

Fig. 6
figure6

Time series of FIDUCEO UTH and CM-SAF UTH. Area-weighted monthly and tropical mean UTH from the FIDUCEO UTH CDR (a) and the CM-SAF UTH CDR (b) for all satellite missions contained in both CDRs. Black horizontal lines indicate a UTH of 27.5% RH. Shaded areas around the monthly means in (a) indicate the measurement uncertainty (±1 standard uncertainty), coloured horizontal bars indicate the time period covered by each satellite mission.

Fig. 7
figure7

Difference of different CDR versions to the original FIDUCEO UTH CDR. Area-weighted monthly and tropical mean UTH from the FIDUCEO UTH CDR (a, same as in Fig. 6) and differences between UTH from the original FIDUCEO UTH CDR and the version generated with the scaling coefficients of the CM-SAF UTH CDR (b), the version generated using all pixels (c) and the CM-SAF UTH CDR (d). Black horizontal lines indicate a UTH of 27.5% RH in (a) and zero difference in (bd), respectively.

Fig. 8
figure8

Geographical distribution of mean differences UTHFID – UTHCM–SAF. Mean distribution of UTHFID in the tropics from the complete time series of NOAA18 (a) and distribution of mean differences UTHFID – UTHCM–SAF for all satellite missions contained in both CDRs: NOAA15 (b), NOAA16 (c), NOAA17 (d), NOAA18 (e), Metop-A (f) and Metop-B (g).

Fig. 9
figure9

Coverage of the different satellites included in the FIDUCEO UTH CDR. Average number of satellite overpasses per grid cell and month (colour shading) for each satellite mission. Grey areas indicate time periods for which no data is available.

Fig. 10
figure10

Impact of cloud filtering on Tb in the FIDUCEO UTH CDR. Difference between area-weighted tropical monthly means of Tbfull and Tbfiltered for all satellite missions (colors). The black line indicates zero difference. For most of the satellite missions the difference is about 0.2 K, but for F14, NOAA15, NOAA16 and NOAA19 the cloud filtering has a stronger impact on Tb.

Traditionally, UTH is defined as a weighted mean of the RH profile, where the weights are given by the humidity Jacobian for the respective instrument channel. Relation 1 is fulfilled with this traditional definition because the altitude levels weighted strongest by the humidity Jacobian correspond to the atmospheric emission layer, i.e. the atmospheric layer contributing to the measured Tb. This Jacobian-based definition has the disadvantage that radiative transfer simulations have to be performed whenever UTH needs to be calculated for given atmospheric profiles of temperature and humidity. Moreover, even though the UTH channels of MW and IR instruments are sensitive to RH in a very similar altitude range, the exact shape of the corresponding humidity Jacobians are different. The resulting difference in the definitions of IR and MW UTH complicate a synergistic use. Therefore, we adapt the traditional UTH definition for the FIDUCEO UTH CDR by eliminating the humidity Jacobian. This new UTH definition will be explained in the next section. The setup used to determine new scaling parameters a and b is described in the following.

Our training data consist of atmospheric profiles from the 137-level sampled ECMWF data set17, which is compiled from the short-range forcast by the Integrated Forecasting System (IFS) spanning the time period of 1 September 2013 to 31 August 2014. The complete data set consists of five subsets with 5000 profiles each, corresponding to the sampling for a specific geophysical variable. For the UTH regression we choose the subset that focuses on a diverse sampling of atmospheric humidity. For this subset 10% of profiles were selected in such a way that differences in specific humidity between the sampled profiles are maximized. The remaining 90% of profiles were selected randomly to include a realistic amount of frequently occurring atmospheric states. Despite this large proportion of randomly sampled profiles extreme humidity cases are overrepresented in the sampled distribution due to the diverse sampling of humidity. However, since we use the profiles to perform a linear regression, a high variability in humidity is as important as a realistic frequency distribution. Therefore, the dataset is a suitable choice for this application. Since the UTH CDR will be restricted to the tropical region we select all tropical profiles from the 5000 profiles in the subset. This leaves a set of 2812 tropical training profiles.

A line-by-line radiative transfer model, The Atmospheric Radiative Transfer Simulator (ARTS) Version 2.218,19, is used to simulate clear-sky 183.31 ± 1 GHz Tb for all viewing angles of AMSU-B and MHS. As our new definition is supposed to work also for IR UTH retrievals, we additionally simulate 6.7 μm Tb. Required geophysical inputs for the radiative transfer simulations are atmospheric profiles of humidity, temperature, ozone, oxygen and nitrogen as well as the surface skin temperature and the surface emissivity. Humidity, temperature and ozone profiles are taken from the training data set. Volume mixing ratios of nitrogen and oxygen are assumed to be constant throughout the atmosphere. Spectroscopic parameters are taken from the high-resolution transmission molecular absorption database (HITRAN) catalogue20. The skin temperature was assumed to be equal to the lowest atmospheric temperature and surface emissivities of 0.6 and 1 were used for the simulation of the MW instruments and HIRS/2, respectively. Due to the strong absorption of water vapour in the UTH channels the surface emissivity only influences the simulated Tb under extremely dry conditions or above high mountains. These extreme cases are filtered out as explained in the next paragraph. To remove such cases conservatively, we assume a rather low value of 0.6 for the MW surface emissivity, which roughly corresponds to the emissivity of an ocean surface.

Surface-contaminated cases are removed using the same methodology as10: Profiles are discarded if the 183.31 ± 7 GHz Tb of AMSU-B is not higher than the 183.31 ± 1 GHz Tb. Under clear-sky conditions this only occurs when both channels see the surface. For all other cases, the 183.31 ± 7 GHz Tb is higher, because the emission originates from lower tropospheric levels. Due to the similarity of the HIRS and AMSU-B UTH channels, it is assumed that surface contamination in the HIRS channel occurs for the same profiles than in the AMSU-B channel. As expected surface contamination is rare for our tropical training profiles. From 2812 profiles only 12 are filtered out by our algorithm.

The remaining profiles are used to determine the scaling parameters a and b from the linear regression of ln(UTH) against Tb. The regression is shown in Fig. 1 for the nadir view and the most off-nadir view of AMSU-B and HIRS. The linear relation (Eq. (1)) is well fulfilled for both viewing angles, indicating that the retrieval error is small. The retrieval performance is assessed in more detail in the validation section of this paper. Regression coefficients for all viewing angles of AMSU-B and MHS are listed in Table 2.

New definition of UTH

The new definition is based on the concept that the atmospheric emission layer for a water vapour channel is bounded by two characteristic amounts of water vapour integrated from the top of the atmosphere downwards21. Using this idea, we define UTH as the mean RH in a layer between two altitude levels z(IWV1) and z(IWV2), at which the integrated water vapour (IWV) above exceeds two viewing angle dependent thresholds IWV1 and IWV2:

$${{\rm{UTH}}}_{{\rm{new}}}(\theta )=\frac{1}{z(IW{V}_{1}(\theta ))-z(IW{V}_{2}(\theta ))}\underset{z(IW{V}_{2}(\theta ))}{\overset{z(IW{V}_{1}(\theta ))}{\int }}RH(z)dz,$$
(2)

where θ is the satellite viewing angle and RH is defined relative to liquid water. The thresholds IWV1 and IWV2 play a similar role in capturing the atmospheric emission layer as the Jacobian in the traditional definition. Since the emission layer is similar for the MW instruments and the HIRS/2 instrument, it is possible to use the same IWV thresholds and hence the same UTH definition for both instrument types.

The IWV thresholds were optimised in such a way that the linear relationship between the Tb and the logarithm of UTH (Eq. (1)) is best fulfilled for the ECMWF training atmospheres. For the optimisation the above-described linear regression was repeatedly performed with different combinations of IWV thresholds in the UTH definition. For each instrument viewing angle the pair of thresholds used for the regression with the smallest root mean square error was chosen to be the most suitable one.

The optimised IWV thresholds are visualized in Fig. 2 for AMSU-B, MHS and HIRS/2. Both thresholds IWV1 and IWV2 exhibit a dependence on the instrument viewing angle. The thresholds decrease as the viewing direction of the satellite moves away from nadir because the path length through the atmosphere increases and a given IWV along the sensor’s line of sight is reached in higher altitudes. Since the IWV above is always defined along the vertical direction, a higher altitude corresponds to a smaller IWV above. Note that the viewing angle dependence of the IWV thresholds implies that the definition of UTH depends on the satellite viewing angle.

The optimal thresholds for the MW instruments AMSU-B and MHS are identical for nearly all viewing angles. This is not surprising since the UTH channels of the two instruments are designed almost identically. For SSMT-2 the optimisation was not performed separately, but it is assumed that the optimal thresholds are similar to those of AMSU-B and MHS due to the similar instrumental design.

The IWV thresholds for HIRS/2 have similar magnitudes and show the same dependence on the satellite viewing angle as the thresholds for the MW instruments. Therefore, MW and HIRS/2 thresholds are averaged to obtain compromise thresholds, indicated by the dashed line in Fig. 2. The use of these compromise thresholds ensures that the final UTH definition is identical for both the MW sensors and HIRS/2. We will show in the validation section that the performance of the UTH retrieval does not suffer from the new UTH definition, confirming that the use of compromise thresholds is reasonable.

CDR processing

The new UTH scaling parameters a and b were applied in the creation of a new level 3 UTH CDR based on 183.31 ± 1 GHz Tb from the level 1c FIDUCEO Microwave FCDR. The three core variables contained in the FIDUCEO UTH CDR are monthly mean 183.31 ± 1 GHz Tb derived from all pixels (Tbfull) and from cloud-filtered pixels (Tbfiltered), as well as UTH derived from cloud-filtered Tb. The variables are mapped on a regular 1° × 1° latitude-longitude grid covering the tropical region. The CDR also provides measurement uncertainties for all quantities. They are propagated from the underlying FCDR.

The processing chain (Fig. 3) consists of three main branches corresponding to the three core CDR variables Tbfull, Tbfiltered and UTH. In the following, we provide a more detailed description of the input data for the CDR and the CDR processing including the propagation of uncertainty. For this description we concentrate on the branch corresponding to the core variable UTH since it encompasses all important processing steps. These comprise a pre-screening of pixels, the transformation from Tb to UTH as well as spatial and temporal averaging.

Input data: The FIDUCEO Microwave FCDR

The input data for the UTH CDR is provided by the FIDUCEO Microwave FCDR Version 4.11,2. It contains Tbs from all instrument channels along with measurement uncertainties for 11 satellite missions carrying either SSMT-2, AMSU-B or MHS between 1994 and 2017. As for all level 1 products the data are provided on pixel level, i.e. the dimensions of the data correspond to the directions along and across the satellite ground track.

Novelties of the FIDUCEO Microwave FCDR include the use of a metrology inspired measurement-equation approach9 for the re-calibration of brightness temperatures and the estimation of measurement uncertainty. For the letter all influences (effects) on the instrumental calibration that lead to errors in the observed signal are taken into account and the associated uncertainties are estimated and propagated to Tb through the measurement equation. See1 for a more detailed description of this procedure. Measurement uncertainty is split into three classes, depending on the type of the underlying effect and the resulting error correlation:

  • Uncertainties due to independent effects (hereafter independent uncertainties): The underlying effects are close to white noise and generate a completely independent uncertainty from pixel to pixel.

  • Uncertainties due to structured effects (hereafter structured uncertainties): Even though the underlying effects are random, the calibration procedure can lead to correlations between pixels that are located close to each other. The resulting correlations have spatial and temporal scales that are typically smaller than one satellite orbit.

  • Uncertainties due to common effects (hereafter common uncertainties): This class includes all effects with associated correlation scales larger than one orbit (often the whole satellite mission).

Pre-screeing of pixels

In a pre-screening procedure, a subset of pixels is selected from the pixel-level 183.31 ± 1 GHz brightness temperatures for further processing.

First, only pixels located close to the nadir view of the satellite are selected. As pointed out earlier, the UTH definition depends on the satellite viewing angle because the UTH layer shifts to higher altitudes as the viewing direction of the satellite moves away from nadir. Since we do not want to mix UTH values defined in different ways, we select only pixels characterized by uniform IWV thresholds (Fig. 2) and hence a uniform UTH definition. This comes at the expense of spatial and temporal coverage but simplifies comparisons with UTH calculated directly from atmospheric profiles, e.g. from model output or radiosonde measurements. For AMSU-B and MHS IWV thresholds are constant for the innermost 28 pixels of the scan line (14 on both sides of the nadir views). This corresponds to the innermost 10 pixels of SSMT-2, assuming that the IWV thresholds for SSMT-2 are similar to those of AMSU-B and MHS.

Second, pixels of low quality are discarded. The FCDR provides quality information for each pixel in the form of an overall quality bitmask and a channel-specific bitmask. Pixels that are marked as “invalid” by the overall quality bitmask are rejected. This can have several reasons, e.g. invalid geolocation or viewing-geometry of the data, invalid pixel acquisition time or invalid sensor status. For a full list of reasons see the description of the quality bitmasks in22. Pixels are also discarded if the specific bitmask of the 183.31 ± 1 GHz channel indicates that calibration was not possible or there was bad data from the Earth views.

Third, cloud contaminated pixels are removed. UTH can only be derived from measurements of Tb that are not contaminated by clouds. A strong advantage of MW measurements over IR measurements is the fact that clouds are nearly transparent in the MW. However, cold ice clouds do interact with MW radiation and can affect the measurement23. The cloud particles scatter radiation away from the sensor’s line of sight and hence cause a reduction in Tb. A cloud filtering has to be performed before retrieving UTH in order to avoid a positive UTH bias in the climatology due to the erroneous interpretation of cloudy scenes as very moist scenes24. This is done using the method suggested in25, which combines two criteria. The first criterion is a viewing angle dependent threshold on the 183.31 ± 1 GHz Tb (240.1 K for the Nadir view). This threshold is based on simulated clear-sky Tb, which were shown to lie above this value. The second criterion uses the differences between 183.31 ± 1 GHz Tb and 183.31 ± 3 GHz Tb. Under clear-sky conditions, the 183.31 ± 1 GHz Tb is colder than the 183.31 ± 3 GHz Tb, because the former is sensitive to a higher region in the troposphere, where temperatures are generally lower. However, in the presence of ice clouds the 183.31 ± 1 GHz Tb can be warmer than the 183.31 ± 3 GHz Tb. Hence, the difference between the two Tb can be used to detect clouds. Additionally, as shown by25, this difference is also a good filter against surface influence in the 183.31 ± 1 GHz Tb.

Transformation of brightness temperature to UTH

After the pre-screening UTH is calculated for each pixel p from cloud-filtered Tb (Tb,filtered) using Eq. (1) and the scaling parameters a and b derived in the UTH retrieval section:

$$UT{H}_{p}=exp(a+b{T}_{b,{\rm{filtered}}}),$$
(3)

For SSMT-2 the scaling coefficients were not derived separately, but the coefficients of the respective nearest MHS views are used. This is a reasonable simplification since the UTH channels of SSMT-2 and MHS have very similar characteristics.

Pixel aggregation and averaging

After the calculation of pixel-level UTHp, data from different pixels are combined in a two-step process. In the first step all pixels from observations of one day are aggregated in 1° × 1° bins on a regular latitude-longitude grid covering the tropics. Subsequently, the aggregated pixels are averaged to get daily averages of UTH for each day d and each grid cell (UTH1°×1°,d):

$$UT{H}_{{1}^{\circ }\times {1}^{\circ },d}=\frac{1}{N}\mathop{\sum }\limits_{p=1}^{N}UT{H}_{p}.$$
(4)

Here, N is the number of pixels that are aggregated within the 1° × 1° region. In a second step, daily averages are combined to monthly means for every grid cell:

$$UT{H}_{{1}^{\circ }\times {1}^{\circ },m}=\frac{1}{{N}_{d}}\mathop{\sum }\limits_{d=1}^{{N}_{d}}UT{H}_{{1}^{\circ }\times {1}^{\circ },d},$$
(5)

where the index m denotes the month and Nd is the number of daily averages entering the monthly average.

Propagation of uncertainty

The uncertainties of pixel-level brightness temperatures from the FCDR are propagated to the gridded and averaged UTH values in the CDR using the Law of the Propagation of Uncertainties (LPU)26, which yields the uncertainty u of a quantity y, which is determined from m other quantities x1, x2, …, xm through a functional relationship f:

$$u(y)=\sqrt{\mathop{\sum }\limits_{i=1}^{m}{\left(\frac{\partial f}{\partial {x}_{i}}\right)}^{2}u{({x}_{i})}^{2}+2\mathop{\sum }\limits_{i=1}^{m-1}\mathop{\sum }\limits_{j=i+1}^{m}\frac{\partial f}{\partial {x}_{i}}\frac{\partial f}{\partial {x}_{j}}u({x}_{i})u({x}_{j})r({x}_{i},{x}_{j})},$$
(6)

where u(xi) are the uncertainties of the input quantities xi. The partial derivatives of f with respect to the input quantities xi describe the sensitivity of f to changes in xi and are therefore also called sensitivity coefficients. The correlation coefficient r(xi, xj) characterizes the correlation between xi and xj. Hence, positive correlations between the input quantities x increase the uncertainty of the output quantity y. To account for the different error correlation properties of independent, structured and common uncertainties, each class of uncertainty is propagated separately.

In a first step, uncertainties of pixel-level Tb,filtered are propagated to pixel-level UTHp. This unit transformation (Eq. (3)) has only one input quantity Tb,filtered and the LPU (Eq. (6)) reduces to:

$${u}_{c}(UT{H}_{p})=\sqrt{{\left(\frac{\partial UT{H}_{p}}{\partial {T}_{b,{\rm{filtered}}}}\right)}^{2}{u}_{c}{\left({T}_{b,{\rm{filtered}}}\right)}^{2}}=\left|bUT{H}_{p}{u}_{c}({T}_{b,{\rm{filtered}}})\right|,$$
(7)

where the index c denotes the class of uncertainty.

Subsequently, uncertainties are propagated from pixel-level UTH to daily grid cell averages UTH1°×1°,d (Eq. (4)). For this averaging process, the LPU (Eq. (6)) takes the following form:

$${u}_{c}(UT{H}_{{1}^{\circ }\times {1}^{\circ },d})=\frac{1}{N}\sqrt{\mathop{\sum }\limits_{p=1}^{N}{u}_{c}{(UT{H}_{p})}^{2}+2\mathop{\sum }\limits_{p=1}^{N-1}\mathop{\sum }\limits_{p{\prime} =p+1}^{N}{u}_{c}(UT{H}_{p}){u}_{c}(UT{H}_{p{\prime} })r(p,p{\prime} )},$$
(8)

where r(p, p′) is the correlation coefficient of two pixels denoted by p and p′. For independent uncertainties the correlation is zero, so the second term under the square root vanishes. Consequently, in this case the averaging process significantly reduces the uncertainty of the grid cell value compared to the uncertainties of the individual pixel values. If all N pixel uncertainties were equal, the grid cell uncertainty would be reduced by a factor 1/\(\sqrt{N}\). The opposite is true for the common uncertainties. For this class correlations may extend over infinite length and time scales and the correlation coefficient r is one per definition. Thus, averaging of several pixel values does not reduce the grid cell uncertainty. In other words, the grid cell uncertainty is obtained by averaging the pixel uncertainties. For structured uncertainties, correlations extend over a certain number of adjacent scan lines in the satellite swath. This results from an averaging of calibration coefficients over several scan lines. In the case of the Microwave FCDR, calibration coefficients are always averaged over seven scan lines22, so the overall length scale over which the correlation decreases to zero is seven scan lines. The FCDR contains so-called correlation vectors ρ, providing the correlation coefficient of two pixels p and p′ as a function of the difference between the scan lines l of these pixels |lp − lp|. Using these correlation vectors ρ the correlation coefficient r in Eq. (8) can be written as

$$r(p,p{\prime} )=\rho \left(\left|{l}_{p}-{l}_{p{\prime} }\right|\right).$$
(9)

The uncertainty propagation from daily averages UTH1°×1°,d to the monthly average UTH1°×1°,m (Eq. (5)) is performed in a very similar way, under the assumption that there is no temporal correlation (r = 0) for independent and structured uncertainties and full temporal correlation (r = 1) for common uncertainties.

Error correlations between the pixel-level input quantities also result in correlations between the final grid cell averages. These “inter-grid cell correlations” are not included in the CDR. However, they must be taken into account in further spatial or temporal averaging and the associated uncertainty propagation should be performed by the CDR user. For independent and common uncertainties the correlations among grid cells behave in the same way as those among pixels; for independent uncertainties there are no correlations, whereas for common uncertainties all grid cells are fully correlated. For structured uncertainties, the correlation structure is more complex and a complete propagation that also yields the covariances of the grid cell averages has to be performed. For one thing, however, this requires large covariance matrices and hence much computational power, and for another, it is a complex procedure to take into account non-uniform inter-grid cell correlation patterns for further propagation of uncertainties. Therefore, we recommend to treat the structured uncertainties in the same way as common uncertainties in an averaging process. This approach provides an upper limit for the structured uncertainty of the average.

Data Records

The FIDUCEO Microwave UTH CDR version 1.227 is freely available from the Centre for Environmental Data Analysis (CEDA) Archive.

Description of data files

The CDR data files are written in NetCDF-4 format, implementing the common CDR file format that has been defined within the FIDUCEO project. Each file contains data for one month and one satellite mission and has a size of about 3.9 MB, resulting in a total CDR size of about 4.0 GB. The filenames follow the FIDUCEO standard and have the following structure:

FIDUCEO_CDR_UTH_{INSTRUMENT}_{SATELLITE}

_{STARTTIME}_{ENDTIME}_L3_v{CDR-VERSION}

_fv{WRITER-VERSION}.nc

where {INSTRUMENT} can be either SSMT2, AMSUB or MHS. {SATELLITE} can be any of the satellites these instruments are flying on. {STARTTIME} is the first second of the first day in the month, with the format year-month-day-hour-minute-second, {ENDTIME} is the last second of the last day in the month. {CDR-VERSION} denotes the version number of the CDR and {WRITER-VERSION} is the version of the NetCDF-writer used to generate the NetCDF files.

Each data file consists of global attributes with general information and a set of data variables with individual attributes. The global attributes provide information on the version of the CDR and the time period covered by the data file. Moreover, they contain the Digital Object Identifier (DOI) of the data set. In order to sustain traceability, a list of file names of all FCDR files that were used to generate the CDR file is also included in the global attributes.

The following variables are contained in each file:

  • lon, lat: Longitudes and latitudes of grid cell centres;

  • BT_full, BT_full_inhomogeneity: Monthly average and standard deviation of 183.31 ± 1 GHz Tb from all available pixels (including cloudy pixels);

  • BT, BT_inhomogeneity: Monthly average and standard deviation of 183.31 ± 1 GHz Tb from all pixels used to derive UTH (excluding cloudy pixels);

  • uth, uth_inhomogeneity: Monthly average and standard deviation of UTH;

  • u_independent, u_structured, u_common: Independent, structured and common uncertainty for monthly averages of Tb and UTH (denoted by suffixes _BT and _uth, respectively);

  • observation_count: Number of pixels entering the monthly grid cell average;

  • overpass_count: Number of satellite overpasses contributing to the monthly grid cell average;

  • time_ranges Earliest and latest time of day of pixel contribution to the monthly average;

    All monthly fields are split into two parts: One from ascending satellite overpasses and one from descending overpasses, indicated by the suffixes _ascending and _descending, respectively, in the variable names. For each variable several attributes are provided. They contain a description of the quantity as well as information on the unit and the dimensions of the variable. Existing dimensions are:

  • x - East-west dimension (size: 360);

  • y - North-south dimension (size: 61);

  • bounds - Dimension defining lower and upper bounds (size: 2);

Another attribute is the fill value, which is placed whenever there are data gaps due to missing FCDR data or due to cloud coverage.

Covered satellite missions and time periods

Similar to the underlying Microwave FCDR, the FIDUCEO UTH CDR covers the time period from 1994 to 2017 and consists of 11 partly overlapping satellite missions carrying either SSMT-2, AMSU-B or MHS. An overview of all included satellite missions and the corresponding time periods is provided in Table 3.

Technical Validation

Performance of the UTH retrieval

To evaluate the performance of the UTH retrieval with the new UTH definition (new retrieval) it is compared to the performance of a retrieval with the traditional definition based on the fractional water vapour mixing ratio Jacobian as in10 (traditional retrieval). We determined the UTH scaling parameters a and b for the traditional retrieval based on the same ECMWF training data set we used for the new retrieval.

To give an impression of the retrieval performance Fig. 4 shows the retrieved UTH (UTHretrieved) from AMSU-B Tb versus the true UTH (UTHtrue) for all ECMWF training atmospheres for both retrievals. The scatter of the data points around the one to one line seems to be approximately equal for both retrievals, indicating that the retrieval accuracy does not suffer from the new definition.

To assess the retrieval performance in more detail, the difference ΔUTH between UTHretrieved and UTHtrue is calculated for all training atmospheres:

$$\Delta UTH=UT{H}_{{\rm{retrieved}}}-UT{H}_{{\rm{true}}}.$$
(10)

Following10 we define the retrieval bias as the mean of ΔUTH of all training atmospheres (\(\bar{\Delta UTH}\)) and the retrieval standard deviation as the standard deviation of ΔUTH (σΔUTH). Relative retrieval bias (\(\bar{\Delta UT{H}_{{\rm{r}}{\rm{e}}{\rm{l}}}}\)) and relative retrieval standard deviation (\({\sigma }_{\Delta UT{H}_{{\rm{rel}}}}\)) are defined similarly, but based on the relative difference between UTHfitted and UTHtrue:

$$\Delta UT{H}_{rel}=\frac{UT{H}_{{\rm{retrieved}}}-UT{H}_{{\rm{true}}}}{UT{H}_{{\rm{true}}}}.$$
(11)

We calculate retrieval statistics by aggregating the data points in 10% RH bins of UTHtrue values. Absolute and relative retrieval bias and standard deviation as a function of UTHtrue are shown in Fig. 5. Overall retrieval bias and standard deviation were calculated only from profiles with UTH ≤ 80% RH since higher values of RH with respect to water hardly occur in nature28 and seem to be a particularity of the ECMWF data set.

Using the new definition, the absolute retrieval bias fluctuates between −3% and 1% for UTH values below 40% RH and increases in magnitude to about −3.5% for UTH values of 70–80% RH. The relative bias is between 0.1% and −4% in all UTH bins below 80% RH. These biases are slightly larger than for the retrieval with the traditional definition for UTH values above 40% RH and slightly smaller for UTH values below 40% RH. With the new definition the absolute retrieval standard deviation in the 0–10% UTH bin is approximately 1% RH and continuously increases to approximately 6% RH at 70–80% UTH. In relative units, the retrieval standard deviation decreases from about 12–13% at UTH values below 20% RH to about 6–8% for UTH values above 40% RH. Compared to the retrieval with the traditional definition, standard deviations are generally lower for UTH below 60% RH and slightly higher for UTH above 60% RH.

For the new retrieval the overall absolute (relative) retrieval standard deviation is 2.9% RH (9.5%), for the traditional retrieval it is 4.2% RH (14.4%). The overall absolute (relative) retrieval bias is −0.6% RH (−1.5%) for the new retrieval and 0.3% RH (0.7%) for the traditional one. Hence, in terms of overall standard deviation the performance of the new retrieval has slightly improved with respect to the traditional one, in terms of overall retrieval bias it has slightly worsened. It is important to note here that the newest version of the CM-SAF UTH CDR, which we will use for comparison in the next section, uses an updated UTH definition based on the RH Jacobian instead of the VMR Jacobian. With the RH Jacobian the retrieval standard deviation is reduced compared to the VMR Jacobian and is similar as with the new definition29. We conclude that the retrieval performance does not suffer from our new UTH definition compared to the two traditional versions used in previous datasets.

Comparison to the CM-SAF UTH CDR

To validate the FIDUCEO UTH CDR, it is compared to the Microwave UTH data record provided by the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) Satellite Application Facility on Climate Monitoring (CM-SAF)30, hereafter called CM-SAF CDR. We chose the comparison to an existing UTH data record over a comparison with in-situ measurements from radiosondes for two main reasons. Firstly, humidity measurements from many types of radiosondes are subject to significant biases in the upper troposphere31,32 and these biases strongly depend on the sensor type33. Hence, combining different sensor types, which would be required to get a sufficient temporal and spatial coverage for the validation of a UTH CDR, is problematic. Even if the quality of the radiosonde data is good, direct comparisons of satellite and radiosonde measurements of water vapour remain difficult due to their different nature and different measurement scales34, which has lead to a large spread in bias results35. Secondly, it is of particular interest how the new aspects of our CDR with respect to existing data records are reflected in UTH. Our CDR differs from the CM-SAF CDR in the underlying FCDR and several aspects of processing, including the UTH scaling parameters (due to a different UTH definition) and the pixel selection.

The CM-SAF UTH CDR is based on a microwave humidity sounder FCDR generated by EUMETSAT within the framework of the European Reanalysis of Global Climate Observations 2 (ERA-Clim2) project. The approach used to reduce inter-satellite biases in this FCDR is a bias-correction based on “Observation minus Background” (O-B) statistics from the ERA-Interim reanalysis and hence differs from the measurement equation based recalibration approach used for the FIDUCEO FCDR. Thus, it can be expected that inter-satellite biases differ between our new CDR and the CM-SAF CDR. To determine the scaling coefficients for the UTH retrieval in the CM-SAF CDR a similar linear regression approach as in the FIDUCEO CDR was used, but UTH was defined in the conventional way as the RH profile weighted with the RH Jacobian. Cloud-contaminated measurements were determined and discarded equally as in the FIDUCEO CDR using the method suggested by25. Instead of selecting only pixels close to the nadir view of the satellite as for the FIDUCEO CDR, all pixels were used for the CM-SAF CDR. As discussed in the section on CDR processing this improves the sampling but results in a mixing of information about UTH from different altitude layers. For the CM-SAF CDR spatial averaging was performed in the same way as for the FIDUCEO CDR: UTH was separated into ascending and descending passes and then binned into 1° × 1° grid cells and averaged. For the comparison of the two CDRs, we calculate monthly averages of UTH from ascending and descending nodes and average these to get one combined monthly average.

When comparing UTH from the FIDUCEO CDR (UTHFID) and from the CM-SAF CDR (UTHCMSAF), we try to trace back the emerging differences to the above-named differences in the underlying FCDRs and in the processing of the two data records. For this purpose, we generated two additional versions of the FIDUCEO UTH CDR by changing aspects of the processing with respect to the original processing chain described in the section on CDR processing. For the first version, the UTH scaling parameters are changed to resemble the parameters used in the CM-SAF UTH CDR. Comparing the resultant UTH (UTHCMSAFcoeff) to UTHFID reveals the effect of the different scaling parameters used for the FIDUCEO CDR (resulting from the new UTH definition). For the second version, the scaling parameters as well as the pixel selection of the CM-SAF CDR are imitated by including all pixels instead of using only near-nadir pixels. The UTH from this CDR version (UTHCMSAFsampling) is used to investigate the combined effect of the new scaling coefficients and the near-nadir only sampling we apply in the FIDUCEO CDR. Differences between the FIDUCEO CDR and the CM-SAF CDR that cannot be explained with the different scaling coefficients or the different sampling are a result of differences in the underlying FCDRs.

Figure 6a shows time series of monthly mean tropical mean UTHFID for all satellite missions that are also included in the CM-SAF CDR: NOAA15, NOAA16, NOAA17, NOAA18, Metop-A and Metop-B. Time series of overlapping satellite missions agree within their uncertainties, which are indicated by the shaded areas around the monthly means. The time series of UTHCMSAF is shown in Fig. 6b. Overall, there is good agreement between tropical mean UTHFID and UTHCMSAF, confirming the validity of our method used to derive UTH. The difference plot in Fig. 7d shows that the absolute difference between tropical mean UTHFID and UTHCMSAF does not exceed 2% RH, except for the first months of the NOAA15 mission. For most satellite missions, UTHFID tends to be about 1.6% RH lower than UTHCMSAF. Exceptions are NOAA15, for which UTHFID is about 3% RH lower than UTHCMSAF in the beginning of the mission and the difference is close to zero for the rest of the mission, as well as NOAA17, for which UTHFID is about 0.8% RH lower than UTHCMSAF.

Inter-satellite biases differ in their magnitudes and signs between the FIDUCEO CDR and the CM-SAF CDR. For example, in the FIDUCEO CDR UTH from NOAA16 is systematically lower than UTH from NOAA17, whereas there is no bias between these two satellite missions in the CM-SAF CDR. In contrast, in our CDR UTH derived from NOAA17 and NOAA18 agree well, while UTH from NOAA17 is lower than UTH from NOAA18 in the CM-SAF CDR.

The difference between UTHFID and UTHCMSAFcoeff (Fig. 7b) is approximately 1.6% RH over all satellite missions, indicating that the use of our new UTH scaling coefficients offsets the monthly mean UTH by a positive, approximately constant value. The offset is slightly larger in months with high UTH than in months with low UTH, but these variations are small (smaller than 0.2% RH) compared to the mean offset.

The comparison of UTHFID and UTHCMSAFsampling (Fig. 7c) shows that near-nadir-only sampling affects the individual satellite missions in different ways, which can in part explain the differing inter-satellite biases in our CDR with respect to the CM-SAF CDR. For most of the missions the difference between UTHFID and UTHCMSAFsampling fluctuates around a mean negative value, indicating that the selection of only near-nadir pixels leads to slightly lower UTH values. This is consistent with the climatological C-shape of the tropical RH profile36, which implies slightly higher RH for off-nadir pixels that sample a slightly higher altitude.

More surprisingly, the mean difference between UTHFID and UTHCMSAFsampling is not uniform among satellite missions, particularly among the NOAA missions. Thus, inter-satellite biases change with a different selection of scan positions and therefore have to be scan-dependent. For NOAA15 and NOAA16 these scan-dependent biases additionally seem to be time dependent. When only near-nadir pixels are selected, for NOAA15 and NOAA17 the decrease in UTH is smaller than for the later instruments including NOAA18, Metop-A and Metop-B. In contrast, for NOAA16 the decrease in UTH is stronger and increases over time. Combined, the weaker decrease of NOAA17 UTH and the stronger decrease of NOAA16 UTH due to the near-nadir pixel selection result in a larger bias between NOAA16 and NOAA17 in UTHFID than in UTHCMSAF. The instruments onboard NOAA15, NOAA16 and NOAA17 are known to suffer from radio frequency interference (RFI) from transmitters on-board the satellite37,38,39. This can explain the emerging scan- and time-dependent biases, since the effect of RFI is both scan- and time-dependent. An RFI correction was applied in the FIDUCEO FCDR to all three instruments39, which was shown to improve the consistency between instruments. However, for NOAA15 and NOAA17 only an early version of the correction scheme could be applied due to the lack of a reference month that is not affected by RFI. Moreover, the effect of the RFI correction has only been tested on the basis of Tb averaged over all scan positions. Thus, it is likely that scan- and time-dependent biases due to RFI still exist.

Figure 8 shows the geographical distribution of the differences between UTHFID and UTHCMSAF. Differences are not distributed uniformly over the tropics. The amplitude of their spatial variations is on the order of 2% RH and hence of similar magnitude as the tropical mean differences (Fig. 7d). A main reason for the spatial variations in the differences lies in the different sampling applied in the production of the FIDUCEO CDR. This is confirmed by the fact that the same spatial patterns are visible in the difference between UTHFID and UTHCMSAFsampling (not shown). The near-nadir-only sampling of FIDUCEO results in a much lower number of observations in each grid cell and rarely occurring extreme events in certain regions that are captured by the CM-SAF CDR might be missing in the FIDUCEO CDR. Due to the non-linear relation between Tb and UTH this can translate into biases between FIDUCEO UTH and CM-SAF UTH.

There is a tendency of biases to be more negative in regions of climatologically high UTH, like the deep convective regions in the Inter Tropical Convergence Zone (ITCZ) and less negative in regions of low UTH, like the subtropical subsidence regions. This is reflected in a weak seasonal dependence of the tropical mean difference between UTHFID and UTHCMSAF, which is apparent in Fig. 7d. Differences are more negative in months with a high average UTH and vice versa. The amplitude of these seasonal fluctuations in the difference is about 0.2% RH and hence small compared to the average difference.

To put the biases between UTHFID and UTHCMSAF into perspective, we can compare them to the state-of-the-art accuracy of space-borne UTH observations. It is impossible to define this accuracy precisely. However, we can take the difference between UTH derived from microwave and infrared sensors as an estimate. For the infrared instrument AIRS and the microwave instrument AMSU-B40 find a mean bias of about 3% RH between 60° S and 60° N (their Table 1) and regional biases on the order of ±10% RH (their Fig. 1). The biases between UTHFID and UTHCMSAF are about half as large. However, for two products from the same sensor type biases should of course be smaller than between products from different sensor types. Therefore, we overall judge the agreement between UTHFID and UTHCMSAF to be in line with expectations.

Our analysis has shown that part of the differences between UTHFID and UTHCMSAF can be explained by the differences in the CDR processing (scaling coefficients and selection of pixels). However, there are additional differences in the inter-satellite biases (e.g. larger biases between NOAA15 and NOAA16 as well as between NOAA17 and NOAA18 in the CM-SAF CDR) originating from differences in the underlying FCDRs. In summary, the underlying FCDRs and the CDR processing each explain about half of the total difference between UTHFID and UTHCMSAF.

Limitations: sources of uncertainty not included in the CDR

A major accomplishment of the FIDUCEO UTH CDR is the fact that it provides estimates of measurement uncertainty for all contained quantities. However, additional uncertainties arise with the level 2 and level 3 processing. They are not included in the CDR since a full understanding and quantification of each uncertainty requires thorough investigations, which should be part of future work. This section provides an overview of these additional uncertainties as well as rough estimates of their magnitude and recommendations for the CDR analysis. A more detailed discussion can be found in the Product User Guide (PUG) provided along with the UTH CDR.

Additional uncertainties at level 2

Uncertainties arising at level 2 are associated with the exponential model used for the transformation of Tb to UTH (Eq. (1)), because there are deviations from this exponential relationship in the real atmosphere. As a rough estimate of the resulting uncertainty in UTH one can use the retrieval standard deviation σΔUTH, apparent as the spread around the identity line in Fig. 4. The overall retrieval standard deviation σΔUTH is is 2.9% RH for UTH ≤ 80% RH. For a more accurate estimation one should take into account that the magnitude of σΔUTH varies over the possible range of UTH values (Fig. 5) and the uncertainty estimate should hence depend on UTH itself. Note that using this uncertainty estimate for monthly grid cell averages of UTH (level 3 UTH) represents a simplified approach since several instantaneous UTH values with different uncertainties entered these averages.

Additional uncertainties at level 3

Uncertainties in the monthly averages of Tb and UTH arise because they are estimated from only a few satellite overpasses. A grid cell in the tropics is typically observed by about 12 to 14 satellite overpasses per month for the newer AMSU-B and MHS missions (Fig. 9). In the earlier SSMT-2 missions (F11, F12, F14, F15), however, large data gaps result in a poorer sampling. There are months with less than two average satellite overpasses per grid cell in all four missions. Hence, the time period before 1999, in which only SSMT-2 observations are available, should be used very cautiously. A related uncertainty results from the fact that a satellite always observes a given point on Earth at the same local time due to its sun-synchronous orbit and therefore always observes the same phase of the diurnal cycle. Thus, the monthly averages derived from these observations are only valid for a certain time of day. The resulting uncertainty in the monthly average depends on the amplitude of the diurnal cycle of UTH in the considered grid cell41,42. found diurnal amplitudes on the order of 1% RH over ocean and 4% RH over land regions using observations from geostationary satellites. These amplitudes can be taken as a rough estimate for the uncertainty due to the diurnal cycle. Estimating the uncertainty more precisely or even correcting for the diurnal cycle would require an exact knowledge of the temporal course of the diurnal cycle. In order to improve the sampling and to get the best estimate of the true monthly average, measurements from ascending and descending satellite overpasses as well as measurements from all available satellite missions with different equator-crossing times should always be combined.

Uncertainties in the long-term trend of Tb and UTH can result from a drift in the satellite orbit. With the exception of Metop-A and Metop-B, which are actively stabilised, all satellites included in the UTH CDR are subject to an orbit drift, which is reflected in a changing equator-crossing time over the course of the satellite mission. As a consequence, the observed phase of the diurnal cycle changes. Such an aliasing of the diurnal cycle can lead to artificial trends when long time scales are analysed. A comparison of the Tbs measured by the stabilised satellite Metop-A and the drifting satellite NOAA18 (Fig. 6) suggests that such artificial trends are small when ascending and descending satellite overpasses are combined to one time series.

Another problem arising at level 3 is a dry bias (or “clear-sky” bias) in UTH25,43 resulting from the cloud filtering, which systematically removes moist pixels. To illustrate this bias in the FIDUCEO UTH CDR, the difference between monthly tropical mean Tbfull and Tbfiltered is shown in Fig. 10. For most missions Tbfiltered is about 0.2 K warmer than Tbfull, corresponding to a fractional bias of about −2% in tropical mean UTH. This only represents an upper limit for the dry bias, because the cloud contaminated pixels included in Tbfull appear colder (moister) than they actually are. More importantly, however, Fig. 10 reveals that for some instruments the difference between Tbfull and Tbfiltered is significantly larger than 0.2 K. Affected instruments are SSMT-2 on F14 (after 2001), AMSU-B on NOAA15 (whole mission), AMSU-B on NOAA16 (after 2006) and MHS on NOAA19 (whole mission). Those missions are affected by strong instrumental noise44, which leads to a broader distribution of measured Tbs. For those instruments, cutting off Tbs below a certain threshold during the cloud filtering leads to a larger change in mean Tb than for instruments with less noise. In order to avoid time dependent biases when a climatological time series is created, the affected time periods should be excluded.

A time series of tropical mean UTH created following the guidelines given above (i.e. combining measurements from ascending and descending overpasses and from overlapping satellite missions as well as excluding time periods affected by strong instrument noise) has shown good agreement with a UTH time series from HIRS measurements45 except from the first six years of the data record, where data coverage is poor.

Code availability

The code used for the processing of the FIDUCEO Microwave UTH CDR is available on GitHub (https://github.com/FIDUCEO/CDR_UTH).

References

  1. 1.

    Hans, I. et al. An uncertainty quantified fundamental climate data record for microwave humidity sounders. Remote Sens. 11(5), 548 (2019).

    ADS  Article  Google Scholar 

  2. 2.

    Burgdorf, M., Hans, I. & Woolliams, E. R. FIDUCEO: Fundamental Climate Data Record of Microwave Brightness Temperatures with uncertainties, 1994–2017, v4.1. Centre for Environmental Data Analysis https://doi.org/10.5285/a8e9f44965434f3b861eba77688701ef (2019).

  3. 3.

    Pierrehumbert, R. T., Brogniez, H. & Roca, R. In The General Circulation (eds Schneider, T. & Sobel, A.) On the relative humitidy of the atmosphere (Princeton University Press, 2007).

  4. 4.

    Held, I. M. & Soden, B. J. Water vapor feedback and global warming. Annu. Rev. Energy Env. 25(1), 441–475 (2000).

    Article  Google Scholar 

  5. 5.

    Soden, B. J. & Held, I. M. An assessment of climate feedbacks in coupled ocean–atmosphere models. J. Clim. 19(14), 3354–3360 (2006).

    ADS  Article  Google Scholar 

  6. 6.

    de F. Foster, P. M. & Collins, M. Quantifying the water vapour feedback associated with post-Pinatubo global cooling. Clim. Dyn. 23(2), 207–214 (2004).

    Article  Google Scholar 

  7. 7.

    Hollmann, R. et al. The ESA climate change initiative: satellite data records for essential climate variables. Bull. Am. Meteorol. Soc. 94(10), 1541–1552 (2013).

    ADS  Article  Google Scholar 

  8. 8.

    Merchant, C. J. et al. Uncertainty information in climate data records from Earth observation. Earth Syst. Sci. Data 9(2), 511–527 (2017).

    ADS  Article  Google Scholar 

  9. 9.

    Mittaz, J., Merchant, C. J. & Woolliams, E. R. Applying principles of metrology to historical Earth observations from satellites. Metrologia 56(3), 032002 (2019).

    ADS  Article  Google Scholar 

  10. 10.

    Buehler, S. A. & John, V. O. A simple method to relate microwave radiances to upper tropospheric humidity. J. Geophys. Res. 110, D02110 (2005).

    ADS  Google Scholar 

  11. 11.

    Robel, J., Graumann, A. & Kidwell, K. NOAA KLM user’s guide with NOAA-N,-N’ supplement. National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Service, National Climatic Data Center, Remote Sensing and Applications Division (2009).

  12. 12.

    Galin, I., Dennis, H. & Martner, G. R. In Proceedings SPIE Vol. 1935 Microwave Instrumentation for Remote Sensing of the Earth (ed. Shiue, J. C.) The DMSP SSM/T-2 microwave water-vapor profiler (SPIE, 1993).

  13. 13.

    Koenig, E. W. In Remote Sensing of Atmospheres and Oceans (ed. Deepak, A.) Performance of the HIRS/2 instrument on TIROS-N. 67–93 (Elsevier, 2012).

  14. 14.

    Shi, L. & Bates, J. J. Three decades of intersatellite-calibrated High-Resolution Infrared Radiation Sounder upper tropospheric water vapor. J. Geophys. Res. 116, D04108 (2011).

    ADS  Google Scholar 

  15. 15.

    WMO OSCAR - Observing Systems Capability Analysis and Review Tool, https://www.wmo-sat.info/oscar/instruments/view/182 (2016).

  16. 16.

    Soden, B. J. & Bretherton, F. P. Upper tropospheric relative humidity from the GOES 6.7 μm channel method and climatology for July 1987. J. Geophys. Res. 98(D9), 16669–16688 (1993).

    ADS  Article  Google Scholar 

  17. 17.

    Eresmaa, R. & McNally, A. P. Diverse profile datasets from the ECMWF 137-level short-range forecasts. NWP SAF Report No. NWPSAF-EC-TR-017, European Centre for Medium-range Weather Forecasts (2014).

  18. 18.

    Buehler, S. A., Eriksson, P., Kuhn, T., von Engeln, A. & Verdes, C. ARTS, the Atmospheric Radiative Transfer Simulator. J. Quant. Spectrosc. Radiat. Transfer 91(1), 65–93 (2005).

    ADS  CAS  Article  Google Scholar 

  19. 19.

    Buehler, S. A. et al. ARTS, the atmospheric radiative transfer simulator - version 2.2, the planetary toolbox edition. Geosci. Model Dev. 11(4), 1537–1556 (2018).

    ADS  CAS  Article  Google Scholar 

  20. 20.

    Rothmann, L. S. et al. The HITRAN2012 molecular spectroscopic database. J. Quant. Spectrosc. Radiat. Transfer 130, 4–50 (2012).

    ADS  Article  Google Scholar 

  21. 21.

    Wu, X., Bates, J. J. & Khalsa, S. J. S. A climatology of the water vapor band brightness temperature from NOAA operational satellites. J. Clim. 6(7), 1282–1300 (1993).

    ADS  Article  Google Scholar 

  22. 22.

    Hans, I. Towards a new fundamental climate data record for microwave humidity sounders based on metrological best practice. PhD thesis, Universität Hamburg (2018).

  23. 23.

    Greenwald, T. J. & Christopher, S. A. Effect of cold clouds on satellite measurements near 183 GHz. J. Geophys. Res. 107(D13), AAC3.1–AAC3.8 (2002).

    Google Scholar 

  24. 24.

    Lanzante, J. R. & Gahrs, G. E. The “clear-sky bias” of TOVS upper-tropospheric humidity. J. Clim. 13(22), 4034–4041 (2000).

    ADS  Article  Google Scholar 

  25. 25.

    Buehler, S. A. et al. A cloud filtering method for microwave upper tropospheric humidity measurements. Atmos. Chem. Phys. 7(21), 5531–5542 (2007).

    ADS  CAS  Article  Google Scholar 

  26. 26.

    Working Group 1 of the Joint Committee for Guides in Metrology (JCGM). Evaluation of measurement data - Guide to the expression of uncertainty in measurement JCGM 100:2008, http://www.bipm.org/utils/common/documents/jcgm/JCGM_100_2008_E.pdf (2008).

  27. 27.

    Lang, T., Buehler, S. A., Burgdorf, M. & Hans, I. FIDUCEO: Microwave Upper Troposheric Humidity and Uncertainties, Climate Data Record, 1994–2017, V1.2. Centre for Environmental Data Analysis, https://doi.org/10.5285/2083b33b5c3d4cf0acb9a49226789caa (2019).

  28. 28.

    Buehler, S. A. et al. An upper tropospheric humidity data set from operational satellite microwave data. J. Geophys. Res. 113, D14110 (2008).

    ADS  Article  Google Scholar 

  29. 29.

    John, V. O., Tsamalis, C. & Good, L. Algorithm Theoretical Basis Document UTH Edition 1.0 - Upper Tropospheric Humidity TCDR CM-14711. Satellite Application Facility on Climate Monitoring (2016).

  30. 30.

    Tsamalis, C. et al. CM SAF Microwave Upper Tropospheric Humidity (UTH) Data Record - Edition 1. Satellite Application Facility on Climate Monitoring, https://doi.org/10.5676/EUM_SAF_CM/UTH/V001 (2019).

  31. 31.

    Elliott, W. P. & Gaffen, D. J. On the utility of radiosonde humidity archives for climate studies. Bull. Am. Meteorol. Soc. 72(10), 1507–1520 (1991).

    ADS  Article  Google Scholar 

  32. 32.

    Miloshevich, L. M., Vömel, H., Paukkunen, A., Heymsfield, A. J. & Oltmans, S. J. Characterization and correlation of relative humidity measurements from Vaisala RS80-A Radiosondes at Cold Temperatures. J. Atmos. Oceanic Technol. 18(2), 135–156 (2001).

    ADS  Article  Google Scholar 

  33. 33.

    Moradi, I., Buehler, S. A., John, V. O., Reale, A. & Ferraro, R. R. Evaluating instrumental inhomogeneities in global radiosonde upper tropospheric humidity data using microwave satellite data. IEEE Trans. Geosci. Remote Sens. 51(6), 3615–3624 (2013).

    ADS  Article  Google Scholar 

  34. 34.

    Bobryshev, A., Buehler, S. A., John, V. O., Brath, M. & Brogniez, H. Is there really a closure gap between 181.31-GHz satellite passive microwave and in situ radiosonde water vapour measurements? IEEE Trans. Geosci. Remote Sens. 56(5), 2904–2910 (2018).

    ADS  Article  Google Scholar 

  35. 35.

    Brogniez, H. et al. A review of sources of systematic errors and uncertainties in observations and simulations at 183 GHz. Atmos. Meas. Tech. 9(5), 2207–2221 (2016).

    CAS  Article  Google Scholar 

  36. 36.

    Romps, D. M. An Analytical Model for Tropical Relative Humidity. J. Climate 27(19), 7432–7449 (2004).

    ADS  Article  Google Scholar 

  37. 37.

    Atkinson, N. C. Calibration, monitoring and validation of AMSU-B. Adv. Space Res. 28(1), 117–126 (2001).

    ADS  Article  Google Scholar 

  38. 38.

    John, V. O., Holl, G., Atkinson, N. & Buehler, S. A. Monitoring scan asymmetry of microwave humidity sounding channels using simultaneous all angle collocations (SAACs). JGA Atm. 118(3), 1536–1545 (2013).

    Google Scholar 

  39. 39.

    Hans, I., Burgdorf, M. & Buehler, S. A. Onboard radio frequency interference as the origin of inter-satellite biases for microwave humidity sounders. Remote Sens. 11(7), 866 (2019).

    ADS  Article  Google Scholar 

  40. 40.

    Milz, M., Buehler, S. A. & John, V. O. Comparison of AIRS and AMSU-B monthly mean estimates of upper tropospheric humidity. Geophys. Res. Lett. 36(10), L10804 (2009).

    ADS  Article  Google Scholar 

  41. 41.

    Tian, B., Soden, B. J. & Wu, X. Diurnal cycle of convection, clouds, and water vapor in the tropical upper troposphere: Satellites versus a general circulation model. J. Geophys. Res. 109, D10101 (2004).

    ADS  Article  Google Scholar 

  42. 42.

    Chung, E. S., Soden, B. J., Schmetz, J. & Koenig, M. Diurnal variation of upper tropospheric humidity and its relations to convective activities over tropical Africa. Atmos. Chem. Phys. 7(10), 2489–2502 (2007).

    ADS  CAS  Article  Google Scholar 

  43. 43.

    John, V. O. et al. Clear-sky biases in satellite infrared estimates of upper tropospheric humidity and its trends. J. Geophys. Res. 116, D14108 (2011).

    ADS  Article  Google Scholar 

  44. 44.

    Hans, I., Burgdorf, M., John, V. O., Mittaz, J. & Buehler, S. A. Noise performance of microwave humidity sounders over their lifetime. Atmos. Meas. Tech. 10(12), 4927–4945 (2017).

    Article  Google Scholar 

  45. 45.

    Shi, L., Schreck, C. J. & Schröder, M. Satellite-Derived Upper Tropospheric Humidity Datasets and Comparison with Total Column Water Vapor. The 22nd International TOVS Study Conferences, http://cimss.ssec.wisc.edu/itwg/itsc/itsc22/posters/6p.11.shi.pdf (2019).

Download references

Acknowledgements

The CDR was created within the project Fidelity and Uncertainty in Climate data records from Earth Observation (FIDUCEO, www.fiduceo.eu), which received funding from the European Union’s Horizon 2020 Programme for Research and Innovation, under Grant Agreement no. 638822. The contributions by M.B. and I.H. were funded by FIDUCEO. This work was a contribution to the Cluster of Excellence’CLICCS - Climate, Climatic Change, and Society’, contribution to the Center for Earth System Research and Sustainability (CEN) of Universität Hamburg. ARTS set-ups to calculate the brightness temperatures and Jacobians were provided by Oleksandr Bobryshev and V.J. The NetCDF writer used to create the data files was provided by Tom Block.

Author information

Affiliations

Authors

Contributions

T.L. developed the code for the CDR processing, carried out the analysis and edited the manuscript. S.B. accompanied and advised the investigation, provided the idea for the new definition of UTH, contributed to the establishement of the CDR processing procedure and co-edited the paper. M.B. provided technical information on the instruments, helped interpreting the validation results and co-edited the manuscript. I.H. contibuted to the validation as well as the development of the processing procedure, in particular the procedure for uncertainty propagation, and co-edited the paper. V.J. co-edited the manuscript, contributed to the validation and was involved in the discussion.

Corresponding author

Correspondence to Theresa Lang.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

The Creative Commons Public Domain Dedication waiver http://creativecommons.org/publicdomain/zero/1.0/ applies to the metadata files associated with this article.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lang, T., Buehler, S.A., Burgdorf, M. et al. A new climate data record of upper-tropospheric humidity from microwave observations. Sci Data 7, 218 (2020). https://doi.org/10.1038/s41597-020-0560-1

Download citation

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing