Background & Summary

Urban ecosystems are the biggest, most dynamic, and most complicated man-made systems, with millions of people interacting and hundreds of governing agencies1,2. Urban modernization has encouraged a significant portion of the global population to move to urban regions. Consequently, the built environment has grown rapidly, and building energy consumption has increased, which augments the release of CO23,4. Improving energy efficiency has been urged by various sectors5. The scientific community has exerted significant efforts to enhance its understanding of the built environment by leveraging a wide range of sensing technologies4. Furthermore, the advent of computational techniques has revolutionized data collection and analysis, enabling us to process vast quantities of information1.

Infrared thermography has been widely used in the built environment for many research purposes, such as urban heat island6,7, building diagnostics5,8,9, and urban heat fluxes10. Infrared thermography can provide images showing the surface temperature of different elements in the built environment with a lower cost and effort compared to similar kinds of sensor networks11. It detects infrared electromagnetic radiation emitted by the inspected objects and is commonly used for non-destructive testing and non-contact diagnostic technology5,8. It is based on an infrared imaging system calibrated to measure surfaces’ emissive power distribution at various temperature ranges. The infrared camera generates a series of two-dimensional, readable thermal images, with different colors and tones representing different temperatures5,8. Infrared systems have been leveraged to analyze the built environment across various scales: Satellites, capable of wide-ranging data collection, are primarily used for city-scale analyses. This typically covers entire cities or large urban regions. Aerial vehicles, possessing a more focused reach, are frequently employed for investigations at both city and district scales. The term district scale indicates a specific urban district or neighborhood. It refers to a geographical area larger than a single building or block but smaller than an entire city or region. This level of analysis allows for considering the collective impact of multiple buildings and infrastructure elements within a defined urban district. Due to their unique vantage point, rooftop observatories are particularly effective for district-scale studies. These offer a detailed view of the urban microclimate within a specific district or neighborhood. Lastly, drones and handheld devices are used for detailed investigations at smaller scales. Their portability and close-range capabilities make them particularly suited for analyses at the building scale (focusing on individual buildings) and human scale (focusing on the immediate surroundings of individuals or small groups)11. Many previous and current studies focus on the city and building scales and could not show the dynamic interactions on the district scale with high-resolution1,4. Furthermore, urban climate studies have been focusing on the temperate climate zones, especially in North America and Europe, and there is a lack of such studies focusing on the tropical climate zones12,13,14.

This work introduces the first thermal observatory deployed in Singapore, a tropical city, with a multi-modal platform with flexibility and high resolution to assess a wide range of dynamic processes in urban systems. The thermal observatory was set up on the roof of buildings that overlooked several educational buildings on the National University of Singapore campus. The platform gathers remote sensing data from a tropical area at a high temporal resolution at approximately 10-second intervals, providing the temperature trends of specific urban elements such as buildings, roads, and vegetation. Demonstration codes were given with data preprocessing, such as segmentation, in order to handle and analyze the generated raw data and allow users to use the data with high flexibility, catering to their research needs.

Methods

Thermal images data collection

Thermal images are taken from a FLIR A300 (9 Hz) thermal camera atop an urban-scale IR observatory, as shown in Fig. 1. The detailed specifications of the thermal camera are listed in Table 1. The thermal camera is protected by a housing against rains with IP67 protection. It was attached to a pan/tilt device that can spin 360 along its horizontal axis to record thermal images in multiple locations. These structures (thermal camera, housing, pan/tilt) were placed on a 2-m-high truss tower to avoid impediments while taking thermal images. Concrete blocks serve as a stabilizing support, and an air terminal serves as lightning protection for the truss tower. Two plugs were added to the truss tower to allow a backup battery in the water tank room to power both the thermal camera and the motor of the pan/tilt device. The backup battery is continually charged from the building’s electrical supply to enable the pan/tilt unit and thermal camera to be operational for up to two hours during a power outage. A laptop with a weatherproof casing in the water tank room was also linked to the pan/tilt device and thermal camera for customizing and reviewing the collection of thermal images.

Fig. 1
figure 1

Observatory installed on the rooftop of the residential area in Singapore and its captured information. Source of the imagery: Google Earth20.

Full size image
Table 1 Information about the FLIR A300 thermal camera.
Full size table

The thermal images are collected from two separate locations, namely Kent Vale and S16. The Kent Vale observatory is located on the rooftop of a 42-m-tall building in a residential area, which is located in front of a university campus consisting of office and educational buildings, in order to access the district-scale analysis. The pan/tilt unit was configured from two separate software to allow the thermal camera to take images from four positions (I, II, III, and IV), as shown in Fig. 2. The first software was installed in a video encoder to control the positions of the pan/tilt unit to take a thermal image. The second software developed by NAX Instruments Pte Ltd was installed in the laptop to control the moment to take a thermal image, which could then be saved either in JPEG or FFF file format. The observatory can capture thermal images of four buildings, as shown in Fig. 1. Building A, known as CREATE, is centrally located at Position I and at a height of 68 meters. Curtain walls cover a significant portion of its facade. Buildings B (E1A) and C (EA) are each centered at positions II and III, respectively. Both buildings are roughly 27 meters tall, characterized by their concrete walls and single-pane windows. Building D, or SDE4, can be observed at Position IV. It is structured with metal grids mounted on a concrete frame and is designed to be a net-zero building. It stands at a height of approximately 24 meters. Surface temperatures of trees are also observed from the above positions. A road with traffic exists in front of Buildings B, C, and D, and the road can be observed in thermal images taken from Position IV. The observatory moved from Position I to IV sequentially and recorded thermal images at various intervals. The detailed time for the image taken is shown in its file name. Then, using a 4 G Internet connection fitted on the laptop, the captured thermal images are stored on the Google Drive repository.

Fig. 2
figure 2

Positions where the observatory capture thermal images on Kent Vale20.

Full size image

The S16 observatory is located on the rooftop of a nine-story building in an educational area on a university campus. The pan/tilt unit was configured to capture images from three distinct targets, denoted as Positions 1, 2, and 3 in Fig. 3, sequentially at different time intervals. The observatory can capture information from three different buildings with thermal images (Fig. 4). The observatory moved from Position 1 to 3 sequentially and recorded thermal images at various time intervals per view. The thermal images obtained in this study elucidate a range of elements, including air conditioning units, glass materials, vegetation, and solar panels. These elements can be systematically analyzed to attain an in-depth understanding of their thermal behavior and consequential impact on the ambient environment. Specifically, this research is poised to investigate the influence of these elements on local temperature variations, energy consumption patterns, and the phenomena of urban heat islands. These insights carry significant implications for urban planning and the development of efficient energy management strategies.

Fig. 3
figure 3

Observatory installed on the rooftop of the university campus in Singapore and its captured information. Source of the imagery: Google Earth.

Full size image
Fig. 4
figure 4

Positions where the observatory capture thermal images on S16.

Full size image

Due to the necessary adjustments to the second observatory’s positioning system, thermal images at the S16 site were not consistently captured from the exact same locations–slight variations to the left or right were noted. Consequently, automated segmentation tools are strongly recommended to account for these minor positional differences. It is acknowledged that these variations may impact the interpretation of temporal patterns in the data. Future enhancements to the observatory system are intended to improve the stability of image capture, leading to more consistent data collection. Furthermore, the positions from which images were captured were altered due to network issues with the observatory system on both Kent Value and S16. In an effort to achieve greater stability in the image capture locations on S16, measures were taken to ensure these positions were closely aligned with each other. Figure 5 depicts this more stable configuration.

Fig. 5
figure 5

Positions where the observatory capture thermal images on S16 after adjustment.

Full size image

The thermal camera comprises an optical system that directs radiation from the scene onto a microbolometer15. Long-wave infrared radiation changes the resistance of the detector, which is translated to apparent temperature (Tobj) readings as follows15:

$${T}_{obj}=\frac{B}{{\rm{l}}{\rm{n}}\left(\frac{{R}_{1}}{{R}_{2}({U}_{obj}+O)}+f\right)}$$
(1)

where Uobj is the total signal response to the incident long-wave infrared radiation on the detector. This response is influenced by the material’s emissivity, denoted by ε, and the current environmental conditions. For a comprehensive computation of the influence factors for Uobj, please refer to Eq. 6. The other terms, B, R1, R2, O, and f, are calibration constants determined by the camera manufacturer in a controlled environment and are stored in the thermal image metadata. However, these parameters usually need to be re-calibrated in an outdoor environment where a significant discrepancy can be observed between Uobj and Tobj. The re-calibrated parameters are presented in Table 2.

Table 2 Calibration parameters of the measurement[20].
Full size table

Emissivity (ε) measures a material’s ability to emit infrared energy. Different materials have different emissivities, which can significantly impact the surface temperature measurements. Therefore, accounting for the material’s emissivity under observation is crucial when processing thermal images. The dataset is presented in a radiometric format to preserve the raw thermal data, facilitating necessary corrections based on specific emissivity values. The total signal response Uobj is also affected by ambient conditions, such as air temperature and humidity. By incorporating local weather data into the calculations, these influences can be corrected, and the accuracy of the surface temperature measurements can be improved. Please refer to the Usage Notes section for further information.

Data preprocessing

Data preprocessing was done before analyzing the images for the Kent Vale dataset. First, data filtering was conducted to manually remove images unsuitable for analysis. Thermal images captured during the rainy period were disturbed by rain droplets and needed removal. Then, a convolution neural network (CNN) classification model was applied to exclude blurred images taken while the pan/tilt unit was moving. The CNN model is also used to classify the building type for the first location. The model was trained on 4316 pre-classified images and then used to classify thermal image datasets of Kent Vale.

Weather station data collection

The weather station locations are shown in Fig. 6. The weather stations measured meteorological data in multiple locations on the university campus around 2 meters above the ground, except for Locations 1 and 7 on rooftops. The six recorded parameters are air temperature (°C), relative humidity (RH, %), dew point (°C), wind speed (m/s) and direction (degree), gust speed (m/s), and solar radiation (W/m2) at a 1-minute interval. For location 9, only the first three parameters were recorded. The weather stations were calibrated before installation. The weather station specifications are listed in Table 3. The weather station installment details are available in Chen et al.16, and Yu et al.17.

Fig. 6
figure 6

Weather station locations16,17,43.

Full size image
Table 3 Specifications of weather station sensors17.
Full size table

Data Records

The thermal image and weather station data for the same periods are published in Zenodo with open access18. The thermal images dataset contains two locations: Kent Vale and S16. A set of 483,915 thermal images were taken from Kent Vale between November 8, 2021, and March 8, 2022, while 882,006 thermal images were taken from S16 between August 3, 2022, and December 14, 2022. The images taken time are indicated in their file names. The data were converted and stored in JPEG format. The thermal images were stored in folders according to locations and views and listed sequentially. The detailed hierarchy of the data folders is listed in the data/README file on GitHub (https://github.com/buds-lab/project-iris-dataset) and Fig. 7. Missing values are intended, as the preprocessing process filters unsuitable images for analysis. The CNN model classified the images taken at Kent Vale into five different groups: CREATE (I), E1A (II), EA (III), SDE4 (IV), and Corrupted. The Corrupted group collected images that were not classified into any of the above four views, which means that these collected images were of low quality and unable to be distinguished; hence, they needed to be excluded. The images taken at S16 were not entirely classified; while some were divided into three different viewpoints (view_1, view_2, and view_3), some were not, requiring further classification. The summary of the provided dataset is shown in Table 4. In Zenodo, the Kent Vale dataset is stored in four independent zip files in sequential order, while the S16 dataset is stored in six independent zip files in sequential order. In each subcategory, the data is zipped in days. The code structure demonstration is also available in the Readme file in the GitHub repository. Examples of the data folder hierarchy include: 2022-08-03/view_1/smap-2022-08-03T13-48-18.19.jpeg; 20220306/create/20220306000001.jpeg. The code demonstration contains three subcategories: data, notebook, and src. The data directory shows all the original and converted/processed data; The notebook directory includes three notebooks containing three preprocess steps; and the src directory runs preprocessing scripts. A more detailed description of the code file organization and explanation is available in the Code Availability section. The weather station data is in Excel format and stored in Zenodo as WS_data.zip. It contains some missing values; therefore, prior data preprocessing is needed.

Fig. 7
figure 7

File structure demonstration example for the dataset.

Full size image
Table 4 Summary of the provided datasets at each location: duration of each location, number of views per group, the number of examples per view, corrupted size per view and dataset size.
Full size table

Technical Validation

Sensitivity analysis

Before evaluating the infrared receptor (Tij) from the observatory, a sensitivity analysis on factors that might significantly impact its variance was carried out. First-order Sobol index19 was used to estimate the contribution of a variable to the variance of Tij. The total Sobol index was also computed to consider the variables’ interactions.

During the sensitivity analysis, the variables and their corresponding constraints are measured and shown in Table 520. The Satteli sampler21 on thermal images taken at the four positions during a sunny day in Singapore was used to calculate the first order and total Sobol indices. Tij was determined for each sample at various times of the day and thermal camera placements. The sensitivity of each variable at different periods was given using the mean indices across the four positions.

Table 5 Constrains of each variable considered during the sensitivity analysis of Tij.
Full size table

Parameters calibration

The thermal images’ surface temperature estimates were calibrated in accordance with the information gathered by contact surface sensors, as shown in Fig. 8 at three different positions. Between December 2021 and March 2022, a heat flux (HF) sensor and a temperature probe were installed at Position a on the surface of a window in Building B. The HF sensor and the temperature probe were connected to a Hioki data logger and measured the surface temperature every 3 seconds. Another type of data logger, the UbiBot WS1 pro indoor monitoring sensor, was used to connect temperature probes at position b and c, respectively.

Fig. 8
figure 8

Locations where sensors were placed in to calibrate thermal images and measure the surface temperature20.

Full size image

The output voltage recorded by the infrared receptor at position ij in the thermal image is represented by an array \({U}_{ij}^{r}\) and a header in these files. Equation 2 can be used to transform \({U}_{ij}^{r}\) into longwave radiations (\({L}_{ij}^{r}\)) between 7.5 and 13 micrometers:

$${U}_{ij}^{r}=c{L}_{ij}^{r}$$
(2)

where the term c is not truly constant but depends on the emissivity of the surface material being observed, with lower emissivities reflecting more radiation from other sources.

The FLIR A300 (9 Hz) thermal camera that was installed in the observatory was calibrated such that the following Eq. 3 holds true for the surface temperature of a target element detected by the infrared receptor (\({T}_{ij}^{r}\)):

$${T}_{ij}^{r}=g({U}_{ij}^{r})=b{\rm{l}}{\rm{n}}{\left[\frac{{r}_{1}}{{r}_{2}({U}_{ij}+O)}+f\right]}^{-1}$$
(3)

where b, r1, r2, O, and f are calibrated parameters.

However, the parameters in Eq. 3 need to be calibrated in order to minimize the discrepancy between \({T}_{ij}^{r}\) and Tij in an outdoor environment. In this case, these values are acquired in a controlled setting where the target element’s actual surface temperature Tij at location ij in the thermal picture is almost identical to \({T}_{ij}^{r}\).

The FLIR A300 camera was calibrated by manually tuning the parameters in Eq. 3 until there was a satisfactory agreement between surface temperature estimations and observations. The agreement was assessed in terms of the Mean Bias Error (MBE) and Root Mean Square Error (RMSE) as shown in Eqs. 4 and 5:

$$MBE=\frac{1}{n}{\sum }_{i=1}^{n}{\left({T}_{S}^{n}-{T}_{S,m}^{n}\right)}^{2}$$
(4)
$$RMSE=\sqrt{\frac{1}{n}{\sum }_{i=1}^{n}{\left({T}_{S}^{n}-{T}_{S,m}^{n}\right)}^{2}}$$
(5)

where \({T}_{S,m}^{n}\) is the surface temperature obtained by contact surface sensors, and \({T}_{S}^{n}\) is the surface temperature calculated from thermal pictures captured by the FLIR A300 camera at time t = t0 + n·Δt. The calibration aims to achieve an MBE as close to 0 with the lowest RMSE. The RMSE and MBE were calculated with different Δt at three different positions to optimize the temporal resolution for each location (30 minutes at Position A and 5 minutes for Positions B and C). At Positions A-C, the RMSE and MBE obtained after calibration are shown in Table 2, achieving an RMSE below 2 degrees Celsius and an MBE below ± 1 degree Celsius. Furthermore, Fig. 9 illustrates a comparison between the estimated and measured surface temperatures after calibration at Positions A-C, offering additional information that enables the data to be freely accessed and tailored to suit specific needs and interests in further research.

Fig. 9
figure 9

Comparison of estimated and measured surface temperatures after calibration of the A300 infrared camera at the observatory20.

Full size image

Usage Notes

Factors consideration

When evaluating Tij from the rooftop observatory, several factors must be considered. First, the buildings, streets, and trees that could be seen from the observatory are exposed to longwave radiations mainly coming from the skydome. Then, in tandem with the longwave radiation released by the element (L) being observed from the observatory, the longwave radiation from the sky (Lsky) is reflected into the air. Both Lsky and L pass through the atmosphere before arriving at the observatory, then together with the longwave radiation emitted by the atmosphere (Latm), transmitting through the window of the housing before reaching the infrared receptor. Along with other types of radiation, the infrared receptor also captures longwave radiation from the window (Lwin). The camera output voltage corresponding to Uij combining the relationship defined in Eq. 3 and the aforementioned factors could be expressed as the following equation:

$${U}_{ij}=\frac{1}{{\varepsilon }_{ij}{\tau }_{ij}^{atm}{\tau }^{win}}{U}_{ij}^{r}-\frac{1-{\varepsilon }_{ij}}{{\varepsilon }_{ij}}{U}^{sky}-\frac{1-{\tau }_{ij}^{atm}}{{\varepsilon }_{ij}{\tau }_{ij}^{atm}}{U}^{atm}-\frac{1-{\tau }^{win}}{{\varepsilon }_{ij}{\tau }_{ij}^{atm}{\tau }^{win}}{U}^{win}$$
(6)

where εij is the thermal emissivity from the element at position ij in the thermal image, \({\tau }_{ij}^{atm}\) is the transmissivity of the atmosphere between the element and the rooftop observatory, and τwin is the transmissivity of the window.

Tij could be calculated through Eq. 3 if Uij is known. εij and τwin could be estimated from the material properties of the element and the window, respectively. \({\tau }_{ij}^{atm}\) can be calculated from weather data collected from a weather station.

Considering building material properties is paramount in analyzing heat fluxes, as various materials exhibit distinct thermal behaviors. Specifically, Building A’s facade comprises two steel walls and one glass wall. In contrast, Buildings B and C predominantly feature concrete walls, while Building D’s facade uniquely incorporates steel and concrete walls. The diversity in materials necessitates a detailed understanding of their emissive properties, which are critical in accurately calculating temperatures. Any oversight in this regard could potentially lead to misleading results. Therefore, the thermal properties, including the corresponding emissivities of the building facades and vegetation, have been meticulously defined. The thermal properties of the building facades and vegetation, including the corresponding emissivities, are defined and presented in Table 6, which has been constructed based on the relevant literature.

Table 6 Thermal properties of building facades and vegetation20.
Full size table

Weather stations data

The weather information obtained from the weather stations could be used together with the thermal images gathered by the rooftop observatory to estimate urban heat fluxes. Eight weather stations measured air temperature, relative humidity, dew point, wind speed and direction, gust speed, and solar radiation.

Weather stations near the observatory camera could be used to calculate \({\tau }_{ij}^{atm}\) in the following equation according to Waldemar and Klecha20,22:

$${\tau }_{ij}^{atm}=x\cdot \exp -\sqrt{{d}_{ij}}\left({\alpha }_{1}+{\beta }_{1}\sqrt{\omega }\right)+\left(1-x\right)\cdot \exp -\sqrt{{d}_{ij}}\left({\alpha }_{2}+{\beta }_{2}\sqrt{\omega }\right)$$
(7)

where dij is the distance between the element and the observatory at position ij of the thermal image, ω is the atmosphere’s water vapor content, α, β, and x are empirical coefficients. ω can be estimated from the air temperature (Tair) and relative humidity (ϕ) measured from the weather stations using the following equation:

$$\omega =\phi \cdot \exp \left[{\gamma }_{o}+{\gamma }_{1}{T}_{air}+{\gamma }_{2}{T}_{air}^{2}+{\gamma }_{3}{T}_{air}^{3}\right]$$
(8)

where γ is another set of empirical coefficients described in Waldemar and Klecha20,22.

Preprocessing

Following the classification of CNN, segmentation of the desired region and extraction of its radiometric data are required in order to evaluate the dataset. Flirextractor python package23 is recommended, as it allows extracting temperature data from regions of interest in the thermal images and storing it in CSV (comma-separated values) format with a simple process. The temperature data from thermal images were converted using Eq. 1. The python package Labelme24 is helpful in the segmentation of regions of interest for analysis. The relevant codes are provided in the Code Availability section.

Image information

The thermal image data are classified and stored based on their positions. The image datasets contain information including time, device positioning, and observatory locations (Kent Vale or S16). The building detail and surrounding environment could be seen based on the corresponding real images in this work. Analysis can be conducted in the time domain, frequency domain, or both, based on the nature of the time series and the degree of information to be retrieved15.

Potential applications

The thermal images provide surface temperatures of urban elements such as buildings, roads, and vegetation. Some example applications of surface temperature data are urban heat island analysis20,25, urban energy monitoring26, and building thermal performance monitoring8,15,27. A detailed application of urban observatory is available in Dobler’s work1. Thermal images collected by the observatory can provide high-resolution data to study dynamic interactions in buildings at the district scale, much smaller than the city scale captured by satellite thermal image data, which are used frequently in relevant fields. Further studies of the thermal properties of non-residential buildings can be conducted and compared with those estimated at the city scale. The image data also lends itself to several digital image processing techniques, including edge detection, image enhancement, segmentation, and statistical analysis. For instance, edge detection could help discern different elements within the urban environment, such as buildings, vegetation, and transportation infrastructure. Image enhancement techniques could enhance the visibility of thermal patterns, and segmentation could be used to isolate areas of interest. The dataset provided here is well-suited for various applications, especially those involving urban heat island effects, building energy efficiency studies, and urban microclimate analysis. However, there remains potential for improving the accuracy of surface temperature measurements. Discussions about the data and innovative approaches to its interpretation and application are welcomed.

Privacy and safety control

These thermal images form the foundation for various research analyses related to urban living environments. For instance, they can be used to study temperature variations within an urban block, understand the effect of building materials on heat retention, or analyze the influence of vegetation on local microclimates. Researchers are encouraged to use various analytical techniques to probe the data, including computational statistics, machine learning, and sequential analysis. Machine learning, for instance, could be used to predict future temperature patterns based on past data, while sequential analysis could help understand temporal changes in the thermal landscape. However, data misuse needs to be prevented1,28. Since IoT cameras can be easily abused29, the dataset collected here should include sufficient privacy protections for relevant individuals and cities1,30. In such case, the images collected in this work do not contain personally identifiable information31 such as facial features, and the individual feature cannot be tracked. All taken images were strictly limited in pixel resolution so that interiors of buildings cannot be seen1.