THERMAL IR IMAGING : IMAGE QUALITY AND ORTHOPHOTO GENERATION

This paper deals with two aspects of photogrammetric processing of thermal images: image quality and 3D reconstruction quality. The first aspect of the paper relates to the influence of day light on Thermal InfraRed (TIR) images captured by an Unmanned Aerial Vehicle (UAV). Environmental factors such as ambient temperature and lack of sun light affect TIR image quality. We acquire image sequences of the same object during day and night and compare the generated orthophotos according to different metrics like contrast and signal-to-noise ratio (SNR). Our experiments show that performing TIR image acquisition during night time provides a better thermal contrast, regardless of whether we compute contrast over the whole image or over small patches. The second aspect investigated in this work is the potential of using TIR images for photogrammetric tasks such as the automatic generation of Digital Surface Models (DSM) and orthophotos. Due to the low geometrical resolution of a TIR camera and the low image quality in terms of contrast and noise compared to RGB images, the TIR DSM suffers from reconstruction errors and an orthophoto generated using the TIR DSM and TIR images is visibly influenced by those errors. We therefore include measurements of the UAVs positions during image capturing provided by a Global Navigation Satellite System (GNSS) receiver to retrieve position and orientation of TIR and RGB images in the same world coordinate system. To generate an orthophoto from TIR images, they are projected onto the DSM reconstructed from RGB images. This procedure leads to a TIR orthophoto of much higher quality in terms of geometrical correctness.


INTRODUCTION
Currently, flights to capture thermal aerial images are typically carried out using airplanes or helicopters.In flight preparation, permanent weather observation of the flight area is necessary to be able to carry out the flight under optimal shooting conditions such as air temperature, wind condition, clouds and time of day.Due to these conditions, the use of airborne thermography is limited and requires a high planning effort.Furthermore, thermal flights by airplane, in particular for small areas of interest are not economically feasible.In this context, the use of an Unmanned Aerial Vehicle (UAV) to capture thermal images is a promising solution.An UAV platform is characterized by high flexibility and low costs compared to thermal flights by a manned aircraft and delivers measurements with high temporal and spatial resolution.UAVs are therefore increasingly used for image acquisition in the photogrammetric 3D data acquisition of objects (Nex et al. 2014, Colomina and Molina et al. 2014, Unger at al. 2014).
Thermal imaging systems were originally developed for military use, however nowadays thermal imaging cameras have migrated into other areas and have found many uses in fields like surveillance activities, as subsidiary tool for firefighters, power line and examination of thermal insulation of buildings.Inspection of building envelopes with TIR images is used to detect heat losses, cracks, thermal bridges, air leakages, insulation problems in walls and roofs, moisture sources and other issues with thermal signature (Maset et al. 2017).Thermal imaging cameras mounted on UAV systems thus allow an easy inspection not only of the facades, but also of the building roof, that cannot be captured by terrestrial images.Despite the outer conditions like sun, daytime, wind and outside tempera-ture, the measured radiance depends on the actual temperature of the inspected structure, the emission coefficient, reflectance of the surface and on surrounding objects that increase the radiance of a facade without an energy loss of the building.This paper consists of two parts.The first part deals with the differences in between acquiring TIR images with or without the influence of sun light.The ambient temperature at night is lower than during the day; this means that same thermal anomalies should be more visible during night.Another factor is the sunlight influence and its long lasting effects.Direct sunlight and shadows might even influence the thermal pattern on a surface many hours after the exposure to sunlight has ended.Madura and Kolodziejczyk (2005) show that sun radiation, even in the far infrared range, can cause significant errors in temperature measurements when TIR cameras are used.On the other hand, being able to fly during the day has many advantages, since navigation is safer and additional sensors such as RGB cameras, which can deliver complimentary information, can be mounted on a UAV.We acquire image sequences of the same object (a building in our case) during day and night and compare the generated orthophotos with respect to different metrics like contrast and signalto-noise ratio (SNR).Our experiments show that acquiring TIR images during the night leads to higher thermal contrast, no matter whether contrast is computed over the entire image or small patches only.Yet, the SNR values for the two flights in homogeneous areas are nearly identical.
The second aspect investigated in this work is the usage of TIR images for photogrammetric tasks such as the automatic generation of Digital Surface Models (DSM) and orthophotos.As TIR images typically have a lower geometrical resolution and suffer from stronger noise compared to RGB images, DSMs from TIR images are of lower quality than those from RGB images.We therefore combine both image types by using GNSS measurements for the position of projection centres to reconstruct the scene using both image types within the same coordinate system.The TIR orthophoto is then generated by projecting TIR images on the RGB DSM instead of using the TIR DSM of low quality.This approach allows us to combine the high resolution and geometric accuracy of RGB images with the thermal information derived from TIR images.In addition, this combination allows us to derive TIR orthophotos from different flights (even at night) using the same RGB DSM, assuming that the absolute orientation of the image blocks is accurate enough and the captured object stays stable.
The paper is organized as follows.In section 2, the related work regarding noise estimation and 3D reconstruction from thermal images is presented.Section 3 briefly discusses hardware and software that were used in the experiments.The methods that are involved in this work are explained in section 4. Experimental results are shown in section 5, while section 6 consists of the conclusion and possible future work.

RELATED WORK
One of the factors that influences the quality of TIR images is the presence of noise.Noise is always present in digital images during image acquisition, coding, transmission, and processing steps.It is common for TIR images that the SNR and the contrast are low, making processing of TIR images difficult.Depending on the IR sensor technology the problem may be connected to photoelectric conversion (in photodetectors) or temperature fluctuation noise (in thermal detectors) as well as the influence of the manufacturing process (Rogalski et al. 2011).The aforementioned problems cause TIR detector non-uniformity, which is revealed by varied response of the pixels.This non-uniformity, if not properly corrected, may be the main source of noise in IR images.Also the environmental conditions during the measurement should be considered.The important problem is thermal stabilization of the TIR camera as stated by Budzan and Wyżgolik (2014).The time required to stabilize the thermal properties of an TIR camera differ according to the camera manufacturer.In most cases, the non-uniformity of the Focal Plane Array (FPA) is reduced already in the camera software.But the output image still can be affected by noise, especially if the object temperature is similar to the environment.To overcome this issue a mechanical shutter is used for a Flat Field Correction (FFC).FFC is a process whereby per-pixel offset terms are updated to improve image quality.This is done by presenting a uniform temperature plane (a flat field) to every detector element.At the same time as capturing this flat field, the camera updates correction coefficients, resulting in a more uniform array output.
One of the key steps in a photogrammetric pipeline is the calculation of the parameters of the interior camera orientation parameters, referred to as camera calibration.Compared to an RGB camera, the main challenge in thermal camera calibration is to produce a high thermal contrast target board.The geometric calibration to estimate interior orientation parameters of a TIR camera has been studied in some works (Luhmann et al. 2013, Lagüela et al. 2011).The usage of a 3D test field with circular reflecting targets instead of a board with heated lamps was found to produce much lower reprojection errors.The effect of reflecting cold sky radiation provides a sufficient image contrast in the thermal spectrum in a test field design.
A method for automatic texturing of building facades from terrestrial TIR image sequences is described by Hoegner and Stilla (2015).A fully automatic method is presented to refine GPS based positions estimating relative orientations of the image sequences including a given building model in a bundle adjustment process.The resulting refined orientation parameters are used to extract partial facade textures from all images and all sequences.Pech et al. ( 2013) analyse the problem of using a TIR camera on an UAV for acquiring multi-temporal thermal images and generating thermal orthophotos.Data acquisition comprises three image flights at different times of day: 7 am, 2 pm and 9 pm.Furthermore, the radiation temperatures from the thermal images were compared to ground measurements to check the correctness of the camera measurements.It was shown that the mission at noon has a significant deviation in measured temperature compared to the ground measurements than missions from the morning and evening.Khodaei et al. (2015) demonstrate that DSM generated from aerial thermal imagery can have comparable accuracy with respect to the one derived from visible images.The thermal camera is geometrically calibrated using a calibration grid based on emissivity differences between the background and the targets.A DSM generation from thermal video imagery is performed in standard photogrammetric manner.For quality assessment of the obtained DSM, two scenarios are proposed: a region wise scenario based on available DSMs, which are derived from visible imagery and a point wise scenario based on the coordinates of GCPs determined from an existing DSM.In the region based scenario, a histogram of the difference between the obtained DSM and the available DSM are computed.In the point based scenario, the positions of GCPs are measured in the obtained DSM and compared with known coordinates.Obtained results show the comparable accuracy of a 3D model generated based on thermal images with respect to a DSM generated from visible images.Boesch et al. (2017) present an investigation of the influence of interior camera orientation, tie point matching and ground control points on the resulting accuracy of bundle adjustment and dense cloud generation with a typically used photogrammetric workflow for UAV based thermal imagery in natural environments.High overlap (>= 80%) and many metal-coated ground control points are required for sufficient tie point matching in vegetation areas.Motion blur in the canopy region and low contrast at the sensor level make photogrammetric processing of thermal images challenging, as image matching for the whole project area may fail.In special cases, a later registration of generated orthophoto to an external registered dataset can help to properly georeference thermal orthophotos.
Due to low spatial resolution of thermal cameras and typically low contrast of a captured scene, several researchers propose fusion techniques with other sensors to overcome those limitations.Hoegner et al. (2014) fuse time-of-flight depth images and TIR images to create an accurate 3D point cloud for subsequent scene segmentation and people detection.Wang et al. (2012) introduce a hybrid method for rapidly measuring thermal data and three-dimensional geometries of buildings using the developed hybrid light detection and ranging (LIDAR) system, allowing "virtual" representations of the energy and environmental performance of existing buildings.
Moreover, the co-registration of images taken with both RGB and TIR cameras can give advantages in terms of accuracy and density of the resulting 3D point cloud (Hoegner andStilla 2016, Hoegner et al. 2016).Maset et al. (2017) show that a commercial computer vision software can be used to automatically orient sequences of TIR images taken from an UAV and to generate 3D point clouds, without requiring any Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS) data about position and attitude of the images.The georeferencing including scaling is done subsequently, exploiting at least three GCPs.Moreover, they use the Iterative Closest Point (ICP) algorithm to register dense clouds of RGB and TIR datasets.As a result, the created 3D model combines high resolution and geometric accuracy of RGB images with the thermal information derived from TIR images.
The motivation for this work is to contribute on the topic of UAV-based photogrammetry using TIR images.Despite the fact that fusion of TIR, RGB and other sensors had been carried out, we show that by using GNSS data of TIR and RGB images, a better result in TIR orthophoto generation could be achieved compared to no RGB data.Further, we are interested to test the abilities (such as FCC and day/night performance) of a specific HW platform to understand its limitation.The knowledge of advantages and drawbacks will assist us in future research tasks.

HARDWARE
In our experiments, we used a DJI Matrice 100 UAV, a vertical take-off and landing (VTOL) quadcopter equipped with GNSS receiver, Inertial Measurement Unit (IMU) and barometer.It has a maximum payload of 1 kg.Two different cameras were in use (see detailed parameters at Table 1): the DJI Zenmuse X3 , consisting of 3-axis gimbal and 4K RGB camera unit, and the FLIR TAU2 640 Long Wave InfraRed (LWIR) camera mounted on another 3-axis gimbal, as it is currently not possible to attach both cameras to the DJI gimbal.In addition, the TIR camera is equipped with a TEAX ThermalCapture 2.0 640 recording device that stores 14 bits raw images with GNSS data that are received from an additional GNSS receiver (not the one of the copter).Agisoft Photoscan was used as tool for the photogrammetric processing for both image types.
The test flights were done above an historic building near the Herrenhäuser Gardens in Hannover, Germany.The covered area is approximately 35x90 meters.In the twocamera setup, the TIR camera was chosen as reference sensor for flight planning, because of lower ground resolution as a result of lower geometrical resolution and corresponding focal length (see Table 1).Based on a pixel size of 17 µm, flying height was set to 30 meters above ground, leading to a ground resolution of 3.9 cm and approximately 2.6 cm on the highest point of the scene per pixel of TIR images.The flight stripes and speed were set to lead to at least 75% overlap in both directions for the TIR imagery.The RGB images have 1.3 cm per pixel ground resolution in this setup.One of the hypotheses of this work is that TIR data captured without sun light interference will yield better TIR images regarding noise and contrast.First, we need to define a quality measurement for TIR images.Different types of noise exist in RGB and especially in TIR images.We selected the SNR as widely used indicator for image quality.The noise is characterized by its standard deviation (Young at al. 1995).The characterization of the signal can differ, thus there exist different definitions of SNR.If the signal is known and lies in between two boundaries, amin ≤ a ≤ amax, then the SNR of image I is defined by equation (1).

Parameters
If the signal is not bounded but has a statistical distribution, then next definitions is in use: with , mean and standard deviation of pixel values Due to the fact that we are dealing with noisy images we extend equation (1).Instead of minimum and maximum, we use the percentiles (see equation ( 3)) as signal boundaries.
with ( , ) percentile of image I, and percentiles values A percentile is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations fall.This approach excludes noisy pixels that are far from the pixel distribution of the pixel values histogram.Figure 1 shows an example histogram of a 14 bits TIR image and different percentile value positions.The red line represents the position of percentiles for different values.It should be mentioned, that when v1 and v2 are set to 100 and 0 respectively, equations ( 1) and ( 3) are equal.It can be easily seen, that the min and max values, percentile 100 and 0 respectively, are far away from the actual pixel distribution.In this particular example 99.5 and 0.5 percentiles represent the distribution in a more realistic way, than the minimum and maximum values.This non-uniformity if not properly corrected, may be the main source of noise in TIR images.
In this work, we focus only on uniform areas of an image for SNR calculation.To find uniform regions the following approach is used and the example of the procedure is shown in Figure 2: 1. Canny edge detection, delivering a logical mask, where edge pixels are set to "1". 2. Morphological erosion on inverse edge mask.3. Pixel blob detection, followed by setting pixels of blobs consisting of less than 500 pixels to zero.The result of this step is shown in the right image at Figure 2.
We use two methods to calculate the SNR of homogenous regions per image.The first one is to calculate a weighted average of region SNRs as described in equation ( 4).The weights are calculated dividing the of pixels in a region by the total number of the pixels in all regions.The second approach is to divide the image into macroblocks (MCU) of size 8*8 pixels and to calculate the SNR according to equation (3) for each block, using the result of morphological erosion (see step 2 above).The macroblocks that do not have all pixels set to "1" do not participate in the calculation (see equation ( 6)).Finally, the average SNR is calculated for the whole image by equation ( 5).In addition to the SNR, we calculate image contrast to compare imaging conditions.We define the image contrast as the span of grey values and once again use percentiles to eliminate pixels that are far from the main pixel distribution, see equation ( 7).
For contrast calculation, we proceed with two methods as for SNR: contrast computation over whole image and average value of contrast computed over MCUs.

TIR and RGB Data Fusion
To be able to benefit from the higher resolution and smaller errors in the DSM reconstructed from RGB data, we combine the TIR data with the RGB DSM.A schematic diagram of the TIR and RGB fusion is shown in Figure 3.For both datasets, a typical photogrammetric workflow using the commercial software package Photoscan by Agisoft is carried out independently: point matching in the images, interior and exterior camera parameter estimation and optimisation by bundle block adjustment.For both cameras self-calibration is used and measurements of the GNSS receivers per camera are introduced as observations for the camera projection centre positions.Therefore, both datasets refer to the GNSS world coordinate frame.Then, for the RGB dataset, dense image matching and DSM generation are performed.Finally, the TIR orthophoto is calculated based on the orientated TIR images and the RGB DSM.For comparison, the dense reconstruction is also done using the TIR data and a TIR orthophoto is generated from purely TIR images.
Figure 3; Schematic procedure flow

EXPERIMENTS AND RESULTS
This paragraph is split into two subparts analogously to section 4: Chapter 5.1 presents the experiments and discusses the results of the comparison among day and night flights, while 5.2 deals with TIR orthophoto generation with and without the use of the RGB DSM.

Image Quality Measurement Results
Investigation of FFC: First, we deal with the influence of the FFC procedure on the TIR image quality.Note that all test flights have been performed with automatic FFC as described at section 2. To examine the FFC influence on the image quality additional data were captured: the TIR camera was placed outdoors and a static scene was recorded.Two uniform regions were selected manually, as shown in Figure 4 by the red and blue boxes, for SNR calculation by equation (1).At the beginning of the sequence, the FPA was at room temperature (upper graph of Figure 5), however during the test its temperature dropped due to low ambient outdoor temperature.As a result of the FPA temperature decline, the image quality degraded.This can be seen by the decreasing value of SNR.Nevertheless, after each FFC event (shown as pink vertical lines in the bottom graph of Figure 5), the SNR came back to its original level.We also observed, that when FPA temperature did not change dramatically (seconds 200-400), the SNR remained almost at the same level.This test gives us a better understanding of the internal process running in the FLIR camera and provides an argumentation in following sections.Image selection from both flights was done accordingly to GNSS position records.Only images close to each other in terms of the projection centre positions and viewing direction were chosen for the SNR comparison to ensure that only similar scenes are compared.Figure 6 shows the results of the SNR calculation based on the macroblock approach defined by the equation ( 5). Figure 7 shows the SNR calculated on homogeneous image regions using equation ( 4).In both examples, it is clear that with lower percentile limits the SNR decreases, because the nominator in equation (3) decreases.
It was expected that, due to lack of sun interference and a lower environmental temperature, images captured night flight would have a higher SNR value.However, our results show that for current data there is no difference between flights in terms of SNR.We attribute this finding partly to the fact that the camera in use, a FLIR TAU2 640, has the internal FFC process for non-uniformity correction, which also eliminates potential differences between day and night flight images.The contrast computation is done over the orthophotos generated by the method described in paragraph 4.2 from the raw 14-bit TIR images.The results of both flights are shown in Figure 9.The overall difference in grey values between two flights comes from the fact that the observed objects had different temperatures.Orthophoto georeferencing allows us to calculate the contrast exactly over the same area in both orthophotos.For the night flight we see a larger contrast, regardless of whether we compute contrast over the whole image or over small patches only (see Figure 8).Another outcome from our experiments is that the night flight provides better contrast independently of percentile values.There is a big drop in contrast between 0%-100% and 0.5%-99.5% percentiles while contrast is calculated over the whole image.The reason for such a drop is that the effect of long tails from the pixel distribution (see Figure 1) is eliminated.This effect does not exist or at least is not strongly dominant when the contrast is calculated over MCUs.In the current work, our main interest is an orthophoto.The generated orthophotos are shown in Figure 10.It can be seen, that the TIR orthophoto produced based on the RGB DSM is geometrically more correct (the roof and other structures have straight lines) than the orthophoto from the TIR data only.Figure 11 shows a couple of zoom-in regions from both orthophotos (TIR only and RGB-TIR orthophoto).It is noticeable that the orthophoto generated from the RGB-TIR model has more straight lines, for example the edge of the roof.However, the RGB-TIR model still suffers from some remaining errors.These errors stem from typical matching difficulties like height jumps in the model, but also from inaccuracies when fusing both datasets using different GNSS receivers.

CONCLUSION
In this paper, we investigate topics regarding TIR images for orthophoto generation.First, we focus on image quality, measured in terms of SNR and contrast.It was expected, that TIR images captured at night would have higher SNR.However, based on our tests, the outcome is that there is no significant difference, probably due to the fact that the FFC procedure implemented in the employed FLIR TAU 2 camera eliminates most of the non-uniformity.However, the night flight, as assumed, does provide a better contrast.
Regarding the second aspect, we showed that a commercial software like Photoscan can be used for photogrammetric processing of TIR images.The TIR orthophoto can be enhanced by the usage of an RGB DSM of the same scene.In addition, the successful combination of the RGB DSM, captured at another epoch, with the TIR imagery shows that TIR data can be captured at conditions where RGB cannot, and can thus profit from the high quality 3D information.
In our experiments, we used two GNSS receivers with low accuracy.That leads to errors in the camera position and orientation estimation.Hence, in future work we will search for corresponding points in between RGB and TIR images to generate more consistent orientation results.

Figure 1 :
Figure 1: Image histogram and percentile example

Figure 2 :
Figure 2: Uniform region detection: left is the raw TIR image, right is the TIR image with detected uniform regions (shown in yellow)

Figure 6 :
Figure 6: SNRmcu of the flights

Figure 8 :
Figure 8: Contrast of orthophotos comparison between flights

Figure 10 :
Figure 10: Orthophotos: upper row -from RGB camera, middle row -from TIR camera, bottom row -after fusion between RGB and TIR data.

Figure 11 :
Figure 11: Thermal orthophoto zoom in: left columnfrom TIR data only, right column -after fusion between RGB and TIR data.