THERMAL TEXTURE SELECTION AND CORRECTION FOR BUILDING FACADE INSPECTION BASED ON THERMAL RADIANT CHARACTERISTICS

An automatic building façade thermal texture mapping approach, using uncooled thermal camera data, is proposed in this paper. First, a shutter-less radiometric thermal camera calibration method is implemented to remove the large offset deviations caused by changing ambient environment. Then, a 3D façade model is generated from a RGB image sequence using structure-from-motion (SfM) techniques. Subsequently, for each triangle in the 3D model, the optimal texture is selected by taking into consideration local image scale, object incident angle, image viewing angle as well as occlusions. Afterwards, the selected textures can be further corrected using thermal radiant characteristics. Finally, the Gauss filter outperforms the voted texture strategy at the seams smoothing and thus for instance helping to reduce the false alarm rate in façade thermal leakages detection. Our approach is evaluated on a building row façade located at Dresden, Germany. * Corresponding author


INTRODUCTION
Due to climate change and energy deficiency, energy efficiency has achieved increasingly more attention in recent years.Considering that the use of building heating or cooling consumes a large part of all produced energy, an increasing demand for inspection of building envelope insulation and improvement of building energy efficiency is required.For this purpose, the usage of thermal infrared (TIR) images is playing an important role, because heat loss from buildings can be easily detected and visualized by thermal cameras.In practice, for detailed inspections of small areas, terrestrial images can be taken manually, while for surveys of larger areas mounting TIR cameras on a vehicle (Hoegner et al., 2016), helicopter (Iwaszczuk and Stilla, 2017) or unmanned aerial vehicle (UAV) (Westfeld et al. 2015) is an alternative option.Terrestrial image acquisition methods can provide detailed documentation of facades visible from the street level, while a flying platform is also able to capture roofs and inner yards.In this study, considering that we mainly focus on faç ades inspection, terrestrial acquisition method is applied.
Thermal texture mapping, which maps thermal images onto existing 3D geometric data, allows for spatial referencing of thermal images and accurate interpretation of thermal patterns.Due to the large image overlaps, each face of a building model usually corresponds to multiple thermal images.Therefore, most of the existing texture mapping methods focus on the automation of texture selection.Bé nitez and Baillard (2009) only took advantage of visibility into texture selection and mentioned that more factors should be taken into consideration.Wang et al. (2008) combined object incident angle and visibility analysis to select textures from oblique images.Frueh et al. (2004) selected the optimal image based on occlusion, image resolution, object viewing angle and coherence with neighbouring triangles.Hoegner and Stilla (2009) investigated a relative orientation method using Nistér's five-point algorithm and image triplets during automatic texturing of building facades with terrestrial thermal images.Hoegner and Stilla (2015) used thermal images taken from different views to detect objects on 3D faç ade models.Iwaszczuk and Stilla (2016) presented a line feature based 3D-2D co-registration method for texture selection from oblique thermal image sequences.They took resolution, occlusion and matching accuracy as quality assessments.Iwaszczuk and Stilla (2017) improved the geometric matching accuracy of their model-to-image registration method by taking uncertainties of 3D building model and image features into consideration.Therefore, for thermal texture selection, most of the existing methods focus on the discussions about the geometric aspects.However, none of them takes radiometric calibration of thermal cameras and thermal radiant characteristics into consideration.Uncooled thermal cameras are easily affected by ambient environment (i.e.wind or temperature change) when used outside, which results in large offset deviations without radiometric calibration.In addition, the effect of thermal radiant characteristics on object surface temperature retrieval is also non-negligible.Furthermore, several publications discussed methods for smoothing seams between adjacent images after texture selection.A voted texture by neighbouring triangles is used in Frueh et al. (2004) while a weighted texture blending using graph cut optimization and Poisson blending is introduced in Sinha et al (2008).However, these methods are proposed for optical images and seams in optical images are mainly due to exposure and illumination differences, while seams for the same material (i.e.concrete) on thermal images are mainly caused by potential thermal leakages, radiometric calibration inaccuracies or a neglection of thermal radiant characteristics.In fact, implementing a temperature correction based on thermal radiant characteristics is helpful to smoothen the seams and thus reduces the false alarm rate in faç ade thermal leakage detection.
Therefore, in this paper, radiometric calibration and thermal radiant characteristics are taken into consideration before any judgement of potential thermal leakages on the building faç ade.In addition, unlike existing thermal texture mapping methods assigning rectangular images to polyhedral models (e.g.CityGML) (Iwaszczuk and Stilla, 2017;Hoegner and Stilla, 2015), point clouds and triangle patches are used as 3D geometries and basic unit textures respectively in this study, thus maintaining the full level of detail of point clouds acquired using structure from motion (SfM) techniques or laser scanning methods.

DATA ACQUISITION
The target building faç ade is located at Nürnberger Str.30, Dresden, Germany.A RGB image sequence was captured on 15 th November, 2017 while a thermal image sequence was taken on 27 th November, 2017.Considering the fact that the RGB camera (Canon 1200D) has a much higher resolution and mostly higher contrast than the thermal camera (FLIR Ax65), the RGB image sequence was used to generate the faç ade point cloud, while the thermal image sequence was used for thermal texture mapping as a basis for faç ade inspection.
The thermal camera used in this study was an uncooled FLIR Ax65 with 13 mm focal length, 640 512 pixels, 17 µm pixel size, 30 Hz image acquisition rate and a temperature measurement range of -25°C to +135°C.As each thermal image only sees a very small part of the whole faç ade, it is necessary to combine and select textures from a large number of images to get a complete scene at good spatial resolution.In fact, the thermal image sequence of the whole faç ade contained 115 frames.

RADIOMETRIC CALIBRATION
In terms of radiometric calibration, two concerns must be taken into consideration: Spatial non-uniformity due to fabrication variations and temporal non-uniformity derived by changing sensor temperature.On the one hand, a vignetting effect, which belongs to spatial non-uniformity, severely degrades the image quality.On the other hand, temporal non-uniformity, which stems from the fact that the camera output depends not only on the object irradiance but also on the time-variant sensor temperature, increases the measurement instability.Therefore, a radiometric calibration procedure (Lin et al., 2017a) is carried out to address these two non-uniformity problems.Firstly, a shutter-less temporal non-uniformity correction (NUC) (Lin et al. 2017b), which takes advantage of polynomial curve fit models and real-time sensor temperatures, is applied to stabilize the unstable outputs.Next, multi-point correction (Tempelhahn et al. 2016) is implemented to get rid of the vignetting effect and to improve the contrast in the thermal imagery.Finally, radiant temperature could be retrieved using a Planck curve after temporal and spatial NUC.However, kinetic temperature, which refers to internal or true temperature, would still be unknown unless the emissivity of the object is known (Vollmer and Möllmann, 2010).
Results of the radiometric calibration are shown in Figure 1.Note that all of the temperature images shown here represent radiant temperatures.However, thermal leakages usually correspond to large kinetic temperature differences on adjacent spaces of the faç ade.The problem is that the kinetic temperature can be derived only if the emissivity is known, see equation ( 1).But the emissivity depends on several factors, such as material (concrete, metal and glass), surface roughness, incident angle etc. (1)

Where
= radiant temperature of the object = kinetic temperature of the object = emissivity of the object Considering that concrete has a relatively high emissivity of above 0.8, while the emissivity of polished metals can reach as low as 0.02, it would for instance be a faulty judgement if the pipelines shown in Figure 1 (b), (c) and (d) would be detected as thermal leakages.In addition, it is usually impossible to measure the temperature of glass using thermal cameras, because glass is always heavily influenced by mirror reflection (e.g.windows located at first or second floor reflect the trees or something else nearby while windows located at upper floors of the faç ade usually reflect the cold sky).On the other hand, the heat loss due to the open windows is relatively easy to see.
Another potential problem that becomes obvious in Figure 1 is that images of the same material (e.g.concrete) taken with nadir view usually have higher temperature values than images taken with tilted view.This is caused by thermal characteristics rules as well as the radiometric calibration errors, which will be detailed explained in Section 6.

3D FAÇADE MODEL GENERATION
Although SfM techniques are widely used for the generation of 3D points cloud and meshes based on RGB images, it is much more challenging in case of thermal images because of low resolution and low contrast.Especially in case of building facades, additional difficulties are the repetitive structures which often cause false matching results.Therefore, in this paper, the 3D faç ade reference model was produced from the RGB image sequence using SfM techniques.And the generated dense 3D mesh model, which consists of 329,099 points and 654,472 triangle faces, is shown in Figure 2.
Prior to thermal texture mapping, registration of the reference model and TIR image sequence is necessary to acknowledge.In this work, as the automatic registration between the 3D model and the thermal image sequence is not the focus, a simple operator is implemented as follows: Firstly, besides the reference model, another 3D model calculated from the TIR images was generated.In the next processing step, the homogeneous points founded in both 3D point clouds are picked up manually and served as the basis for point cloud registration.Afterwards, once both point clouds are registered, all the used images including RGB images and TIR images were transformed as well and their corresponding orientation parameters were automatically calculated by back projection.As a result, both generated 3D point clouds, as well as both image sequences share the same coordinate frame.Thus, the orientation information is obtained and can be used to orient thermal images directly to the reference 3D model.All of these procedures are implemented in Agisoft PhotoScan.And a future topic includes the automatic registration of the RGB point cloud and the thermal point cloud.

TEXTURE SELECTION
As most of the 3D object meshes are covered by several thermal images, the goal of texture selection is to choose the best image for each mesh.In this study, a best texture selection procedure based on resolution, occlusion, object incident angle and image viewing angle is proposed.
(1).Occlusion Before the texture selection, for each triangle on the 3D surface model, a list of visible images has to be established.Considering that each thermal image captures a small part of the whole faç ade in this study, a Z-buffer algorithm with spatial subdivision is implemented here to detect the occlusions.Firstly, all of the 3D points are allocated to different boxes according to their X coordinates (X axis is the main direction of the facade shown in Figure 3).Then the projection centre of each image could also be distributed into these boxes using its own X coordinate.Subsequently, for each image, the general visible 3D points could be determined by searching several boxes near its own box rather than the whole faç ade, which can dramatically improve the efficiency of the traditional Z-buffer algorithm.Afterwards, for each pixel of an image, only one object which is closest to the projection centre is defined as visible.Finally, for image and triangle , the occlusion parameter is calculated by the ratio between the number of visible pixels to the total number of back-projected 2D pixels.Here, the visibility of triangle for image is set by comparing with a threshold in equation ( 2).In the task of texture mapping with oblique aerial images, a lot of existing papers claimed that strictly requiring 100%-visibility would eliminate some "good" candidate images (Frueh et al. 2004), due to the fact that aerial images with perpendicular view and a few occluded pixels usually look better than images taken from extremely oblique view.However, in this close-range terrestrial application, occlusion rules could be set more strictly, therefore, is set to 1.0 in this paper. (2) (2).Resolution As images are taken under different viewing directions and distances, the same 3D triangle has varying resolutions on different images.In order to select the most detailed texture, the resolution of a triangle in an image is defined as the number of pixels within its projected 2D triangle.

(3). Object incident angle and image viewing angle
Object incident angle and image viewing angle are shown as and in Figure 3 respectively.According to the thermal radiant characteristics, the ideal thermal image used for texture mapping is taken from a nadir view.This is mainly due to the following three reasons.Firstly, the radiant power that is emitted by an object and received by an image depends on the distance and relative orientation of the two areas with respect to the viewing vector, which is shown in equation (3) (Vollmer and Möllmann, 2010).This explains why images taken with shorter object-image distances and nadir view are recorded with higher temperature values shown in Figure 1 (c) and (d).Secondly, reflection from other hot spot sources, which is called undesired noise radiation, is also dependent on the object incident angle.Thirdly, for nearly all practically important surfaces, the emissivity will only be nearly constant from the normal direction 0° to 40°, while the emissivity would change and different materials behave differently when confronted with larger angles (Vollmer and Möllmann, 2010).This means kinetic temperature calculation acquired from more than 40° oblique view needs to consider a correction of object emissivity, although the kinetic temperature retrieval would be a future topic.Note that the calculated temperatures mentioned in the following part of this paper are all related to the radiant temperature in equation ( 1).
( As stated above, larger incident angle (including and ) would lead to the camera capture a lot of reflection from other objects rather than the radiation from the targeted object, which increases the difficulty of the temperature retrieval.Therefore, for application of thermal building inspection, images taken from nadir view are better than oblique view.Thus, in order to determine the optimal image for a 3D triangle , a criterion is designed here.Firstly, in terms of each triangle (taking as an example), a score for image is calculated using equation ( 4).Then the image with highest score is chosen as the preliminary texture for the target triangle. (4) In terms of equation ( 4), in order to decrease the values of the parameters and during texture selection, other forms such as the square root of and are also tested, but we have empirically observed that the above equation could achieve the best result for this test data.More data need to be testified in the future.Texture mapping results using the above selection criterion is shown in Figure 4. Thermal energy losses due to the opening of the windows or doors are clearly visible in Figure 4.The reason why the triangles located inside of the balconies cannot be textured is because these triangles are occluded by the lower prominent parts when confronted with terrestrial images taken from the street using an up-looking view.A similar situation also shows up at the intersection area of the roof and faç ade.The reason why the area located at the bottom and left part of the faç ade has no proper texture (area surrounded by a white box in Figure 4) is due to the missing thermal images.Furthermore, another problem is the salt and pepper effect.This is caused by noise of the 3D photogrammetric point cloud, which is derived with limited accuracy and results in the unexpected directional changes of surface normal vectors affecting the object incident angle.In addition, considering that the RGB image sequence and the thermal image sequence were acquired at different days, some 3D objects (e.g. a car surrounded by a black box in Figure 4) are not included in the 3D model, but visible in the thermal images and thus textured onto the mesh.

SEAMS SMOOTHING
Another problem visible in Figure 4 is the visibility of seams.Two different strategies (voted texture, temperature correction and filtering) are applied and compared here.

Voted Texture
The first strategy is a voted texture strategy, which is frequently used in aerial oblique texture mapping using optical imagery (Frueh et al. 2004).The main idea of this method is to use as few images as possible based on the fact that seams usually show up at the boundary between triangles textured with different images.Therefore, this method is in favour of assigning neighbouring triangles with textures extracted from the same image.Specifically, for a triangle, firstly, a kd-tree is utilized here to determine its Nth level neighbor list.Then a voted texture favoured by the majority of its neighbours can be found.Afterwards, the preliminarily assigned texture would be replaced by the voted texture when the preliminarily texture has more seams than the voted texture.Therein, a seam is described as a difference of chosen images between the target triangle and one of its neighbours.Thus texture mapping result using the above voted texture is shown in Figure 5.
As shown in Figure 5 (area surrounded by a red box), the voted texture strategy turns out to be not effective in reducing seams.
The reason why this strategy does not work, is in the fact that this method is originally proposed for texture mapping using airborne oblique optical images.Compared with terrestrial images, oblique view images are usually acquired with much longer object-image distances and thus capture much larger area.Then when fewer images are used for texture mapping, the number of seams in small area could be reduced effectively.However, in this study, each thermal image corresponds to a rather small part of the facade.As a consequence, the number of seams cannot be largely reduced using this method.Furthermore, as shown in Figure 5, the reason why concrete regions located at the first floor of the faç ade are recorded with higher temperature values than the other concrete regions located at the upper floors, is believed not caused by thermal leakages, but mainly due to the thermal radiant characteristics shown in equation ( 3) as well as radiometric calibration errors.Therefore, we tested an alternative method to smooth the seams by taking thermal radiant characteristics into consideration.

Temperature Correction and Filtering
Considering that the object radiant temperature is related to the distance from object to image, object incident angle and image viewing angle, therefore, it is possible to correct the temperature of selected textures using equation ( 5) (Vollmer and Möllmann, 2010).
(5) Where = original radiant temperature = corrected radiant temperature = constant reference distance, which is in the order of magnitude of the distance from 3D triangle to image projection center = distance from the projection center of image to the center point of 3D triangle Thus texture mapping result using equation ( 5) is shown in Figure 6.
Compared with the results when temperature correction is not applied (shown in Figure 4), Figure 6 presents a better texture mapping result with a more consistent temperature distribution.The texture mapping result after temperature correction and Gaussian filtering is shown in Figure 7. Based on visual judgment, temperature correction plus Gaussian filtering shows better results than the voted textures strategy in this study.In spite of a smoothing and blur effect, Figure 7 not only reduces the block effect defect in Figure 5, but also overcomes the salt and pepper effect in Figure 6.

CONCLUSIONS AND OUTLOOK
A thermal image texture selection and correction method for thermal faç ade attribute provision based on thermal radiant characteristics is proposed in this paper.A 3D faç ade model is generated from RGB image data by applying SfM and dense matching techniques, and thermal attributes are obtained from TIR camera imagery.For texture selection, factors of resolution, occlusion, object incident angle and image viewing angle are taken into consideration.Compared with the voted texture strategy widely used in optical images, temperature correction plus Gaussian filtering proved to be more effective on smoothing the seams and thus achieving a more consistent and reliable result.
In general, this faç ade is believed to be free from the thermal leakages (besides open windows or doors), and the slight visual seams are due to the limited radiometric calibration accuracy of this low-cost uncooled thermal camera.In fact, because of the limited accuracy of radiometric calibration, it is still hard to acknowledge the number and specific locations of the thermal leakages.
Future challenges include the further accuracy improvement of radiometric calibration, otherwise it is rather hard to distinguish whether the temperature difference is coming from radiometric calibration error or a real thermal leakage.Another promising work is to use classification methods to classify the target object into different materials, which is helpful to further retrieve the kinetic temperature values of object in large area automatically.Furthermore, according to the thermal radiant characteristics, a UAV-based imagery acquisition method, which guarantees that all parts of the building could be captured under nadir view, is obviously the best method for thermal building inspection.However, sometimes the usage of UAV is legally prohibited in residential areas.To thoroughly check the quality of the thermal attribute data obtained by the data processing chain, thermal ground truth hat to be included.
Figure 1.Four exemplary radiant temperature images (°C) after radiometric calibration

Figure 2 .
Figure 2. 3D mesh generated from RGB image sequence

Figure 3 .
Figure 3. Illustration for thermal radiant characteristics

Figure 6 .
Figure 6.Texture mapping results after temperature correction