3D SURFACE GENERATION FROM AERIAL THERMAL IMAGERY

Aerial thermal imagery has been recently applied to quantitative analysis of several scenes. For the mapping purpose based on aerial thermal imagery, high accuracy photogrammetric process is necessary. However, due to low geometric resolution and low contrast of thermal imaging sensors, there are some challenges in precise 3D measurement of objects. In this paper the potential of thermal video in 3D surface generation is evaluated. In the pre-processing step, thermal camera is geometrically calibrated using a calibration grid based on emissivity differences between the background and the targets. Then, Digital Surface Model (DSM) generation from thermal video imagery is performed in four steps. Initially, frames are extracted from video, then tie points are generated by Scale-Invariant Feature Transform (SIFT) algorithm. Bundle adjustment is then applied and the camera position and orientation parameters are determined. Finally, multi-resolution dense image matching algorithm is used to create 3D point cloud of the scene. Potential of the proposed method is evaluated based on thermal imaging cover an industrial area. The thermal camera has 640×480 Uncooled Focal Plane Array (UFPA) sensor, equipped with a 25 mm lens which mounted in the Unmanned Aerial Vehicle (UAV). The obtained results show the comparable accuracy of 3D model generated based on thermal images with respect to DSM generated from visible images, however thermal based DSM is somehow smoother with lower level of texture. Comparing the generated DSM with the 9 measured GCPs in the area shows the Root Mean Square Error (RMSE) value is smaller than 5 decimetres in both X and Y directions and 1.6 meters for the Z direction.


INTRODUCTION
Thermal imaging sensors have been recently applied for object detection and identification in wide range of applications, such as surveillance and reconnaissance (Leira et al. 2015).Nevertheless, due to low geometric resolution and low contrast of thermal imaging sensors, it is mostly used for interpretation and monitoring purposes (Berni et al. 2009).However, thermal mapping is getting more important in civil applications, as thermal sensors may be applied in foggy weather and night times which is not possible for visible camera (Kuenzer and Dech, 2013).For high-resolution mapping and specific products generation from aerial thermal imagery, high accuracy photogrammetric process is necessary.Therefore, geometric calibration and 3D mathematical modelling of thermal imagery becomes evident for all precise applications.Camera calibration is a key aspect to obtain geometrical information through photogrammetry process.Therefore, wide range of literature focus on geometrical calibration of thermal imaging sensor.The main difference of them is the designed calibration field (Luhmann et al. 2013;Vidas et al. 2012;López et al. 2015).A wooden plate with the warm up lamps (Luhmann et al. 2013) or foil targets (López et al. 2015) is common test field.However the quality of the targets is quite poor and the centre of the targets cannot be measured with high precision.Vidas et al. proposed a mask based approach that placed an opaque mask in front of warm background (Vidas et al. 2012).According to simplicity and accuracy of mask based calibration test field, this method is chosen for implementation.

* Corresponding author
The applications of remotely sensed thermal imagery has been investigated.Berni et al. (2009) use thermal and visible cameras mount on UAV for monitoring of vegetation.Detection of people and vehicle is another application of thermal camera (Gąszczak et al. 2011). Pless et al. (2012) in German aerospace center (DLR) generate orthophoto and 3D surface model based on thermal imaging.Two thermal cameras with cooled detector (mid-wave and long wave) are tested for photogrammetric process.Moreover geometric and radiometric calibration is performed before flight.Another study proposed the method to generate multi-temporal thermal orthophoto from UAV data which can be used for spatial analysis of temperature distribution.It shows SIFT is well suited for feature detection in low resolution thermal imagery (Pech et al. 2013).Oblique thermal imagery are also used for the automatic extraction of building geometry.Both the geometry and the thermal information are used in the generation of building 3D models (Lagüela et al. 2014).The classification of roof surface for the installation of solar panel is performed by visible and thermal data fusion (Lopez et al. 2015).
The purpose of this paper is to generate 3D object surface using thermal images acquired with a small UAV equipped with an infrared thermal camera.In following the 3D surface generation methodology including geometric camera calibration, flight planning, bundle block adjustment and dense matching are described.Then experimental results obtained from an industrial field are presented and discussion and conclusion over results are presented.

METHODOLOGY
The proposed method includes three main steps: geometric thermal camera calibration before flight, flight planning and imaging and 3D surface generation form acquired thermal video.

Thermal Camera Calibration
In the first step, thermal camera is geometrically calibrated in the laboratory.In this study, the calibration grid is constructed based on emissivity differences between the background and the targets.In this way, target detection can be conducted automatically.
The calibration pattern consists of a grid of regularly sized circles cut out of a thin opaque material which is held in front of a powered computer monitor.By this way, the calibration pattern is easily identifiable in the thermal domain and geometric camera calibration is performed (Vidas et al. 2012).
For camera calibration the corrected image coordinates   and   can be calculated from the measured coordinates   and   via equation ( 1): where  = √  +   ,  =   −   ,  =   −   ,   and   are principal point offsets in x and y image coordinate direction and dr is determined by equation ( 2) as bellow: is the term of order (2 + 1) ℎ for radial distortion correction. 1 and  2 are the coefficients of decentering distortion (Fraser, 1997).
Constructed test field consists of a number of circular and coded targets which their centres must be found automatically.In order to improve center detection of targets, contrast enhancement is performed.Then targets centers are automatically detected and calibration parameters are determined by self-calibration.

Flight planning
Flight planning is the pre-request step of the photogrammetric mission.There are some parameters to be identified including flight height and speed, camera focal length, capturing rate, etc.These parameters is defined based on required map scale, topography, etc (Mouget et al 2014).
In standard photogrammetry at least 60% overlap along the flight axis and about 20% sidelap of flight strips should be covered.In the case of UAV missions, due to instability of the platform, its better to increase overlap and sidelap to 80% and 60% respectively.

3D Surface Generation
In order to generate 3D surface from thermal video, sequences of frames are extracted from video, then tie points are generated by SIFT algorithm.Bundle block adjustment is then applied and the camera position and orientation parameters are determined.Finally, multi-resolution dense image matching algorithm is used to create 3D points cloud of the scene.

Frame Extraction:
Data processing starts with thermal frames extraction from acquired airborne video.According to the small size of the thermal frames, to prepare the required overlap and sidelap, the rate of the frame extraction should be high.

Interior
where Z is the disparity image, The first term is the sum of all pixel matching costs,  is a priori weighting coefficient, V is a neighbourhood of the pixel (usually 4 or 8 neighbourhood) and  ,  is a weighting function in the neighbourhood V.
This minimization problem is solved based on multi-resolution implementation of a Cox&Roy optimal flow image matching algorithm (Roy & Cox, 1998).This multi-resolution approach is necessary for achieving reasonable processing times on extended areas and improving robustness by restraining matching ambiguities.Finally DSM is obtained by dense matching on thermal images.

3D Surface Evaluation:
For quality assessment of the obtained DSM, two scenarios are proposed: region wise scenario based on available dense DSM which derived from visible imagery and point wise scenario based on the coordinates of GCPs determined on the available DSM.In the region based scenario, histogram of the difference between the obtained DSM and available DSM are computed.In the point based scenario, the position of GCPs are measured in the obtained DSM and compare with known coordinate, then statistical analysis is performed and the RMSE and Mean Absolute Error (MAE) are computed to quantitate the accuracy of thermal derived-DSM.(Hobi & Ginzler, 2012).

EXPERIMENTAL RESULTS
Potential of the proposed DSM generation method is evaluated based on thermal imaging cover an industrial area.

Dataset
Thermal video is acquired by TC688 camera which is mounted on small UAV.The specifications of camera are presented in Table 1.The altitude of the flight was 100 m above the ground which results in a ground sampling distance of about 7 cm.The flight lines were planned for the images to have sidelap of 60% between the image stripes.According to the frame rate of the camera, flight velocity, flight height and the 80 % overlap, the frames are extracted.Figure 1 demonstrates some extracted frames.
Figure 1.Frame extracted from thermal video

Geometric Calibration
Geometric calibration of thermal camera with planar pattern is performed in laboratory.The test field consists of a grid of 28 regularly sized circle with 4 coded targets cut out of a thin nonconductor material.The size of mask pattern is 44×24 cm 2 , diameter of squares are 20 mm which spaced with 50 mm separation.Figure 2 shows the thermal imagery acquired from mask based calibration pattern.The pattern is held in front of a power on computer monitor.Self-calibration is performed to compute camera calibration parameters.The results of camera calibration are presented in Table 2. 2.10540e-007 3.0611e-007 K5 -1.89923e-009 3.1147e-009 P1 -7.2475e-004 3.984e-004 P2 -6.5166e-004 6.079e-004

3D Surface Generation
In order to orient extracted frames, tie points are extracted between unordered frames.For these purpose each frame is considered with respect to all other frames to determine tie points.Figure 4 shows two samples of tie points extracted from thermal imagery based on SIFT algorithm.The next step is to compute the absolute orientation using some measured GCPs.Here the GCPs are extracted from an available ortho-image.Figure 6 shows the positions of GCPs on the orthoimage.
Figure 6.GCPs on ortho-photo After absolute orientation step, the dense image matching procedure should be done to generate final products.These products are thermal DSM, thermal ortho-photo and thermal points cloud.Figure 7 shows the generated thermal ortho-photo.For each part, the final thermal DSM and the histogram of height differences between visible DSM and thermal one is represented (Figure 9).For point-wise assessment of the thermal derived DSM, 9 check points are considered.Figure 10 shows the planimetric error vector of check points by comparing thermal and visible orthophotos.
Figure 10.Planimetric error vector of the check points The obtained results show the comparable accuracy of orthophoto generated based on thermal images with respect to visible ortho-photo.The statistical analysis on check points coordinates are shown in Table 3.Four statistical parameters are computed.These parameters are the minimum, maximum, MAE and RMSE of check point's error in X, Y and Z directions.As presented in Table 3, the planimetric position is more accurate than vertical coordinate.There are 2 points with large error values which seems to be the blunders.In order to improve the results, blunder detection should be applied to remove such blunders.

CONCLUSION
In this paper the potential of thermal video imagery in 3D surface generation was inspected.For this purpose three steps method consists of geometric camera calibration, flight planning and 3D surface generation presented.Obtained results show the comparable accuracy of 3D model generated based on thermal images with respect to DSM generated from visible images, although thermal based DSM is somehow smoother with lower level of texture.Results show the RMSE of 0.44m, 0.8m and 1.62m in X, Y and Z directions, respectively.
Although the blurry appearance and low resolution of the thermal images cause a bit vagueness in thermal products, further investigations should be carried out in combination of the thermal and visible outputs for mapping applications.

Figure 3 .
Figure 3. Automatic detection of targets centres

Figure 4 .
Figure 4. Tie points extraction Relative orientation is performed based on tie points.The initial value of orientation parameters are determined by essential matrix whereas bundle adjustment is used for relative orientation.Obtained results of relative orientation for four flight strips are illustrated in Figure 5.

Figure 5 .
Figure 5. Results of bundle adjustment

Figure 8 .
Figure 8. Dividing area to some separated parts.
Figure 9.The comparison of the thermal DSM with visible DSM.
Orientation: Geometric calibration parameters computed in camera calibration step are applied and thermal images are resampled to distortion-free images.In order to solve bundle adjustment of the selected frames, tie points are extracted by SIFT.It is well suited for the feature detection and description because it is able to handle the special characteristics of the input data such as low image contrast as well as rotation, translation and scale differences due to a more instable platform.For computing exterior orientation parameters, two arbitrary images with corresponding tie points are selected for relative orientation computations.For outlier detection and relative orientation computations, Essential matrix with RANSAC algorithm are applied.Next image then added to oriented subset images till all images are oriented relatively.Bundle adjustment is regularly applied on the already oriented images to avoid divergence.After relative orientation of all images, the absolute orientation is performed based on the GCPs and Helmert transformation(Pierrot and Clery, 2006).
2.3.3 3D Surface Generation:Dense image matching is carried out to generate DSM.However, low spatial resolution, low contrast and low level of textures in thermal imagery, poses some challenges in the procedure.For dense image matching, surface reconstruction problem formalized as a minimization of an energy function.The Energy function E(Z) is represented in the equation (3).() = ∑((, , (, )) +

Table 1 .
Thermal camera specifications

Table 2 .
Thermal camera specification

Table 3 .
Statistical assessment of check point accuracy