POTENTIAL OF UAV-BASED LASER SCANNER AND MULTISPECTRAL CAMERA DATA IN BUILDING INSPECTION

Conventional building inspection of bridges, dams or large constructions in general is rather time consuming and often cost expensive due to traffic closures and the need of special heavy vehicles such as under-bridge inspection units or other large lifting platforms. In consideration that, an unmanned aerial vehicle (UAV) will be more reliable and efficient as well as less expensive and simpler to operate. The utilisation of UAVs as an assisting tool in building inspections is obviously. Furthermore, light-weight special sensors such as infrared and thermal cameras as well as laser scanner are available and predestined for usage on unmanned aircraft systems. Such a flexible low-cost system is realized in the ADFEX project with the goal of time-efficient object exploration, monitoring and damage detection. For this purpose, a fleet of UAVs, equipped with several sensors for navigation, obstacle avoidance and 3D object-data acquisition, has been developed and constructed. This contribution deals with the potential of UAV-based data in building inspection. Therefore, an overview of the ADFEX project, sensor specifications and requirements of building inspections in general are given. On the basis of results achieved in practical studies, the applicability and potential of the UAV system in building inspection will be presented and discussed.


INTRODUCTION
Conventional building inspection requires special heavy vehicles, is time consuming and cost expensive.Due to the high number of buildings, which need rehabilitation measures (Radke, 2014), an effective usage of limited resources, such as time and equipment, is important.This requires a highly up-to-date and effective monitoring system for all important constructions of public life.For carrying out the monitoring tasks, a flexible and valuable tool such as UAVs with special sensors can be very valuable.There are many investigations for using light-weight sensors on unmanned aircraft.Mainly RGB cameras are mounted on UAVs.Such an digital image-based building inspection and monitoring is shown in (Eschmann et al., 2012) by using a micro aerial vehicle and a high resolution digital camera.The results were a digital fac ¸ade reconstruction based on image mosaic and automatically detected cracks.Morgenthal and Hallermann (2014) discuss the flight properties of UAVs and the resulting influencing factors on the image quality.The authors in (Roca et al., 2014) utilized a combination of a light-weight laser scanner and an UAV for generating 3D point clouds for building assessment.An application for the combination of an UAV and a thermal camera is presented in (Lagüela et al., 2014).There, thermal data of hard-to-access areas such as higher fac ¸ades and roofs was captured to complete data from a handheld camera on the ground.The systems as stated above normally consist of one UAV with a special sensor, which covers a specific application.However, a wide variety of damage types, which are not detectable with one kind of sensor, occur on buildings.For this purpose, an adaptive low-cost system (Figure 1) consisting of three flying robots and a ground station was realized in the ADFEX project (Adaptive Fed- erative 3D Exploration of Multi Robot Systems).Related to the facts mentioned above, the goal was to develop a time-efficient and flexible alternative for object exploration, object monitoring and damage detection.The three robots are equipped with standard sensors for navigation and obstacle detection (GNSS, IMU, ultrasonic sensors, navigation cameras) as well as special sensors (laser scanner, high resolution camera, near infrared camera, thermal camera) for data acquisition, which are distributed to the three platforms.This approach offers the advantage of larger operational flexibility in mission planning and execution.In detail, the platforms will have a lower weight, a longer flight duration and the needed sensor can be used on specific spots of damage.With such a system an inspection of an unknown object can be performed as sketched in Figure 3.The user has to mark the area of interest via a polygon of GNSS-coordinates or a boundary box   in a provided map.Firstly the unknown area will be explored by UAV #1 equipped with a laser scanning range finder.The resulting geo-referenced rough 3D point cloud is well suited for the following mission planning by the user and in addition for an obstacle map.Subsequently, the data acquisition with the RGB, near infrared and thermal camera mounted on UAV #2 and UAV #3 will be carried out.By processing this data, the generation of dense and accurate 3D point clouds with multispectral attributes, the visual inspection based on geo-referenced images and the creation of a map of damages is possible.Further information can be found on the website http://www.adfex.eu/.
This contribution shows the development and construction of an UAV based tool for building inspection exemplary for German industrial standards.The first part deals with the utilized platform and sensors, the requirements of building inspection, as well as a short excursion into the calibration and sensor alignment.Afterwards, it will be focussed on practical studies carried out and the placement of the developed system in building inspection tasks.Finally, the paper will finish with a conclusion and proposals for the next steps in future work.

REQUIREMENT PROFILE OF BUILDING INSPECTION
Within the framework of public service, civil engineering structures like roads and ways are obliged to inspect periodically.The key components of the road building management are the security and the preservation of the serviceability.Special attention needs to be paid for road bridges.The bridge monitoring includes the detection, localization and rating of damages and defects with regard to structural stability, traffic safety and durability.After de-termining the cause of damage, suitable rehabilitation measures have to be provided.In Germany, the procedure and amount of bridge monitoring is regulated by the German industry standard (DIN1076, 1999) complemented by the directive (BASt, 2007).
It establish rules for planning of conservation measures, consistent acquisition and evaluation of inspection results as well as for carrying out economic feasibility studies.In addition, buildings have to be inspected in general even if they are excepted from inspection and monitoring obligation by law.Inspections or maintenance at regular intervals and rehabilitation measures at an early stage ensure the fulfilment or a significant extension of the intended economic life of engineering works (Figure 4).Due to a wide range of light-weight high-performance imaging sensors, an UAV equipped with such a sensor can be an alternative to support or to replace the conventional procedure.Detailed inspection procedures and potential threats for the test engineer can be minimized.The monitoring is grouped into the individual parts of the bridge.Study areas are separated into the single bridge components such as the concrete, reinforcing steel or joints and bearings.Every area is further subdivided into study features relating to the area.Concrete features to be examined are holes, water emergence, cracks, rust marks or mechanical damages.For a complete bridge monitoring, the study features have to cover all potential sources of defects.In case of bridge bearings, the condition, position and damages must be checked.More information can be found in (Schach and Weller, 2015).
Most of the named study features have to be evaluated visually on-site by a skilled engineer.Through the application of an UAV, there is no need to enter the building.This leads, in case of multitemporal condition assessment, to a reduction of effort and risk for the test engineer.With the conventional method, these periodical generated image data at a fixed position is time-and costintensive.

Unmanned Aerial Vehicle -basic equipment
The three flying robots are identical in construction.The CAD-MIC Goliath Coax 8 frame has four cantilever arms each with two coaxial mounted rotors (Figure 2a).The basic sensors configuration of each robot includes a single frequency GNSS-receiver u-blox LEA-6T, an ADIS16407 IMU module and two cameras with fisheye lens, which are oriented upwards and downwards for determination of the UAV's position and orientation by using a visual SLAM navigation filter (Klix et al., 2014).Additionally, there are ultra-sonic sensors integrated to detect obstacles up to a distance of about 6 m arround the octocopter.Furthermore, all UAVs have additional special sensors for special tasks or applications.

Special Sensor UAV #1
The first UAV is equipped with the low-cost 2D laser scanning range finder (LSRF) Hokuyo UTM-30LX-EW (Figure 2b), which works according to the time-of-flight principle and measures distances in a range of 0.1 m up to 30 m with an accurracy given by the manufacturer of 30 mm to 50 mm .The LSRF captures 1080 distances in a plane during 1/40 s with an angular resolution of 0.25 • resulting in a field of view of 270 • .Important properties are the low weight of 210 g and the compact size (62 mm × 62 mm × 87.5 mm), which predestines the LSRF for using on UAVs.

Special Sensor UAV #2
The RGB camera Prosilica GT3300C from Allied Vision Technologies (Figure 2c) is equipped with a 3296×2472 px CCDsensor and a pixel size of 5.5 µm.With 14.7 images per second, high-resolution images in a sufficient frame rate are guaranteed.A F-mount camera bayonet allows a flexible application concerned choice of the lens.The dimension of the camera body with 121 mm × 59.7 mm × 59.7 mm and a weight of 314 g is well suited for use on the ADFEX-copter.

Special Sensors UAV #3
The multispectral camera system (Figure 2d) consists of the RGB camera Mako G-419C, the near infrared camera Mako G-223NIR and the thermal camera FLIR A65, all fixed rigidly on a bar.All cameras have a compact size and a low weight, whereby the integration of the camera bar does not present a problem.The detailed specifications of the sensors are given in Table 1.

CALIBRATION AND ORIENTATION
A sensor calibration and alignment of the entire system is indispensable for high-quality results.The calibration strategy includes the determination of lever arm and boresight angles between the sensor coordinate system (specified itself) and the platform coordinate system (given by the IMU) as well as the interior sensor parameters.In detail, there are the focal length, principal point and lens distortion for cameras as well as distance and angle correction for LSRF.An integrated self-calibration strategy was developed for a simultaneously estimation of all unknown parameters to guarantee a correct calculation with different observation types (distances, angles, image point measurements with different sensors).For this purpose, an adaptive fully integrated and modularly structured bundle adjustment was implemented (Mader et al., 2014).A variance component estimation ensures an optimal weighting of the observation groups and additional restrictions (Koch, 1999).This approach was extended with ge-Figure 5: Calibration field for self-calibration approach with cones and thermal markers as reference targets for laser scanner and multispectral image point measurements.
ometric and stochastic models for estimation of lever arm and boresight angles for all sensor-UAV combinations as well as each camera of the multispectral camera system.Due to the different types of sensors, a special target field was necessary for linking the observation types.Figure 5 shows the reference field consisting of distributed cones with markers as geometric primitive for the LSRF and a target field with thermal markers, which are measurable with all cameras (RGB, near infrared, thermal infrared).

PRACTICAL STUDIES
The focus of this work will be the potential and placement of UAV-sensor-systems in building inspection.For this purpose, practical studies were done and will be shown in this section.

Obstacle map and mission planning based on a rough point cloud
For evaluating the potential of the combination of light-weight LSRF and UAV using a single-frequency GNSS, the following aspects will be interesting: (1) precision of the relative position of neighbouring strips, (2) the quality of the point cloud in comparison to reference data and (3) the accuracy of the absolute georeferenced 3D point coordinates.The first two aspects were evaluated by practical studies.The accuracy of the absolute position was not investigated and should correspond to most manufacturer specifications with usually 2 m to 2.5 m in lateral direction.The accuracy of the GNSS height component does not matter by using a relative barometric altitude measuring.
For the first aspect, a lighthouse was captured by an UAV (similar to the ADFEX UAV) and a Hokuyo LSRF.The lever arm and the boresight angles were manually estimated by a skillful sensor arrangement, because the calibration strategy was not available at that time.The remaining inaccuracies can be neglected with regard to the GPS-and IMU accuracy as well as a low flight altitude.Six overlapping strips (two along and four across the lighthouse) with a total of 470 000 points were recorded and analysed.The unaligned strips show obvious discrepancies, whereby the lateral deviation is larger than those in the height (Figure 6a).This effect should be checked by a registration of the strips to strip 2 to 1 3 to 1 4 to 1 5 to each other with strip 1 as reference.Firstly, a coarse alignment was carried out with some features of an overlapping area with recognizable building geometries (roof, ridge, chimney).For the fine registration the iterative closest point (ICP) algorithm was used.The resulting point cloud is shown in Figure 6b and represents the lighthouse well.The alignment of strip 6 to strip 1 was not possible, because of sparse overlap of suitable features for a reliable registration.In Table 2 the transformation parameter (translation and rotation) of the strips 2 to 5 related to strip 1 are given.On the one hand, the transformation parameter confirm the observed effect, that the lateral deviations (X, Y) are larger than the height deviation (Z).And on the other hand the discrepancies seem to grow with increasing flight time, which can be explained with a changing of satellite constellation.This is confirmed by the fact, that the individual strips captured in a short time duration show good geometric characteristics.In Figure 6c  the strips can be assigned to the low accuracy of the used IMU, which is designed for UAV navigation only.
In a second campaign, the LSRF was used on the ADFEX UAV for capturing a bridge with a length of 135 m (Mader et al., 2015).Scans alongside the bridge with low flight speed led to 20 000 scan lines in a flight-time of about 10 min.The resulting point cloud represents the bridge sufficiently and shows a root mean square errors (RMSE) of point deviations after ICP of about 31.7 cm compared to a terrestrial laser scanner point cloud as reference.
Problems with the IMU were responsible for larger discrepancies.
Regarding the problems, which were encountered in the bridge and the lighthouse campaign, the data in both cases is sufficient for obstacle maps and an interactive mission planning by the user on-site.The utilization of real time kinematic (RTK) and differential GNSS (DGNSS) method as well as a higher-quality IMU would improve the results significantly.Regarding this, (Eling et al., 2015) presented an UAV with high-performance positioning and navigation sensors as well as its potential of accuracy.

Visual inspections based on multispectal image data
The visual damage detection is an important aspect of building inspections.The quality, degree of detail and the completeness of the captured datasets are the basis for an appropriate assessment of the test engineer in regard to the further rehabilitation measures.For optimizing the resulting data the most influential parameters or factors are listed in the following: • illumination of the region of interest • exposure time

• gain value
• resolution
For a good visibility of important building features a sufficient illumination of the damage spot is important.Deficits regarding this can be compensated with an integrated spotlight or by adapting the exposure time, the gain value and the aperture.The exposure time is restricted by blur effects caused in UAV vibration and motion.A too high gain value leads to a quality reducing noise and an open aperture reduces the depth of field.It is important to find a good balance between this parameters.The degree of detail is mainly influenced by the sensor resolution and the focal length.The resolution is given in the most cases by sensor specification, so the lens have to be adapted to the application.For a complete dataset a correct aligned UAV-sensor-system (lever arm and boresight angles) is necessary to guarantee the correct sensor orientation to the object.
To evaluate the potential of the sensor, some interesting damage spots were captured (Figure 7).In general, the majority of damage types or study features can be detected by using RGB image data.Figure 7a shows an efflorescence on the bottom side of the bridge, which indicates water emergence or moisture penetration.Spallings and cracks in the coating or in the concrete itself can be an indication for a settlement of the building.In this case, a multitemporal measurement of the crack width is necessary to prevent static problems.Figure 7c shows, according to keep the safety distance, that crack widths of 0.2 mm are clearly identifiable.In Figure 7b a spalling on a bridge bearing (hard-to-access area) is recognizable.It can be assumed that other obvious damages such as contamination, concrete holes, rust marks, exposed rebars or foreign matters are also detectable in high-resolution RGB images.
The distinction of surface contamination by foreign matters and plant cover like moss, presents a challenge in RGB image data.Therefore, the near infrared camera is used for the identification of living vegetation.Figure 7d presents a partially vegetation covered sandstone bridge, whereby the vegetation appears bright because of their high reflectivity in the near infrared spectrum.Due to this fact, vegetation can be better identified.By using band ratio parameters (BRP; Rouse et al. 1973, Kriegler et al. 1969 based on the camera channels, vegetation or other properties will become more visible.Some building defects become visible by temperature differences on the building surface.Water-bearing cracks or wet spots, caused by a malfunctioning drainage system, have an other temperature than the surrounding dry surface and can be detected in thermal infrared image data.In the case of Figure 7e, the wall was heated by the sun, so that the wet spots had a lower temperature as the surface and appears as darker spots.An other damage is the replacement of the coating from the building surface in an early stadium.This effect is not recognizable in RGB images (Figure 7f), but causes in temperature differences, which are visible as bright spots in thermal data (Figure 7e).

Imaged based 3D point clouds
This part of the contribution deals with 3D point clouds purely derived from image data.Three studies (RGB, thermal, multispectral imagery based point cloud) will be shown with regard to their accuracy and properties such as density and attributes.The reconstruction of all point clouds was carried out by using SfM tools.The principle of SfM-techniques in general can be found in further literature, e.g.(Koenderink and Van Doorn, 1991), (Beardsley et al., 1997) and(Pollefeys et al., 2000).In a first investigation, a bridge was imaged by using the RGB camera mounted rigidly on an UAV (Mader et al., 2015).For this purpose, about 500 images were captured with an overlap of at least 80 %.On basis of this data the bridge was reconstructed by using SfM-tools, in detail with the commercial software package Agisoft PhotoScan.As result, a dense point cloud with about 13 million points of the entire bridge was generated.Problems were gaps and inaccurate points on poorly-lit areas under the bridge.
A comparison with a terrestrial laser scanner point cloud showed RMS values of point deviations of 4.9 cm, whereby the majority of the points had deviations under 4.1 cm.Westfeld et al. (2015) showed the generation of dense 3D point cloud purely based on thermal imagery.The complex fac ¸ades of a courtyard building with dimensions of 50 m × 50 m × 20 m were captured in a raster pattern.On basis of about 15 000 images the reconstruction by SfM-tools (VisualSfM, Agisoft PhotoScan) and the dense point cloud generation were realized.Extensive accuracy analyses showed that the achieved results with purely thermal images processed were good.The coordinates of reference target points were calculated by a spatial intersection method on basis of measured image coordinates and with SfM-tools determined camera orientations.In lateral direction the RMSE was between 4 mm and 5 mm and in depth direction the RMSE was 18 mm.Concerning the utilization of the system in building inspection, potential of improvements of geometric accuracy and object characteristics as well as the point density should be investigated.Further the usage of purely thermal images can be problematic on objects which are poor in thermal features.An extension with a RGB camera would be useful to improve the quality of the results.Additionally, a NIR camera completed the multispectral camera system to cover the near infrared spectrum.As stated above, all cameras are mounted rigidly on a bar.For generation of multi-attributed 3D point cloud, in a first step the building is simultaneously captured with all cameras.Subsequently, the scene reconstruction and the dense point cloud generation is carried out by using SfM-techniques.The processed RGB camera orientation, the calibrated lever arms and boresight angles (see section 4) are the basis for calculating the camera poses of the near infrared and thermal camera.Finally, the near infrared and thermal attributes are mapped to each point of the 3D point cloud.In addition, it is possible to calculate band ratio parameter for the entire point cloud on basis of the spectral attributes.This workflow was tested in a practical study exemplary on a damaged wall with dimensions of 4.37 m in width, 2.0 m in height and 0.5 m in depth.Firstly, the calibration of the camera parameter and the alignment of the cameras to each other were performed.The results of the lever arm and boresight angles are given in Table 3.The reconstruction was carried out on basis of 19 camera positions (approximatly rastered in a 3 × 6 pattern) with an overlap of 70 % to 80 % in horizontal and about 70 % in vertical direction.For this study, the camera bar was mounted on a tripod.To simulate the situation on an UAV as realistic as possible, an exposure time of 1/500 s, an aperture of 5.6 , the auto-gain-mode and distances to the object with 4 m to 5 m were realized.The processed point cloud consists of about 2.9 million points (only wall) resulting in a point density of about 321 000 points/m 2 .Furthermore, the orientation parameter of the near infrared and the thermal camera were calculated and their attributes were mapped to each point of the point cloud (Figure 8a -8b).To demonstrate the potential of combination the multispectral bands, a BRP for detection of living vegetation (Figure 8d) was calculated by using the gray values of the red and near infrared channel with: where N IR : grey value of near infrared camera red : red channel of RGB camera This BRP and other combination can be added to each 3D point with multispectral information.
For an accuracy test, coded targets were attached to the wall and the 3D coordinates were estimated on basis of a convergent multiimage set, which was taken with a DSLR camera.The image data was processed with the commercial software package AICON 3D Studio.The coordinate system is oriented as follows: the surface of the wall forms the X-Z-plane and the Y-axis points into the wall.For the reference coordinates an accuracy of 0.10 mm in lateral direction and 0.12 mm in depth direction were achieved.target points were calculated on basis of measured image coordinates, determined camera pose and calibrated camera parameter from near infrared and thermal camera by using spatial intersection.The RMSE of the coordinates are shown in Table 4.The result of the thermal camera is in an acceptable range with regard to the intended application and a ground sample distance (GSD) of 5.88 mm related to an object distance of about 4.5 m.The situation is different to the RMSE of the object coordinates resulting from near infrared image data.An additional analysis of the results, in detail a 3D Helmert transformation of the determined target field coordinates (with fixed scale) to reference coordinates, shows a systematic error in this data.In Table 5 the RMSE after the transformation are presented.Therefore, it is considered that the position of the near infrared camera has changed related to the other cameras in the time between the calibration and the measurement (few days).Consequently, the calibrated lever arm and boresight angles of the near infrared camera were not correct at the time of measurement.This fact is confirmed by carrying out the same procedure (used to the image data of the wall) with the multispectral image data of the calibration procedure.The result is given in Table 6 and shows that the procedure works and the reachable accuracy of this system.

CONCLUSION AND PLACEMENT IN BUILDING INSPECTION
The paper presents a time-and cost-efficient UAV based system for building inspection tasks.On three flying robots special sensors, in detail a laser scanning range finder, a high resolution RGB camera and a multispectral camera system, were distributed for flexible mission execution.For evaluating the performance of each UAV-sensor combination, several practical studies were carried out.The results of the LSRF are sufficient for use as basis for on-site mission planning, obstacle maps and deriving geometrical properties of the investigation object.Further the object-as well as geo-referenced images of the high resolution RGB camera, the near infrared camera and the thermal camera fulfil the requirements for detection and localization of the majority of damages on objects surface (visual inspection).Moreover the generation of geometrical high accurate and dense point clouds with different attributes was shown.The data is the basis for orthophotos or indices, which can be considered for an examination of the building condition.Besides the advantages of the UAV based monitoring as well as the versatile and flexible applications, the following technical, environmental and legal aspects for UAV systems should be respected: • need of flight permission • location and environmental conditions (weather, traffic, close development...) • economic feasibility • financial risk in the event of a crash • free line of sight between UAV and monitoring object.
An other fact is, that is not possible to solve all required inspection tasks.Image data is able to map all study features on the surface of the construction multispectral and in high resolution.
But for example defects of the reinforcing steel will only be visible due to surface spallings or rust marks on the concrete.Nevertheless, UAV-based damage detection is a helpful and flexible assisting tool in building inspection.The multispectral analysis of building structures significantly facilitate the visual interpretation of feature conditions and damages.

FUTURE WORK
The results of the practical studies realized with the UAV system, indicate the potential of UAV-based building inspection.However, technical limitations were pointed out.The accuracy of the rough point cloud based on LSRF data depends directly on the accuracy of determined UAV pose.Integrating RTK or DGNSS as well as higher-quality IMU for navigation solution on the UAVs would increase the quality of laser scanner point cloud significantly.A second aspect is to evaluate the improvement of image quality by using a spotlight.That is a important fact to guarantees a complete inspection also of poorly lit building areas.Further analysis of single point precision regarding the multispectral point cloud needs to be continued in future.Finally, the usability of appropriate indicies and orthophoto mosaic for building inspection must be investigated.

Figure 1 :
Figure 1: System design of ADFEX project.The capturing system consists of three UAVs equipped with special sensors and a ground station.

Figure 3 :
Figure 3: Scenario from an unknown area or building (top) via the first exploration with a laser scanning range finder mounted on an UAV (mid) to the fully explored building (bottom).Results of the exploration may be dense point clouds and detail images for damage detection.

Figure 6 :
Figure 6: 3D point cloud of the lighthouse on basis of data from laser scanning range finder mounted on an UAV.(a) Unaligned point cloud with large lateral discrepancies.(b) Aligned point cloud.(c) The separate strips (in this case strip 2) shows the lighthouse with good geometries.(d) A side view of the complete strip 2.
Figure 7: damage spots of: (a) efflorescence (RGB), (b) spalling on bridge bearing (RGB), (c) crack with a width of 0.2 mm (RGB), (d) vegetation on a sandstone bridge (NIR), (e) wet spot in circle 3 and replacement of coating from surface in circle 1 and 2 (TIR), (f) wet spot (circle 3) and replacement (circle 1 and 2) identified is not clearly visible in RGB image data.

Table 1 :
Sensors of multispectral camera bar mounted on UAV # 3

Table 2 :
Transformation parameter of strips 2 to 5 related to the first strip, all in the north-east-down coordinate system.The Xand Y-coordinate forms the lateral component and the Z-axis is similar to the height component.

Table 4 :
RMSE of marker coordinates based on image point measurement, external and interior orientation of near infrared as well as thermal camera.

Table 6 :
RMSE of marker coordinates of the calibration field.