LOCATION-BASED INFRASTRUCTURE INSPECTION FOR SABO FACILITIES

Infrastructure asset management is a framework for achieving sustainable infrastructure. Based on this framework, although we often generate a three-dimensional (3D) geometrical model as a base map in management, it is not easy to acquire details of asset attributes in 3D measurement. Therefore, we focus on field-based investigation and inspection using mobile devices, and aim at assisting investigators in infrastructure asset monitoring with location-based applications. In this paper, we propose and evaluate our location-based investigation application as follows. First, we propose an inspection flow suitable for field-based monitoring. Second, we develop a Web GIS application for field-based investigation with mobile devices. Third, we propose base map generation suitable for sabo facilities using UAV and terrestrial laser scanner. We conduct an experiment in a sediment-retarding basin consisting of dikes, bridges, and debris barriers, and explore some issues in infrastructure asset monitoring using mobile devices.


INTRODUCTION
Infrastructure asset management is a framework for achieving sustainable infrastructure, such as roads, bridges, railways, and water treatment facilities. A conventional flow for groundbased infrastructure inspection is shown in Figure 1. During infrastructure inspection, we generally refer to the latest inspection documents to determine an inspected position, as follows. First, the structure to be inspected is detected after the inspector's arrival in the inspection area. Next, an inspected point is detected in the structure. Then, the condition of the inspected point is recorded and compared with the latest inspection. After that, a geo-tagged photo is captured at the inspected point. Generally, the management focuses on the low life-cycle cost in a process of construction, maintenance, rehabilitation, and replacement. Based on this framework, a 3D geometric model is often generated based on construction information modeling (CIM). Moreover, asset attributes, such as deterioration, condition, and age are acquired. To check the position of structures and structural elements and collect data related to these structures in frequent monitoring, there is a need to refer to maps, engineering drawings, databases, and technical documents (Garrett et al. 2002). Reliability, completeness, efficiency, and cost are significant indices in monitoring.
Reliability, completeness, and efficiency can be satisfied using terrestrial LiDAR, a vehicle-borne mobile mapping system (MMS), and aerial photogrammetry using an unmanned aerial vehicle (UAV), as shown in Figure 2. In the current state, although 3D scanners can acquire high resolution data, it is not easy to acquire details of asset attributes with 3D measurements. Therefore, we focus on ground investigation and inspection using mobile devices . Field-based inspection requires some location-based applications, such as geo-tagged image acquisition (photography), database interface, and navigation (Hammad et al. 2006). Mobile devices, such as tablet PCs, smart phones, and global positioning system (GPS) cameras, have the potential to assist inspectors in infrastructure asset monitoring because of their built-in sensors and components that include cameras, assisted GPS receivers, gyro sensors, Wi-Fi, microphones, speakers, vibrators, and large storage. We aimed to assist investigators in infrastructure asset monitoring with location-based applications. The control of erosion and sediment is called Sabo. The Sabo is one of significant topics in infrastructure inspection. In this paper, we propose and evaluate our location-based investigation application for Sabo facility management.

METHODOLOGY
Our proposed methodology for Location-based infrastructure inspection is described in Figure 3. Our methodology consists of inspection operations with mobile devices and 3D mapping with images and point cloud to improve conventional inspection approaches.

Mobile inspection application
The functions and performance of infrastructure inspection assistance with a mobile device, such as a tablet PC equipped with GPS, are summarized in Table 1. Category A indicates essential functions and category B indicates additional functions. In addition, we propose a data model for our Web GIS-based mobile inspection application to satisfy the above-mentioned functions, as shown in Figure 4. An inspection work is subdivided into several activities, such as geotagged image acquisition, adding a postscript to a photo, and adding a postscript to an engineering drawing. Geotagged data generated from these works are managed with Extensible Markup Language to automate file export using an inspection template prepared by municipalities and a combination of managed data, such as maps, images, and movies, using position data as a retrieval key in inspection navigation. Acquired GPS data are mainly used for the management of location and time data. The location data included represent the position of structures, camera position data, and camera azimuth and rotation data.

Location data management
The required positioning accuracy is dynamically changed by each inspection work. For example, a closed photograph requires the same position (with approximately 1 cm accuracy) and direction (with approximately 1 degree accuracy) in the latest inspection to achieve automation of image registration for detection of any change in an infrastructure inspection (Nakagawa, Katuki, Isomatu and Kamada, 2013). On the other hand, inspection point detection requires lower positioning accuracy, from approximately 10 cm to 1 m. Moreover, in structure detection, positioning accuracy is allowed to be approximately 10 m. In addition, 100 m positioning accuracy is sufficient for an inspector's arrival in an inspection area. Thus, a definition with several steps or spatial resolutions is effective in location data management. In this research, these steps are represented as levels of details (LODs), such as LOD1: address, LOD2: structure, LOD3: inspection point, and LOD4: photography, as shown in Table 2.

Base map generation
In the CIM, base maps and 3D data are required to manage processes of construction, maintenance, rehabilitation, and replacement. Online maps, such as Google Maps and OpenStreetMap, are useful for infrastructure inspection in urban areas. However, in rural areas and mountainous districts, the online maps are often insufficient for infrastructure inspection to recognize the details of natural features. Thus, base maps and 3D data should be prepared before the inspection.
In an open-sky environment, aerial photogrammetry and Structure from Motion (SfM) using UAV is more effective than ground-based scanning. On the other hand, when environments include natural obstacles, such as trees, terrestrial LiDAR is more effective than UAV and MMS. In our research, we apply both approaches to prepare digital surface models (DSM) and digital elevation models (DEM) as base maps. Particularly, we expect that LiDAR measurement can generate colored DEM from colored point cloud acquired with a laser scanner, as shown in Figure 5. However, when we acquire point cloud around riversides which include many water surfaces, many missing points will exist because of laser reflection problems. Thus, we apply a randomized algorithm for quickly finding an approximate nearest-neighbor matches between image patches (Barnes et al. 2009), as shown in Figure  6.

EXPERIMENT
We conducted experiments involving the daily and annual Sabo infrastructure inspection work in a sediment-retarding basin consisting of dikes, bridges, and debris barriers in Fukushima, Japan (see Figure 7).

Attribute data acquisition
In attribute data acquisition, we record conditions of infrastructures, such as cracks, damages and displacements, based on checklists distributed by Japanese Ministry of Land, Infrastructure, Transport and Tourism (MLIT). We assigned these checklists to meta-data and main data, as shown in Figure  8. Then, we input text data and images to record the conditions of infrastructures with some mobile devices, as shown in Figure  9. In addition, omni-directional images are also acquired to record attribute data of the conditions of infrastructures. These images are used to improve the integrity in infrastructure inspection with augmented reality applications in office works. We used two types of cameras, such as THETA m15 (RICOH) and QBiC PANORAMA (Elmo). These cameras were mounted on a monopod, as shown in Figure 10 and Figure 11. We also used a GPS logger (N-241, HOLUX) to get position data with omni-directional images. Acquired omni-directional images were stitched to be panoramic images and movies. These images and movies are viewed with a head-mount display (Oculus Rift), as shown in Figure 12.

Attribute data acquisition
In our experiment, 213 images were acquired with mobile devices. Using geo-tag data, these images are reverse-geocoded into a map with GPS position data, as shown in Figure 16. Then, acquired images are grouped into 36 viewpoints. In a manual work, it took 3120 sec. On the other hand, it took 4 sec in our position and azimuth filtering. Therefore, we confirmed that our application drastically shorten a work time for the image retrieval, as shown in Figure 17.

3D map generation
We generated a colored DSM with a ground resolution of 4 cm from the UAV. As shown in Figure 18, we have confirmed that ground surfaces, such as roads, gravels, trees, and water surfaces, were reconstructed well. Figure 18. DSM generated with images from UAV Moreover, we generated a colored DEM with a ground resolution of 5 cm from point cloud acquired with the terrestrial laser scanner. As shown in Figure 19, several missing areas exist in the DEM due to laser reflection problems. Thus, we applied the PatchMatch to the colored DEM from the laser scanning data. We have confirmed that missing areas were reconstructed as well as an aerial image, as shown in Figure 20.

DISCUSSION
Positioning from LOD1 to LOD3 requires from 100 to 1 m accuracy. Thus, single GPS positioning is suitable for position data acquisition. However, LOD4 requires 1 cm accuracy with precise positioning, such as a real-time kinematic GPS (RTK-GPS). Generally, low-cost inspection restricts the use of expensive devices such as an RTK-GPS. In low-cost inspection, the performance of satellite positioning is generally improved by assisted-GPS, differential GPS and multi-GNSS positioning using GPS, GLONASS, and QZSS. Data fusion of GPS and dead reckoning also improves the performance of positioning. However, although these approaches improve availability, they have almost no effect on positioning accuracy improvement (Inaba et al. 2013). In this research, satellite positioning was assumed to have 1m accuracy, even if we could apply improvement approaches to positioning accuracy. Thus, a location data management approach using movies was applied in LOD4 (precise positioning). This approach assists inspectors to determine a position in a photography using a movie that was captured in the latest inspection and the attached approximate position data acquired with GPS.

Integrity of location data
In this research, we focused on a mobile application for infrastructure inspection, and location data management for inspection navigation. In fact, the availability of an application using location data depends on the integrity of the location data (Yabuki, 2013), and the integrity of the location data strongly affects location data browsing, and inspection recording and log browsing. Here, to improve the integrity, we focused on a combination of acquired data, such as location, photo and movie data. We classified three combinations among these data types, as follows.
a) Position (latitude and longitude), azimuth, and elevation angle Several minutes were required for detection of the previous inspection position. The most suitable procedure for the detection of the previous inspection position in our experiment was as follows. First, we referred to longitude and latitude to detect the position. Second, we referred to the azimuth angle using magnetic sensor data taken from a tablet PC. Third, we referred to an elevation angle using gyro sensor data taken from the tablet PC.
b) Position (latitude and longitude), azimuth, elevation angle, and photograph We determined the position for inspection with position, azimuth, and elevation angle data. Moreover, we could reconfirm the position with a photo taken in the previous inspection. However, when a difference existed between the position determined from position, azimuth, and elevation angle data and the position estimated from a geotagged photo, inspectors were unable to determine the true position for the inspection.
c) Position (latitude and longitude) and movie We did not conduct an experiment related to video capture, which provides the shortest capture time and efficiency for navigation. However, we confirmed that an effective approach for inspection navigation was to capture a movie that shifted from a position that was far from an inspected point to one that was close. Moreover, we confirmed that a movie was a better approach than a picture for change detection in infrastructure inspection when an inspection point was in a complex environment, such as craggy places and Sabo sites.

Performance of operation using tablet PCs
We qualitatively confirmed that automation of location and time data recording is more reliable than manual paper-based recording in infrastructure inspection. On the other hand, paperbased recording offers an advantage for documentation in an outdoor location, because text input with a mobile PC is timeconsuming work. Moreover, we confirmed that raindrops worsen the performance of the touch interface, even when a waterproof tablet PC is used. Position data acquisition depends on single GPS positioning. Although our study area consisted of open-sky environments and structures, GPS positioning was insufficient for positioning in LOD3 (inspected position detection) in an area surrounded by mountains or under a bridge. On the other hand, we have confirmed that geotagged movie was effective in estimating the LOD3 and LOD4 position data. Even if position data included a positioning error caused by low dilution of precision and multipath transmission, an inspection position could be detected using movie guidance. Moreover, we could also focus on geotagged omni-directional camera data to detect an inspected position. However, although we used Google Maps and OpenStreetMap as the base maps in our preliminary experiments, there was a difficulty in managing the frequent map updates. Therefore, we prepared DSM generated using images from UAV and DEM generated from a laser scanner as a more reliable base map. Although we prepared high resolution data, we used them as ortho images because it is not easy to view 3D data with tablet PC.
In addition, we confirmed that inspection work using a tablet PC held with both hands was dangerous on bad roads, in riverbeds, and in craggy places. Therefore, we propose to use hands-free applications using wearable devices and voiceguided applications with geofencing techniques to improve safety in inspections using a mobile device.

SUMMARY
In this paper, we focused on ground investigation and inspection using mobile devices. We aimed to assist investigators in infrastructure asset monitoring with locationbased applications. We proposed and evaluated our locationbased investigation application for facility management based on CIM. Through our experiment, we explored several issues in infrastructure asset monitoring using mobile devices. Integrity in positioning should be improved to achieve more reliable and effective inspection works. Therefore, we proposed an LOD definition for positioning data management in inspection works. Moreover, we proposed combinations of base maps and several types of data acquired with a mobile device in inspection works to improve reliability, completeness, and integrity in positioning.

ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant Number 26870580. Moreover, our experiments are supported by Fukushima City and Fukushima River and National Highway Office, Tohoku Regional Development Bureau, Ministry of Land Infrastructure and Tourism.