FUSION OF MOBILE LASER SCANNING AND PANORAMIC IMAGES FOR STUDYING RIVER ENVIRONMENT TOPOGRAPHY AND CHANGES

In recent years, laser scanning measurements have been widely used to detect topographic and urban features. In this article, we present a method to integrate mobile laser scanning (MLS) data and panoramic images for producing a textured surface model within a fluvial environment. We also describe the use of the textured surface model to characterize and interpret changes caused by high discharges. The accuracy of MLS-based digital terrain models (DTM) and change detection is evaluated by using static terrestrial laser scanning (TLS) measurements as a reference. The laser scanning data was measured with the mobile mapping system developed in co-operation with the Finnish Geodetic Institute and Aalto University. The boat-based system (BoMMS) has been specially developed for fluvial geomorphology and river mapping. We created panoramic image mosaics by combining sequences of digital images. These methods are suitable for collecting detailed data when traditional methods are unable to gather the data needed for fluvial modelling. The paper also presents a process chain for combining these data sources.


INTRODUCTION
Airborne laser scanning (ALS) measurements and digital imagery techniques have been widely used in research on topographic and urban features because they provide an accurate way to create 3-D models.Mobile mapping systems (MMSs), or mobile laser scanning (MLS), have become more popular techniques when high resolution and details are needed to model large areas from a terrestrial point of view.Laser scanning and photography equipments have also been mounted on a number of mobile platforms, including cars (e.g., Petrie, 2010;El-Sheimy, 2005;Manandhar and Shibasaki, 2001;Kukko et al., 2007), boats (e.g., Alho et al., 2009) and unmanned aerial vehicles (UAV) (e.g., Jaakkola et al., 2010).These systems have been developed for road inventories, fluvial geomorphology, forestry, and environmental modelling (Holopainen et al., 2010).
In a river environment, MLS data offers a variety of interesting possibilities, such as 3-D mapping of the point bars.Compared to traditional ALS, MLS is better suited to mapping smaller areas with a higher point density (Alho et al., 2011).The method can also be used to acquire very precise multi-temporal data for change detection (e.g., Vaaja et al., 2011).The geometry of MLS is dependent on the location of the sensor, and the pulse density varies as a function of the range.The difference between terrestrial and mobile laser scanning is in the evenness of the data and the perspective.In MLS, the point cloud is evenly distributed along the driving direction, and the viewing direction to the target remains constant (Hyyppa et al., 2009).The accuracy of MLS is almost equal to terrestrial laser scanners.In general, the quality of MLS is limited mainly by the GPS/IMU georeferencing system.
Terrestrial panoramic techniques make it possible to capture high-resolution wide-angle images (Kekre and Thepade, 2008;Haggren et al., 2004).With MMSs, panoramic images are usually collected with multi-camera systems, such as Google Street View, or with a CCD-line panoramic camera (Reulke et al., 2004).
The fusion of laser scanning and image texture improves the interpretation and detection of features and objects (Ronnholm, 2010).The result can be a geo-referenced panorama image, textured surface, building facade, or virtual environment (Zhu et al., 2009;Alamus et al., 2008;Brenner and Haala, 1998).The textured DTM can also be used to assist a laser-based change detection model.When considering data fusion, both images and laser scanning data must be registered in the common coordinate frame.The main approaches for registration include determining the sensor orientations separately using ground control features, solving the relative orientation between data sets, or making a system calibration of the integrated imaging system (Ronnholm et al., 2007).
An overview of the most common applications for fused data is provided in a study by Ronnholm et al. (2007).The integration of panoramic images and laser scanning has been utilized in the forest inventory (e.g., Haala et al., 2004), in colorizing 3D laser point clouds (e.g., Scheibe et al., 2004), in building facade reconstructions (e.g., Pu and Vosselman, 2009), in improving the field-of-view of images for more accurate registration (e.g., Ronnholm et al., 2003), and in calibrating the interior geometry of laser scanners and images (e.g., Parian and Gruen, 2005;Schneider and Maas, 2007), to name a few.data, 2) relative orientation of the panoramic images, and 3) visualisation.The process chain includes laser scanning with MLS, the creation of panoramic images, the registration and integration of data sets, the visualization of 3D data and the analysis of temporal changes to the river bank.
Integrating panoramic images with terrestrial laser scanning (TLS) has been commonly implemented at the instrument level (Wendt and Dold, 2005), enabling an easy approach for colorizing point clouds and also assisting with 3D measurements.However, most of the MMSs acquire single images instead of panoramic images.Cameras and laser scanners in MMSs are mounted on the same platform and the system is calibrated.In addition, navigation information comes from direct orientation sensors (e.g., Schwarz et al., 1993;Grejner-Brzezinska, 1999), including GPS, the inertial system, and sometimes also the odometer.The accuracy of the direct orientation sensors is sufficient for purposes that do not require the highest accuracies, such as the visualisation of virtual models.However, if better accuracy is needed, additional orientation using ground control features should be utilized.
The aim of the study was to integrate panoramic images and mobile laser scanning data from a river environment and to apply the merged data for change detection of river topography.In further studies, textured, high-detailed 3-D models will be coupled for fluvial processes such as sediment transportation capacity.

Study Area
The study area, located on the Pulmanki River, is a 58 km-long tributary of the River Teno (Tana) in the subarctic.It flows near the border of Finland and Norway at latitude 69.95 N and longitude 28.10 E, where Lake Pulmanki divides the river into two parts.On the Finnish side, the river is about 10 km in length and builds up a small delta that flows into the lake.The river has eroded a 30 m deep and 20-50 m wide channel.The river is characterized by steep banks, it is highly sensitive to erosion and relatively flat, and has large point bars and bushtype vegetation.The target area of our experiment was one of the large point bars (13 600 m2, Fig. 1).Test data for our experiment was collected in the late summers o f2008 -2009.
Figure 2. Workflow for combining laser data and panoramic images after data acquisition

Laser Data Acquisition
We mounted the ROAMER mobile laser unit (Kukko et al., 2007) on a boat (BoMMS, Boat-based Mobile Mapping System, Alho et al., 2009) and a cart platform, abbreviated as Cart Mobile Mapping System, CartMMS (Fig. 3).We used the latter system in the river environment, especially for mapping the point bars.

Workflow
The workflow for the processing and combining of the laser data and panorama images is presented in Fig. 2. The process is divided into the following three stages: 1) preparation of the The operation of the ROAMER system is based on GPS and inertial measurements for localization, and a time-synchronised laser scanner (FARO Photon80), which measures three dimensional points from the surrounding objects in profiling mode.The scanner was tilted 60 degrees from the horizon level in the cart.The system parameters used in the laser data collection (Table 1) were a 30 Hz scanning frequency, a 120 kHz point measurement rate, and a point spacing of 15.7 mm at a range of 20 m The navigation system was the Novatel SPAN (DL 4plus GPS and Honeywell HG1700 AG11 IMU).The system recorded GPS data at 1 second intervals, and it recorded the altitude and acceleration information from the IMU at a frequency of 100 Hz.We set up a GPS reference station at a known point to provide correction data for the trajectory determination.During data collection, the length of the baseline from the reference station to the system varied from 0.5 km to 1.5 km.We carried out the synchronization of the scanning and imaging sub systems using bi-trigger synchronization, and the GPS receiver recorded the event log.We computed the trajectory during the post-processing phase and parsed the event log to provide temporally synchronized georeferencing for each of the laser points.The raw laser points were subsequently transformed into a 3-D point cloud in a mapping coordinate system according to the trajectory.Additionally, we used the terrestrial laser scanner Leica HDS6000 to collect reference data.

Laser Data Processing
The purpose of laser data processing was to create as detailed a digital terrain model (DTM) for our study area as possible (Fig. 4).The laser point clouds were georeferenced in a 3-D geocentric WGS84 (World Geodetic System 1984) frame.Furthermore, we filtered the point clouds using isolated and intensity routines that removed noise points from the air and below ground (Soininen, 2011).
Figure 4.A detailed DTM from our test area We created the DTMs for 2008 and 2009 from filtered point clouds using a regular grid approach, which calculated elevation for points as the average height within a selected grid size.We achieved a detailed DTM with this approach, but the point density was reduced at the same time.Usually, the density of mobile laser data is high and it is necessary to reduce the amount of data for the post-processing phase.We selected a grid size of 0.1 m with 1.3 million points.This model had enough details for riverine change detection and was easy to process (Fig. 5).
Figure 5.Comparison of the detailed features presented with a 0.1 m grid size of the point cloud (above) and the panoramic image (below).The white arrow indicates the same boulder cluster in both figures

Panoramic Images Acquisition
In our study, the panoramic terrestrial images provide additional information for the DTM.Panoramic techniques allow for the possibility to capture high-resolution wide-angle images, and the images can be comfortably achieved under the same circumstances with laser data.However, the terrestrial point of view typically suffers because of obstacles that are in the way, thus it requires several camera stations.
We created panoramic images by combining a panoramic mosaic of a sequence of calibrated digital images taken using a standard tripod and ball head.Even if image sequences were not acquired by means of concentric imaging (Haggren et al. 2004), the projection centre of the camera was close enough to the rotation centre that errors in the final panoramic images were negligible.The images were captured with a Micro-Nikkor 105mm f2.8 objective mounted on a Nikon D700 dSLR camera.The original image size was 4256x2832 pixels, whereas the long edge of the panoramic images varied from between 15596 and 31379 pixels.We used a more than 60% overlap to ensure a strong joint geometry when stitching the panoramic image.
There were ten camera stations on the opposite side of the point bar (Fig. 1).In order to cover the distance between the nearest and farthest objects in the point bar, we had to set the aperture to f/32.The narrow aperture caused diffraction, but the details were still sharp enough.We used the ISO speed (200), which allows the Nikon D700 camera to take the best quality pictures.The sub-frames were developed to tiff-format.The illumination was flat due to cloud cover, which allows for texture capturing without shadows.On the other hand, diffuse illumination may complicate the visual interpretation of the images.
We used PTGui software to stitch the series of overlapping images into a complete panorama image.PTQui is a graphical Panorama Tools front-end for Windows platforms, which automatically detects tie points.We mapped images to a common plane by using rectilinear projection.

Relative Orientation of DTM and Panoramic Images
Relative orientation of the DTM and panoramic images were solved in order to add texture to the DTM.We registered images directly by determining the orientation of images relative to laser scanning data using natural targets.The lower resolution close to the edges of the panoramic images can sometimes cause difficulties in determining tie points evenly and reliably over the area.Moreover, the interpolated DTM grid of 10 cm limits the accuracy of detecting tie points.We applied two different techniques to solve relative orientation between the DTM and panoramic images.
In the first method, we performed the registration of the panoramas in Geomagic software.The successful orientation required minimum of 20 manually identified tie points between MLS data and images.We confirmed the success of the registration only visually due to lack of numerical information in Geomagic.
The second approach was an image-based orientation.In this method, we created virtual image using a laser-derived DTM.
The projection centre of virtual image was selected in such a way that it was approximately at the same scale with optical images.Finally, we solved the relative orientations between optical and laser-derived virtual images by including all images into bundle block adjustment.

Visualisation of DTM
We created a detailed and textured DTM using a fusion of mobile laser data and panoramic images.Laser measurements provide the geometry of point bars and panoramic images provide information on land cover types.We created orthoimages using images and the laser-derived DTM.Using orthoimages is a straightforward process if texture mapping is needed for the DTM.Rendering software offers many alternatives for visualising 3-D models.For the textured DTM models, we preferred an open source software Blender for the rendering process (Fig. 6).

Accuracy of MLS data
We evaluated the accuracy of the MLS data (Table 2) by comparing its DTMs with TLS points on three of the same kinds of non-vegetated point bars.After correcting the systematic errors, we calculated the RMSE (Root Mean Square Error).

Topography Changes Using Textured Model
We obtained the vertical difference between the MLS-based DTMs in 2008 and 2009 (Fig. 7).We enhanced the limited information on land cover in the DTMs using texture based on the panoramic images (textured DTM).When comparing the change detection model based on the plain DTM and the textured DTM, the changes can be characterized and interpreted.We can evaluate the change source (for example, human activities or a natural phenomenon, such as a flood), classify the land cover, find the random error in change model or estimate visually the sediment grain size or vegetation.
We extracted the most significant erosion and deposition areas from the change detection model and projected them onto the textured model (Fig. 7).The eroded areas are depicted with blue colour, whereas red colour is used on the deposition areas, where the estimated ground surface has increased by over 0.2

MLS-based Change Detection Accuracy
We estimated the accuracy of change detection by comparing the differences in BoMMS and TLS DTMs obtained in 2008 and 2009.We found the square of the sample correlation coefficient (R2) between the elevation changes to be 0.93; there was a standard deviation of error 0.034 m (Figure 8).Volume analysis of the TLS data revealed that the erosion within the study area in 2009 was 12.5 m3, while the deposition volume added up to 29.9 m3.The BoMMS data analysis showed equal behaviour and the corresponding values were 11.5 and 32.5 m3.In this paper, we present a method for integrating panoramic images with high-density terrestrial mobile laser point clouds from a river environment which is highly sensitive for floods.
Our test site was one of the largest point bars along the river.The fusion of data sets contains detailed information on landform development caused by flowing water, such as rivers and streams.Our experiment revealed that textured DTMs allow for advanced interpretation of land cover and, thus, provide more information about fluvial geomorphology and river dynamics.Our method supports the visualization of erosion and deposition areas and the characterization of these areas.

DISCUSSION AND CONCLUSIONS
The mobile mapping system and panoramic image method used in this study proved to be useful when a close viewpoint, dense point clouds and resolution, and high-ranging accuracy were needed.The MLS applications (BoMMS and CartMMS) enable a detailed DTM production of riverine topography.These systems can also be used for measuring multi-temporal data that allows for precise change detection studies of the river.Regional coverage of BoMMS is limited due to the low measuring perspective, thus methods such as CartMMS or TLS are also needed for collecting high-resolution topographic data.Overall, 3-D laser scanning data on the point bar provides a data source for various environmental studies, including river dynamic studies, hydraulic modelling and visual interpretation of fluvial geomorphology.For example, heights and the sediment size of dunes on the point bar can be detected based on the approach.
Applying the regular grid calculation approach to the MLSbased DTMs, in order to reduce the amount of data, typically causes the loss of details.However, a lower level of details might lead to difficulties in identifying and searching for tie points between images and the DTM.Combining panoramic images and a laser-based DTM can be a relatively laborious process, but the fusion allows for a more detailed interpretation of the fluvial geomorphology and river dynamics.
We found panorama images to be useful for our evaluation of the quality of the laser data.The images allowed us to check whether or not the filtering of the laser point clouds had been successful.It was also possible to see the errors for the ground surfaces and to check on how the vegetation stands out from the ground.Because the laser beam passes through a few centimetres of water, it was possible, by using images, to separate the border between the water and the ground.
There is an obvious need for further research in order to improve the accuracy of the relative orientation of images and laser-based DTMs.It is essential to study the effect of the image geometry and the selection method of tie points.Moreover, the orientation accuracy can be improved by using targets.The quality of the orientation is also dependent on the filtering and processing methods for the point cloud.Therefore, the different filtering methods for the DTMs should be evaluated.
An unmanned aerial vehicle (UAV) may also provide improvements for close range photogrammetry and a terrestrial When we directly compared the MLS-based DTMs with the TLS-based reference DTMs, the RMSEs varied between 0.023 and 0.076 m.In addition, the square of the sample correlation coefficient (R2) between TLS-and BoMMS-derived elevation changes was 0.93 and the standard deviation of error was 0.034 We found that panoramic images could feasibly be integrated with laser point clouds.The integrated data allowed for more information and excellent visualization possibilities.In addition, change detection using MLS-based data was successful in terms of illustrating the temporal changes to the point bars caused by flooding.
In further studies, textured, high-detailed 3-D models will be coupled with fluvial processes such as sediment transportation capacity.We will also consider using UAV and a bathymetry for completing the regional coverage of measurements in further studies.

Figure 1 .
Figure 1.The studied point bar was approximately 200 m in length and it mainly consists of coarse sand

Figure 3 .
Figure 3. Mobile mapping systems used in this study: a) BoMMS and b) CartMMS

Figure 7
Figure 7. a) Top view of the change detection model; b) extraction of erosion and deposition areas, where the changes are largest; c) boundaries of erosion and deposition areas on textured DTM

Table 1 .
Laser scanning system parameters of the MLS and TLS