THE TRIPLET MEASURED BY AERIAL CAMERA USING LINE SEGMENTS LINE MATCHING-BASED RELATIVE ORIENTATION USING TRIPLET CAMERA

In urgent observations after disasters, we can mention that the image matching processing is an essential technique to establish more stable and rapid 3D data generation. Particularly, multi-images taken from various viewpoints are useful in the disaster monitoring. Thus, feature and corresponded point detection would be designed for a multi-image matching. Recently, Structure from Motion (SfM) is often applied to generate 3D data. The SfM is useful approach to generate 3D data from images of random viewpoints. However, Scale-Invariant Feature Transform (SIFT) requires a plenty of time to detect feature points and corresponded points from multi-images. Therefore, we proposed a methodology to improve triplet matching and SfM with line segments extracted from images. Moreover, we evaluated our methodology using multi-images taken from aerial triplet camera.


INTRODUCTION
Recently, 3D data in urban area are generated by using highresolution aerial camera, laser scanner, RTK-GPS, and inertial measurement unit (IMU).However, 3D data acquisition with aerial LiDAR and Photogrammetry are affected by disasters, because these methodologies depend on ground control points and electronic reference points.In the Great East Japan Earthquake, many electronic reference points were lost or shut down in the wide area.Moreover, positions of reference points were considerably moved by diastrophism after the earthquake.Thus, it was difficult to keep high absolute accuracy in 3D data generation for disaster observations.On the other hand, we have confirmed that high relative accuracy in camera orientation for 3D measurement can be kept with image matching processing and single GPS positioning.Therefore, in urgent observations after disasters, we can mention that the image matching processing, such as feature and corresponded point detection, is an essential technique to establish more stable and rapid generation of 3D data.Particularly, multi-images taken from various viewpoints were useful in the disaster monitoring (Chikatsu, 2005a), (Oga, 2009b).Therefore, feature and corresponded point detection would be designed for a multiimage matching.The multi-image matching can be applied to images taken from airborne-triplet camera, mobile mapping systems and unmanned aerial vehicles.Recently, Structure from Motion (SfM) (Mitsugami, 2011c.)(Asai, 2010d) is often applied to generate 3D data.The SfM is point-based matching and 3D modeing using corresponded points estimated with Scale-Invariant Feature Transform (SIFT) (Fujiyoshi, 2011e) (Takagi, 2009f).The SfM is useful approach to generate 3D data from images of random viewpoints.However, the SIFT requires a plenty of time to detect feature points and corresponded points from multiimages.Thus, it is not easy to achieve a rapid processing.In this study, we focus on a geometrical camera model for multi-image matching using a triplet camera, which is consisted of three cameras to cover urban area from various viewpoints.The triplet camera can concurrently obtain images in three directions consisted of forward, nadir, and backward.
Moreover, we proposed a methodology to improve triplet matching and SfM with line segments extracted from images.In addition, we evaluated our methodology using multi-images taken from aerial triplet camera.

METHODOLOGY
The proposed methodology consists of a feature line extraction line matching, and feature point matching.Firstly, feature lines are extracted from triplet images.Next, extracted line segments are corresponded to be used as candidates for feature point matching.Then, feature points are extracted with SIFT using corresponded line segments.In this processing, corresponded line segments are adjusted based on a triplet matching with exterior orientation parameters taken from GPS/IMU as initial values to achieve a relative orientation.as shown in Figure 3.In Figure 3, orthogonalize line segments to the flight direction are better than paralleled line segments to a flight direction to achieve higher 3D measurement accuracy.Although the paralleled line segments to the flight direction are easier to find matching candidates based on epipolar constraint, it is difficult to determine matching point precisely.Thus, the orthogonalize line segments to the flight direction are selected from triplet images.Additionally, long line segments are selected to achieve more stable matching.The second parameter is color information of both side areas along a line segment.Conventional edge matching is difficult to select corresponded line segment from paralleled edges, as shown in Figure 6.When the line segments have color information along their lines, image matching, such as template matching, can be applied to line segments to avoid of uncertainty in the edge matching.
Figure 6.Line segment matching with color information

Line matching correction with SIFT and triplet matching
The SIFT is an image processing describes the invariant features to change scale and rotation of an image.The SIFT processing can extract corresponded points in images automatically.In this study, SIFT processing is used to improve integrity in our line segment matching.Moreover, when corresponded line segments are obtained, initial external orientation parameters can be estimated.With these parameters, line matching results are corrected based on triplet matching, such as back-projection from a nadir image to forward or backward images.Finally, fine external orientation parameters are estimated after matching error rejection.

Experiment
In our experiment, we used a multi-camera system (SAKURA III, NNK-DCS4H003, Nakanihon Air Service).This system consists of GPS/IMU and a triplet camera, as shown in Figure 5 and Table 1.Moreover, this system can acquire triplet images (nadir, forward, and backward)

Study area
Nagoya city was selected as our study area.This area includes buisiness buildings, high-rise buildings, wide roads, highways, and railways, as shown in Figue 8.

Tripret images
We used 3 images (nadir images, forward images, and backward images), as shown in Figure 9~11.Each image size was 4904 × 3678 pixels.Figure 16 shows our matching result with each length of line segments.Figure 7 shows our matching result with color information line segments.

DISCUSSION
Line segment matching was processed well between in combination of nadir images.However, many mismatching occurred in the line segment matching in combination of triplet images (nadir, forward and backward images).We have confirmed that occlusion caused by buildings affected line matching with triplet images.Although triplet camera can cover details of urban features, there are many hidden parts in triplet images.
Figure 18.Hidden areas in centric camera projection In mismatched results, there are mainly a paint of crosswalk, road markers and building windows.Each object is consisted of similar features aligned in images.We have confirmed that these features are difficult to be corresponded with length and color of line segments, even if conventional SIFT is applied to these features.

Figure 19. Mismatched points in corresponded point detection results
For a mismatching reduction, line matching with repeated features, as shown in Figure 20, should be firstly improved.For example, when repeated segments are grouped into a cluster, these can be assumed as a texture and pattern.Relative positions of repeated features in images would be useful clues for line matching based on the pattern recognition.In urban area, there are many repeated features such as road features, as shown in Figure 21, and Figure 22.

CONCLUSION
In this study, we focused on a triplet camera model to improve triplet image matching in urban area.We proposed a feature matching based on line matching with SIFT to find corresponding line segments.In our experiment, we used an airborne triplet images acquired in Nagoya city.Moreover, we evaluated our methodology using these images.

Figure 2 .
Figure 2. Input image (top image) and extracted line segments with LSD (bottom image) Moreover, extracted line segments are classified with their directions and lengths into several groups.In image matching using triplet images, 3D measurement accuracy and integrity depend on geometrical model based on epipolar constraint[8],as shown in Figure3.In Figure3, orthogonalize line segments to the flight direction are better than paralleled line segments to a flight direction to achieve higher 3D measurement accuracy.Although the paralleled line segments to the flight direction are easier to find matching candidates based on epipolar constraint, it is difficult to determine matching point precisely.Thus, the orthogonalize line segments to the flight direction are selected from triplet images.Additionally, long line segments are selected to achieve more stable matching.

Figure 3 .
Figure 3. Triplet matching(Yanagi, 2011h) 2.2 Line segment matchingTwo parameters are used in line segment matching, as shown in Figure4.The first parameter is a length of line segment.Each corresponded paralleled line segment to the flight direction changes its length in triplet images considerably.On the other hand, each corresponded orthogonalize line segment to the flight direction has almost same length in triplet images, as shown in Figure5.Although edges should be successfully extracted from nadir, forward, and backward images, the length of line segment is one of useful parameters in line matching.

Figure
Figure 8. Study area

Figure 12 .
Figure 12.Extracted line segments in nadir image