TRUE ORTHOPHOTO GENERATION USING MULTIVIEW AERIAL IMAGES

Usually aerial imagery are acquired through perspective projection which results in relief displacement in the map. Therefore, orthophotos from large scale aerial images over urban area often suffer from double mappings caused by sudden elevation changes in the scene. In the generation of true orthophotos, one aims at removing these perspective displacements and occlusions. Hence, knowledge of the surface shape is needed, as it has great impacts both on the orthorectification and occlusion identification. Mostly, the relief displacements are caused by buildings. In this paper, we present a method for the generation of true orthophotos from overlapping aerial images of known orientation, especially in built-up areas. Apart from the aerial images, the method does not depend on additional data sources. In particular, we focus on the derivation of building roof outline model using image segmentation and edge detection techniques under the guide of the classification result of point cloud from image matching . Experiment results on real data show the feasibility and the performance of the method.


INTRODUCTION
Aerial images are subject to perspective distortion and therefore cannot be used directly as a map with homogeneous scale.Therefore, orthorectification is carried out.For smooth terrain, the traditional digital orthorectification works well as there is no rapid elevation change.However, for large scale imagery of urban areas with complex buildings, the distortion produces unavoidable artifacts in the form of double mapped areas, therefore reduces the usefulness and reliability of the orthomap.Nowadays, the term true orthophoto is generally used for an orthophoto where surface elements that are not included in the digital terrain model are also rectified to the orthogonal projection (Amhar et al., 1998).
For generation of a true orthophoto two aspects need to be considered.Firstly, the correct geometric surface description is necessary.Secondly, if the image ray has more than one intersection with the objects, including the terrain, resulting in hidden areas and double mapping.
For the visibility analysis, additional description of the objects on the terrain is required.The quality of the man-made objects model influences the result because it shows how well the occlusion can be detected, thus how well the quality of true orthophoto can be achieved.Hence, in many cases, in addition to the terrain model, the Digital Building Model (DBM) need to be introduced.The larger the image scale is, the more objects must be included in the surface model (Amhar et al., 1998).Consequently, the more preprocessing and more expensive price it requires for the true orthophoto production.There are different DBM levels.Concerning the building shape and the aim of visibility analysis, only the building roofs matter, as only the tops shall be mapped.Therefore, a simple building model including the roof information is sufficient.
Such artifacts can be removed by identification of occlusion caused by buildings.Generally existing methods for occlusion detection can be divided into the z-buffer based method (Catmull, 1974, Amhar et al., 1998, Rau et al., 2000, Rau et al., 2002), polygon based method (Kuzmin et al., 2004) and angle based ray tracing method (Bang et al., 2007).After that, the true orthophoto can be obtained by filling up the hidden area by overlapped image pixels.
The contribution of this article is that we • exploit the capabilities of current image matching algorithms to get dense point clouds describing the visible surfaces, • classify these point clouds to obtain buildings, and finally • refine these building models with color information from the original perspective images.
In order to obtain a true ortho photo mosaick additionally occlusion detection based on a state of the art method is applied.

Overview
The entire process can be divided into four main steps: First, the point clouds from dense matching of multi-view aerial images are obtained, and gross errors have already been eliminated.Secondly, the point clouds are classified in order to get the points on the building top.With the approximate building roof outlines, the extent of buildings on the original aerial images can be sketched out.Then the better fitting roof edges are created from the aerial images by color space segmentation.The third step is to derive the surface model from the improved point cloud dataset consisting of image matching points and refined building edge points.The fourth step detects the occlusion of the surface on the object space, draws the visibility map and identifies the hidden area.The orthorectification for each image is done individually.In the end we fill up the hidden area pixels by picking up neighboring visible pixels in the same location according to the visibility map.The workflow can be described in the following chart (see fig. 1).
Figure 1: The entire processing of producing true orthophoto.

Coarse building outline acquisition
We take point clouds from dense matching of multi-view aerial images as input.The program package SCOP++ (Pfeifer et al., 2001) provides a robust hierarchic filtering algorithm to filter and classify point clouds.We extract points on buildings from the classification (see fig. 2).
The problems are that points on building facades might also be collected and that points have not been grouped according to the building roof they belong to.Therefore, we construct a Delaunay triangulation on the building points dataset and build constraints for the triangles to eliminate the facade points and group the roof points to each building roof.First, The normals of triangles consisting of points on the wall are not perpendicular to the ground.These points and triangles should be deleted.Secondly, the distance between points from the same roof is usually smaller than from different roofs, these triangles are in a form of narrow triangles.Narrow triangles and points are removed.
These outlines (see fig. 3) cannot directly be used for the building models because the point clouds from matching are not necessarily distributed well on the roof and edges and so far the processing is quite rough so that these point sets reflect the true building outlines only coarsely.However, they can still provide an approximate extent for each building.

Simple building model refinement
As mentioned above, we aim to get more accurate building roof outlines.As the outline acquired in such a coarse way is merely approximate, they should be refined in the following processing.The color information contained in the aerial images shall not be disregarded as it provides potentials to improve the outlines.The building outlines from the last step can be used in the determination of search range for the building edges in the aerial images.
In order to derive precise outlines, we project the building roof points back to image space using the known camera orientations.We aggregate the points within such an outline region for each building.Then, we apply a buffer distance to each region in order to make sure that it fully contains the respective building, resulting in a building mask.Applying the unsupervised K-means segmentation on each region, we determine areas of similar color.We hence pick up the largest segment of the result.
There are two cases.First, the largest cluster occupying a large area of the searching masked region (we set the 'large' threshold, i.e. 75%) is treated as a correct roof detection.Second, the largest area of clusters is not large enough (less than 75% of the extent area), which reflects that probably the roof pixels have been segmented to more than one cluster (this mostly happens for complex roofs such as ridge roofs), we merge segments close to the center of each such area, resulting in a large segment that represents the building roof.Finally, we eliminate small holes due to Figure 5: a sample of points originally classified as on buildings (magenta) vs. higher quality building outlines (cyan) found by image segmentation of unmasked regions.
e.g.chimneys and roof-lights by morphological operations.In that way, the correct spatial extent of building roofs is detected.
In the middle column of fig. 4, one can see that both simple and complex buildings can be detected.
The boundary (i.e. the last column of fig.4) is hence obtained.For each pixel along the boundary of a building roof region, we find the corresponding point in the point cloud from image matching as the one with the smallest distance when projected into image space.Via these procedures, we obtain a simple building model with roof outlines.
Figure .5 shows the roof points from dense matching and the building roof edges from image segmentation according to the method we propose.

Visiblity analysis and true orthophoto generation
Now that an improved 3D building outline estimation is available, we can also refine the building roof points from classification in the first step.Points within the outline are kept as correct building roof classification result, while those outside of the outlines are removed from the dataset.Now the surface model can be derived based on the new point cloud.The grid size is set according to the ground sample distance of the original aerial image.In the meantime, each image is rectified to get the corresponding orthophoto.
The occlusion detection is achieved through the angle based ray tracing algorithm (Habib et al., 2007).The angle based algorithm identifies occlusions by checking the off-nadir angle to the line of sight connecting the perspective center of the imaging sensor and the DSM cells.This algorithm has several advantages over the traditional z-buffer method: it is not sensitive to the sampling interval of the surface model and it does not require a detailed DBM.Afterwards, we generate the true orthophoto by filling up the occluded areas with information from the neighboring orthophotos according to the visibility maps.

EXPERIMENTS AND DISCUSSION
In order to verify the performance of the proposed method, experiments are carried out using real data.The data consist of four digital aerial images which were captured by a DMC camera over Vaihingen, Germany, in the course of a DGPF-project (Cramer, 2010).The ground sampling distance is 8 cm.The interior orientation parameters of the camera and the exterior orientation parameters of the imaging sensor are available.Fig. 6 shows the DSM derived from the image matching points and refined building boundary points with the aerial image as background.
By picking up the roof manually over three sample areas, we testify how complete and how well the outlines fit.Results are listed in table 1.It reveals that for buildings which are detected, the mean of the errors in the horizontal direction are less than 12 cm, which is less than 1.5 times of the ground sample distance of the images.The mean of the errors in the vertical direction are of less than 3 times of the ground sample distance.The standard deviation for both in the horizontal and vertical directions are less than 15 cm.This approach handles complex building roof structures, because image texture matters most.Due to the same reason, it performs poorly on roofs covered or shadowed by trees.
With the simple building roof model based DSM, occlusion detection is carried out and in the end the true orthophoto is produced.Basically, a true orthophoto should meet several requirements: correct positional information, no false visibilities / occlusions and smooth building boundaries.Fig. 7 shows the respective area of the original aerial image, the orthophoto and the true orthophoto.By a closer look at them, one can notice the double mappings.The existence of such effects reduces the reliability of an orthophoto.We can see from the result that the relief displacements are largely removed.For our four overlapping aerial images, up to 99.5% of the pixels on the ortho area can be found in at least one aerial image, which means that the hidden areas have been compensated well.Column 4 in Fig. 7 is the result using a manually captured DBM with a z-buffer method to produce the true ortho photo.Compared to the method presented in this paper, the edges are clearer and not wavy any more.

CONCLUSION
The generation of orthophotos from multi-view images is a process of coordinate transformation and geometric correction (Baltsavias, 1996).True orthophotos aim to eliminate the double mappings caused by orthorectification and gives a better fit in application.In this research, we present a new automated process for producing true orthophotos from multi-view aerial images without additional digital building model.
The possibility of using building points from image matching in guiding the edge detection and outline refinement is discussed and tested.Results show that to a large extent objects are corrected to the right location and double mapping effects are detected and compensated.

Figure 2 :
Figure 2: Points on the building roof due to the classifcation of SCOP ++ in a 3D view.

Figure 3 :
Figure 3: Roof outline extent after eliminating the wrong points.

Figure 6 :
Figure 6: The overlap view of shading DSM and aerial image is underneath.

Figure 4 :
Figure 4: two examples of detecting edges within the masked region from coarse building outline acquisition.

Table 1 :
The statistic of detected building outline compared with manually reference data