RECONSTRUCTING THREE-DIMENSIONAL SPECIFIC CURVE BUILDING MODELS FROM A SINGLE PERSPECTIVE VIEW IMAGE

This paper presents a systematic approach to reconstruct 3D models of complex buildings with specific curve structures from single perspective view images. The proposed method begins with the automatic extraction of straight feature lines and corner points. Three mutually orthogonal vanishing points are then calculated automatically and robustly for reconstructing three-dimensional regular (planar structure) models. For curved segments, they are extracted and processed separately. Three types of curved segments are considered in this study, including cylinder shapes, surfaces of revolution and free-form curve models. Cylinder shape structures can be defined directly by calculated length of major and minor axes. As for surfaces of revolution, the model can be reconstructed using a fitted curve contour line and determined symmetry axis. Free-form curve structures are reconstructed using measured boundary curves for the interpolation of curve surfaces. Regular and curved models are then merged based on their shared feature points. Finally, the level of detail of reconstructed models can be increased using rectified façade texture rendered from single-view images. The output imagery of a computer-simulated model and close-range photographs of real buildings are used in this study to test the performance of the developed algorithms. Quantitative evaluations of the results compared with ground-based surveying and visualized comparison with raw images indicate that the developed algorithms can successfully extract 3D information and reconstruct 3D specific curve building models.


INTRODUCTION
Three-dimensional (3D) building modeling is an important issue in both computer vision and close-range photogrammetry, and has important applications in 3D-GIS, cyber city, augmented and virtual reality, digital documentation of buildings, monuments and sites, architectural surveying and others.For the purpose of 3D model generation, direct measurement or surveying such as using LIDAR, total stations or well-calibrated cameras are practical methods for ground-based construction.Although the quality of direct measurement can achieve centimeter or even millimeter level, these types of approach often require expensive equipments, professional operators and cost long time for post-processing.In addition, traditional photogrammetry-based reconstruction often requires multiple views of objects and known camera parameters.The majority of vision-based 3D reconstruction has focused on binocular vision (Furukawa and Ponce, 2010) and other algorithms requiring multiple images, such as structure and motion estimation (Nistér, 2005) and estimating depth from defocus (Wöhler et al., 2009).
Single view reconstruction (SVR) is another r image-based method, which has a potential to extract 3D geometry information from an image without knowing the priori parameters, and should provide a more economic and efficient way for building model reconstruction.Furthermore, non-existing buildings can also be reconstructed from historical images or paintings with perspective projection, which may be difficult, if not impossible, to achieve using traditional reconstruction methods or direct on-site measurements.
Researchers in the fields of computer vision and close-range photogrammetry have investigated in single view reconstruction.A continuance triangular surface model is one of the popular output formats in computer vision.Hoiem et al. (2005) assumed that the scene consists of horizontal and vertical planes and than classified the image into to several vertical regions and based ground plane to build up a scene.Therefore, this method does not apply to scenes that are not made up only of vertical surfaces standing on a horizontal plane.Saxena et al. (2008) proposed a method by over-segmenting an image into small homogeneous regions (superpixel) and using a Makove Random Field (MRF) to infer the 3D position and orientation of each superpixel.They improved previous method to handle scenes with significant non-vertical structure and also increased correctness, but lack of geometry constraint and the classification results heavily related to training datasets.Koutsourakis et al. (2009) described a way to use shape grammars with MRF framework in order to retrieve a fixed derivation semantics tree and the geometry of challenging facades.T¨oppe et al. ( 2011) considered a variational approach by searching for a weighted minimal surface which is consistent with the marked silhouette.Zhang et al. (2001) and Prasad et al. (2006) addressed the problem of reconstructing free-form curved surfaces by combining a sparse set of user-specified space topology cues, such as surface position, normals, silhouettes and creases.Their method can generate high-quality and visually pleasing 2.5D mesh surface model manually.
Another popular single view reconstruction approach is using single, two or three vanishing points to obtain approximate 3D information of objects.Horry et al. (1997) used the cues of different scales along the line to the single vanishing point.The vanishing point is the most distant point from the observer; other points away from the vanishing point will having lager scale.If two of the vanishing points in the horizontal direction can be found in the image, a vanishing line can be constructed and used for modeling scenes (Kang et al., 2001) or camera calibration (Grammatikopoulos et al., 2007).Colombo et al. (2005) proposed a method to recover the 3D structure of a generic surface of revolution object and its texture from a single uncalibrated view, based on the projective properties expressed through planar and harmonic homologies.Jiang et al. (2009) exploited constraints that derive from shape symmetry, including both bilateral and rotational symmetry, which are prevalent in man-made architecture.After camera calibration, an interactive architecture modeling interface was proposed to recover 3D information and enhanced the texture quality.In general, the most popular algorithms for scene reconstruction is to using vanishing points which are corresponding to mutually orthogonal directions in the world space.Hartley and Zisserman (2003) explained the geometric meaning of these three vanishing points that are related to camera parameters, including focal length, principle points, and camera pose.Based on the vanishing points and vanishing lines, polyhedral models can be constructed from measuring lines and other features as well as topological and geometric constrains, such as planarity, parallelism, alignment and angles topology of objects in a single image (Van Den Heuvel, 1998;Criminisi et al., 2000;Wang et al., 2005).
Most SVR methods rely heavily on manual operations and are not systematically implemented.In addition, vanishing points detection might be stable, but vanishing points are imaginary points, thus are difficult to assess the accuracy.To address this issue, this research extends a previous work (Chang and Tsai, 2009) to develop a systematic approach for feature detection and increasing the level of automation for 3D building model reconstruction from a single perspective view image.Moreover, in addition to planar facades, three types of curve structures can also be reconstructed with the proposed algorithms.The developed algorithms are based on vanish point metrology and require only a single image with perspective projection; no camera and other parameters are needed.This is one of the most economical means to extract 3D information, because no expensive equipments and professional operators are required.The data can be acquired with consumer digital cameras or even downloaded from the Internet.

METHODOLOGY
Figure 1 illustrates the general procedure of the developed reconstruction method.The processes begin with the pre-processing, feature edge detection and feature line extraction from the input image.Extracted straight line segments are separated from curved segments and used to select feature points and to obtain vanishing points.Coordinates of the feature points are calculated from three vanishing points.Height equalization and vanishing points fine-tuning processes are developed to reduce the systematic and random errors, in order to provide a reliable location of the vanishing points.Curved structures are reconstructed with obtained parameters and combined with planar models generated from extracted feature points.

Feature lines detection
Figure 2 shows the procedure of feature line detection.Feature edges can be extracted using Canny operators.Image enhancement might be helpful for improving edge detection, and morphology could also be useful for merging discontinuous lines.By transferring extracted feature edges to the normal distance and normal angle (ρ-θ) space, straight feature lines can be detected automatically.For the best results, this study uses the inverted pyramid pattern iterative calculation for ρ-θ parameters, and the iteration stops when vanishing points are stable.A voting scheme is used to select candidate peaks from the accumulated histogram for collinearity detection in ρ-θ space.Afterward, similarity rectification is applied to identify different line segments.Then use least squares methods to trace line groups iteratively by adjusting the threshold of histogram peaks, until the number of line groups is satisfied.

Vanishing points and feature points localization
A vanishing point is the point in the perspective space where a group of parallel lines in the object space converge.In practice, each two lines will converge into a point in the perspective 2D space due to existing errors, so the convergent points will be dispersed within a certain areas.This study uses k-means cluster to separate convergent points for three mutually orthogonal directions, and uses the standard deviation as the threshold to determine the most possible position of a vanishing point as indicated in Figure 3. Two conditions for obtaining stable vanishing points are considered.One is the number of convergent points; i.e., adjusting line group number threshold to prevent most detected lines from pointing to a certain direction.The other is that the representative vanishing points should be stable under different ρ-θ parameters.Modifying ρ-θ parameters can increase the reliability for vanishing point calculation.In this step, the initial location of the vanishing points can be quickly obtained.Coefficients of radial symmetric lens distortion can also be calculated using vanishing points and feature lines.Short segments and small closed polygons from detected segments can be removed because most of them are windows, patterns or minor structures.Use the rest segments to calculate intersected corner points as candidate feature points, and geometry constrains can be used for filtering candidate points.An important issue is to define a reference plane with the reference origin and vanishing points along x and y axes.The reference origin is the intersection point from the bottom edges along x and y axes of the main structure.Candidate feature points below the reference plane or collinear to others can also be removed.

Feature points calculation
As illustrated in Figure 5, vanishing line (L v ) is the line passing thought the vanishing points V x and V y ,, and it forms a reference plane with the origin.Feature point F r is at a known (or reference) distance H r to B r from the reference plane.
Another feature point F is with an unknown distance H above the reference plane.Points B and B r are the base points for vertical distances on the reference plane.The four points I, F, B, V z are collinear.Point I can be obtained from eq. ( 1), which projects point F r on to the line linked by F and V z .Once a reference height, H r , is defined, every feature points' height, H, can be calculated from eq. ( 2).If there is no reference information available, this algorithm can still provide a relative model.Horizontal information can be obtained by projecting feature points onto X-Z and Y-Z plans and measured by their scales.(2) (1)

Curve structure reconstruction
Curved segments are separated from the extracted straight lines in the previous procedures.Curve fitting is used to fit the discontinuous curve segments due to occlusions on the image.Three types of curve structure are separated for different reconstruction methods, including cylinder shape, surface of revolution (SOR) and free-form surface.
For cylinder shape reconstruction, first find two intersection points of straight lines and the curve line as the endpoints of the major axis, as indicated in the general procedure of curve structure reconstruction (Figure .6).The line passing thought the center of the major axis and corresponding vanishing point will intersect curve line to indicate the position of the arc apex.The minor axis is the line linked from apex to the center of the major axis.Next, project the image on to the vertical and horizontal planes according to the vanishing points respectively to calculate the length of the major and minor axes.The cylinder shape structure can then be generated from the endpoints and the length of major and minor axes.
As for the SOR structure, the image is rectified and projected on to the vertical plane using vanishing points.The contour line is generated from projected contour points, while the symmetry axis is the line that passes through the center of rotation and the vanishing point in vertical direction.According to the contour line and symmetry axis, SOR structure can then be reconstructed.Finally, combine the curved structure with the main body to form a complete building model with curved structures.To reconstruct free-form structure, inflection points on the surface boundaries will be selected and fitting with the endpoints of curve line to form a boundary contour.After obtaining the boundary contour, model surface can be generated by surface interpolation.If the surface is in convex or concave form, additional control points will be needed to simulate the shape.
Curve segments

Endpoints detection
Major and minor axis calculation

Cylinder shape reconstruction
Contour fitting

Surface of revolution reconstruction
Curve fitting

Boundary curve fitting
Free-form surface reconstruction Figure 6.General procedure for curve structure reconstruction

EXPERIMANTAL RESULTS
This study used two types of single-view images to test the performance of the developed algorithms, computer-simulated model and close-range photograph of buildings with fine perspective projection.Quantitative evaluations by ground surveying and reference data were also performed to validate the reconstructed building models.

Computer simulated geometric structure image
Three computer-simulated curve structure images including a cylinder shape, SOR and flying eaves structure were used to verify the proposed methods.These test cases were generated with known size and perspectively projected onto an image plane.By using these case studies, complete reference data can also be used for quantitative evaluation.Figure 7 shows a few steps of reconstructing the cylinder shape structure model.Figure 7 (a) shows the result of transferring pre-processed image to ρ-θ space with iterative threshold detected peaks.Vanishing points examination is illustrated in Figure 7 (b) after filtering.Figure 7 (c) displays detected feature points (blue points) and their corresponding base points (green points).Table 1 lists the elevation of the reconstructed model using height of feature point a as the reference.The result indicates that the overall RMSE is about 1.32%.Table 2 lists the evaluation of the reconstructed cylinder shape structure.The errors in the major and minor axises are 2.5% and 4%, respectively.The result of this example indicates that the developed algorithms can accurately reconstruct cylinder-shape structures.Flying eaves is a common building style in Asia, and the hip of the roof which connected to, is usually a non-linear segment.Table 4: Validations of free-form shape structure parameters.

Close-range photograph of buildings
Figure 10 (a) is a photograph of a building taken from an overlooking position.Figure 10 (b) illustrates the position of the vanishing points, which were calculated iteratively until the given thresholds were satisfied.The reconstructed 3D building model using the proposed method is displayed in Figure 11, which contains a cylindrical structure on one side of the building and a SOR dome shape object on the top of it.A high-precision CAD model of the building generated from stereo aerial photographs and ground-base LIDAR surveys was used as reference data to evaluate the reconstructed result with the developed algorithms.The evaluation of the result in the X-Y plane is displayed in Figure 12.In Figure 12, numbers in blue and italics are measured from reference data and numbers in black are from the experimental results.The reference height is 38 m from the top of the roof to the building foot point.Validations of cylinder shape parameters are listed and compared with ground truth data in Table 4.As displayed in the figure and the table, most of the errors are only a few centimeters, except the minor axis of the cylinder.The average error is approximately 1.36%, further proving that the developed single-view reconstruction algorithms can achieve high accuracy for reconstructing building models with curved structures.Projected façade textures can be used to identify the windows, extrusions or intrusions on the surface.The position and outline of the objects can be directly extracted from the projected texture, and the depth cues can be measured according to the vanishing points' geometry.Figure 13 are the results after editing the reconstructed model, which can provide higher level of detail and more visually pleasurable.

CONCLUTIONS AND FUTURE WORKS
This study developed algorithms requiring only a single image with perspective view and good geometry to reconstruct 3D building models with curve structures without internal or external camera parameters.With the developed algorithms, vanishing points, feature points and corresponding base points can be extracted automatically with defined thresholds.In addition to planar building structures, cylinder shape, surface of revolution and free-form structures can also be reconstructed effectively and with reasonable accuracy.Finally, the level of detail of reconstructed models can be increased using rectified façade texture and geometry cues.The developed approach is one of the most economical means to extract 3D information, and it can be applied to close-range photos, images obtained from Internet, and even real paintings with perspective views.Experimental results demonstrated that reliable accuracy in the reconstruction of 3D building models can be achieved with the proposed methods.Future improvement of the developed system will focus on the reform of reconstructing free-form curved structure, and the ability for curve segment classification.Self-occlusion is also an issue which needs to be addressed more rigorously in order to improve the modeling capability.
Figure 1.Building model reconstruction procedure

Figure 2 .
Figure 2. Procedure for feature line detection.

Figure 3 .
Figure 3. Systematic procedure for vanishing point calculation and feature point extraction.

Figure 4 Figure 4 .
Figure4is an example of base point prediction, in which feature points are marked as blue circular dots.According to the vanishing points along Y and Z or X and Z directions, feature points are projected onto the X-Z or Y-Z plane, and the projection path is recorded.The projections for the feature points are marked as red triangles in Figure4(b).The same procedure is applied to vanishing point along Z direction to project them onto Y or X axis noted as green squares at the bottom of Figure4(c); then record and link to the vanishing point along y or x direction as the dash lines shown in Figure4(d).The intersection point to the line linked from feature points to the vanishing point along z direction (vertical line in Figure4(e)) are the base points (red hollow circular points).In Figure4(f), green solid lines represent the target heights between the feature points and their corresponding base points to be determined in the next step.

Figure
Figure 5. Feature points calculation.

Figure 7 .Feature
Figure 7. Reconstructing computer simulated geometric structure model.(a) Feature lines extraction in ρ-θ space (b) vanishing points examination (c) feature points (blue) and their correspondent base points (red).

Figure 8
Figure8(a) shows the result of reconstructed simulated SOR structure.The blue curve is the contour line fitted from the contour points, and the red line is the symmetry axis.Figure8 Figure 9. (a) Reconstructed free-form shape structure (b) curved hip parameters.

Figure 11 .
Figure 10.Close-range photograph of a building (a) and vanishing points (b).

Figure 12 .Figure 13 .
Figure 12.Validations of the close-range photograph case in X-Y plane

Table 1 :
Validations in elevation of cylinder shape computer simulated geometric structure case.

Table 2 :
Validations of cylinder shape parameters.

Table 3 :
Validations of SOR structure parameters.