A FIELD-BASED METHOD FOR ESTIMATION OF OVERLAP FOR CONVERGENT IMAGES FOR VIEW PLANNING OF BUILDINGS

Overlap between two convergent images for close-range terrestrial photogrammetry is a pre-requisite for view planning of building corners. Available tools determine overlap of images that are acquired by normal geometry at a constant distance. However, for the convergent images lengths of image footprints vary according to camera position. The paper proposes a field-based method that requires to measures only geometric dimensions of image footprints in field for assessing overlap fractions for convergent images. The paper first derives the overlap fraction of convergent images as a function of image footprints, which depend upon the camera position, object geometry, and camera FOV. Experiments are conducted in field for two building corner sites. The proposed method provides conservative estimates of overlap fractions compared to that provided by image-based methods. The errors in the overlap fractions are contributed by three sources, namely, the approximations of the proposed method, uncertainty in camera positions and alignment of camera optical axis, and placement of markers in field. Experimental results suggest that the proposed method can be used confidently in field for overlap estimation for convergent images for the view planning. Images acquired for the view planning of the two corners successfully generated 3D models.


INTRODUCTION
Close-range terrestrial photogrammetry (CRTP) is becoming one of the popular methods for various applications of social and commercial interests. CRTP acquires overlapping images with ease and better control using low-cost consumer grade cameras and yet delivers high resolution surface models of above ground objects situated over small areas, where data acquisition is challenged due to unavailability of GPS signals, or occluded objects and terrain surface. Recent studies highlight applications for 3D mapping and modelling of building structures, geomorphologic structure, artefacts, heritage structure, tree diameter etc (Matthews, 2008;García-Gago et al. 2014;Abbaszadeh and Rastiveisa, 2017;Mokroš et al., 2018;Baramsyah and Rich, 2019).
Above studies have mentioned that high overlap is an essential requirement for CRTP for both convergent and normal geometries between a camera and given object. For a nontopographic application, Fu et al. (2017) has found that high overlap and high resolution are necessary for the higher density and higher accuracy of photogrammetric point cloud. For most of topographic CRTP applications related to 3D model generation of non-coplanar and irregular surfaces, high overlap in range of 80-90% between two adjacent images is a prerequisite (Haneberg, 2008;Krajnak et al., 2011;James and Robson, 2012;Westoby et al., 2012;Prieto and Ramos, 2015;Hidayat and Cahyono, 2016;Förstner and Wrobel, 2016;Micheletti et al., 2018). Similarly, for 3D models of building structures by CRTP using surface from motion (SfM) approach, a large distance between camera positions and building is selected such that each image occupies complete building information by convergent geometry (García-Gago et al., 2014). This arrangement of convergent image acquisition achieves high overlap though, yet the arrangement compromises on high resolution. Förstner and Wrobel (2016) mention view planning exercise that considers each building corner as an individual element of a building. Further, the view planning configures camera positions at constant distance around a building corner. A block of cameras around a corner acquire images ensuring both high overlap and high resolution. Among three major goals of CRTP for 3D model generation by view planning, the 'completeness of photogrammetric information' for a 3D model can be achieved by high overlap for two non-coplanar surfaces of a corner. Therefore, if overlap is examined and evaluated for a building corner, it can work as precursor indicator and one more criterion to guarantee the 3D model generation.
Overlap can be defined as the ratio of common portion of two image footprints to footprint of any of the two images. For normal view geometry, overlap is generally measured by imagebased method in field as well as in lab. Image based method overlays two images and calculates overlap using feature matching. With recent advances in algorithms of image and feature matching, sophisticated algorithms like scale invariant feature transform (SIFT) are also employed for detecting and identifying highly accurate key points and then overlap is estimated using the common key points in images (Xang et al., 2010;Li et al., 2018). However, a manual method, as described by Wolf (2018), that uses stereoscope with images is adopted with photographs.
Unlike normal images acquired by camera positions parallel to building surface, convergent geometry requires multiple camera positions on a circular path around a building corner such that camera optical axis at each camera position points to object corner. Due to this, overlap estimation for the convergent images poses some challenges. First, the convergent images should be projected with a same viewing direction for overlap estimation. However, the involved projective transformation results to large deformations for images that are acquired at angularly distant camera viewpoints. Also, for adjacent camera positions on circular path, the image footprint and consequently the image overlap vary for convergent images even at the constant radial distance from object corner. On the other hand, image-based method for overlap estimation also necessitates unique knowledge and some level of expertise for the use of sophisticated resources (sophisticated algorithms, involved techniques, and computer hardware). Considering these facts, this paper proposes and derives a field-based method for overlap measurement for the view planning of two non-coplanar surfaces of a building corner. The method requires geometric measurements and image acquisition for the surfaces of building corner in field. The paper is organized in four sections: the introduction in the first section is followed by detailed explanations of view planning and derivations of the image footprints and consequently overlap fraction in second section. Experiments and results of the proposed method is validated and discussed in section 3. In addition, the section 3 also shows the 3D models generated from the acquired images by view planning of the building corners. Conclusion is presented in section 4.

Convergent Image Acquisition for Building Corner
A building corner consists of walls meeting at a specific angle.  In above figure, lines OP and OQ are of equal lengths. Line OP and OQ are normal to left and right wall surfaces, respectively. The corner O is described by an exterior angle ,  which is subscribed between lines OP and OQ. Point C is a camera position measured by angle ψ from line OP on the circular path on ground defined between points P and Q. Lengths OD and OE show footprint of the image captured at point C. Figure 2 below illustrates planimetric view of a building corner O and acquisition of a convergent image at point C, which captures photogrammetric information of plane wall surfaces OU and OV. Similarly, the line OQ is a reference line for measuring a camera position in right half of the circular path. Field of view (FOV) of a camera is generally divided in two equal parts symmetrically on either side of the optical axis. However, we are considering L  and R  for left and right parts of the FOV around the optical axis (as shown in figures in this paper).

Image Footprint of a Convergent Image
For a convergent image, such as acquired at camera position C in figure 2, the image footprint occupies object surface on both sides of point O as lengths OD and OE. From the above figure, sine rule for triangle DOA gives: Similarly, for triangle BOE sine rule derives expression of length OE as: From equations 1 and 2, it is evident that the image width, expressed in two parts for a convergent image, is a function of camera FOV ( L  , R  ), camera position ( ψ), and building corner geometry (i.e. exterior angle  at a corner). For an image The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France captured at C, if a camera position ( ψ) is located in left half of the circular path, length OD is less than length OE and vice versa. At the center point of the circular path, OD and OE are equal. Mathematical expressions of OD and OE also confirm that when exterior angle of corner (  ) is less than 90°, the image widths OD and OE are smaller compared to that of the exterior angle values more than 90° as camera rays intersect with the planes OU and OV at a less distance from O in former case. In forthcoming discussion, expression of overlap is derived as function of image widths OD and OE.

Overlap Fraction
Due to convergent geometry, for given combination of camera, building corner, and constant distance ( d ), the image width varies with the camera position ( ψ ). As a result, overlap fraction ( CC η ) between the two convergent images depends up on their camera positions ( ψ ), FOV of camera ( L  and R  ), and exterior angle (  ) of building corner. Figure 3 shows schematic representation of the proposed method to calculate the overlap fraction of two convergent images. The convergent image acquired at point C has image width DE (shown in figures 2 and 3). Figure 3 illustrates two camera positions, C1 and C2, which are located at ( ψ ) and ( ψ +∆ ψ ) angles, respectively. The former and latter camera poistions acquire image footprints as DE and D E   , respectively. To calculate the overlap fraction, footprints of both convergent images are projected to a line D E   , which is parallel to A B   (line D E   is shown by red dotted line in figure 3). As a result, the projected points on the line D E   are: 1 D from point D, 1 E from point E, and 1 E from point E .
Overlap fraction, as mentioned before, is defined as the ratio of the common portion between the footprint projections of first and second images to the projected footprint of the first image on line D E   . As shown in figure 3, on line D E   common length is 1 1 D E and projected length of the first image is 1 1 D E .
Therefore, overlap fraction ( CC η ) between the convergent images can be written as: From the figure 3, numerator term   1 D E   and denominator term   1 1 D E of above expression can be written as: Equations (4) and (5) modify the expression of overlap fraction as: Equation (6) (7), (8), and (9) in equation (6) gives: In the above expression of CC η , term OD DD    is equal to OD . Thus, CC η is modified as: Equation (10) provides the expresssion of overlap fraction between two convergent images. It should be noted that the left image in a pair of convergent images is consisdred as a The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France reference image and the variables OD and OE are appearing in the denomiator of the mathenatical expression (10). Figure 4 below expresses a simplified pictorial representation of the overlaps for the two convergent images along a straight line in field.  (10) can be measured in field for immediatiate verification of overlap fraction. On the other hand, theoritical value of overlap fraction can also be calculated by equation (10) by substituting values of OD , OE and OE . From equation (2), OE can be written as: Substituting the values of OD , OE , and OE in equation (10) gives: Equation (12) is a mathematical expression for calculating the overlap fraction of two convergent images which are acquired at camera positions ψ and ψ +∆ ψ on the circular path.

EXPERIMENTS AND RESULTS
The method for overlap estimation described above is applied for two corners of 55° and 90° exterior angles. Figures 5 and 6 show images of the two corner sites.   Corresponding to above camera positions for a corner, four images are numbered as 1, 2, 3, and 4. In order to mark the camera positions on ground, two perpendiculars are drawn to both the facades of the walls for making the circular path. On these normal lines, reference points P and Q are marked on ground at shooting distance ( d ) from the building corner. Figure 7 shows the schematic view of the field procedure for marking a camera position for the 90° corner. A circular arc of radius d , centered at the building corner and defined between P and Q, is the circular path of camera positions. As shown in the figure 6, for a point that is located on the circular path at an angle j ψ from line OP, chord length between points, P and Cj is calculated as: A camera position on the circular path is marked by intersection of length d from the corner O and the length L j from point P.
Following this procedure, all camera positions are marked. To acquire the convergent images on these camera positions, Canon 7D Mark-II camera (64° FOV, 5472 x 3648 pixels) is used. The camera is mounted on a pole and aligned horizontally with the help of level and images are captured at 6m radial distance from the corner O.
Image widths on the walls are measured by marking edges of images on walls manually. Lengths of the facades covered in the images are measured in the field using a 30m tape (least count 1 cm). For both corners, sufficient lengths of planar wall surfaces are not available for image acquisition. Therefore, for demostration of overlap assesssment in field, the images are curtailed by known number of pixels and image footprints of modified images are considered for calculations. For the curtailed images, the variables L  and R  are determined and used for theoritical calculations of lengths OD and OE using equations 1 and 2, respectively. The results of OD and OE for different camera postions for the two cases are given in Table 2 and Table 3. In the tables, an image pair i-j indiates that the i th and j th images are acquired from their respective camera positions from the left edge of the circular path. Moreover, the i th image is reference image for calculations of overlap fractions by equations (10) and (12). Furthermore, length OE of the j th image or latter image of the image pair should be considered as OE , which can also be calculated by equation (11). For example, the image pair 1-2, describes that the 1 st and 2 nd images are acquired from camera positions 1 ψ and 2 ψ , respectively, and the 1 st image is the reference image for overlap fraction calculations. For the calculations of variable ∆ ψ , one should select 1 ψ < 2 ψ . Theoretical estimate of the overlap fraction can be calculated in multiple steps using equations (1), (2), (10), and (11). On the other hand, equation (12) provides the overlap estimate in one step. In tables 2 and 3, estimated values by formula and measured values in field for two lengths (OD and OE) are shown for the two corner sites. Using estiamtes and field observations of image widths (variables OD and OE), the overlap is estimated by the proposed method. For validating the results of the proposed object-based method, an image-based method demonstrated by Xing et al (2010) is adopted. Accordingly, an open source image-based soft copy method developed by Garg 2018 is used. The algorithm performs image matching and detects common matches. Coordinates of common matches are determined and overlap is calculated by conventional approach, which is used for stereo images, i.e. overlap is estimated as if two convergent images are stereo images. For 90° and 55° corners, Table 4 and  Table 5 Tables 2, 3, 4 and 5 confirm that the image footprint size and overlap fractions for convergent images vary according to camera positions. In Tables 2 and 3, the difference of the theoretical values and field measurements for image footprints are in range of 28-46 cm and 0-37 cm for 90° and 55° corners, respectively. The errors are biased towards the right side of the images. Moreover, higher errors can be observed for larger image footprints. On the other hand, these error values are equivalent to 30-110 average pixels for the given image characteristics. For a camera of 5472 pixels, the maximum amount of the errors in variables OD or OE is approximately 5% of total pixels in half portion of an image. Sources of errors are contributed by centering of camera positions, orientation of camera axes (tilt of camera frame, tilts of optical axis in normal and vertical direction), and marking of image edges as well as length measurements for OD and OE variables in field. In the field experiments, marking of camera positions by the intersection process has also contributed to errors for camera positions, which are in addition to errors for centering the camera in field at a location. Amongst all sources of errors, maximum errors are contributed by optical axis orientation and camera position as the former error cannot be controlled in object surface and latter one originates from marking procedure adopted. While performing experiments in field, raw estimates confirm that camera position may be incorrect in range of 2-10cm and optical axis may show deviation in range of 0.25-1.5°. These values of errors should be accounted for analysis while estimating the error budget for this method.
Results in Tables 4 and 5 express that overlap fractions calculated by field measurements and mathematical expressions are in close agreement. On the other hand, the results of overlap fraction by the proposed method are conservative or underestimate of the overlap fraction calculated by conventional approach of image-based method. Convergent images acquired for two corners are used for 3D model generation by pix4D software. Figures 8 and 9 showcase the generated 3D models. These figures validate the results of view planning performed for the two corners.   figure 9, homogenous surface or uniform texture lacks the features and thus the model contains voids in right wall. In addition, geometry of 90° corner allows less overlap compared to 55° corner. Consequently, smaller lengths for the building surfaces is obtained in 3D model for the 90° corner.
Authors have also conducted experiments for both corners with higher number of images (staring from 3 to 10 convergent images on the circular path). Experiments confirm that higher number of convergent images increases the overlap fraction between adjacent images and consequently spatial extent of 3D model of the corner increases. In addition, higher values of point cloud density of the 3D model are achieved.

CONCLUSION
This paper proposes an object-based approach for overlap estimation in field for convergent images for view planning of building corners. The paper considers the convergent geometry of image acquisition for a building corner and derives mathematical expressions for estimating the overlap fraction both in laboratory and field. The expressions for overlap calculations in field are simple as well as elegant for use with an electronic calculator or developing a monograph for a given camera and building corner geometry. For two corner sites of exterior angles 55° and 90°, experiments are conducted in field. Image footprints and overlap fractions are measured and results are validated. The overlap fractions for convergent images should not be assumed constant as the overlap fractions for convergent images vary according to camera positions. The experiments, theoretical calculations, and validation show close agreement among overlap fraction values for both corner sites. Reliable results suggest that the field-based method is an alternate to the image-based methods in lab for convergent images. Moreover, the method can be implemented in field with minimum expertise because one needs to measure only geometric dimensions of convergent images (or footprints) for a building corner in field. Furthermore, generated 3D model of the two corners sites also confirm that field-based measurements of overlap fraction demonstrated in this paper can be used for view planning exercise for building corners consisting of two non-coplanar surfaces. Moreover, repeating this exercise for each building corner can create seamless 3D model of the building by convergent images. Authors also envision that the similar methods can be developed for validation of overlaps in field for various CRTP applications for irregular surface formed by multiple non-coplanar surfaces.