PHOTOGRAMMETRIC 3D RECONSTRUCTION IN MATLAB: DEVELOPMENT OF A FREE TOOL

: This paper presents the current state of development of a free Matlab tool for photogrammetric reconstruction developed at the University of Padova, Italy. The goal of this software is mostly educational, i.e. allowing students to have a close look to the speciﬁc steps which lead to the computation of a dense point cloud. As most of recently developed photogrammetric softwares, it is based on a Structure from Motion approach. Despite being mainly motivated by educational purposes, certain implementation details are clearly inspired by recent research works, e.g. limiting the computational burden of the feature matching by determining a suboptimal set of features to be considered, using information provided by external sensors to ease the matching process.


INTRODUCTION
Thanks to its accuracy and reliability, during the last two decades aerial and terrestrial Light Detection and Ranging (LiDAR) have been considered the state of the art of surveying and digital model generation.However, the recent quick development of photogrammetric reconstruction techniques based on the Structure from Motion approach (originally developed by the computer vision community (Agarwal et al., 2010, Byröd and Aström, 2010, Hartley and Zisserman, 2003, Ma et al., 2003)) have attracted attention by the surveying, remote sensing and geomatics research communities, making photogrammetry a very frequently considered surveying solution (recently even combined with the use of drones and/or integrated with laser scanning techniques) (Remondino et al., 2005, Masiero et al., 2017, Aicardi et al., 2016).
Ease of use of 3D reconstruction software, possibility of using cheap instrumentation (e.g. standard consumer cameras, which are usually much cheaper than laser scanners) have been key factors for the success of the method and of the related commercial and free software solutions, e.g.Agisoft PhotoScan, Pix4Dmapper, MicMac, SURE, 3DF Zephyr, VisualSFM just to cite some.Furthermore, in certain applications the possibility of easily moving the camera on the environment (e.g.just hand-carrying the camera or mounting it on a drone) can allow the generation of clouds with a more uniform distribution of the points therein with respect to laser scanners.
This paper aims at presenting the current state of development of a free Matlab tool for photogrammetric reconstruction (implementing the Structure from Motion approach) that is being developed by the Interdepartmental Research Center of Geomatics (CIRGEO) at the University Padua, Italy.The main goal of such free tool is that of providing a software particularly well suited for educational purposes, i.e. which allows to check and validate step by step the reconstruction procedure, allowing the user to fully understand strengths and weaknesses.
Both graphical interface and shell commands can be used to use the implemented tools.Despite being mainly developed for educational purposes, this tool also implements certain interesting functionalities which can make it of interest also for certain research purposes.For instance, since Matlab is typically not the ideal tool for efficiently deal with large amount of data, multiresolution approach and techniques for decreasing memory requirements can be considered in the implementation (Lingua et al., 2009, Martinez-Rubi et al., 2017).Integration of external information (e.g.provided by an Inertial Measurement Unit) are also allowed in order to improve tie points matching (Kurz and Benhimane, 2011, Troiani et al., 2014, Masiero and Vettore, 2016, He and Habib, 2016).
Performance, in terms of reconstruction accuracy, is compared with that of commercial software (e.g.Agisoft PhotoScan) in a case study with data acquired in the Agripolis campus of the University of Padua.Since computational burden is clearly of major interest when implementing software with a programming language as Matlab, several strategies have been implemented in order to reduce the computational time (partially reducing the reconstruction accuracy).
The developed sotware actually takes advantage of several functions already implemented in Matlab (e.g.feature extraction by means of BRISK), and of other Matlab libraries ppreviously developed by other authors, i.e.SIFT feature matching with VLFeat (Vedaldi and Fulkerson, 2008), CVLab computer vision library (Fusiello, 2013), SFMedu Structure from Motion tool (Xiao, 2014).Furthermore, a future integration with web-based view tools of 3D models will also be considered (Scianna and La Guardia, 2017).

FEATURE MATCHING BETWEEN IMAGE PAIRS
Feature matching for the computation of a set of tie points (between each image pair), and the subsequently bundle adjustment solution of the Structure from Motion problem, is done as described in the following: • Appearance-based feature matching • Geometry estimation and outlier rejection

Geometry estimation and outlier rejection
Once matching is done exploiting information summarized by descriptors, geometry shall be used in order to reduce the number of wrong matches.RANdom SAmple Consensus (RANSAC) is used to estimate from the obtained matches the Fundamental matrix (Ma et al., 2003) summarizing the geometrical relation between two images.
According to the RANSAC estimation procedure, several guesses are done in order to obtain a reliable estimation of the Fundamental matrix.The developed tool allows to compare the results obtained by varying the number of guesses in the RANSAC procedure and to evaluate the variability of the obtained estimate.
Intrinsically, the RANSAC procedure provides both an estimation of the geometry between the two images and the outlier matches (i.e. to be discarded).

External information
The recent spread of systems integrating several sensors allowed the possibility of exploiting information provided by such sensors in the matching procedure.
For instance, mobile mapping systems, based either on terrestrial/aerial vehicles or on human carried devices (Remondino et al., 2011, Al Hamad and El Sheimy, 2014, Chiang et al., 2012, Ballarin et al., 2017, Masiero et al., 2015, Fissore et al., 2017b), which have been used in the last dozen of years in order to map and monitor areas of interest, typically integrate information provided by laser scanners, cameras with position and orientation information provided by GNSS/IMU (Inertial Measurement Unit) or other positioning sensors, e.g.(Gikas andPerakis, 2016, Goel et al., 2017).
Here integration of orientation information has been implemented as described in (Masiero and Vettore, 2016), i.e. by integrating information about orientation provide by the IMU.
Nevertheless, the integration of other strategies will be considered in the future.For instance, the use of prior information about the trajectory can be exploited as shown in (He and Habib, 2016).

SPARSE RECONSTRUCTION
Geometry between pairs of images, estimated at the previous step, is used as initial guess for the bundle adjustment procedure: Bundle adjustment is used in order to obtain a sparse reconstruction of the scene and the relative positions/orientations of the cameras.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W8, 2017 5th International Workshop LowCost 3D -Sensors, Algorithms, Applications, 28-29 November 2017, Hamburg, Germany Motivated by recent works on the reduction of the number of tie points, in order to reduce the computational burden of this step only a subset of the original tie points is considered in the bundle adjustment.Currently the tie points to be used are randomly selected, however a more reliable strategy will be implemented in the future in the system, e.g. as described in (Martinez-Rubi et al., 2017) .

DENSE MATCHING
Nowadays, semi-global matching (Hirschmuller, 2008) is widely used in order to solve the dense-matching step.Nevertheless, a simpler approach is considered in this tool in order to reduce the computational complexity, accepting a possible reduction of the accuracy: since the main goal of this tool is that of being used for educational purposes, reducing the computational burden is of fundamental importance in order to make it quickly usable by the students.
This motivated the choice of the following strategy: • First, local interpolation on a dense set of points obtained from the sparse point cloud provides an initial estimate of dense 3D points.
• Each of the previously estimated points is processed: "altitude" of the point along the orthogonal direction to the local orientation of the interpolated surface is determined in order to obtain the best pixel match, in a vertical line locus (VLL) matching fashion (Linder, 2006).
In order to ease the matching procedure, similarity cost function is currently the zero mean normalized cross-correlation.

RESULTS
Fig. 4 (obtained by using Cloud Compare) compares the cloud points of the fac ¸ade of a University building considered here as case study.It is worth to notice that the presented software actually was not able to completely reconstruct all the fac ¸ade: areas close to the uncompleted region are those with larger difference with respect to PhotoScan reconstruction.
Agisoft PhotoScan and the presented software provided point clouds with similar cardinality of the fac ¸ade (4-5 Millions of points).According to Cloud Compare comparison, average distance between the two point clouds is approximately 12 cm.
Despite the reconstruction obtained with the proposed software is less regular than that of PhotoScan, it was also obtained by requiring much less computational burden (by a factor 10, approximately).

CONCLUSIONS AND FUTURE WORKS
This paper presented the current status of development of a photogrammetric 3D reconstruction tool that is under development at the University of Padova.
The presented tool will be improved in our future work in order to integrate other functionalities: • In order to reduce time to solve bundle adjustment tie points are currently randomly subsampled.A smarter rule for tie points selection shall be implemented to reduce the estimation error, for instance similarly to the method suggested in (Martinez-Rubi et al., 2017).
• Implementation of all the functionalities shall be completed as soon as possible.
• The software has been developed to mainly deal with airborne photogrammetry (or for different cases but in similar working conditions, e.g.(Fissore et al., 2017a)).However, its extension and validation on a larger variety of cases shall be considered, for instance related also to the reconstruction of closed shapes (e.g. as statues), which is quite common in cultural heritage applications (Costantino and Angelini, 2012, Faresin et al., 2012, Kersten et al., 2004, Spangher et al., 2017).
• Despite Matlab is widely used, it is a commercial product.The development of this tool in a free Matlab-like programming language shall be considered in the future.
Once completed the main parts of the software it will be available through the website of the University of Padova.

2. 1 Figure 1 .
Figure 1.Feature positions for a pair of images.

Fig. 3 Figure 3 .
Fig.3shows the matches of Fig.2.after outlier removal.Yellow lines in Fig.2.link matching points in the images.The presence of mostly horizontal yellow lines ensure a quite good reliability of the obtained result.
Figure 4. (a) Orthophoto of the building fac ¸ade considered as case study.(b) Comparison between cloud points generated by Agisoft PhotoScan and the presented Matlab tool.