PHOTOGRAMMETRIC PROCESSES AND AUGMENTED REALITY APPLICATIONS USING MOBILE DEVICES

: This contribution shows the first results of a research in fieri , which aims to introduce low-cost instruments for the continuous monitoring of architectures. The use of these devices in the architectural field is of great interest to the scientific community and therefore, with the aim of researching rapid mapping methodologies, the integrated camera on the new iPhone 12PRO is analyzed and then compared with the NikonD5000 reflex camera, whose use is more consolidated. In the era of digitalization and industry 4.0, smartphones have made significant progress and these devices are establishing as ideal solutions, thanks to their technical characteristics, costs and portability, compared to other acquisition techniques. The study is aimed at experimenting with image-based methodologies with the use of low-cost sensors where the three-dimensional models will constitute the basic element on which, through augmented reality applications, it will be possible to implement strategies aimed at documentation, conservation and monitoring. The experiment, reported therein, was conducted on the facade of the Quadriportico of the Cathedral of San Matteo in Salerno, Italy.


INTRODUCTION
The wealth of themes that affect the field of conservation and monitoring of Cultural Heritage requires the development of increasingly advanced analysis and investigation methodologies. One of the main challenges that the scientific community has set is to investigate new solutions to monitor cultural heritage, using alternative techniques of survey with the experimentation of lowcost sensors. In recent years, the effects of the digital revolution are increasingly evident through the widespread development of technological devices which have amended the usual techniques of acquisition, communication and perception of reality. In the era of digitization and industry 4.0, smartphones represent a new frontier compared to traditional sensors used in the disciplines of survey and representation such as laser scanners, action cameras, digital cameras, etc. In this context, technological development has made significant progress thanks to the continuous implementation of advanced multicore processors, increasingly high-performance digital cameras and integrated LiDAR sensors (Light Detection and Ranging) -such as those incorporated into the PRO version of the iPad and iPhone of the Apple -which open up new application possibilities to augmented reality, virtual reality and three-dimensional reconstruction (Santachiara et al., 2018;Scianna et al., 2019;Carrion-Ruiz et al., 2019;Campi and Falcone, 2021*;Falcone et al., 2021;Spreafico et al., 2021). The use of these devices is of great interest in the field of geomatics and, in this contribution, the integrated camera on the new iPhone 12PRO is analyzed and then compared with the reflex camera, whose use is more consolidated, with the aim of researching methodologies of rapid mapping-image based with low-cost sensors. As a case study the facade of the Quadriportico of the Cathedral of San Matteo in Salerno (Italy) was investigated (M.C. and A.d.L.).

STATE OF ART
Over the last few years, several researches have been dealing with the issue of rapid mapping survey for the purposes of documentation and monitoring of architectures. The techniques and sensors used are different and the state of the literature highlights how the most up-to-date studies are focusing, in the historical-architectural field, on the validation of new low-cost instruments by comparing them with the now consolidated acquisition techniques based on active sensors, such as terrestrial laser scanning (TLS) and passive sensors such as terrestrial and aerial photogrammetry and mapping mobile systems (Cera, V. and Iovane, D., 2015;Chiabrandro et al., 2015;Bitelli et al., 2017;Nocerino et. al., 2017;Campi et al., 2019;Muthanna et al., 2018;Pulcrano et al., 2019;). Each of these techniques has its advantages and disadvantages in terms of accuracy of the metric, colorimetric data, instrumentation costs, acquisition times, data processing, etc. However, image-based methodologies are establishing themselves as ideal solutions for rapid mapping activities and this result has been achieved both for the development of increasingly advanced sensors, manageable and reduced size and cost that for the continuous implementation of image matching algorithms in Structure from Motion software. In this context, mobile devices play a key role, which aim at acquiring data in shorter times. Therefore, the paper investigates image based and augmented reality techniques on smartphones with the aim of verifying the potential in the field of Cultural Heritage (M.F.).

METHODOLOGY
The experiment was conducted on the facade of the Quadriportico of the Cathedral of San Matteo in Salerno. The methodology has been articulated in three main phases. The first phase related to the analysis of the bibliographic sources has made it possible to know the architectural interventions over the centuries that have modified the original structure of the facade. In addition to the analysis of historical documentation, a survey campaign with image-based technology and the acquisition of morphometric data from smartphones and digital cameras was conducted as a second phase. Based on the results obtained, it was possible to compare the levels of accuracy of the data acquired by the two different devices, assessing their limits and potential. Subsequently, after the survey phases, we proceeded to the third and last phase which concerned the three-dimensional realization of the facade, the model of which was imported into an augmented reality display system using the ARki application installed on smartphone.

Case history: the façade of the Quadriportico of the Cathedral of San Matteo
The facade of the Quadriportico of the Cathedral of San Matteo was chosen as a case study on which to apply the abovementioned methodology. This choice was dictated by the desire to investigate innovative rapid mapping methodologies on this site with the use of low-cost sensors and, in this context, this contribution reports the first results of an ongoing research that is taking place in the field of the PhD in Architecture at the University of Naples Federico II. The Cathedral of San Matteo was built between 1080 and 1085. After the rapid period of the foundation, which led to the construction of the basilica and the crypt, where the spolie of San Matteo are kept, the works continued slowly and ended only towards the middle of the 12th century with the completion of the large atrium, called Quadriportico, and the bell tower. Today, the entrance, the result of several alterations that have taken place over the centuries, is preceded by a high façade in neoclassical style marked by two pairs of pilasters symmetrically arranged on the sides of the two smooth columns that frame the entrance portal (figure 1). Access to the portal is through the monumental staircase with double lateral ramp that was built in the second half of the eighteenth century by Archbishop Isidoro Sanchez De Luna. This intervention replaced, as described by Marsilio Colonna in 1580, the Romanesque entrance preceded by a semicircular staircase of 12 steps that extended the extension of the optical axis from the central nave of the church to the square in front of the entrance. The facade was decidedly lower than the present one and, in all probability, it was the same as that of the lateral arms of the Quadriportico. Today, the portal called Porta dei Leoni remains of the ancient facade due to the presence of two marble statues on the sides of the jambs, depicting a lion (on the right) and a lioness with a lion (on the left) (figure 2). Both have their heads turned downwards and are sculpted in a naturalistic way. On the architrave, above the portal, there is a lunette with the epistyle carved from a vine with animal decorations: a monkey and a dove pecking at dates. In this representation, clear references to the symbologies of early Christian architecture are evident while on the edges of the same an inscription in Latin, in imitation of a Roman portal, recalls the peace between Roberto il Guiscardo and his nephew Giordano and, therefore, the alliance between the principalities of Salerno and Capua. An ancient mosaic was previously placed in the lunette, replaced in 1768 by the current fresco depicting Saint Matthew (Capone, 1927;Carucci, 1989;Pace, 1997;Braca, 2003;Braca, 2018;). In the middle of the last century, on the facade of the Cathedral were present, below the trabeation, four circular oculus that were buffered during the restoration of the Second World War. Also, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China on the northern side, behind the facade was the garden of the Palace of the barons of Santomango (figure 3).

-Photogrammetric survey from digital camera
The geometric-dimensional data of the object of study were acquired by integrating two different sensors. This methodology is not totally new in the disciplinary sector of the survey, but the use of the sensor present in the smartphone is innovative.
The survey campaign was conducted with an image-based technique, based on photogrammetric algorithms in which the three-dimensional data of the facade was recorded with two types of low-cost instruments: Nikon D5000 digital camera and iPhone 12PRO. In both cases, before proceeding with the acquisition of the frames, the dataset of photos was planned and, to cover the entire facade, 15 station points were provided for an optimal distribution of the images. These images were recorded in both RAW and JPEG formats, while using a photographic tripod in order to reduce vibration during shooting. Finally, the main characteristics of the two tools used together with the specifications of the photogrammetric process are shown in figure 4 and figure 5.
In the first survey phase, set an average GSD (Ground Sample Distance) of 4 mm, a Nikon D5000 reflex camera was used with a lens with a focal length set at 24 mm for the acquisition of the first dataset, divided into 229 frames. To obtain the recognition of homologous points, an average overlap between the acquired frames of about 70% was guaranteed with the parallel axis and convergent axis shooting technique. This procedure made it possible to recover spatial information on different levels of depth and to limit distortion errors. Due to the variable lighting conditions, it was advisable to carry out a radiometric pre-processing of the images acquired during the first dataset of photos. The processing was carried out using the Metashape-Agisoft software, based on the use of image matching algorithms for the automatic orientation of the frames.
To implement the calculation, the camera calibration certificate was used in order to refine the correction parameters of the internal aberration of the optics. After the phase of automatic alignment of the frames, the sparse point cloud of 382,627 points was generated followed by the construction of the dense cloud of 31,781,783 points scaled thanks to the metric data acquired through direct survey methodology. Finally, in the last two stages of the process, the textured three-dimensional model was built.
Average quality parameters have been set for the entire workflow. Figure 5. Photogrammetric process in Metashape-Agisoft.

-Photogrammetric survey from iPhone 12PRO
The second dataset saw the use of the smartphone as the protagonist. The new Apple iPhone 12PRO mobile device is one of the most technologically advanced IOS models on the market and has a 6.1-inch display with a resolution of 2532 x 1170 pixels. The camera has been significantly integrated, compared to previous models, with the redesign of two of the three sensors present inside the rear photographic module. The wide-angle lens has been improved and is equipped with a seven-element lens with an f / 1.6 aperture capable of capturing 27% more light and in this case the optical stabilization sensor has been integrated, a technology reserved for reflex cameras. digital. Instead, the ultrawide angle with f / 2.4 aperture and 120 ° angle of view, offers the possibility of taking shots with the night mode (https://www.Apple.com). For the recording of the frames the smartphone has been set in manual mode, with a focal length of 18 mm. Retracing the same station points, 215 images were recorded with the consolidated overlapping rules. The subsequent processing phase followed the photogrammetric pipeline described previously, once again setting average quality parameters and from the image processing a sparse point cloud of 350,457 points and a dense point cloud of 29,659,773 points were obtained.

COMPARISON AND ANALYSIS
Completed the processing phases of three-dimensional data, the results obtained by the two different sensors used were examined at different scales of detail, with the comparison of the two dense clouds, from general to particular. In the last case, a small area of the marble jamb was analyzed as a portion, including the statue of the lion, located to the left of the access portal. A first phase of evaluation of the 3D models was the result of a critical analysis carried out by the authors in which some incomplete areas were highlighted. In fact, in both point clouds, as imagined, incomplete areas were found on the part above the entablature, which was not processed by the software, and this lack is due to the restricted acquisition space and the high of the facade. Also, a small portion of the dataset from the iPhone 12PRO was not captured due to some vehicles obstructing the view. The colorimetric characteristics of the photogrammetric cloud acquired by smartphone improve (figure 6). Considering the objective of this contribution, for the comparison operations between the two instruments, the first analysis was carried out by aligning and calculating the deviation between the cloud acquired by NikonD5000 digital camera (A) and that acquired by iPhone 12PRO (B). The alignment process was carried out in the open source application CloudCompare where it was necessary to align the photogrammetric dataset B with dataset A, used as a reference cloud for the registration procedure with the ICP (Interactive Closest Point) algorithm. For the alignment procedure, ten homologous points, characteristic of the façade and arranged on several levels of the floor, were manually collimated. At this point a rigid roto-translation was applied, finding an RMSE (Root Mean Square Error) alignment error between the two data sets equal to 0.04 m. Finally, the cloud / cloud distance values were calculated. This first comparison showed discrepancies in the upper part of the entablature, due to the difficulty in shooting the frames, and an adherence of 2 cm for the rest of the facade (Figure 7). Subsequently, two cross sections were extrapolated ( Figure 8) in which each profile corresponds to the point cloud generated by the two sensors. In both cases we can be satisfied with the result obtained and note how the cloud acquired by smartphone (in red) is quite uniform to the profile of the photogrammetric cloud, acquired by reflex camera (in blue), finding only near the corner parts of the columns and of the pilasters a lower geometric precision.  As a final analysis, roughness value was analyzed, a tool implemented in CloudCompare, on the same portion of the two clouds and, once again, the two datasets were compared. With this function the deviation of distances of point samples with the value of r = 0.1 m were calculated. In comparison, this type of analysis reported the following results: the point cloud A ( figure  9) is characterized by a noise level equal to 1.2 cm with a homogeneous distribution of the points while the dataset B has slightly higher noise values which are around 1.8 cm with a sparser distribution of the points but which in any case allow to read geometric and detailed information (figure 10).

RECOSTRUCTION AND AUGMENTED REALITY APP
The evolution of augmented reality in the architectural field is closely related to the potential of industry 4.0. In this sector, the technological scenario of augmented reality enables numerous application horizons: it is possible to implement strategies aimed at documentation, conservation, continuous monitoring and design, supporting domain experts in the decision-making process. The effectiveness of these systems is mainly due to the potential to generate immersive and instant experiences with an overlap of information ranging on different levels of reading.
For this reason, the last phase of the methodology involved the three-dimensional modeling of the facade with the implementation of the model in an AR display system. Before proceeding with the import on the ARki application, installed on the iPhone 12PRO, the mesh obtained from the photogrammetric process was decimated by keeping a greater number of polygons near the curved parts and edges in order to obtain a geometric reconstruction rather faithful to the real data. Therefore, it was possible to build the 3D model in the Rhinoceros application, with a NURBS type modeling. Once the model was created, it was necessary to export it from Rhinoceros, in fbx format, and import it into ARki (figure 11).

CONCLUSIONS
In conclusion, the results obtained from the comparison of the photogrammetric clouds acquired by low-cost sensors have been reported. The focus of the contribution was to validate the new photographic and augmented reality sensor in the iPhone 12PRO for rapid mapping surveys. As can be seen from the analyzes carried out, the results of this new photographic sensor, even if with slightly lower values than the professional camera sensor, are satisfactory, offering quality in terms of geometric and radiometric accuracy. Therefore, we believe that this new smartphone, thanks also to the integration of the Lidar sensor, will be widely used in the field of cultural heritage with further technological advances (M.C. and A.d.L.).