CONNECTING INSIDE AND OUTSIDE THROUGH 360 ° IMAGERY FOR CLOSE-RANGE PHOTOGRAMMETRY

Metric documentation of buildings requires the connection of different spaces, such as rooms, corridors, floors, and interior and exterior spaces. Images and laser scans have to be oriented and registered to obtain accurate metric data about different areas and the related metric information (e.g., wall thickness). A robust registration can be obtained with total station measurements, especially when a geodetic network with multiple intersections on different station points is available. In the case of a photogrammetric project with several images acquired with a central perspective camera, the lack of total station measurements (i.e., control and check points) could result in a weak orientation for the limited overlap between images acquired through doors and windows. The procedure presented in this paper is based on 360° images acquired with an affordable digital camera (less than 350$). The large field of view of 360° images allows one to simultaneously capture different rooms as well as indoor and outdoor spaces, which will be visible in just a picture. This could provide a more robust orientation of multiple images acquired through narrow spaces. A combined bundle block adjustment that integrates central perspective and spherical images is here proposed and discussed. Additional considerations on the integration of fisheye images are discussed as well.


INTRODUCTION
Most projects in the field of architecture and engineering require the connection between different rooms, different floors, and interior and exterior spaces.Laser scanning has become very popular for the ability to acquire the entire scene around the station point.Registration procedures based on targets or natural features allow one to process several scans acquired from different locations.Large and complex buildings are therefore typically surveyed using laser scans.Additional total station measurements are useful to improve registration results, especially in the case of large sets of scans.The survey of complex buildings with photogrammetry is more complicated for the challenge in connecting different rooms.For this reason, photogrammetry is not the first choice for measuring buildings with many rooms and many floors.The typical photogrammetric approach based on the acquisition of different sets of images which are independently processed, could be critical for the lack of a suitable overlap between images acquired through doors and windows.Different photogrammetric projects can be registered using control points measured with a total station, which is an optimal measuring tool to provide a stable reference system for the whole project.Similar considerations still hold for the case of surveys with fisheye lenses, which allows surveyors to capture a larger field of view, reducing the number of images.This paper aims at presenting a novel solution for the survey of buildings using photogrammetry.The proposed solution is automated and allows the user to capture rooms in a traditional photogrammetric way, joining then the different blocks using a second set of images acquired with a 360 camera (Fangi, 2017;Matzen et al., 2017).Then, all images are simultaneously processed to obtain more reliable orientation results.
The problem discussed in this paper is summarized in Fig. 1a.Two sets of images are required to obtain a full reconstruction of both sides of the wall.Two doors do not allow a robust connection based only on the acquired central perspective images, making two or more station points necessary to join the photogrammetric projects with some control points.Recent work (e.g., Kwiatek andTokarczyk, 2014, 2015;Aghayaria et al., 2017;Abate et al., 2017;Barazzetti et al., 2017, 2018, Mandelli et al., 2017) has demonstrated that images acquired with affordable 360° cameras can be used for photogrammetric projects, notwithstanding metric accuracy is about 4-6 time worse than results achievable using central perspective cameras (Fig. 1b).Although dense point clouds can be generated from 360° images, the resolution (point density) is still not sufficient for digital reconstructions with a very high level of detail.360° images could results useful for long and narrow spaces when traditional central perspectives would provide a vast number of images.The approach proposed in this paper is a combined adjustment of central perspective and 360° images, as illustrated in Fig. 1c.The idea is to obtain a metric reconstruction from two independent sets of central perspective images, that are connected in a single inside-outside project through 360° images.In other words, 360° images are used for the orientation phase, but they are not included in the additional steps of the processing workflow: dense point cloud generation, mesh production, texture mapping, and orthophoto generation.The proposed procedure is fully automated since the used software (Agisoft PhotoScan in this work) can match corresponding points in the images, notwithstanding they are based on different camera models.The method limits the use of total stations for connecting the various projects so that a single photogrammetric project can be created for rooms, corridors, indoor and outdoor spaces.
Different results and experiments are described and discussed in the paper to explain the pros and cons of the proposed method.Results with varying image combinations are reported to make the reader aware of the potential and limitations of the proposed method.It is important to mention that the method can also integrate fisheye images, as illustrated in Fig 1d .On the other hand, tests with fisheye lenses (Fassi et al., 2018) are not illustrated in this paper, except for a preliminary analysis reported in the last section.

CONSIDERATIONS ON 3D MODELING BASED ON 360° IMAGES
As mentioned in the previous section, a block of spherical images can be automatically processed to generate accurate 3D models.Results by different authors have demonstrated that the method cannot achieve the accuracy of projects based on central perspectives or fisheye lenses (Barazzetti et al., 2018).
The method based only on 360° images becomes very useful in the case of long and narrow sequences, or when a limited metric accuracy is required, and a fast data acquisition and processing are of major importance.Fig. 2 shows the results for a sequence of images acquired around a building.The sequence is made up of 104 spherical images acquired with a Xiaomi Mijia 360.
Besides, some images were acquired inside the building, entering from two doors on two opposite facades (Fig. 3).As can be seen, the inside/outside connection is feasible but the different illumination conditions result in some problems with front-and rear-facing images, which are also visible in the final spherical (equirectangular) projection.Data acquisition took about 15 minutes.Processing was carried out on an Intel i7, 32 GB RAM, NVIDIA 1060 GTX graphic card.The idea behind this paper is to integrate such images in an adjustment with a sequence of central perspective images.As can be 360° images are able to connect different spaces with an accuracy of about 20 mm.On the other hand, the metric resolution is too low to generate accurate orthophotos of the facades at a scale 1:50, as requested in the current project.

360° IMAGES AS A LINK BETWEEN DIFFERENT PHOTOGRAMMETRIC PROJECTS
As mentioned in the previous section, the idea is to process sequences and blocks of central perspective images, using the 360° images as a link between different rooms.Different walls are considered different photogrammetric projects.
The approach becomes similar to the traditional work carried out with a total station placed inside a room, measuring control points on the different walls to register photogrammetric projects in a single reference system.The link between different spaces is achieved with the total station, which can be repositioned using traditional surveying schemes (geodetic networks, traversing).Here, the total station is replaced by images acquired with the spherical camera.Short baselines are required to orient the images, so multiple images are taken walking in different rooms.Then, central perspective images are oriented using points automatically extracted from both central perspectives and 360° images.The experiments carried out with Agisoft PhotoScan have revealed that it is possible to process such images in a single adjustment.Fig. 5 (top) shows an example between two rooms.The same figure (bottom) shows a detail.The wall separating the two rooms has been captured with two sequences of central perspectives images collected with a calibrated Nikon D610 with a 20 mm lens.Then, a sequence of 360° images was captured to link the rooms.Four control points were placed on the fourth walls of the first room, whereas the other four points in the second room were used as check points.Points were measured with a total station.Then, image orientation allowed one to estimate exterior orientation parameters of all the images in a single step.Table 1 shows the results obtained for the project.As can be seen, residual errors on control points are significantly better than check points (about 5 times).This indicates that the sequence was affected by an error that becomes larger depending on the number of images.The error cannot be neglected for real applications requiring detailed photogrammetric survey, making the method still not completely mature for producing accurate deliverables.Another consideration is related to the worse accuracy obtained for planar coordinates, whereas Z errors are smaller.However, the reader should be aware that control and check points were measured only in 360° images, in which the points cannot be detected with the same precision of traditional images.In other words, this error could also be caused by an error in the manual collimation of control points.For this reason, a second test was carried out acquiring some laser scans inside the two rooms, comparing laser scans in the room with with the point cloud generated from central perspective images.
The laser scanner used is a Faro Focus 3D HDR.CloudCompare provided a very small discrepancy between the two point clouds (Fig. 6), i.e. 0.002 m mean distance and 0.001 m standard deviation.This results is quite interesting, especially because of bias was expected, at least in the normal direction to the wall (the direction parallel to the wall has less influence since the wall is quite flat).Such results require future investigations since the tests using total station check points, and laser scans (in the same room) give contradictory results concerning metric accuracy.It is the authors' opinion that the market of 360° camera is going to provide new sensors with a better resolution, that is still quite limited for images with an angle view of 360°.This makes the method very attractive especially for those operators that have not the opportunity to use a total station for connecting different rooms or indoor/outdoor spaces of buildings.

360° IMAGES TO CONNECT ROOMS AT DIFFERENT FLOORS
Another test was carried out to connect two floors of a building.The main issue in this experiment is the progressive accumulation of errors in the orientation of a long sequence in narrow spaces (corridors).In addition, problems are expected along the staircase.In other words, the aim is to check the total error caused by the concatenation of several images.Such error can be significantly more significant than the traditional metric accuracy required in photogrammetric projects for building reconstruction.
A sequence of 96 images was captured walking with the 360° camera from a room to the corresponding room upstairs.Figure 7 shows two spherical images of the sequence.They were taken in the room downstairs and along the staircase.As can be seen, the texture of the scene is not optimal for photogrammetric projects.
Figure 7.Some images of the sequence acquired between two floors.
Different targets were placed in the two rooms.Coordinates were measured with both the 360° camera and a traverse of 5 stations, using a total station Leica 1200.Images were then oriented obtaining the results shows in Figure 8.All targets were used as control points to check the total deviation of the 360° image sequence.Results in table 2 demonstrate that the discrepancy is about 5 cm.A second test was carried out using the coordinates of the room upstairs as control points, whereas targets in the second room were used as check points.The error on check points is shown in table 3, and it is about 12 cm.This confirms that the method is still not suitable for the production of accurate deliverables at typical scales 1:50 or 1:100.The camera can be used only when measurements with limited accuracy are required, such as the estimation of room volumes.

CONCLUSIONS AND OUTLOOKS
The paper has proposed a novel approach based on images acquired inside different rooms, different floors, and exterior / interior spaces.The combined use of a central perspective camera and 360° images allows the user to connect those rooms through doors and windows.The results have proved that metric accuracy is still not comparable with typical results achievable with a total station, which remains the best solution to create a set of control points in a unique reference system.Results obtained with the proposed solution has to be considered as the preliminary outcome of future work with spherical images captured by low-cost cameras.The commercial market of 360° cameras is evolving very fast, providing new sensors with better geometric and radiometric resolution.From this point of view, hardware and algorithmic improvements are expected, making the method attractive for users who do not have the opportunity to use a total station.Future work will include the analysis of photogrammetric blocks made up of central perspective, spherical and fisheye images.First results with fisheye images incorporated into the combined adjustment have revealed that the approach is feasible (Strecha et al., 2015).
An experiment related to a combined project with fisheye images is illustrated in Figure 9.The test was carried out using a wall with a good texture (bricks) and some targets measured with a total station.Ground control points are coded targets automatically matched using the available tools in PhotoScan.
Targets were matched on all the images, notwithstanding the different camera model and their corresponding "distortion" in the images.Probably, as the target has a limited size, the local distortion due to a different camera model is not significant and does not prevent correlation algorithms to match specific features.The software was able to orient all images, demonstrating that all camera models can be processed in a single adjustment.The achieved RMS on control point coordinates provided a metric accuracy better than ±2 mm, that is consistent with the average resolution of the images.Results confirm the feasibility of the proposed approach, notwithstanding more experiments will be carried out in future work to assess the accuracy and level of automation of the proposed method.Limitations of the proposed approach are also related to the need of suitable object for photogrammetric reconstructions.
The proposed method has all the typical limitations of pure photogrammetric projects, i.e. the need of objects with a good texture as well as good lighting conditions, which are not simple to obtain for 360° images.

Figure 2 .
Figure 2. Some examples of the 104 images acquired inside and outside a building.

Figure 3 .
Figure 3.The original images acquired from front-and rearfacing cameras (top) and the final spherical image generated by stitching the previous images (bottom).

Figure 5 .
Figure 5. Image orientation results for central perspectives and 360° images in the two rooms (top), and a detail for only the second room (bottom).

Figure 6 .
Figure 6.Comparison between a laser scan and the photogrammetric point cloud generated from central perspective images in the room with check points.

Figure 8 .
Figure 8. 3D visualization of the computed camera poses.

Figure 9 .
Figure 9.The test where spherical, fisheye and central perspective images were simultaneously oriented.

Table 2 .
Results using all targets as control points.

Table 3 .
Results using only targets in the first room as control points.Targets in the second room are used as check points.