INDOOR PHOTOGRAMMETRY AIDED WITH UWB NAVIGATION

: The subject of photogrammetric surveying with mobile devices, in particular smartphones, is becoming of signiﬁcant interest in the research community. Nowadays, the process of providing 3D point clouds with photogrammetric procedures is well known. However, external information is still typically needed in order to move from the point cloud obtained from images to a 3D metric reconstruction. This paper investigates the integration of information provided by an UWB positioning system with visual based reconstruction to produce a metric reconstruction. Furthermore, the orientation (with respect to North-East directions) of the obtained model is assessed thanks to the use of inertial sensors included in the considered UWB devices. Results of this integration are shown on two case studies in indoor environments.


INTRODUCTION
In the last decades, photogrammetry has been widely used for producing 3D representations of reality.In particular, it has become even more popular after the development of several commercial software based on the implementation of the Structure from Motion (SfM) approach.Indeed, the typical easiness of use of such kind software is enabling the production of photogrammetric 3D models to not so specialized persons as well.
Photo-based 3D models typically requires external information in order to be properly scaled, i.e. to become a metric reconstruction: control points and GNSS measurements are often used for such purpose (where the latter can be used for georeferencing the 3D model as well).Despite this procedure is widely used and it is well known to ensure reliable and accurate results, the use of GNSS and control points can be difficult in certain operating conditions, e.g. when surveying areas difficult to reach for humans/terrestrial vehicles (Bendea et al., 2008), in indoor environments (Tucci et al., 2018, Dabove et al., 2018) (recently scale estimation has been investigated also for smartphone-based indoor photogrammetric reconstructions, typically exploiting inertial sensor information (Mustaniemi et al., 2017, Ham et al., 2014, Alsubaie et al., 2017)).Thanks to the need of providing 3D models also in such conditions, 3D modelling without the need of control points (e.g.direct georeferencing (Chiang et al., 2012, Pfeifer et al., 2012, Lo et al., 2015, Masiero et al., 2017)) is recently becoming a quite hot topic.Motivated by the above considerations and to the potential application of 3D reconstruction without control points to other research fields as well (e.g.robotics, autonomous), this paper consider a solution based on the use of Ultra-Wideband (UWB) sensors in order to achieve metric reconstruction.Interestingly, use of UWB sensors shall be simultaneously used for navigation purposes as well.
UWB sensors are radio transmitters/receivers which enable the real time position estimation of an UWB rover.Such position is obtained by means of trilateration, by combining information of range measurements collected from a set of UWB devices fixed at constant locations (such devices are usually called anchors).A proper UWB sensor calibration can be considered in order to improve positioning performance, e.g.systematic error has been modelled as a constant time lag (e.g.due to device synchronization) (Hol, 2011) and as a polynomial term, as a function of distance (Dierenbach et al., 2015, Toth et al., 2015).Since positioning is based on the use of a collection of range measurements, best accuracy of the method is achieved when all measurements are collected in clear line of sight (CLOS), i.e. when obstacles are not in the line connecting two UWB devices.Actually, multipath and non-clear line of sight measurements sometimes are not avoidable, and hence they might affect positioning accuracy.
In this work, UWB rover is rigidly attached to the camera: their relative position and orientation is supposed to be constant during all image acquisition.Consequently, metric reconstruction can be achieved by estimating the reconstruction scale from the UWB camera-rover positions and the corresponding locations in the photogrammetric reconstruction.Since the considered rover is also provided with an Inertial Measurement Unit (IMU), its measurements can also be used in order to orienting the 3D model according to the East-North directions.
First aim of this paper is that of assessing scale estimation ability of the proposed system in two case studies in indoor environments.In order to ease the usage of the system, positions of UWB anchors are estimated by means of an ad hoc self-positioning procedure.Thanks to such procedure, once UWB anchors are properly positioned in the area to be surveyed, data collection can start without the need of surveying their positions.Actually, scale estimation accuracy obviously depends on the UWB self-positioning error.This paper investigates the obtained self-positioning error in different anchor configurations.
Furthermore, the performance of the reconstruction orientation, with respect to North-East directions, (based on information provided by the IMU) is investigated.This step clearly highly depends on the reliability of measurements provided by the IMU.This work actually presents new results obtained evolving the strategy already presented in (Masiero et al., 2018, Masiero et al., 2017)).

UWB POSITIONING SYSTEM
Local positioning was obtained by using a Pozyx UWB positioning system (Fig. 1).In this system each device is a radio transmitter and receiver that provides ranging measurements once connected to another UWB device.Ranging is obtained by the time of flight of the radio signal, even if this clearly causes a decrease of ranging accuracy when devices are not in clear line of sight (i.e. when obstacles are along the line connecting the two devices).A set of UWB devices, named anchors, are fixed to constant positions.Then, UWB rover position is obtained by trilateration of range measurements provided by UWB anchors.Furthermore, Pozyx rover is also provided with an IMU, which will be exploited in the following in order to estimate the 3D reconstruction orientation.Pozyx UWB devices have certain interesting properties: they are very portable (low weight and small size, see Fig. 1(b)) and quite low cost ($150 per device, approximately).Maximum range is over 100 m, however the number of available measurements (i.e.successful communications between UWB devices) significantly decreases as the distance between the devices becomes larger.Ranging error is affected by several factors, and, in particular, it is environment dependent.Given the quite complex error behavior, a more detailed description can be found in (Pozyx Labs, Pozyx positioning system, n.d., Masiero et al., 2017), and the goal of validating the system in conditions easily reproducible by standard suers, the system has been used in this work without any ad hoc calibration procedure.
The rationale is that any common user can obtain a metric 3D reconstruction simply by distributing UWB anchors over the area of interest and start taking photos (with UWB rover attached to the camera).Self-positioning of UWB anchors, based on their own range measurements, can be done automatically just before starting to take photos.Then, UWB rover tracks camera positions while images are acquired, and this information combined with image-based reconstruction leads to a metric 3D model.

UWB anchor self-positioning
Since UWB rover position is computed based on trilateration of range measurements provided by the anchors, the knowledge of anchor positions is a sine qua non information for tracking the rover during image acquisition.
Despite surveying anchor positions (e.g. with a total station, or with a professional GNSS receiver if in outdoor environments) is surely a viable solution, this implies the need for external instruments and the presence of trained personnel.The approach proposed in this paper aims at: (i) avoiding the need of personnel with specific expertise, (ii) use only quite low-cost easily portable devices (small and lightweight), (iii) being usable in a wide range of conditions, e.g. both indoors and outdoors.
Given the above motivations, an anchor self-positioning procedure is used in order to avoid the need of surveying anchor locations.Self-positioning is done by collecting with a rover the range measurements between anchors: a person holding the rover moves across the area of interest in order to collect measurements from all the anchors, then anchor positions are computed in a centralized way.This kind of acquisition procedure allows (self)positioning also when not all anchors are visible with each other (which is quite common when surveying a quite complex/large area), and it limits the requirements of information exchange between anchors.
Let a set of three anchors be an admissible network if (and only if) all the three range measurements between them are available.In this case it is possible to define a local coordinate reference system based on such three anchors, for instance using a procedure similar to the one described in (Pozyx Labs, Pozyx positioning system, n.d.).
Let R be the set of all available range measurements.Given two anchor networks, the conditions to be able to join them are: • there exists at least three range measurements ri 1 j 1 ∈ R, ri 2 j 2 ∈ R, ri 3 j 3 ∈ R such that {i1, i2, i3} are anchor indexes from the first network, whereas {j1, j2, j3} are from the second one (it is worth to notice that an index value can be repeated, e.g.i1 can be equal to i2).
• if the cardinality of both the networks is greater than one, then there should be at least two different index values in both {i1, i2, i3} and {j1, j2, j3}.
• Anchor cardinality of the network obtained joining the two sets has to be at least 4.
Let C and L be the set of currently available anchor networks and the set of pair of networks which can be joined, respectively.
Once anchor-to-anchor ranges have been collected, the following procedure, based on the assumption that anchors are positioned on a plane (i.e.2D positioning), is used in order to estimate anchor positions in a local reference system.
1. Compute C = {all admissible three-node networks, and all single anchor nodes} 2. Compute L = {list of networks in C which can be combined} while L = ∅ do 3.1.Randomly select an element l of L 3.2.Join the networks corresponding to l 3.3.Optimize anchor positions in the joined network 3.4.Update C 3.5.Update L end while 4. Compute range fitting error It is worth to notice that certain criteria different from a completely random choice can be applied in order to make Step 3.1 more effective.For instance, it might be convenient to give a higher priority to joining just one anchor to a larger networks than joining networks with higher cardinalities.It might also be convenient to give a higher priority to joining networks that lead to a smaller range fitting error (which however is possible only if fitting errors are pre-computed).
Step 3.3 is done by nonlinear optimization of anchor positions in the joined network with respect to the resulting range fitting error (e.g. in least squares sense).
Step 3.4: Once the two selected networks have been joined they are inserted in C. Instead, all other networks in C containing at least a node of the newly joined network are deleted from C.
Step 3.5: L is updated according to the new networks in C.
Finally, the procedure computes the range fitting error corresponding to the computed network.
The above procedure is repeated N times, where N is a design parameter, in order to have a high chance of determining a (closeto) optimal solution (since this is a non-convex optimization problem, optimization is sensitive to the initial condition, hence repeating the above procedure N times aims at reducing the probability of reaching just a local minimum).It is worth to notice that this optimization is typically done before the user starts acquiring the images, hence its computational time requirements are not so stringent.

UWB rover positioning
Once anchor positions have been computed, rover position can be estimated in real-time, for instance by using an Extended Kalman filter (Anderson and Moore, 2012) to integrate information provided by the anchors (e.g.acquiring anchors-rover range measurements and solving the trilateration problem).Rover dynamic can be conveniently modeled as a random walk on the velocity equation (Masiero et al., 2018).
If rover positions are not necessary during image acquisitions, then smoothing shall be considered instead of filtering.
Furthermore, rover altitude with respect to the ground typically varies among a relatively small interval of values.Hence, rover altitude can usually be set to a constant value in the filtering (or smoothing) algorithm, i.e. reducing the tracking to a 2D positioning problem.

METRIC 3D RECONSTRUCTION
Rover is rigidly attached to the camera during image acquisition.Lever-arm is usually quite small, and however both lever-arm and boresight angles should be estimated by calibration in order to improve the performance of the system (Hol et al., 2010).
If camera and positioning system are synchronized, then rover positions at time instants of image acquisitions can be introduced in the photogrammetric reconstruction procedure as priors for the camera locations.
Otherwise, scale (actually roto-translation and scale estimation) and synchronization estimation shall be done simultaneously, by minimizing the fitting error between camera positions computed by means of photogrammetric reconstruction and UWB positioning (Masiero et al., 2017).
Orientation of the obtained metric reconstruction can be estimated by exploiting measurements provided by the IMU embedded in the Pozyx UWB rover.
First, IMU measurements collected during image acquisition are used in to compute an approximate calibration model (actually, ad hoc calibration, if available, can surely lead to a better system performance, furthermore, stochastic modeling of inertial sensor errors can also improve the obtained results (Radi et al., 2018)).
Then, calibrated magnetometer and accelerometer measurements, {arover,i} and {mrover,i}, in correspondence of image acquisitions are considered.During image acquisition the camerarover system is assumed to be still, hence {arover,i} corresponds to a measurement of the gravity, whereas (if the system is not close to any ferromagnetic material and to any magnetic field source) {mrover,i} is a measurement of the geomagnetic field.A rough outlier rejection can be done for instance by neglecting measurements with absolute value much different from the expected one (which can be, for instance, the effect of magnetic field deviations due to the presence of metals quite close to the device).These measurements, taken in the rover reference frame, are transformed in the camera reference frame (exploiting camera-IMU calibration), in the photogrammetric coordinate system (thanks to the camera position and rotation estimates), and finally in the UWB metric reference system, obtaining {a uwb,i } and {m uwb,i }, respectively.Hence, estimates of the North and vertical directions, with respect to the UWB coordinate system, can be easily computed for instance taking the averages (or the medians) of {m uwb,i } and {a uwb,i }.
The method presented above for estimating the reconstruction orientation is based on averaging the IMU measurements (weighted averaging, if any information about the reliability of each measurement is taken into account) in order to reduce the noise effect.A similar but different approach has been recently proposed in (Alsubaie et al., 2017), where the comparison of IMU-camera measurements to determine which IMU measurements are more reliable: only most reliable ones are considered for the orientation estimation.
Finally, georeferencing (i.e.expressing the reconstruction in global mapping frame (e.g.WGS84, UTM32 in our case studies)) can be obtained by means of a GNSS receiver, if at least part of the surveyed area is outdoors.

RESULTS
In this section we will present some results on the UWB selfpositioning (subsection 4.1) and then we will assess the performance of the overall system on two case studies, namely the Impossible bastion (Fig. 2

UWB self-positioning
Performance of anchor self-positioning is clearly related to UWB range error measurements, and to anchor network geometry.Since anchor positions are computed by solving trilateration problems, the influence of anchor network geometry is similar to the case of GNSS positioning (but in this case we are considering 2D positioning), e.g.homogeneously distributed anchors typically provides better self-positioning performance.
In order to investigate the practical influence of anchor network on the performance of 2D self-positioning, we considered a specific configuration of anchors (see for instance red dots in Fig. 4(a)), and we repeated self-positioning increasing the distance of anchors along the x axis (while anchor coordinates along the y axis were constant during all the experiments).More specifically, distance dx along the x axis between two successive anchors varied from 0.8 m (Fig. 4(a)) to 2.0 m (Fig. 4(b)).The obtained selfpositioning results are summarized in Table 1.It is worth to notice that in all the considered cases all anchor-toanchor ranges were available.Furthermore, it has already been shown (see for instance (Masiero et al., 2017)) that UWB error (among these range values) is quite independent of the anchor distances.Consequently, the clear increase of self-positioning error should be due to the network differences in the four cases, i.e. measurement error is better compensated in Fig. 4(a) than in Fig. 4(b) (poor anchor distribution along the y direction in the latter case).Furthermore, Fig. 5 shows anchor self-positioning results in the two case studies.

Case study 1: Impossible bastion
Data acquisition in the bastion was carried out on 25 July 2017.Images have been taken by using a Canon G7X camera (20.2 MPix), with settings fixed at constant values (1/60-s shutter speed, f/1.8 aperture, 8.8-mm focal length, i.e., 35 mm equivalent: 24 mm).Five hundred and seven images have been collected in approximately one hour varying camera position and orientation.Portable spotlights were used in order to properly illuminate the bastion during image acquisitions.Agisoft PhotoScan performed reconstruction with camera self-calibration.Error characteristics are summarized in the two cases in Table 3.It is worth to notice that errors for the anchor self-positioning case in Table 3 are significantly larger than those in Table 2.This is mostly due to the scale estimation error (6% error on scale factor estimation in the second case study).Such error is also related to the specific choice made for Kalman filter parameters (i.e.tuning of the filter).For instance, self-positioning error reported in Table 3 has been obtained by using exactly the same parameter values used in case study 1, however, changing such values (e.g.just increasing the weight of measurements) error reduced to: 20.2 cm average error, 21.6 cm RMS error, 90.1 cm maximum error.
Finally, Fig. 10(b) shows the estimated model orientation with respect to the correct one.

CONCLUSIONS
This paper presented the use of photogrammetry combined with UWB sensors in order to provide metric reconstructions.The results shown in the two case studies show that the proposed method provides results with decimetric-level accuracy (actually not comparable yet to that which can be obtained with state-ofthe-art surveying methods) for what concerns relative error.The main advantages of the proposed method is that it is quite low cost, very portable and it can be used in a wide range of conditions, e.g. also when GNSS is not available/reliable.
Actually, UWB positiong has a very important role in scale estimation, and the obtained results have been shown to be sensitive to the choice of the parameter values used in the Kalman filter.Since positioning results in this case are not really necessary in real time, future investigations will be dedicated to the optimization of off-line positioning (e.g.smoothing) and to the integration of other sensor information in the positioning algorithm (e.g.inertial sensors, WiFi signals (Ham et al., 2014, Widyawan et al., 2012, Saeedi et al., 2014, Masiero et al., 2014)) Future investigation foresees also the use of different techniques for estimating model orientation (e.g. with respect to North-East directions) (Alsubaie et al., 2017), and more advanced inertial sensor error modeling (Radi et al., 2018) 6. ACKNOWLEDGMENTS
, subsection 4.2) and Savonarola gate (3(a) and (b), subsection 4.3), both located in Padua, Italy.Both the case studies are examples of indoor mapping, where GNSS signal is typically not available.Furthermore, natural illumination in these places is typically insufficient, hence external illumination has been used (when possible) in order to improve reconstruction results.(a) Figure 2. A section of the tunnel of the Impossible bastion.(a) An internal view of Savonarola gate.(b) 3D reconstruction of the internal part of Savonarola gate.

Figure 4 .
Figure 4. Comparison of UWB anchor positions: surveyed (red dots) and obtained by means of the self-positioning procedure (blue dots).Case studies: (a) Impossible bastion, (b) Savonarola gate.

Figure 5 .
Figure 5.Comparison of UWB anchor positions: surveyed (red dots) and obtained by means of the self-positioning procedure (blue dots).Case studies: (a) Impossible bastion, (b) Savonarola gate.

Figure 6
Figure 6(a) shows the map of point-to-point distances between TLS and photogrammetric point clouds in a local coordinate system (relative error), whereas Figure 6(b) reports the corresponding error distribution.Scale of photogrammetric point cloud shown in Figure 6(a) has been obtained with UWB anchor self-positioning.

Fig. 10
Fig. 10(a)  shows the estimated model orientation with respect to the correct one.
Photogrammetric 3D model accuracy assessment: comparison with the TLS 3D model.Point clouds registered in local coordinates with the ICP algorithm.(a) Map of point-to-point distances.(b) Histogram of distances shown in (a).

Figure 8
Figure 8(a) shows the map of point-to-point distances between TLS and photogrammetric point clouds in a local coordinate system (relative error), whereas Figure 8(b) reports the corresponding error distribution.Scale of photogrammetric point cloud shown in Figure 8(a) has been obtained with UWB anchor self-positioning.Similarly, Figure 9(a) shows the map of point-to-point distances between TLS and photogrammetric point clouds in a local coordinate system (relative error), and Figure 9(b) the corresponding error distribution, but in this case scale of photogrammetric reconstruction has been set to the optimal value.

Figure 7 .
Figure 7. Photogrammetric 3D model (with optimal scale factor) accuracy assessment: comparison with the TLS 3D model.Point clouds registered in local coordinates with the ICP algorithm.(a) Map of point-to-point distances.(b) Histogram of distances shown in (a).

Figure 8 .
Figure 8. Photogrammetric 3D model of Savonarola gate (with UWB anchor self-positioning) accuracy assessment: comparison with the TLS 3D model.Point clouds registered in local coordinates with the ICP algorithm.(a) Map of point-to-point distances.(b) Histogram of distances shown in (a).

Figure 9 .
Figure 9. Photogrammetric 3D model of Savonarola gate (with optimal scale factor) accuracy assessment: comparison with the TLS 3D model.Point clouds registered in local coordinates with the ICP algorithm.(a) Map of point-to-point distances.(b) Histogram of distances shown in (a).

Table 2 .
Reconstruction error in case study 1

Table 3 .
Reconstruction error in case study 1