EXPLOITING MIRRORS IN 3D RECONSTRUCTION OF SMALL ARTEFACTS

: 3D reconstruction of small artefacts is very significant in order to capture the details of the whole object irrespective of the documentation method which is used (Ranged Based or Image Based). Sometimes it is very difficult to achieve it because of hidden parts, occlusions, and obstructions which the object has. Hence, more data are necessary in order to 3D digitise the whole of the artefact leading to increased time for collecting and consequently processing the data. A methodology is necessary in order to reduce the collection of the data and therefore their processing time especially in cases of mass digitisation. So in this paper, the use of mirrors in particular high-quality mirrors in the data acquisition phase for the 3D reconstruction of small artefacts is investigated. Two case studies of 3D reconstruction are presented: the first one concerns Range-Based modelling especially a Time of Flight laser scanner is utilised and in the second one Image-Based modelling technique is implemented.


INTRODUCTION
Recently, 3D modelling has become a necessity for the conservation of small Cultural Heritage objects.Often, it is very crucial to capture the details of the whole object, especially in the case of museum artefacts.Sometimes this is very difficult because of hidden parts, occlusions and obstructions of parts of the object.This leads to increased time for capturing and consequently processing the data in particular when mass digitisation of artefacts is the case.Mirrors, whose use for photogrammetric purposes was firstly mentioned fifty years ago (Mikhail, 1968), have been utilised in many projects concerning 3D reconstruction.In this paper, a front surfaced mirror is used in order to investigate if it positively affects the process of data acquisition for the 3D reconstruction of small artefacts.The rest of the paper is organised as follows: section 2 describes the state of art about using mirrors in photogrammetry and 3D reconstruction.Section 3 presents the kinds of mirrors available and used in the industry.Also, it describes the reflection theory and the development of the Mirror Transformation application.Section 4 describes the implementation of the algorithm in real conditions.Section 5 shows the final results and Section 6 presents some comparisons.Finally, section 7 presents the concluding remarks and some future perspectives for using mirrors in 3D modelling.

STATE OF THE ART
Mirrors in photogrammetry were firstly reported by Mikhail (1968), who mentions that different perspective views can be obtained without moving the camera by placing a mirror close to the object.The position of the camera must be in such a way to photograph both object and its reflection in the mirror(s).He presented four case studies in which it is assumed that the camera axis is horizontal and the object space coordinate system has its origin at the camera perspective center, its X axis toward the * Corresponding author mirror, which is at its side and its Y axis along the optical axis.A four-stereo system including a single camera with the use of planar, ellipsoidal, hyperbolic and parabolic mirrors was proposed by Nene & Nayar (1998).Their proposed system is able to acquire two or more projections of the scene in the same image with each projection is from a single viewpoint.The results were compared with conventional stereo systems.Mirrors were also used in close range photogrammetry especially in medical and railway applications (Tokarczyk & Mikrut, 2000).Especially the medical application concerned overload monitoring for the purpose of medical rehabilitation and the railway application was about location points on cross-sections.In both cases, images were calibrated in a laboratory.Putze (2005) described a 3D motion data analysis with a system which consisted of a highspeed camera and a four-headed mirror system in order to determine the coordinates of an object.Also, this system was employed for generating virtual cameras.In another project, mirrors were used in 3D reconstruction from 2D perspective images (Mitsumoto et al. 1992).Especially mirrors were utilised in order to find symmetric relations between a direct and its mirrored image.The 3D shape of an object can be reconstructed by finding correspondences by means of plane symmetry recovery method using vanishing points.It is also observed that even when a mirror is used, invisible parts may still remain.But when multiple mirrors are utilised occluded parts will decrease or disappear and, in this way, a full 3D reconstruction of an object will be possible.A system consisting of a stable camera and a mirror was presented by Hu et al. (2005).Specifically, they proved a theorem for the mirror's orientation and they determined the distance between mirror and object.A laboratory test was designed in order to analyse the arising noise when standard and high-quality mirrors were used in terrestrial laser scanning procedure (Riquelme et al. 2017).They mentioned that the order of magnitude of the errors obtained when mirrors are used is below the accuracy of the instrument.In addition, mirrors are used in planar catadioptric vision sensors consisting of a pinhole camera which observes a scene reflected on two or more planar mirrors.Doubek & Svoboda (2002) used catadioptric images in SfM procedure in order to reliably reconstruct a large indoor scene.Mariottini et al. (2011) explored a scenario in which a robot manipulator was equipped with a pinhole camera on its end-effector and was observing an unknown 3D scene both directly and reflected through multiple mirrors.Akay & Akgul (2014) mentioned the problem of using RGB-D cameras and they proposed a solution which uses mirrors in order to introduce these cameras as virtual cameras into the system.In another project reflected images on planar mirrors which were acquired by one camera were used for calibration and 3D reconstruction with the volumetric representation approach (Martins & Dias, 2004).

Kinds of Mirrors
In industry, there are several kinds of mirrors or in general reflective surfaces and they are used in many applications.So it is essential to analyse the various reflective surfaces and their reactions.The most common type of mirror found in everyday applications (such as home use etc.) is Second or Back Surface Mirrors.These mirrors consist of a thin layer of a highly reflective metal, which most commonly is aluminum, coated on the back of a glass surface and protected from corrosion by a special varnish.The main disadvantage of this type of mirrors is that any given light beam has to go through a thick layer of glass before being reflected on the metal.As a result, the phenomenon of refraction is implemented, once when the beam is inserted into the glass and secondly when it exits.This leads to miscalculated coordinates of every point of the reflected object.Front or First Surface Mirrors are commonly used in industrial and scientific applications such as in the construction of telescopes.They are constructed in the opposite way than the Back Surface Mirrors, having the metallic reflective surface in front and the glass layer in the back.These mirrors produce the best optical and metrical results during all of the experimental procedures, as there is no refraction of the light since it directly interacts with the reflective material.However, they are very sensitive because the reflective surface is directly exposed in the environment.Figure 1 shows an example of a Back and a Front surface mirror.In both cases the pen is touching the surface of the mirror.A ghosting effect is visible in the case of using a back surface mirror (First Surface Mirror, 2018) Despite the fact that a front surface mirror produces the best experimental results, its main disadvantage is its rather high cost.
Therefore, more research had to be done using different types of reflective surfaces such as energy efficient glass, reflective membranes, and plexiglass, in order to reduce the commercial cost of the studied method.
According to the results of the experiments, all these types of surfaces brought up different kinds of distortions and/or even absorbed the laser scanner's beam or completely altered the colours of the reflected images.Hence the use of expensive mirrors seems unavoidable.

Reflection Theory
Further study about the theory of reflection of objects is considered useful at this stage.The major questions that needed to be answered were about the position of the reflection of an object in relation to the mirror's surface and if the mirror changes the size of the reflected object.
In order to answer these questions, basic reflection theory is applied.According to the geometric approach, the distance of any point on the object from the reflective layer of a mirror is equal to its corresponding reflection on the image (Figure 2).Moreover, as every point is situated exactly opposite from its reflection on the image, and the imaginary line that connects them is exactly vertical to the surface of the mirror, it can be easily extracted that the size of the image is the same as the original object itself (Figure 3).

Transformation development
In order to understand and develop the proper transformation which should be applied to the reflected object's point cloud, a basic question had to be answered: If and how the mirror changes the reference system of an external observer e.g.laser scanner, camera etc.The theoretical approach of geometric optic physics suggests that the reflection of an object is a left-right inverted image of the original; for example, the reflection of a right hand is a left hand.This is a phenomenon which is generated by the use of mirrors.
As far as the external observer is concerned, the intervention of the mirror in the procedure does not change the orientation of the reflected object (only its image) and therefore the reference system remains the same (Figure 4).Given the fact that the reference system remains the same and every reflected point on the surface of the image is connected with its corresponding one on the original object, with a straight line vertical on the plane of the mirror, the most suitable type of transformation is the translation.In order to reconstruct the original surface of a reflected object, every point of the reflection must be moved vertically to the surface of the mirror at a distance twice as its distance from the mirror surface (Figure 5).To achieve connection between the real point cloud and the reflected one, the plane of the mirror has to be calculated in the reference system of the scanner or the camera.In the case of a laser scanner four special reflective (blue & white) targets are used which are recognisable by the scanner's software.These targets were measured in the laser scanner's system and used for the definition of the mirror's plane.On the other hand, in order to calculate the mirror's plane in the case of Image-Based Modelling coded black & white targets recognisable by the selected software were used and placed on the mirror.Figure 6 shows the location of the targets on the mirror's surface.As a result, it was possible to define the plane in the system of the scanner or camera.The next step was to reconstruct the reflected point cloud according to the transformation mentioned above.Given A as a single point from the point cloud, with coordinates XA, YA, ZA, the coordinates XM, YM, ZM of its vertical trace M on the plane of the mirror needed to be found at first (Figure 7).The vector  ⃗⃗⃗⃗⃗⃗ is parallel to the characteristic vector  ⃗ of the plane.This means that: where  = any real number.Solving the above equations for XM, YM, ZM and replacing the results in equation ( 1), the only unknown parameter is .The final solution for  is given by the following equation: By replacing  in equations ( 3) to ( 5) the coordinates of point M can be found.The next step, was to calculate the coordinates of the reconstructed point, let it be B, XB, YB, ZB (Figure 8).Due to the fact that the points A, M and B are on the same line, it is: and by replacing the coordinates in the above equation the final solution for XB, YB, ZB is: (  ,   ,   ) = (  ,   ,   ) + 2 * (  −   ,   −   ,   −   ) (8) It is also remarkable that in the case of Image-Based Modelling technique the produced point cloud has colour information.This information does not affect the transformation procedure because colour information is not transformed.

Mirror transformation application
The amount of points in a point cloud is generally enormous, ranging from a few thousand to millions.This means that the gathered information (3D coordinates) is also too much to process.Therefore, the process of the calculations had to be automated.For this purpose, a cross-platform application was developed written in Python TM version 2.7.6 (Python 2.7.6, 2018) using wxPython version 2.8 (wxPython, 2018) as Graphical User Interface (GUI) library.The software is called Mirror Transformation (Thomaidis, 2014).Figure 8 shows a flowchart of the Mirror transformation application.The source code of the software was modified in order to include colour information of every point.Also, an additional modification was carried out in the case of Image-Based Modelling.Especially the coded targets were not placed on the mirror's surface but on the plane that the mirror was put.For this reason, the models were not able to register correctly because there is a small difference between the targets and the mirror's plane.This distance was measured 3.22mm with the use of a digital caliper, with a measuring uncertainty of 0.01mm.This distance was inserted in the source code of the application specifically in the Z coordinate in which the amount of 6.44mm (2x3.22) was added because the distance is double as it appears in equation ( 7).The program is Free Software and is distributed under the terms of version 3 of the GNU General Public License (GNU General Public License version 3). Figure 9 shows a screenshot of the Mirror Transformation application interface.

Time of Flight (ToF) Pulse Scanner
At the first case study, the object, a box (Figure 10) with dimensions 45x14x28 cm, was scanned with a Leica ScanStation 2 ToF pulse laser scanner.The first surface mirror used was placed in all necessary positions in order to capture all of the object without moving the scanner.The orientation of the mirror was oblique in relation to the scanner-object system (Figure 11) in order to reduce the noise at the produced point cloud.Also, the object was scanned by changing the position of the scanner without using the mirror in order to compare the results.So in this case study, three colorless point clouds were created: one of the whole object, the halfobject, and its reflection.All point clouds were georeferenced in the reference system of the scanner with the use of targets recognisable by the scanner's software.

Image-based modelling technique
In the second experiment, the object used was a small (approx.10cm) statue representing an owl (Figure 12).
Figure 12: Case study object The object was photographed with a Canon EOS 1Ds Mark III camera that has a CMOS full frame sensor and a Canon fixed focal length 50mm EF L USM lens providing an f/1.2 maximum aperture.Special lighting equipment consisting of two Bowens Duo Lite studio flashlights and their control panel (Bowens lights, 2018) with 40x50cm soft boxes were utilized to light the scene.The object was stable and the camera moved around it (Figure 13).In order to create the 3D point clouds of the object, imaging procedure was implemented in two parts: the first one in front of the object (half perimeter of shooting circle / 0-180 o ) and the second one behind the object (second half of shooting circle / 180-360 o ).In the first part the shooting included both the object and the mirror, so the model was based on real object images in the front and the images of the reflection for the back side.In the second part the mirror was removed taking images behind the object, so the second 3D model was produced purely by the real object images from the front and the back without using any mirror.The studio flashlights were positioned so as not to appear or to reflect in the mirror.In particular, one of them was put on a pole directed from above.The luminance power was adjusted to the maximum, aiming to succeed the smallest aperture value (f/16), as well as, the largest depth of field.The focus point during the first half shooting perimeter (in front of the object) was performed manually in order to focus both the real object and its reflection in the mirror.For the second half shooting, behind of the object, the focus point was straight to the real object for all the frames.

Transformation
The transformation of the reflected point clouds was performed in Mirror Transformation application.As it was mentioned above, the first step of the transformation is the insertion of the coordinates of the targets.In the case of the ToF laser scanner the coordinates of the targets recognisable by the laser scanner software were used.Figure 14 shows the point clouds before and after the transformation.In the case of the Image-Based Modelling technique, the source code of the application was modified in order to include colour information and to resolve the problem that coded targets were not in the same plane with the mirror.Figure 15 presents the results.In order to extract safe and correct results about the use of mirrors in range based modelling, the results were compared with results which were derived by conventional methods.For this reason, the object was measured with three different methods.The first method made use of a steel measuring tape.The second one used the conventional method of scanning and the third one used scanning with the use of a mirror.  2 shows that the differences between the measured sizes of the point cloud of the object are absolute compatible both with measuring tape's measures also and between them.It is observed that the differences are within the accuracy limits of the specifications of the laser scanner.Also, the scanning method with the use of mirror fulfills the accuracy specification which was established because all the differences in the measured distances are smaller than 1cm.This means that scanning method with the use of mirrors is able to be implemented in geometric documentation of various objects.

Image Based Modelling
Comparisons were also carried out between the point clouds produced by Image-Based modelling technique.Table 3 shows the number of points in the sparse and dense point clouds in the cases of using or not using a mirror.In the front part of the object, the differences are under 0.5mm which is expected because the front part is the same in both cases, i.e. with mirror and without mirror.On the other hand, in the back part of the object, the differences are more obvious and they range from 1mm to 3mm.This part is not similar because in one case the point cloud resulted from images of the reflection and was transformed with Mirror Transformation application.In the other case, the 3D point cloud resulted from images taken directly at the back part of the artefact.

CONCLUDING REMARKS
From the two experiments presented above it is obvious that the use of a mirror in the process of data acquisition has some potential.It may be used either in the case of range based or image-based modelling enabling the acquisition of occluded or hidden parts of the object with definitely less effort.It remains to be established whether the resulting point clouds may be considered accurate.This will be done by comparing the resulting point clouds with one with better accuracy.It is expected that, although the difference in scale and consequently in GSD size has a negative effect on the final accuracy, the point clouds produced through mirror reflections are satisfactory for mass digitisation cases.
More mirrors may also be deployed around the object in order to speed up the process.In addition, it is proposed to automate the transformation by using clever targets and incorporating to the software their automatic recognition and decoding.

Figure 2 :
Figure 2: Reflection of a single point

Figure 3 :
Figure 3: The mirror does not change the size of the original object

Figure 6 :
Figure 6: Target placement around the mirror In order to calculate coordinates of the targets in the case of image-based modelling technique, distances between the targets were accurately measured.The trilateration procedure was utilised for the coordinate's calculation.The equation of a plane in 3D space is:  *  +  *  +  *  =  (1) where a, b, c = coordinates of characteristic normal of the plane d = constant The calculation both of the characteristic vector of the plane and the constant was performed from the measurements of the targets.As a result, it was possible to define the plane in the system of the scanner or camera.The next step was to reconstruct the reflected point cloud according to the transformation mentioned above.Given A as a single point from the point cloud, with coordinates XA, YA, ZA, the coordinates XM, YM, ZM of its vertical trace M on the plane of the mirror needed to be found at first (Figure7).The vector  ⃗⃗⃗⃗⃗⃗ is parallel to the characteristic vector  ⃗ of the plane.This means that:

Figure 8 :
Figure 8: Flowchart of the Mirror transformation application

Figure 9 :
Figure 9: Mirror Transformation Interface 4. IMPLEMENTATION Two case studies were carried out with the use of a First Surface Mirror.In the first, a Time of Flight (ToF) laser scanner was used and in the second Image Based Modelling technique was employed.

Figure 13 :
Figure 13: Image-based modelling experiment setup The creation of 3D point clouds was performed in Agisoft Photoscan software.Three 3D point clouds were created: a point cloud of the whole object, a point cloud of half of the object and a point cloud of its reflection.Point clouds were georeferenced with the use of four coded targets which the software can recognise automatically.

Figure 14 :
Figure 14: Point clouds before and after transformation.ToF case study

Figure 15 :
Figure 15: Point cloud before and after transformation.Image Based Modelling case study 5. COMPARISONS 5.1 Time of Flight (ToF) Scanner Number of a point in Sparse and Dense point cloudsThe significant difference is probably caused by the scale difference of the imaged object in the mirror, although care was taken to position the mirror as close to the back of the object as possible.The difference in taking distance is approximately 20%.In addition, a comparison was performed to establish the differences between the point clouds.CloudCompare software was implemented for this procedure in which 3D point clouds were inserted and Cloud to Cloud distance function was used.It is noted that at the moment no ground truth was available, i.e. a more accurate point cloud with e.g. a structured light scanner.Hence, the comparison results only to relative differences and no qualitative conclusions may be drawn.Figures16 and 17show the differences that point clouds have in the front (Figure16) and back part (Figure17) of the object.

Figure 16 :
Figure 16: Point cloud comparison (front part of the object)

Figure 17 :
Figure 17: Point cloud comparison (rear part of the object)

Table 1 :
Table1shows the measured sizes of the object for each method.Measured sizes per method Table 2 presents the differences between each method which was used.

Table 2 :
Absolute & Relative precision between the methodsTable