INTEGRATION OF PHOTOGRAMMETRY, COMPUTED TOMOGRAPHY AND ENDOSCOPY FOR GYROSCOPE 3D DIGITIZATION

Cultural heritage preservation via 3D digitization is becoming more and more important. Besides conventional buildings and landmarks, many technical instruments and artifacts, which belong to tech heritage (TH), are also of great importance, historically and didactically. Gyroscopes, which can be dated back for 200 years, are fascinating instruments with complex structures and different working principles. With such properties, any 3D digitization of Gyroscopes could not be realized by simply using conventional solutions of photogrammetry or laser scanning. In our work, we introduce photogrammetry, endoscopy and computed tomography (CT) for an integrated 3D digitization solution. Though photogrammetry has been widely used for the purpose of cultural heritage preservation, 3D reconstructions using the other two sensor systems have their own challenges. For an endoscope, a pre-calibration solution has been put forward and the Structure-from-Motion (SfM) process has been optimized to deal with the drift caused by a long imaging trajectory. Regarding the CT 3D reconstruction, we mainly focus on the 3D representation’s completeness and the denoising process. In the section of data integration, we designed different methods according to the characteristics of the objects as well as the 3D models from different sonsors. In case of limited overlap between the pair of point clouds, the Gauss-Helmert model with manually picked control points is applied for the estimation of the transformation matrix. CT point clouds, which hold only the intensity values representing the material attenuation, could be integrated with photogrammetry data via a surface color mapping method using the photogrammetric images or the primitive based corresponding virtual control points. Through our research, the concept of integrating photogrammetry, endoscopy and CT for 3D digitization of Gyroscopes is validated. Furthermore, advantages and disadvantages involved in the complete process are discussed and a solid foundation has been laid for further research.


INTRODUCTION
3D digitization is an important tool for cultural heritage preservation. With the development of different sensor technologies, new possibilities and application scenarios are being put forward. In addition to the widely-used photogrammetry and laser scanning techniques, as recently reviewed by (Hassan and Fritsch, 2019), passive sensing for macro photography such as endoscopy could also be used. Furthermore, active sensing such as CT could also be applied for the internal structure 3D visualization (Morigi et al., 2010, Kastner andHeinzl, 2015). Both of them originate from medical imaging, but are now applied also in other fields such as technical applications. Regarding the 3D digitization objects, beside the traditional relics, many other technical instruments also have the necessity to be digitized. Even if they have a much shorter history, they witnessed the origins of the technology as well as its development. A collection of gyroscopes in the University of Stuttgart represents a set of such fascinating examples, which have been used for the purpose of routing and navigation. Nowadays, gyroscopes are used even wider in smartphones and cars, but the form and size are largely different from the original ones. Nevertheless, this collection has seen the evolution of gyroscopes and therefore important for historical and didactical research purposes. The implementation of complete 3D digitizations could take those objects out of the black-box to be seen by wider range of people, with not only offering a 3d meshed and textured surface but also * Corresponding author the internal structure. Thus, the working principles are easier to be understood by the public.
To fulfill the goals above, the Gyrolog project, funded by the German Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung -BMBF), has introduced photogrammetry, endoscopy and CT for an integrated 3D solution (Fritsch et al., 2018. First of all, a 3D reconstruction process of each sensor should be solved. Secondly, the integration solution is to be developed according to the different characteristics of the 3D model obtained from different sensors.
Photogrammetry is one of the earliest methods that have been used for the application of cultural heritage 3D preservation. Thanks to the mature research foundation and application exploration, photogrammetry is able to generate high accuracy textured 3D surface models via multi-view imagery with sufficient overlaps. Using a special camera system, the workflow of endoscopy 3D reconstructions is of the same principle as that of photogrammetry. Endoscopy has been mostly applied for surgeries, while it has limited application in the industrial field. Due to its special structure (Leiner, 2015), an endoscope has difficulties in taking images and also suffers from the low image quality in in comparison with regular photography (Xia et al., 2018). Therefore, many research has introduced an optical tracker system for the pose tracking , Yamaguchi et al., 2004 or stereo design for depth information (Wu, 2010, Parchami and. However, few references apply endoscopic images for a SfM process. The reason is that it is challenging to guarantee the image overlap, which happens generally only for neighboring epoch images. The problem for such application is similar to visual simultaneous localization and mapping (V-SLAM), which is likely to have an accumulated drift if the imaging trajectory is too long (Cornelis et al., 2004). The drift can happen due to the camera calibration accuracy and also the SfM algorithm feasibility. Besides the problems above, the insufficient overlap between a locally reconstructed area by an endoscope and other global reconstruction results could lead to difficulties in registration. CT is an technique used to reconstruct internal structures of an object that is not visible from outside. There are mainly two problems for CT 3D reconstructions. The first one is the incomplete representation of multiple materials which correspond differently with the X-ray. The second one is the noise on the surface of the 3D models.
We put forward corresponding solutions to the whole integrated 3D digitization framework. For endoscopy, our onging research is working on the calibration as well as the optimization of the SfM process. Once the satisfied 3D endoscopic model is obtained to be registered with the photogrammetric 3D model, we apply a Gauss-Helmert model to estimate the transformation matrix with manually picked control points from both datasets. Regarding the CT 3D reconstruction, an assumption is made that the objects could be successfully reconstructed without influence of noise and containing complete material representation. To combine the advantages of CT and photogrammetry data, we have proven the two-step voxel-to-point cloud processing in (Zhan et al., 2020), when coloring voxel clouds using photogrammetric hull textures. Furthermore, if the CT surface has a incomplete representation, we are working on a primitive-based virtual control points registration, which will be published in the following work.
This work has validated the conception of integrating photogrammetry, endoscopy and CT for a complete gyroscope 3D digitization. The reconstruction process of each sensor has fully studied and demonstrated. In section 2 all reconstruction and registration related fundamental theories are illustrated. Section 3 introduces the detailed experimental process as well as shows results. In the last section 4, we discuss the advantages and the disadvantages of the workflow.

Stability analysis of camera calibration
Camera calibration is an important step in the 3D reconstruction process to obtain accurate camera intrinsic parameter. The most popular calibration method for a single camera is Zhang's method (Zhang, 2000), which is easy to implement in practice. However, due to the correlation nature of the calibration parameters in the model as well as the high requirement of imaging configuration, the result is likely to be unstable when multiple calibrations are implemented (Semeniuta, 2016). Therefore, considering the random effect caused by the single image in the calibration, it is beneficial to use far more images than necessary to analyze the statistical behavior of the calibration as well as to define the well-configured sufficient number of images for a calibration. In the process of gaussian fitting of calibration parameters from many randomly selected image subsets, the error distribution will follow the rule of the central limit theorem. More details could refer to (Zhan et al., 2021b).

Photogrammetry
With the accurately calibrated camera, photogrammetry uses multi-view images with enough overlap and coverage to recover the 3D shape. In the first step, the pose of the images is estimated by the principle of SfM (Structure-from-Motion). Then Dense Image Matching (DIM) follows to generate a dense point cloud of the object surface. In the last step, the meshing and texturing will make a more realistic model by unwrapping and calculating optimal texture from the imagery. The workflow can be summarized in Figure 1.

Endoscopy
Endoscopy, as a special optic structure, shares the same principle with photogrammetry for 3D reconstruction. However, the special characteristics of an endoscope introduce challenges. Generally, it could be divided into flexible endoscopes and rigid endoscopes. In our work, the latter one is investigated. Among all steps, proper data acquisition delivers the biggest difference ompared with normal camera applications. Rather than just for visualization, more factors of endoscopy should be taken into account such as accuracy, image quality, appropriate imaging block, etc. In traditional 3D reconstruction using photogrammetry, the overlap between neighboring images should be over 80 percent. Because of the imaging distance and resolution, the regular camera can fulfill the requirement without too much effort. However for endoscopy, the situation is illustrated in Figure 2. Suppose the imaging distance is d, the coverage of the object space can be calculated via Equation 1.
According to the specifications of the endoscope, the overlap requirement is quite demanding for the 3D reconstruction purpose. Another new degree of freedom, the rotation between the scope cylinder and the camera head is designed to ensure a wider viewing field. However, we keep the unchanged structure between the scope cylinder and the camera head for a more The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China stable intrinsic information for the purpose of metric information delivery.

CT 3D reconstruction
It is an imaging technique taking a large number of radiographic measurements from different angles, in order to generate slices of the object. All reconstructed slices are then integrated into a uniform 3D coordinate system to construct the complete 3D volumetric representation.
To be able to register with the other two sensor's reconstructed 3D data, the volumetric 3D model is transferred into a point cloud by replacing each voxel with the coordinates of its center point. The value stored in each voxel, which represents the accumulated attenuation, is assigned to the corresponding point as intensity. The whole process could be illustrated as Figure 3. However, based on different principles with photogrammetry, the active sensing CT technique has different difficulties in the 3D reconstruction process. On the one hand, the completeness of the 3D representation is needed to be considered when the scanned objects consist of multiple materials. Some parts of the scanned objects, which have low atomic number leading to low attenuation, can be simply not visible during the visualization as a result of the high X-Ray energy spectrum. On the other hand, the X-ray image may be noisy which can come from the nature of the monochromatic X-Ray source causing beam hardening artefacts and scattering as indicated in (Krumm et al., 2008).

Data Integration
2.3.1 CT surface coloring Under the assumption that the CT surface representation is complete and the noise is sufficiently eliminated, we could combine the advantages of CT and photogrammetry data attributions. A photogrammetric 3D model contains surface geometry as well as texture, while the precision of the geometry depends largely from the homogeneity of the surface as well as the redundancy of the images. A common situation is that homogeneous surfaces, boundaries as well as corners are rough or less sharp than the other areas. However, though without texture information, the CT data has advantages over photogrammetry regarding the smoothness and the geometry precision of the surface. Therefore, to combine both datasets, the texture could be mapped to the CT surface point cloud from the corresponding photogrammetry 3D model or even directly from the 2D images. A complete workflow is shown in Figure 4.
In addition, if the assumption of surface completeness representation is not met, the method above will lead to errors by projecting the texture onto wrong places. In this case, due to the rich regular shapes within the gyroscopes, a geometric primitive based virtual control point registration is to be developed, which will be published in (Zhan et al., 2021a).

Gauss-Helmert model for transformation estimation
The registration of pair-wise point clouds is achieved normally by the iterative closest point (ICP) algorithm or some variants. But first of all, a coarse registration is necessary to provide an initial position for the source and the target point cloud. If automatic registration methods are difficult to implement, because of the special characteristics of the 3D data, manual control points together with a least-squares based method will definitely work for our application. Here we are applying a Gauss-Helmert model, which takes the errors in both target and source systems into consideration as indicated in (Ge and Wunderlich, 2016). The mathematical foundation is illustrated from Equation 2 to Equation 7, where X and x are coordinate vectors of control points from the source and target point clouds respectively, X0 contains the translation parameter (X0, Y0, Z0) , µ is the scale and R is the 3 × 3 rotation matrix. This transformation can be reformulated as a linear model through Taylor expansion in Equation 3 ∆x = B · t (3) t = (tx, ty, tz, dα, dβ, dγ, dµ) (4) where t denotes differential changes of translations, rotations and the scale.
When applying the well-known least squares Gauss-Helmert model for the determination of the unknown parameters x, its first order model can be formulated as Equation 6, The second order model or the dispersion model D is as Equation 7 with v the vector of inconsistencies, P the weight matrix of the observations and σ 2 the variance of the unit weight.

EXPERIMENTS AND RESULTS
In this section, the whole workflow of an integration of photogrammetry, endoscopy and CT for gyroscope 3D digitization has been investigated. The current progress contains finished developments as well as some undergoing research. The following section will describe the implementation and show experimental results regarding camera calibration, 3D digitization and point cloud registration.

Devices and experimental objects
The photogrammetric setup based on a turn table and CT setup using X-Ray tube are shown in Figure 5 and Figure 6 respectively. More details can be referred to (Zhan et al., 2020) Figure 5. Photogrammetry Setup. The model type of endoscopic camera head is 20212030 TELE-COM from Karl Storz with 1/2 inch CCD chip and 752 * 582 pixels resolution as indicated in the left side of Figure 7. The left image of Figure 7 also shows a self-designed imaging assistant setup for the endoscope, which is consisting of three directional cross slides for translation and a ball head on the top for rotation movement. On the right side of Figure 7 is the rigid scope with 140mm length, 2.7mm diameter, 30 • direction of view, and the field of view is from 50 • to 105 • .

Camera Calibration
As stated in section 2.1, we accomplished the experiment with four cameras : A Sony α 7R III with Zeiss Loxia 25mm lens, and a Leica Q (Typ 116) camera with Leica Summilux 28mm lens. In addition, two smart phone cameras are also investigated: a Galaxy Note 8 from SAMSUNG and an iPhone 7 Plus from Apple. For each camera, more than 800 images from different perspectives are collected. The number of subsets is ranging from 5 to 50, which are randomly selected from the 1000 images. Taking the Sony α 7R III as an example, the Gaussian fitting result is shown in Figure 8. The other three cameras have the similar behavior as the Sony α 7R III. In addition, Figure 9 indicates the standard deviation of the Gaussian fitting process when the image number of the subset is changing.  It can be concluded from the results that the random effect of single images corresponds well with the Gaussian function.
More results and analysis can be found in (Zhan et al., 2021b). The experiment of the endoscope calibration stability is also under investigation and will be published soon.

3D Reconstruction
According to the methodologies introduced in section 2.2, 3D digitized models from different sensors of partial gyroscopic instruments are displayed in this section.
3.3.1 Photogrammetry With the setup displayed above, objects are placed on the turn table which will rotate in a reasonable interval during the imaging process with a stationary camera. The working principle can be illustrated from Figure  10 with the green dots representing the positions of estimated cameras. The camera will also be adjusted vertically to obtain The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China sufficient perspectives from all directions as shown in Figure  11. The object in Figure 10, called Honeywell Golden Gnat, is a miniature rate gyro and was mostly part of guidance and control systems during the cold-war era. The object in Figure 11 is a pneumatically driven direction gyro produced by Ternstedt Manufacturing Div, GM Corp in USA.  3.3.2 CT As stated in section 2.2.3, an example of a X-Ray image of the Golden Gnat is shown in Figure 12. The corresponding 3D CT volumetric model of the pneumatically driven direction gyro introduced in section 3.3.1 is indicated in Figure 13. More examples of CT scanned 3D model could be referred to (Zhan et al., 2020.

Endoscopy
There are many challenges, including camera calibration, image pre-processing and SfM optimization, for the endoscopic 3D reconstruction. The research is undergoing and will be published soon. Due to the fact that an endoscope is applied for local details reconstruction, Figure 14 gives a global view of the reconstructed object, which is a direction gyro produced by BEZU, France. The red circle shows the reconstructed parts in the following figures. Figure 15 shows the endoscope poses and a sparse point cloud recovers by densely overlapped Figure 13. ISO surface of a denoised filtered back projection (FBP) CT scan. endoscopic images and videos. In Figure 16, an improved reconstruction using manual control points to overcome the drift is displayed.

3D Shape Registration
3.4.1 CT surface coloring When the CT 3D model contains a complete surface regarding different materials, a CT coloring example using the method mentioned in section 2.3.1 is shown in Figure 17.

Gauss-Helmert model based point cloud registration
Except the method indicated in section 3.4.1 and other automatic registration methods, manual picked control points will always work regarding arbitrarily oriented 3D data captured from different sensors, on different time-scales and at different resolutions and have a quality assessment according to the Gauss-Helmert model estimation. Figure 18 and Figure 19 shows the cross section of the integrated CT and photogrammetry point cloud based on such method.

CONCLUSION AND OUTLOOK
This work demonstrates the concept of an integration of photogrammetry, endoscopy and CT for 3D digitization of gyroscopes. This solution is a creative trial in the field of 3D preservation of cultural heritage combining conventional photogrammetry and medical originated CT and endoscopy technology. However, the challenges come from the creativity, i.e. the 3D reconstruction of the CT and the endoscope as well as the data integration from different sensors. Regarding the CT 3D reconstruction, the threshold selection for the complete representation as well as the de-nosing methods are necessary to be improved. For the endoscope, the low image quality and the small field of view while taking images are to be overcome for Figure 19. A cross section view of the integrated model of the pneumatically driven direction gyro.
the application of endoscopic 3D reconstructions. The data integration could always be solved by a Gauss-Helmert model with manual picked control points for a satisfying transformation estimation as well as a quality assessment. Furthermore, more automatic and robust registration algorithms are to be developed according to characteristics of the different sensors.
Our work is one of the first trials of 3D digitization for tech heritage and have achieved good results. All the digitized models have been published in the Goobi and DaRUS platform of the University of Stuttgart, Germany. In the future, the optimization of 3D reconstruction with each sensor will be made and more general integration methods will be developed.