OVERVIEW OF PHOTOGRAMMETRIC MEASUREMENT TECHNIQUES IN MINIMALLY INVASIVE SURGERY USING ENDOSCOPES

This contribution provides an overview of various photogrammetric measurement techniques in minimally invasive surgery and presents a self-developed prototypical trinocular endoscope for reliable surface measurements. Most of the presented techniques focus on applications regarding laparoscopy, which mean endoscopic operations in the abdominal or pelvic cavities. Since endoscopic operations are very demanding to the surgeon, various assistant systems have been developed. Imaging systems may use photogrammetric techniques in order to perform 3D measurements during operation. The intra-operatively acquired 3D data may be used for analysis, model registration, guidance or documentation. Passive and active techniques have been miniaturised, integrated into endoscopes and investigated by several research groups. The main advantages and disadvantages of several active and passive techniques adapted to laparoscopy are described in this contribution. Additionally, a self-developed trinocular endoscope is described and evaluated. * Corresponding author


INTRODUCTION
In medical practice, there are different kinds of surgeries, various ways of treatment and plenty of difficulties.Depending on the injury of the patient, surgeries can be performed conventionally (open) or minimally invasive.Compared to conventional surgery, the great advantage of minimally invasive surgery (MIS) is the reduction of injuries needed for operation procedure (e.g.smaller incisions).Less injuries lead to faster wound healing, less pain and infection risks, and leave fewer scars.In order to cause merely minor injuries during intervention, MIS requires special surgical instruments and devices that are usually designed for specific applications.The most important device is an endoscope that is inserted through small incisions and provides a view onto the treatment area.Depending on the application, endoscopes are designed with a rigid or flexible shaft and can vary in diameter.Endoscopes designed for operations in the abdominal or pelvic cavities are called laparoscopes and have a rigid shaft with a diameter of about 10 mm.In laparoscopy, additional instruments such as miniaturised forceps or scissors can be inserted through further incisions (Figure 1).

Figure 1. Minimally invasive surgery using an endoscope
In order to cause even less injuries, MIS has been enhanced by the so called NOTES technique (Natural Orifice Transluminal Endoscopic Surgery).NOTES avoids external incisions and scars by accessing the treatment area through natural orifices (e.g.mouth) and subsequent internal incisions (Höller, 2010).
Since endoscopy is a very demanding task for surgeons, a lot of experience, good skills and spatial imagination is required.In order to reduce endoscopic challenges and to improve procedures, various assistant systems have been developed in the field of computer-assisted surgery (CAS).For example, the spatial position and orientation of endoscopes or other surgical instruments can be determined by optical (e.g.AXIOS, 2017) or electromagnetic (e.g.Fiagon, 2017) tracking systems.Additionally, 3D models derived from preoperative MRI-or CT-scans or patient specific CAD models can be registered to the actual treatment area during intervention.Therefore, however, corresponding target points or entire surfaces of the 3D model and the patient lying in the operating room are required.
Intra-operative point or surface measurements using endoscopes may allow for registration of 3D models and would offer further applications for measurement purposes.Thereby, the endoscope or measurement system and the analysis procedures must deal with very difficult conditions in object space like soft and continuously deforming tissue, low textured regions, specular highlights or reflections (e.g. by liquids), moving obstacles (e.g.surgical instruments) and image distortions (e.g. by gas or liquids).Figure 2 shows endoscopic images of a potential surface measurement application for laparoscopic partial nephrectomy that contains many of the challenges mentioned.(Simon et al., 2008) Further requirements are real-time processing (e.g.video frame rate or 25 Hz) to allow for continuous surface reconstruction, a good construction stability to maintain constant calibration parameters during operation and a construction size similar to or smaller than conventional endoscopes.These requirements and the complex imaging conditions make surface reconstruction one of the main challenges in CAS.
Various optical surface measurement techniques have already been investigated in research for endoscopic purposes.A classification of measurement techniques is shown in Figure 3.This paper focuses on contact-free optical measurement techniques, which can be divided into active and passive methods.Appropriate applications are discussed in the following sections.

Monocular
Conventional flexible or rigid monocular endoscopes without any modifications can be used for surface reconstruction by single camera techniques.Once calibration parameters are determined, 3D measurements can be assessed by structure from motion (SfM), simultaneous localisation and mapping (SLAM) (Thormählen et al., 2001;Mountney et al., 2006;Grasa et al., 2011) or shape from shading (SfS) (Collins et al., 2012).The first two methods require image sequences and a non-deforming surface.Since medical surfaces typically deforming continuously, deformable SfM (DSfM) approaches have been tested, too.Malti et al. (2012) published a template-based DSfM approach combined with SfS for laparoscopic purposes.In an initial step, the undeformed surface is reconstructed using SfM from multiple camera positions and serves as template surface for subsequent steps.During operation, the deformed surface is continuously approximated by image measurements and the template surface, which is manipulated using elastic constraints.
In areas that do not provide reliable point correspondences, the surface model is refined by SfS.An accuracy assessment using simulated object deformations and ideal measurement conditions yield 3D RMS errors in a range of 0.1 and 0.4 mm depending on the extensibility.The algorithms is also tested using real data.Due to missing reference data, a 3D comparison on real data for accuracy assessment cannot be provided.However, it could be shown that the Deformable-Shape-from-Motion-and-Shading (DSfMS) algorithm from Malti et al. (2012) outperforms the standalone approaches of SfS and DSfM in terms of visual smoothness and meaningfulness (Figure 4).The result from SfS is quite bumpy due to specularity and unmodelled physical phenomena.The result from DSfM provides only a sparse point cloud due to lacking image correspondences in deformed regions.The combination of both, the DSfMS, results in a finer and more meaningful 3D reconstruction.
The main advantage of single camera techniques is that present hardware does not need to be manipulated by further sensors.
The techniques can be applied to the ordinary monocular endoscopic image stream.On the other hand, SfM fails in deforming surfaces and SfS fails in unmodelled phenomena such as bleeding or occlusions.Combinations of different single camera techniques and implementation of elastic surface models might bring an essential advantage regarding deformations and unmodelled phenomena (Malti et al., 2012).However, the correspondence problem in SfM is hardly solvable in poor or repetitive textured regions that frequently occur in laparoscopic images.Furthermore, SfS and elastic models require many assumptions that often cannot be fulfilled in practice.

Binocular
As described in section 1, endoscopic operations may be very challenging to surgeons and especially to beginners since conventional monocular endoscopes only provide a flat view of the three-dimensional operation scene.Stereo endoscopes have been introduced in order to provide better spatial perception to the surgeon by a 3D view.Various studies comparing monocular and binocular endoscopes have shown that the 3D vision improves the depth perception and thus the hand-eyecoordination with surgical instruments.Hence, microsurgical tasks can be performed more easily and safety with 3D view (Taffinder et al., 1999), (Sinha et al., 2013).
Furthermore, stereo endoscopes allow 3D measurements using classical photogrammetric stereo procedures.However, it has not yet been intended in daily practice.Nevertheless, some approaches have been researched.Conen et al. (2015) investigated the 3D measurement capabilities of the stereo laparoscope EinsteinVision® from Aesculap.The endoscope cameras are calibrated and the relative pose and orientation parameters are determined by a bundle adjustment over a three-dimensional test field.Thus, it allows for metrically correct 3D measurements.A rigid skull is observed by the stereo endoscope and a 3D fringe projection system with ~60 µm depth accuracy is used for comparison.The object surface is reconstructed by the rectified endoscopic image pair using the OpenCV semi-global block matching algorithm.Due to the good matching conditions and full-HD resolution, the resulting point cloud is very dense (≈ 480 points per mm²).Both datasets are registered using an iterative-closestpoint algorithm.The root mean square error of the distances between the surfaces is determined to 0.45 mm (Figure 5).Although this investigation is rather simplified, it shows the potential of stereo endoscopes for 3D surface measurements.However, some drawbacks like high image noise and changing orientation parameters during warm-up phase could be detected.2013) state that stereoscopic 3D reconstruction is a mature method for laparoscopic applications since many powerful and well-tested algorithms already exist.However, the most existing algorithms are not developed for surgical purposes and do not handle application specific problems.Thus, the main challenge is to ensure high robustness and reliability, which is sufficient for practical medical purposes.Maier-Hein et al. ( 2013) assume that 3D measurement applications using stereo endoscopes have the greatest chance of being integrated into operating rooms, since stereo endoscopes are already established in some areas.

Trinocular
To improve the reliability especially in regions with repetitive or low texture, a trinocular endoscope has been developed by Conen et al. (2016).A third camera could easily be integrated into current stereo endoscopes and should bring a higher reliability and accuracy for surface reconstruction.To our knowledge, a trinocular endoscope for 3D measurements has not been taken into account by other literature.However, Hassanfiroozi et al., (2016) proposed an endoscope with three lenses that is not specially developed for measurement purposes (cf.section 2.4).Trinocular camera systems with larger scale have been investigated for various applications, e.g.robotics (Ayache et al., 1991), particle tracking velocimetry (Maas, 1992) or 3D modelling (Heinrichs et al., 2007).
The prototype from Conen et al. ( 2016) (Figure 7) consists of three miniaturised cameras, which are arranged in an equilateral triangle to achieve the smallest possible construction size.This arrangement also provides, according to Maas (1992), the optimum configuration for minimal ambiguities in trinocular correspondence analysis.Especially in case of repetitive texture, the trinocular correspondence analysis should significantly reduce the number of ambiguous matches.The interior and relative orientation parameters are determined by a bundle adjustment over a three-dimensional test field.
Figure 7. Miniaturised cameras and trinocular mounting For surface reconstruction, a rectification and a semi-global matching procedure according to Hirschmüller (2005Hirschmüller ( , 2008) ) and Heinrichs et al. (2007) was developed for image triplets.
Corresponding point triplets are calculated very efficiently using epipolar transfer and a zero-mean formula for the similarity measure normalised correlation coefficient (NCC).The computational most expensive part is the semi-global optimisation which is applied once during the process using the aggregated NCC of a corresponding point triplet.A comprehensive description of the algorithm is given in Conen et al. (2016).
The trinocular system is evaluated by comparing measurements of the trinocular system with measurements of a 3D fringe projection system of ~60 µm depth accuracy.Both datasets are registered by an iterative-closest-point procedure.The remaining differences between the datasets represent the measurement precision of the trinocular system.Figure 8 shows the original left image of the sample objects: a rigid sculpture, soft tissue in form of a piece of meat and a rigid, partly low textured cartilage of a pig´s knee joint.The well textured sculpture surface has been captured from a distance of about 60 mm.A best-fit alignment between the reconstructed surface by the trinocular system and the 3D model of the fringe projection system leads to an average distance of about -1 µm and a one-sigma standard deviation over all distances of about 64 µm.The top left picture in Figure 9 shows the deviations in a range of ±0.4 mm by colour-coded points.The grey surface in the background represents the 3D model acquired by the fringe projection system.The bounding box of the point cloud reconstructed using the trinocular system is about 2×1.5×1.5 cm³.
The piece of meat has been captured from a distance of about 140 mm resulting in a reconstructed point cloud with the dimensions of 6.5×5.5×3cm³.The average distance and the standard deviation after best-fit alignment are determined to be -0.05mm and 0.57 mm, respectively.The colour-coded points in Figure 9  The average distances are determined to be 0.21 mm for the binocular and 0.29 mm for the trinocular approach.The standard deviations are determined to be 1.48 mm for the binocular and 1.14 mm for the trinocular approach.The coverage of the model from the 3D fringe projection system is determined to be 84 % for the binocular and 96 % for the trinocular approach.The unmeasured area in the binocular result seem to be due to a lowly textured surface region.The trinocular results were in general more complete and errorfree.The improvements became more visible with low textured objects.Without semi-global optimisation, the results from trinocular matching were significantly more reliable and errorfree.Further details will be published by Conen et al. (2017).

Multi-ocular
Camera systems that are able to record more than three images simultaneously have also been investigated.Such multi-ocular endoscope systems can be realised with the help of micro lenses.Yaron et al. ( 2006) developed a plenoptic endoscope that creates a stereoscopic view by micro lenses in front of a single image sensor (Figure 10).However, the authors are not intended to generate more than two views.
Figure 10.Single-chip stereo endoscope using micro lenses (Yaron et al., 2006) The light rays passing through the pupils (P) are emitted as parallel beams behind the image lens (L).These rays are focused by the micro lenses to the left (black) or right (white) pixel depending on the angle of incidence.Thus, the left and right images can be divided with a single image sensor.Since each lens can capture a perspective image of the 3D scene from a slightly different viewpoint, 3D reconstruction is enabled.However, no further investigations are made regarding 3D reconstruction and measurement accuracy.Hassanfiroozi et al. (2016) proposed an advancement of the hexagonal endoscope.The novel lens system consists of three multi-functional LC lenses and a larger single LC lens behind it.Unlike the hexagonal system, this system also allows for adjusting the focus in 2D mode.Furthermore, the outer diameter could be reduced to 2.5 mm. Figure 12 shows sample images with different settings.A photogrammetric calibration for accurate 3D measurements may be an unresolved issue due to the variable and individual focal lengths.As in the contribution before, the authors did not provide further investigations regarding camera calibration, 3D reconstruction and measurement accuracy.

ACTIVE METHODS
Active measurement techniques are enabled when modifying conventional endoscopes by additional sensors or components.Thereby, requirements such as construction size, heat emission, energy consumption, stability and so on have to be complied.

Structured Light
In structured light approaches, special patterns are projected onto the observed surface and thus enables for reconstructing surfaces with low texture.Albitar et al. (2007) used a geometrycoded pattern which can be determined very robust in object space (Figure 13).However, due to the limited pattern size, the resulting point cloud is quite sparse.Since the system could not be calibrated, an accuracy assessment could not be conducted.The system is suited for very narrow bent tubes and canals.
Colour-coded stripes are projected onto the inner surface of hollow objects and allow for 3D reconstruction by triangulation.
The RMS deviation regarding a reference object was determined to less than 0.1 mm (Figure 15).The main challenge for structured light techniques is the realisation of high-contrast and bright patterns through the small endoscope shaft.Additionally, projected patterns complicate colour reconstruction and affect the conventional endoscopic view for the surgeon.

Time-of-Flight
An endoscope with a time-of-flight (ToF) sensor was developed by Penne et al. (2009) and improved by Groch et al. (2012).The ToF technique does not require a baseline or a projection unit and is thus relatively compact.In addition, no correspondence problem has to be solved.Groch et al. (2012) compared the results of a ToF endoscope and a stereo endoscope with ground truth measurements of a CT scan and obtained better performance for the stereo endoscope (Figure 16, 17).The averaged RMS distances between the measured 3D points and the ground truth data is determined to be about 4.5 mm for the ToF endoscope and about 3 mm for the stereo endoscope.Moreover, the stereo endoscope offers a larger field of view and a higher resolution in object space.Figure 17 shows the mean surface distances over the RMS distances of various sample measurements using the ToF enoscope, the stereo endoscope, and additionally a ToF camera and a full-HD endoscope that reconstructed the surface using SfM.However, active methods perform well even in difficult clinical environments.The challenge for passive methods is to ensure reliable and highly accurate results in complex situations.Additionally, LC lenses may have an open issue to determine reliable camera parameters needed for highly accurate 3D measurements.Integration into operation rooms can only be considered when measuring systems meet the application specific and clinical requirements and bring significant benefits.The latter is already the case for many of the applications shown in this contribution.

Figure 3 .
Figure 3. Classification of measurement techniques

Figure 5 .
Figure 5. Top: calibration of endoscope (left) and endoscope tip (right).Bottom: original endoscope image and disparity map (left), 3D comparison between endoscopic reconstruction and model from fringe projection (right).(Conen et al., 2014) Röhl et al. (2011) published an approach for real-time surface reconstruction from stereo endoscopic images.The interior and relative orientation was determined by a common computer vision approach using a checkerboard pattern.For dense image matching a modified census similarity measure and a hybrid

Figure 6 .
Figure 6.Top: original endoscope image and disparity map (left), reconstructed point cloud (right).Bottom: 3D comparison between endoscopic reconstruction and reference model of silicone liver.(Röhl et al., 2011) Maier-Hein et al. (2013) state that stereoscopic 3D reconstruction is a mature method for laparoscopic applications since many powerful and well-tested algorithms already exist.However, the most existing algorithms are not developed for surgical purposes and do not handle application specific problems.Thus, the main challenge is to ensure high robustness and reliability, which is sufficient for practical medical purposes.Maier-Hein et al. (2013) assume that 3D measurement applications using stereo endoscopes have the greatest chance of being integrated into operating rooms, since stereo endoscopes are already established in some areas.

Figure 8 .
Figure 8. original left images of sculpture, meat and cartilage (from left to right) (top right) show deviations in a range of ±2.0 mm.The pig knee joint has been captured from a distance of about 210 mm and is reconstructed (Figure 9 middle) using the trinocular approach (right) and the binocular OpenCV semiglobal block matching algorithm (left).For the binocular reconstruction, only the left and right images are used.The colour-coded points show the deviations in a range of ±4.0 mm.A histogram of deviations is also shown in Figure 9 (bottom).
Decker et al. (2015) investigated the plenoptic camera Raytrix R12 in terms of clinical measurement applications.Test measurements could performed from optically challenging tissue surfaces with a near millimetre precision and accuracy.However, the authors have not yet integrated this technique into an endoscope.Another technique realising multi-ocular endoscope systems has been proposed byHassanfiroozi et al. (2015) using liquidcrystal (LC) lenses and a single image sensor.LC lenses are thin, tuneable in focus and refractive index by applying voltage and can electronically switch between 2D and 3D mode.In 3D mode, the LC lenses are turned on and focus the light rays on different sensor areas.In 2D mode, the LC lenses are turned off and just act like simple glass that would pass light rays without focusing.Figure11shows the endoscope tip with seven LC lenses in a hexagonal arrangement (left) and the resulting image in 3D mode (right).The outer diameter of the endoscope tip is only 3.5 mm.

Figure 15 .
Figure15.Sample measurement from 32 mm of an artificial cavity.3D comparison to ground truth CAD data with average error of 0.09 mm.(Schmalz et al., 2012)

Figure 16 .
Figure 16.Measurement of lung surface.From left to right:original lung, 3D comparison with results from ToF endoscope, and stereo endoscope.The colour-code shows the 3D deviations in range 0 -8.8 mm for TOF and 0 -11.7 mm for stereo.(Grochet al., 2012)

Figure 17 .
Figure 17.Mean surface distance averaged over the RMS distance of every sample measurement.TOF-ENDO: ToF endoscope, TOF-CAM: ToF camera CamCube 2.0 (PMD Technologies), STEREO-PAL: stereo endoscope 3D Endocam 5535.901(R. Wolf), SFM-HDTV: Aesculap Full HD 3CCD camera system PV440 (B.Braun).(Groch et al., 2012)4.CONCLUSIONDifferent active and passive reconstruction techniques for endoscopy have been researched in the last decade.A big challenge for active techniques is the miniaturisation and integration into endoscopes.Structured light approaches may affect the natural view onto the surface and ToF endoscopes need to be enhanced in terms of precision and resolution.However, active methods perform well even in difficult clinical environments.The challenge for passive methods is to ensure reliable and highly accurate results in complex situations.Additionally, LC lenses may have an open issue to determine reliable camera parameters needed for highly accurate 3D measurements.Integration into operation rooms can only be considered when measuring systems meet the application specific and clinical requirements and bring significant benefits.The latter is already the case for many of the applications shown in this contribution.