PHOTOGRAMMETRY IN 3 D MODELLING OF HUMAN BONE STRUCTURES FROM RADIOGRAPHS

Photogrammetry can have great impact on the success of medical processes for diagnosis, treatment and surgeries. Precise 3D models which can be achieved by photogrammetry improve considerably the results of orthopedic surgeries and processes. Usual 3D imaging techniques, computed tomography (CT) and magnetic resonance imaging (MRI), have some limitations such as being used only in non-weight-bearing positions, costs and high radiation dose(for CT) and limitations of MRI for patients with ferromagnetic implants or objects in their bodies. 3D reconstruction of bony structures from biplanar X-ray images is a reliable and accepted alternative for achieving accurate 3D information with low dose radiation in weight-bearing positions. The information can be obtained from multi-view radiographs by using photogrammetry. The primary step for 3D reconstruction of human bone structure from medical X-ray images is calibration which is done by applying principles of photogrammetry. After the calibration step, 3D reconstruction can be done using efficient methods with different levels of automation. Because of the different nature of X-ray images from optical images, there are distinct challenges in medical applications for calibration step of stereoradiography. In this paper, after demonstrating the general steps and principles of 3D reconstruction from X-ray images, a comparison will be done on calibration methods for 3D reconstruction from radiographs and they are assessed from photogrammetry point of view by considering various metrics such as their camera models, calibration objects, accuracy, availability, patient-friendly and cost. * Corresponding author


INTRODUCTION
Using photogrammetry brings us an accurate powerful noncontact and non-aggressive tool in medicine for extracting spatial data about human body structures from medical images with high speed.Therefore, various researches have been done on diverse applications of photogrammetry knowledge in medicine.Medical photogrammetry is applied for 3D body measurements since about one hundred years ago.Extracting 3D spatial information from X-ray images is one of the oldest and important applications of photogrammetry in medicine.Stereoradiography and roentgen stereometric analysis (RSA) called X-ray photogrammetry by several authors, such as F.A. van den Heuvel (2002).Applying personalised 3D precise models improve considerably the results of orthopedic surgeries and processes.However, usual 3D imaging modalities, computed tomography and magnetic resonance imaging (MRI), have some limitations for several applications.Computed tomography (CT) is an accurate 3D imaging modality but it has high irradiating dose for the patients.CT imaging is expensive and it cannot be applied in standing and weight bearing positions.The global shape of bone structures of patients is changed considerably in lying and standing positions for example in Cobb angle of spine.Due to these limitations, CT is not suitable for 3D evaluation of bone structure in several structure pathologies.Methods based on MRI does not have the problem of inducing high irradiating dose but they are less accurate for bones and they are much more expensive than CT imaging.On the other hand, they are not sufficient for patients with ferromagnetic materials in their bodies because of the potential risks associated with artifacts and movement of metallic objects in MRI.MRI is more used for soft tissue, and it is adopted in lying position not weight bearing position similarly.Therefore, conventional 3D imaging techniques, MRI and CT scan are not sufficient for 3D accurate models of bone structures in some pathologies and many researches have been done on introducing alternative techniques.3D reconstruction methods from biplanar X-ray images has been considered as reliable alternative methods for achieving accurate 3D information with low dose radiation in standing positions and it has been applied by different researchers since early seventies.The information can be obtained from biplanar radiographs by using photogrammetric principles.This is used for diverse applications such as follow-up exams of bone diseases, study of micromotions of orthopedic implants, analysis of locomotions and 3D reconstruction for diagnosis, surgeries, and treatments.The 3D reconstruction of the human bone structures from radiographs is important for different medical applications such as surgery planning, diagnosis, treatment and personalizing finite element models for biomechanical studies and 3D evaluation and follow-up of bone structure diseases such as genu valgum, genu varum, and scoliosis.3D reconstruction from X-ray images is widely applied for diverse bone structures such as rib cage, lower limb (femur, tibia, and pelvis), upper limb (such as radius and humerus), vertebrae and spine.
The essential step for 3D reconstruction of human bone structure from medical X-ray images is calibration in which photogrammetry has an important role.In this paper, after demonstrating the steps of 3D reconstruction from X-ray images, several existing methods applied for the calibration steps of 3D reconstruction from radiographs are assessed from photogrammetry point of view by considering various metrics such as their camera models and calibration objects.

STEPS OF 3D MODELLING FROM RADIOGRAPHS
General steps of 3D reconstruction from radiographic images can be considered as followed:

Acquisition of at least two radiographs
After the network design which is done by considering requirements of the application, at least two X-ray images will be captured by one or several commercial standard radiographic devices.Also, calibrated biplanar radiographic images can be obtained by specified solutions such as EOS imaging system if needed equipment is available.Various types of standard X-ray imaging systems widely applied for 3D reconstruction of bony structures are as followed: (a) Film-screen radiography: In filmscreen radiography, at first the film must be scanned.In these images, the quantum noise is negligible when it is compared to the scanning noise, (b) Digital radiography: digital radiographs can be classified into different types considering their detectors and conversion processes as it is shown in figure 1, and (c) Fluoroscopy: In fluoroscopy there is a continuous beam of radiation.The relatively low exposure in fluoroscopy produces higher noise.Various types of X-ray images have different characteristics and levels of noise which are distinct from optical images.Therefore, the specified restoration methods should be applied by considering the noise behaviour in X-ray images.
The captured images can contain markers or not, with regards to the application and the required accuracy.Markers are artificial objects.These markers, which are usually small balls, have different types such as those implanted into the bone by surgery which are rigidly fixed to the bones or those simply affixed on the skin or attached to frames.Using internal rigidly fixed markers brings us accuracy but, at the same time, it has some serious disadvantages such as inconveniency for the patients and costs which cannot be neglected.

Image enhancement
Restoration is important for achieving acceptable results according to the nature of X-ray images.X-ray images have high quantum noise because of the limited photons involved.Gaussian modelling is not accurate for photon-limited images because of the nature of signal-dependent quantum noise.X-ray images are generally modelled by Poisson distribution.Denoising of X-ray images should be considered as an important pre-processing step in 3D reconstruction, more specifically to increase the accuracy of the segmentation process.Hence, an appropriate denoising method should be selected in order to improve the results of 3D reconstruction.For this purpose, different researches have been done on assessment of performance of denoising methods on X-ray image such as (Hosseinian et al, 2016) and it is concluded that PURELET, BM3D and regularization based methods can achieve reliable results.In addition, it should be considered that the quality of X-ray images is affected by the superimposition of bony structures.

Calibration
Calibration is a very important step and it should be done to generate 3D information from 2D coordinates and it includes correction for image distortion, calculation of the focus position and calculation of the relative position and orientation of the images.For this step, principles of photogrammetry have been taken into consideration.Nowadays, calibrated images are available by means of some specified solutions such as EOS which can acquire calibrated stereo radiographic images simultaneously.But because of limitations in availability and cost constraints, calibration methods which can be applied by standard radiographic systems in usual clinics are considered more especially for developing countries.In order to propose calibration methods which can be used by standard radiographic systems, calibration scope should be evaluated precisely.For this purpose, in section 3 various outperforming calibration methods for stereoradiography are explored.

3D reconstruction
After the calibration step, 3D reconstruction can be done using appropriate methods with different levels of automation.3D reconstruction methods can be assessed based on different measures such as level of automation, primitives and models which are used, and their applications.3D reconstruction methods can have different levels of automation for data extraction and optimization and redefinition of generic models and obtaining specified 3D model.Based on the level of automation, these methods can be manual, semi-automatic and automatic methods.Different types of deformable models can be applied by reconstruction methods, such as geometric deformable models and statistical models.Diverse 3D reconstruction methods from X-ray images have been proposed using different primitives such as points and contours and surfaces.Also, various deformable models such as geometric deformable models and statistical models are applied for achieving more accurate 3D models in shorter time.In (Hosseinian et al, 2015) these methods are evaluated and their advantages and disadvantages were described.By considering the used primitives and deformable models, 3D reconstruction methods from X-ray images can be classified as followed:

Point-based methods:
These methods apply points which can be only stereo corresponding points (SCPs which are visible on various radiographs) or both stereo and non-stereo corresponding points (NSCPs which are visible only on one radiograph) for 3D reconstruction of bony structures.Applying both of these points makes it possible to Use more information from the radiographs and to obtain a more refined and detailed 3D geometry by deforming generic models using an epipolar geometry.Point-based methods are applied by different authors such as Stockes et al, 1981;Pearcy, 1985;Dansereau et al, 1988;Andre et al, 1994 for 3D reconstruction.These methods have low reproducibility because of their dependency on the skill of the expert operator for identifying exact points.These methods are unusable for featureless bones such as femur and they need complex manual identification.Besides, they are time consuming and so they cannot be applied for clinical application.However, they are widely applied because of their simple process usually as an initial solution.

Contour based methods:
These methods reduce user intervention and their needed time is less than point-based methods but still they are time consuming for clinical applications.The method's principle is elastic 3D model deformation with regards to 2D contours available on multi view radiographs.This method can be adapted to any structure as long as the requirements are fulfilled and it has been applied successfully for 3D reconstruction by authors such as Laporte et al (2003) for distal femur and Mitton et al (2006) for pelvis.The Requirements of this method are as followed: (a) accurate calibration of the x-rays environment (b) a location and shape initial solution of the object to be reconstructed.These methods have acceptable accuracy.

Statistical shape model based methods:
These methods apply statistical deformable models.Statistical shape model (SSM) is a model containing information of the mean shape and its variations.These methods require a large learning database of shapes including normal and pathologic shapes of subjects.Principal Component Analysis (PCA) is used on the learning database in order to find model shape variations.These methods are widely used for 3D reconstruction of different objects such as Vertebrae and spine (Benameur et al, 2003(Benameur et al, , 2005)), Femur (Baka et al, 2011) and Pelvis (Yao et al, 2003, Sadowsky et al, 2006).Statistical shape model based methods have higher level of automation.They apply more information about bony structures and their pathologies.These methods provide accurate 3D reconstruction especially for bony structures such as vertebrae and they need less time.However, the total needed time were not reported precisely.

Hybrid methods:
These methods can have different properties considering their combination.They can improve the results of 3D reconstruction if they are consistent to application.Kadoury's method (2009Kadoury's method ( , 2015) ) is a sample of these methods for 3D reconstruction of spine.They used a large database.They achieved acceptable accuracy.However, only computation time was reported.

CALIBRATION METHODS
Accurate camera calibration and orientation procedures are necessary prerequisites for extraction of precise and reliable 3D metric information from images.In photogrammetry a camera is considered calibrated if the principal distance, principal point offset and distortion parameters are known.In many applications, especially in computer vision (CV), only the focal length is recovered while for precise photogrammetric measurements all the calibration parameters are generally employed (Remondino et al, 2006).Similarly, essential prerequisite of 3D reconstruction from X-ray images for achieving accurate 3D coordinates from 2D coordinates of Xray images is calibration.Here, the purpose of camera calibration is calculation of Interior parameters of radiographic systems such as principal distance and principal point offsets (and image distortion for higher accuracy) and exterior parameters consisting of position and orientation of the radiographs.The principal distance in radiography is the distance between X-ray source and the detector as it is shown in the figure 2. The geometry of a radiographic system consisting of the X-ray source and the detectors with regard to the patient (object) is shown in figure 2.
Figure 2. Geometry of biplane radiography using X-rays Calibration methods for 3D reconstruction from X-ray images can be classified by different metrics.Calibration methods can be divided in to two classes by considering their sensor models.
They can be based on projective or perspective models.In the following, outperforming calibration approaches for stereoradiography are evaluated by consideration of their sensor models.

Projective model based methods
Projective models support projective reconstruction and they need at least 6 control points.Direct Linear Transform (DLT) method is an implicit method proposed by Abdelaziz and Karara in 1971 based on projective model.This linear method has been used widely for calibration in 3D reconstruction from radiographs.However, its limitations should be considered such as high correlation between parameters.Dansereau and Stokes in 1988 proposed a calibration system based on Direct Linear Transform (DLT) for 3D reconstruction of rib cage from X-ray images.They presented a large cage of calibration device consisting of 55 steel balls on two sheets (figure 3) which accommodating and surrounding the patient's body during the image acquisition in order to avoid extrapolation errors because of using DLT algorithm.Correspondence between views must be established based on the epipolar geometry inferred from the stationary calibration object and error is introduced by involuntary patient motion with respect to the calibration object during the delay of approximately 25 seconds between X-ray exposures (Cheriet et al, 2007).This device was installed at the Sainte-Justine Hospital in 1992 and it has been used for 3D reconstruction of diverse bones from X-ray images by different authors such as Aubin et al (1997), Mitton et al (2000), Mitulescu et al (2001Mitulescu et al ( , 2002) ) and Benameur et al (2003Benameur et al ( , 2005) ) for vertebrae and spine.This approach had appropriate performance for 3D reconstruction of manually identified anatomical landmarks of spine.The superimposition of the markers and bony structures should be considered in this approach.

Perspective model based methods
Another camera model is perspective model which is based on collinearity equations and it needs more than five control points.Considering the nonlinear equations applied, estimations of initial values of parameters are needed for bundle adjustment.
Dumas et al in 2003 developed a simplified explicit calibration system for 3D reconstruction based on perspective model.Their method was limited to only a specific case of biplanar radiography.Figure 4 shows the developed calibration device.This device was installed at the Saint-Vincent de Paul Hospital (Paris, France) using the ELC (Explicit Linear Calibration) protocol.It was attached to the stereoradiographic system (directly integrated on the turntable) in order to facilitate clinical applications (Dumas et al, 2003).They considered perspective transformation and they applied collinearity equations for the calibration.In order to exploit the collinearity equations, they presented a specific calibration device integrated on the stereoradiographic device and attached to the turntable (see figure 4).It is composed of lines of control points of which the coordinates are known (Dumas et al, 2003).The control points are distributed along the three axes using two vertical lines composed of 32 markers, three horizontal lines composed of 8 markers and 11 markers embedded in radiotransparent bars.A subset of these control points was visible on the radiographs depending on its dimensions and position (Dumas et al, 2003).This calibration device is more comfortable for patients and it has less superimposition (markers and bones) in comparison to Dansereau's system.Cheriet et al proposed a self-calibration approach based on the perspective model for 3D reconstruction of spine and rib cage from X-ray images in 2007.In this method the calibration is done by minimisation of the retroprojection error of landmarks.This method involves explicit use of the description of the calibration matrices with the geometrical parameters of the radiographic system and it is based on the iterative nonlinear optimization process.It was demonstrated that this system allows the patient to adopt a normal attitude without any constraint, compensating for her displacement between exposures (Cheriet et al, 2007).Their system was installed at the Sainte-Justine Hospital.They used collinearity equations and Levenburg Marquardt optimization algorithm.They applied two calibration objects, a rigid calibration object consists of 6 coplanar radio-opaque pellets of known 3D coordinates which define a global reference plane and a non-rigid vest containing 16 radio-opaque pellets worn by the patients during imaging.This system is shown in figure 5.The validation study showed similar accuracy to those obtained with the standard DLT technique.Kadoury et al used this calibration approach in 2009 for 3D reconstruction of spine from X-ray images.Currently, a different method is applying specified solutions such as EOS to provide simultaneous calibrated radiographs.EOS imaging system was developed in 2005 by Dubousset et al.This system captures two simultaneous calibrated X-ray images with lower radiation dose than standard radiographies and known orientation.This system has linear array detectors (shown in the figure 6).This system relies on the high sensitivity of a detector (multi-wire chamber) invented by Georges Charpak which gave him the 1992 Nobel Prize (EOS imaging website).EOS is commercialized by the French company EOS imaging for orthopaedic applications.Nowadays in many developed countries EOS imaging system is applied in several hospitals and many researchers have used this system for 3D reconstruction of diverse bone structures such as Mitton et al (2006) for Pelvis, Baudoin et al ( 2008), Chaibi et al (2011), andQuijano et al (2013) for Lower Limb, Lebailly et al (2012) for Upper Limb, Aubert et al (2014) for Rib cage, Humbert et al ( 2009) and Rehm et al (2017) for Spine.However, this system is still unavailable in many developing countries and medical centres because of some restrictions such as costs.Beside the advantages of these system such as capturing two calibrated simultaneous low dose radiographs with no patient movement between acquisitions of radiographs, there are some limitations of using EOS such as high cost, availability for clinics, high distortion in one direction, and movement of patient during imaging time (25sec).
There are other calibration approaches such as the algorithm proposed by Moura based on (Zhang, 2000)'s method using checkerboard phantom for calibration.Although using planar phantoms is simple and widely applied for calibration, it leads to correlation between parameters.Table 1 shows mentioned calibration methods in this paper for 3D Reconstruction from X-ray images in different applications.In this table, the calibration algorithms and their devices, the accuracy, the reconstruction methods, validation references and imaging systems in different studies for 3D reconstruction from X-ray images are presented for various bony structures.Table1: Evaluation of Calibration methods for 3D Reconstruction from X-ray images

DISCUSSION AND CONCLUSION
In this paper, after demonstrating the steps of 3D modelling of bony structures from radiographs, more applied calibration approaches for 3D reconstruction from biplanar radiographs are explored and their characteristics and their advantages and disadvantages have been expressed.It should be mentioned that there are various calibration devices and methods but in this paper only methods which achieve acceptable accuracy and widely used are evaluated.Also we do not consider calibration methods for other modalities such as fluoroscopy in this paper because of the extent of issue.As it is expressed and it can be seen from the EOS has valuable benefits of capturing two calibrated simultaneous radiographs with lower radiation dose with no patient movement between acquisitions of radiographs, but there are some limitations of using EOS such as high cost, high distortion in one direction, unavailability for usual clinics, occupying large space, and Movement of patient during imaging time.
methods: These methods can provide a fast and robust initial solution for other 3D reconstruction methods.These methods specify a Simplified Parametric Model to represent the interest objects using geometric primitives.The geometric primitives are described by geometric parameters called descriptors.In these methods, Instead of using the full set of points as proposed by SSM modelling, Statistics are performed on anatomical descriptive parameters extracted from the surface of interest.They improve the robustness and convergence of the procedures and they bring us enhanced results with acceptable reproducibility with reducing user intervention especially for lower limb bony structures.These methods can achieve acceptable accuracy in shorter time in comparison with other methods.However, for using these methods the requirements of the application must be considered.Different parametric algorithms have been proposed for 3D reconstruction of various bony structures such asBaudoin et al (2008) for 3D reconstruction of the proximal femur,Quijano et al (2013) andChaibi et al (2011) for lower limb, andPomero et al (2004) scoliotic vertebrae andLebailly et al (2012) for upper limb.

Figure 3 .
Figure 3.The calibration device developed by Dansereau et al in 1988.

Figure 4 .
Figure 4.The calibration device developed by Dumas et al in 2003.

Figure 5 .
Figure 5.The calibration device developed by Cheriet et al in 2007.
table 1, Dansereau's calibration device can produce acceptable and accurate results despite of limitations of DLT algorithm.Also, this device is relatively expensive and unavailable for many clinics.Movement of patient between imaging and markers overlaying with anatomical body structures are important challenges of Dansereau's calibration device which can cause conflict and it should be considered.This devise also occupies large space which may cause some limitations in the environment.The other calibration device proposed by Dumas et al uses perspective model and it applies simplified collinearity equations reduces some of the problems of Dansereau's device such as correlation between parameters and uncomfortability and markers superimposition with bones to some extent but still it needs a large area and it has high cost and unavailable for many clinics.The self-calibration approach proposed by Cheriet et al in 2007 can bring us accurate results with higher comfortability for patients and less markers superimposition with bones.