ICE DETECTION ON AIRPLANE WINGS USING A PHOTOGRAMMETRIC POINT CLOUD: A SIMULATION

This study describes some tests carried out, within the European project (reference call: MANUNET III 2018, project code: MNET18/ICT-3438) called SEI (Spectral Evidence of ice), for the geometrical ice detection on airplane wings. The purpose of these analysis is to estimate thickness and shape of the ice that an RGB sensor is able to detect on large aircrafts as Boeing 737-800. However, field testing are not available yet, therefore, in order to simulate the final configuration, a steel panel has been used to reproduce the aircraft surface. The adopted methodology consists in defining a reference surface and modelling its 3D shape with and without ice through photogrammetric acquisitions collected by a DJI Mavic Air drone hosting a RGB camera and processed by Agisoft Metashape software. The comparison among models with and without the ice has been presented and results show that it is possible to identify the ice, even though some noise still remains due to the geometric reconstruction itself. Finally, using 3dReshaper and Matlab software, the authors develop various analysis defining the operative limits, the processing time, the correct setting up of Metashape for a more accurate ice detection, the optimization of the methodology in terms of processing time, precision and completeness. The procedure can certainly be more reliable considering the usage of the hyperspectral sensor technique as future implementation.


INTRODUCTION
In the aircraft industry, ice accumulation on lifting or control surfaces of an airplane represents one of the central aspects of the flight performance reduction. Since ice buildup mainly on aircraft wing and fuselage, aerodynamic efficiency is altered thus increasing risk for people on board occurs. Airflow distortion over the wing, degradation of the control authority, drag addiction, maximum lift decreasing are primary fallouts due to the ice accretion and they have brought about many disasters (Addy, 2000;Bragg, 1981). The SEI (Spectral Evidence of Ice) (Falcone et al., 2019) project proposes to provide a solution that includes experiments about spectral sensor fusion techniques of data acquired by Autonomous Aerial Vehicles (AAVs) during autonomous aircraft pre-flight inspection without the intervention of expert operators. Currently, ice recognition is not an easy problem because there are various complex aspects to consider:  Physical characteristics of ice;  Safety of aircrafts and travelers;  Limits of airport environment;  Variable atmospheric conditions in different survey.
Generally, two main types of ice have been identified: rime ice and clear ice. Regarding to appearance condition, the former is more visible to the human eye because of its milky white look, while the latter has a transparent behaviour so in order to identify it sometimes, in addition to a near visual examination, operators need to touch the section concerned. Consequently, in light of these problems in characteristics identifying, taking into account * Corresponding author the risk that both arise at the same time, it is not always clear in which areas ice should be removed. Targeted operations are not comfortably achievable, thus a high quantity of de-icing liquid is usually involved in the cleaning process incurring in high costs and considerable damage to the natural environment. The use of an AAV based devices for the aircraft inspection would improve safety both for operators and machines regarding reliability of results at the end of the process. Furthermore, the identification of the areas covered by ice would allow to reduce the high amount of de-icing liquid usually involved in the process, so it would be possible to minimize the impact on the environment. Another benefit of the proposed method is a significant reduction in defrosting time, depending on aircraft size, extent of frost coverage and ambient conditions. For a standard de-icing procedure, pre and post inspections made by AAVs is expected to result up to 30-35% time savings (Falcone et al., 2019). Moreover, reducing the amount of the de-icing liquid there would be a reduction of high costs, so it could be developed a fast time consuming and low cost procedure for the ice identification. The SEI project establishes the objective of building a platform that could be able to locate the aircraft, identify the ice, report to the operator the areas to clean and finally verify that they have actually been cleaned. Therefore, the implementation of an AAV platform, the selection of sensors put on board to identify the ice and the communication systems with ground platforms and with operators are topical subjects that must be treated inside the project.
Regarding the now available sensors, in recent decades, several ice detection technologies have been developed. At the beginning, most used devices were those able to measuring signals as radio frequency (Abaunza, Donnangelo, 1998), electro-mechanical (DeAnna, 1999, electro-optical (Gregoris, 1996), electro-magnetic (Bassey, Simpson, 2007), vibration (Daniels, 1988;Barre et al., 2006) or ultrasonic (Hansman, Kirby, 1986;Peterson et al., 1995;Gao, Rose, 2009). Even though these technologies are capable to detect the presence and the thickness of the ice, in some cases they have complications about physical implementations and, mostly, all of them cannot reach a precise estimation of the ice shape and geometry on the aircraft (except for multi-sensors systems architecture). Nowadays, to partially overcome this constraint, remote sensing (Gagnon et al., 2012) and photogrammetry (Chiabrando et al., 2012) come to the aid. Thanks to radiometric and geo-referencing information (Cina, Piras, 2015), both active (i.e. LiDAR, Gregoris et al., 2004) and passive (i.e. RGB, multispectral and hyperspectral images) sensors are able to get three-dimensional information about the environment in form of point clouds in a quick and easy way (Castagnetti et al., 2013) and high levels of details can be reached (Balletti et al, 2015, Barsanti et al., 2016. Obviously, being mounted on AAVs, these last sensors don't need to be in direct contact with the aircraft, so previous issues about physical implementations don't occur.
Inside the SEI project, in order to understand which could be the best technology to acquire 3D geometry of airplane shape and to detect the ice, and consequently which sensor will be mounted on the AAV, both radiometric classification and photogrammetric analysis are performed, respectively with hyperspectral camera and RGB camera. However, in this paper, the study is focused in a certain sense on the selection of proper sensor to achieve the goal and, more precisely, on performing and showing a simulation that verifies if a photogrammetric geometrical ice identification with the RGB camera sensor is possible and reliable. This kind of sensor and its related technology are selected because, from project requirements, the lower cost architecture must be analyzed as first.
Using different strategies, including stereo matching and the Structure from Motion (SfM) approaches, it is possible to reconstruct three-dimensional point cloud from RGB 2D images. Moreover, the increasing automation of image matching algorithms and the great potential, as well as the spread of these sensors in the mass-market, have increased the use of passive sensors in different applications.

METHODOLOGY
The photogrammetry purpose is to obtain geometric properties of objects and physical model information through the contents of the 2D images. This aim can be achieved by reconstructing the camera poses during the acquisition of the data, thus obtaining the relationship between the image and the object being analyzed. Today, the use of digital images, allowed the application of automatic algorithms deriving from computer vision (CV) techniques, to solve the images orientations. The major part of the available software for the extraction of 3D information from images (like Agisoft Metashape and Pix4D) exploit SfM techniques based on CV algorithm. SfM is the simultaneous recovery of the 3D structure of the object and the pose of the cameras from image correspondence. The image orientation is based on image matching techniques to extract a set of points and find correspondences between them. After the recovery of the matches between images, a bundle block adjustment technique can finally be performed to estimate the image coordinates and to recover the camera parameters. If no initial External Orientation Parameters (EOPs) were provided (for example from the Global Navigation Satellite Systems (GNSS) data collected during the flight), the introduction of the coordinates of Ground Control Points (GCPs) is, here, required to precisely georeference the model in the selected reference coordinate system. It is necessary to specify that within the SEI project, the AAV will be equipped with a suitable GNSS sensor and proper inertial platform that will allow to know the position and attitude of the acquired images with high precision (cm, 0.1 deg). In this way the introduction of GCPs will not be required and it will be possible to save time on the photogrammetric triangulation phase by proceeding with a direct photogrammetry approach. Finally, in order to produce detailed and accurate 3D model of the environment, on which perform measurement and analysis, at least a dense structure is required. Densifying a point cloud means to try to extract information involving all the pixels of the input images and not only the selected ones as in the sparse method. These methods are based on a similarity approach or photo consistency measures, which compares pixel values between images. Going more deeply in the adopted methodology, it consists in defining a reference surface and modelling its 3D shape with and without ice. In order to evaluate the performances of the photogrammetric techniques in the geometrical identification of the ice, a testing methodology was adopted.
The steps related to the evaluation are: 1. Measuring the GCPs used by the total station in order to create the model; 2. Acquiring a reference 3D surface without the ice (without ice surface, WIS); 3. Performing a photogrammetric acquisition of the surface with different structures of ice thanks to the AAV; 4. Generating the 3D surface model with ice using the GCPs (GCP iced surface, GIS) ; 5. Extracting the External Orientation Parameters (EOPs); 6. Affecting the EOPs of a Gauss white noise: around 1-2 cm (for position) and 0.1deg (for orientation) in order to simulate a real time GNSS/IMU solution; 7. Generating a new 3D surface model using the affected EOPs (affected EOPs iced surface, AIS); 8. Comparing the models with (AIS) and without the ice (WIS) in order to identify the ice.
Getting point clouds needs fully photogrammetric acquisition of the upper surface of the aircraft but, since the project AAV is not available at this moment, the tests have been carried out simulating the final configuration. If this configuration turns out not enough, the integration of a different kind of sensor will be evaluated.
In order to evaluate the best configuration of data acquisition useful to obtain the most performing ice detection, all the steps of the methodology are classified in three different cases:  Only nadir images, 62 images (NI);  Only oblique images, 65 images (OI);  Both nadir and oblique images, 127 images (NOI).
Finally, during the analysis of the obtained results, the methodology can be assessed through two different approaches:  Internal evaluation: estimating noise, density and time processing of the point clouds;  External evaluation: estimating the difference of the point cloud with ice (AIS) respect to the reference one (WIS) in terms of thickness.
The whole procedure has been validated considering different 3D models generated by Agisoft Metashape (version 1.6.0).

TEST
Test part is simply split in three different sections, each one related to fundamental aspects of the methodology:  The acquisition, for the planning of flight;  The processing, for a proper data perturbation for performing the following photogrammetric procedure;  The analysis, for setting suitable requirements to understand which configuration generate the most accurate, complete and less noisy point cloud.

Data acquisition
Before starting the acquisition procedure, it is fundamental to define a priori the achievable product, the resolution and accuracy in terms of Ground Sample Distance (GSD) and block orientation precision. These parameters have to be considered in the flight planning.
In remote sensing, GSD is the distance between two consecutive pixel centers measured on the ground and it's an important parameter because it affects the accuracy of a survey. That's because it is related to the altitude of flight, so the higher is the flight altitude, the bigger will be the GSD number. Consequently, as bigger the number of the GSD, the lower the spatial resolution of the image and the less visible details, but the shorter will be the flight time because, with higher flight altitude, it needs less images to cover same area. For these reasons during an inspection the surveyed object cannot be more accurate than the GSD value, it means that it could be impossible to measure precisely objects within one centimeter of actual dimension if GSD will be two centimeters.
For what concern the AAVs, the EuroSDR (EuroSDR), which is a not-for-profit organization linking National Mapping and Cadastral Agencies with Research Institutes and Universities in Europe for the purpose of applied research in spatial data provision, management and delivery, defined the empirical formulas for project planning: where σXY = planimetric accuracy σZ = height accuracy σxy = measurement accuracy of image points M = scale factor Considering also the formula to estimate the GSD value: where Z = flight height c = focal length dp = pixel size Sh, Sw = sensor height/width Ih, Iw = sensor height/width Taking into account data from Table 1, GSD can be easily computed and it's equal to 1.7 mm at 5 meters. Moreover, from empirical formulas, usually = 3 • 10 −4 • and = . Therefore, when obtained results will be discussed, knowing the accuracy from the point cloud analysis, it will be possible to verify also that minimal requirements for project planning are fulfilled. Coming back to the acquisition part, as defined in the abstract, the tests have not been conducted on a real aircraft due to occurred airport policy problems. So, to simulate the final configuration, or at least to move close to that, a common AAV platform, in particular a DJI Mavic Air able to house an RGB camera, and a steel panel, to reproduce the aircraft surface, have been used. Over the panel, different types of ice have been modelled and they are quite visible in the Figure 1.
In common AAV photogrammetry, in order to reach a high level of detail, it is suggested to perform both linear and cross stripes during the nadiral image acquisition, such that a better overlap arises. In last years, the scientific community is also evaluation the use of oblique images (which means camera axis with an angle of 45° respect to the object) that can improve the final results of the photogrammetric triangulation by introducing a different point of view. In the case of AAV photogrammetry the advice in the literature is to use between 60% and 90% of forward overlap (usually 80%) and between 20% and 80% of side overlap (usually 60%). In the study case, photogrammetric acquisitions have been made around the panel acquiring both nadir and oblique images, while overlapping works around 80% for forward case and a very good The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) 85% for side case for nadir acquisitions but it decreases to 60% and 40%, respectively forward and side case, for oblique acquisitions.
Furthermore, as Figure 1 shows, in order to estimate the EOPs (since the on-board positioning system is not enough accurate), 12 artificial markers were placed around the panel and measured by a total station in order to have their coordinates with an accuracy of few mm in a local reference system.

Data processing
In photogrammetry, the developed algorithms for image calibration are based on two analytic models: that of the collinearity equations and that of the direct linear transformation (DLT). The second is quite simple and fast, but generally cannot handle lens distortions; the first is more rigorous and accurate. According to this approach, the unknown are the 6 external orientation parameters (EOPs, position of the acquisition center, x0, y0, z0, and camera orientation, ω, φ, k) and the 3 geometric parameters (c, ξ0, η0) and the radial (k1, k2, k3, k4), the tangential (p1, p2, p3) and the sensor distortion parameters (c1, c2). The bundle block adjustment technique provides the simultaneous estimation of all parameters and their accuracy. The solution of the system using the least squares algorithm is carried out in an iterative way starting from the approximate parameters necessary for the linearization of the collinearity equations. Coming back to the study case, it's important to remember that, in the final AAV platform of the project, the RGB camera will be integrated with several modules to ensure the most accurate positioning creating redundancy of pose data. GNSS receiver with Real-time Kinematic (RTK) positioning solution, an Inertial Measurement Unit (IMU) and Ultra-Wide Band (UWB) module have been considered for this purpose.
In real configuration, thus, markers can be avoided; nonetheless during this simulation they are still present due to on-board positioning system not really reliable. However it can take advantage of this situation, indeed, firstly, a model has been achieved using GCPs coordinates. In addition, Table 2 shows the errors in the identification of the GCPs positions inside the images. Then, accomplished the GIS model, it could be possible to extract EOPs and use them to generate a new model in which the idea is to reproduce an ordinary GNSS solution with its limitations in order to simulate a real positioning system. For this reason, starting from the EOPs, a Gaussian white noise is added to them through a dedicated Matlab script. It means that a random value, simulating a precision of 1-2 cm for the projection centre position and 0.1deg for the attitude angles, affects the EOPs and shift slightly their values. The dedicated Matlab code is the function:

Count
where awgn = function that allows to add white Gaussian noise to input vector Out = output vector (in this case the affected EOPs) In = input vector (in this case the original EOPs) snr = signal-to-noise ratio used to impose a specific value of noise and a specific multiplier to random values that affect EOPs coordinates In order to run this script the user has to add the "Communication Toolbox" on Matlab. Afterward, these noisy EOPs have been used to generate a new model, without GPCs, in which a GNSS/IMU solution is reproduced. The configuration selected as the most proper will act as the reference model for the final evaluation of the points clouds related to different approaches. Therefore, for each approach it is possible to perform the image alignment and create sparse and dense point cloud and evaluate the best configuration in terms of quality-processing times through setting, in Metashape software, one of the following values to processing parameters: In total, the following 27 dense point clouds related to noisy images coordinates were processed: 9 dense clouds obtained from only nadiral images; 9 dense clouds obtained from only oblique images; 9 point clouds obtained by inserting both nadiral and oblique images. Obviously, to implement a precise and rigorous analysis, all configurations must keep at least one variable constant during all the photogrammetric process. Thanks to a Python script, in fact, the geometry of region of interest for all procedures, called bounding box (more than a region as surface, it's a volume), can be maintained fixed. For completeness, the specifications of the computer used for the data processing part are reported here:

Data analysis
As a means to achieve best configuration among several ones, the data analysis is structured in observing three significant aspects collected by the photogrammetric procedure. A first screening is performed examining the number of points in both sparse and dense point cloud and, consequently, inspecting the impact of point clouds density on the study. Looking at the Table 3, two features can be principally noticed: as first, oblique approach has more difficulties in the alignment process. Considering that the sample is resting on a flat a particularly unstructured surface, although there are markers, probably insufficient overlapping can increase complexity in extraction and matching common features. Secondly but not less crucial, the substantial divergence among combined case and the other two occurs in alignment procedure because the former case has twice images with respect to latter cases, so it has more possibilities to find common and relevant features. Indeed, although numbers of points in dense clouds cases are more similar, thanks to alignment part, combined approach presents more high-quality features, as it can be shown in Figure 2. Examples of dense point cloud of both three cases referred to highest alignment and ultra-high dense cloud parameters are represented and it is easily visible that combined dense cloud is the most accurate and clean while that referred to oblique acquisition presents intense level of noise.
Finally, it should be noted that those very high values relating to the highest alignment case are probably due to the possibility that the oblique images, being inclined, have to reach a wider field of view and consequently to "see" more details respect to the nadiral case. This behavior is more evident in this case also because, due to how the algorithm is made, it is recommended for sharp surface that are presented in the environment around the sample. Due to the focus on density value, just combined case data will be considered in following observations and moving to them it will be evident that tradeoff situations in the selection of best configuration will be more frequent.
The second fundamental aspect that has been analyzed is the processing time. In this case, before reading obtained results, it needs to consider that reported times are referred as:  Alignment: matching time + alignment time;  Dense Cloud: depth maps generation + dense cloud generation.
As it's shown in the Table 4, it depends on algorithm complexity but also on density, thus on point cloud complexity. From algorithm point of view, choosing better parameters increases chances to obtain more accurate camera position estimation for alignment process and more detailed and correct geometry for dense cloud reconstruction, but both require longer time for processing. From the other side, higher density allows to collect more features and then more data from the environment, but with the drawback that more points must to be processed and consequently, also here, more time processing is needed. Tradeoff between time consuming and acquired quality results, but in this case the decision could be quite obvious. In order to maintain a certain level of density and quality data the High alignment parameter is still the best option, but about the dense cloud parameter it could be not smart consider the best quality option (Ultra-high value), because its time processing is too much big for our purpose. Therefore, after time consuming analysis, so far, N+O_S_H_H is considered as the best configuration.

RESULTS
This section presents the results of the geometrical ice detection procedure for all tests performed. Due to internal evaluation, estimating noise, density and time processing of the point clouds has conducted to the most complete and less noisy configuration. In Figure 3, a section analysis is performed on N+O_S_H_H dense point cloud through 3DReShaper to display another evidence that it is possible to identify the ice with RGB camera. As result, in Figure 4, profile of ice cube can be clearly seen inside red rectangles.
About the external evaluation, estimating the difference of the point cloud with ice (AIS) respect to the reference one (WIS) in terms of thickness is reported in two analysis. At first, in order to have an automatic procedure a specific code has been implemented. Starting from the reference dense cloud with ice and the other one related to the panel without ice, through a dedicated Matlab code, the detection of the ice is automated and a color map is added to show better the results. The used function allows to create a comparison between 2 dense point clouds. In particular: where pc1: dense cloud related to steel panel with ice pc2: dense cloud related to steel panel without ice In order to run this script the user has to add the "Computer Vision Toolbox". An example of detection of three ice cubes is shown in Figure5 While a second approach is implemented in 3DReShaper; the configuration used is still the same but this time mesh models based on point cloud information respectively from (AIS) and (WIS) are compared. As illustrated in Figure 6, surfaces are pretty cleaner than those one related to dense point cloud case, but both of them validate this study because they get same result about ice thickness. Thanks to this last analysis on thickness, it can be possible to verify that minimal requirements for project planning are fulfilled. From the Table 6 it can be noticed that, at 5 meters, the  The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) DJI Mavic Air camera can see 2.4 GSD and therefore it has a good behavior about accuracy details. At 10 meters, instead, it can't allow a good relation between ice thickness and GSD because its ratio is smaller than 2 but, however, it could be possible using this camera just increasing the focal length value or considering ice thickness from 6.8 mm (exactly 2GSD) on.

CONCLUSIONS
In order to evaluate the possibility to use a geometrical approach to identify the ice, a simulation procedure has been set up. The experiments have been related to the use of an AAV and the acquisition of different images in order to evaluate the reliability of a simple RGB camera for the geometric identification of ice, under different parameters, different images configurations and processing options. All the configurations can give a complete coverage and a good reconstruction of the panel, but the case with both nadir and oblique images is able to generate the most accurate, complete and less noisy points cloud. The achievable accuracy, reliability, and data density could be decisive to effectively survey the needed information.
In particular, the dense cloud obtained setting a High alignment and a High dense cloud parameter has been identified as the best configuration. This configuration has been compared with the model of the panel without ice and results show that it is possible to identify the ice. However, RGB sensor proves to be not enough because some noise still remains due to the geometric reconstruction itself. The procedure can certainly be more reliable with the usage of the hyperspectral sensor technique. For this reason, within the SEI project, the next steps will concern about investigating the use of data from (multi/hyper) spectral sensors for the complete inspection of the aircraft.