USING THE OPEN SKIES MULTI SENSOR SYSTEM FOR MILITARY OBSERVATION MISSIONS AND CIVILIAN APPLICATIONS

: In this paper we aim to show the workflow and the technical systems used for Open Skies missions as well as the use of this Multi Sensor System for civil applications. While missions conducted and analyzed under the force of Open Skies Treaty is a technical and administrative one, the very complex sensor system enables various civil missions for a wide range of tasks. Especially the combination with LiDAR gives new opportunities. We describe typical Open Skies work steps besides civil tasks that need as calibration, accuracy estimations and adjusted workflows especially with the LiDAR data and different camera setups.


INTRODUCTION
The complex aerial system designed to be used by Romania in Open Skies missions (RoDAS) is composed of 8 Phase One cameras, configured in the way to obtain three theoretical flying heights at the same GSD -"low", "middle", "high" (Figure 1).

Figure 1. RoDAS mounted in the Romanian Open Skies aircraft
Even this system is customized and specially design for Open Skies missions, it can be used in the civilian domains as well.

OPEN SKIES MISSIONS
Data collected during an Open Skies mission over a state-party along a specific route are prior assigned between the observed state and the observing one. Using a flight planning software, the observing state will set up a corridor that defines the planned Open Skies mission which will be uploaded on the digital aerial system installed in the aircraft.
In the day when the mission takes place and after the inspection of the certified aircraft and aerial system, there will be acquired a big amount of aerial data in raw format. After that, the data must be processed on-site or in a location of the observing state. For this operation a limited time is defined by the Treaty.
The workflow to obtain the final product implies several steps that requires lots of hardware and software resources, as well as trained personnel stuff. At the end, the product is obtained in a non-editable image format according to the Open Skies Treaty. Only these final products are kept on media storages by the state parties involved in the mission, while all the acquired, additional or intermediary data stored during the process must be deleted from all the equipment used in the mission, together with the software and the operating system.
To plan an Open Skies mission, it AeroTopoL software in corridor mode is used. The flight plan is then transferred to the onboard PC, where during the mission execution, the same application as FMS is used and controlles the entire system. On the other hand, cameras settings are done using IxCapture software developed by PhaseOne (Figure 2).

Figure 2. AeroTopoL and IxCapture software
After the data are acquired, the removable storages are extracted from the aircraft, inserted via docking station and connected to powerful processing workstations. The raw iiq images are then transformed in OSDDEF format files using a convertor software developed by Intergraph Romania, based on the PhaseOne SDK (Figure 3).

Figure 3. Romanian convertor used in Open Skies missions
All the resulted OSDDEF images are then duplicated and transferred on media storages to be shared between the mission implied nations using Beyond Compare software and will be furthermore analyzed by IMINT (Imagery INTeligence) specialists from the observation state.
Finally, all data are erased from the media storages used during the mission by using Parted Magic software, thus the storages are prepared for the next mission by restoring the operating system and all the Open Skies mission applications using Macrium Reflect software (Figure 4). Only after these processes are done, the system is considered to be "safe" and therefore legalized for the next mission.

SYSTEM CALIBRATION FOR THE UNION HIGHWAY PROJECT
The strict acquired data processing and the customization of the digital aerial system of Open Skies it is not an impediment to take advantage of this complex and accurate equipment in other domains (Motz et al., 2021). Any of the configurations utilized in Open Skies missions can be used to collect data for specific civilian purposes. Such an example it is the use of the RoDAS "middle" configuration in a national civilian application named "Union Highway", where 15 cm resolution images have been successfully combined with Lidar data into a very well elaborated project.

General workflow
To make use of the RoDAS System in a civil application with high precision, it requires a new workflow. This also includes a more precise and far more complex calibration of the RoDAS system, identifying new post-processing procedures and software solutions.
We decided to use the Open-Skies system in combination with aerial LiDAR system for a highway infrastructure project. In Figure 5 we present the workflow process that was used in this case. The infrastructure project is about a highway named "Union Highway" because this will connect two Romanian historical provinces: Moldavia and Transylvania. The highway begins from Târgu Mureș and it ends at Târgu Neamț, with a length of 214 km. The relief varies from 250 to 1500 meters and it is verry well wooded in the mountain areas. This kind of terrain impose the use of the optical data together with aerial scanning. To realise the design project for a highway like this it is necessary that the geodetic data has to have a good precision (e.g., GSD of 15 cm, aerotriangulation and LiDAR error of maximum 20 cm). To obtain this kind of precision it is necessary to realise an accurate calibration of the Open Skies system.
For this project we used the two nadiral iXM-RS150F cameras (RGB and NIR) from the RoDAS "middle" configuration, that is also having another two iXM-RS150F cameras, both RGB, in oblique position. All four cameras are equipped with a 40 mm lenses. The flight plan was done in AeroTopoL, with a 15 cm GSD, 65% overlap and at least 40% side lap. Then, for flight execution, we used AeroTopoL together with the PhaseOne software, IX Capture.

Open-Skies System Calibration
For indirect calibration, the entire flown area must be covered with Ground Control Points (GCP). Calibration includes focal length and radial distortion calculation, the camera-to-IMU offsets calibration, the boresight calibration between the IMU and camera, and GPS antenna lever arms as well.
The processing system of the navigational data is the IMU systems. The calibration of the Open Skies System consists of determining the correlation between the IMU system, the airplane (vehicle/body system), the GNSS center of phase and the optical sensor. To bring the Body (airplane) system into the IMU system we will use an Euler intrinsic rotation, respecting the right hand rule (Novatel development team, 2019a). The angular rotations are made in the Z, X, Y order. After the calibration, the rotations are implemented in the processing software of the navigational data, Novatel Inertial Explorer. The offset between the centre of IMU system and the phase center of GNSS antenna are measured along the IMU axes ( Figure 6). The offsets were implemented in the processing software of the navigational data. The calibration between the IMU sensor and the optical camera were measured and used to export the images orientations and centre of projections. This calibration is exemplified in Figure 9. Figure 6. Calibration of the IMU to GNSS offsets (Lever Arm) and the Body (airplane) to IMU rotations After the calibration of the system is done, the next step is consisting in the post-processing of the navigational data.

Precise post-processing of the navigational data
The aerial mission was planned to be post-processed from the take-off from the airport of Bucharest to the landing point on the same airport (Figure 7). At the beginning and at the end of the mission was recorded also statical GNSS data for at least 10 minutes to improve the post-processing accuracy of the trajectory (Novatel development team, 2019b).

Figure 7.
Post-processing trajectory mission with Novatel Inertial Explorer -from take-off to landing The precise post-processing of the navigational data was done with Inertial Explorer from Novatel. To ensure a quality precision, and also a uniform accuracy, we use 16 base stations (Figure 7). All the base stations were considered fixed and were chosen only from the national network, Rompos. The same network was used to determine the GCP for aerotriangulation or for LiDAR geoid application. The post-processing of the LiDAR trajectories involved base stations only from Rompos. Using a single unitary network ensure a high precision for all the results, images and LiDAR.
The trajectory post-process was made in two steps. The first step was to compute the GNSS trajectory and use the results in the second step to post-process inertial trajectory. This method is known as Loosely Coupled and it is a method developed by Novatel in Inertial Explorer (Novatel development team, 2022a). In the end of trajectory post-processing, the precision and the accuracy results are expressed by graphics (Figure 8). In Figure 8 the first graphic expresses the 3D positional precision (North, East, Up), completed by the second graphic concerning the accuracy (between forward and reverse processing). The third and fourth graphics illustrates the precision and the accuracy of the IMU data (Roll, Pitch, Heading).

Initial image orientation
After the trajectory post-processing of the GNSS and IMU data, the next step is to transfer the positional and attitude data to each image. There are two important aspects to be considered. The first one is the camera calibration in relation with the IMU sensor. As we explained above, the camera calibration consists in determining the offsets from the center of IMU coordinate sensor to the optical center of the digital camera (Figure 9 left). The second important aspect is the exporting parameters in Novatel Inertial Explorer. The most important is the projection system used to export the orientation data of the images. Even if the trajectory post-processing was made in a global geodetic system (ETRS89), the export of the orientation data must be made in the projection system where the images will be used. The export was done in the national projection, Stereographic 1970, which has associated the Krasovski 1940 ellipsoid. Even if the transformation from ETRS89 to Stereographic 1970 is not so precise, it is essentially to define the type of the projection, the pole of projection and other essential parameters as shown in Figure 9 right. Applying this method, the attitude data will be more precise exported, so we will have more accurate apriori data for the aerotriangulation input. To eliminate the deficiency of the lack of precision when transforming the coordinates, we transformed the coordinate of the center of images from ETRS89 to Stereografic 1970 with the official application TransDatRo, elaborated by the National Center for Cartography.

COMBINING OPEN SKIES OPTICAL DATA WITH LIDAR POINTS IN THE UNION HIGHWAY PROJECT
Because of the natural relief and the very forested areas in the corridor of the Union Highway it is necessarily to use LiDAR technology to penetrate the dense vegetation. Combining with images data from Open Skies System, it will result in a comprehensive and a complex 3D model of the entire area affected by the project.

LiDAR point cloud acquisition
The LiDAR scanning missions were made with a civil airplane, managed by the team of Prosig Expert company. It was used a LiDAR sensor, model LMSQ780 from Riegl. It was necessary two missions to cover the entire corridor of the future highway.

Precise post-processing of the navigational data
For the precise post-processing of the trajectories, we use also the two steps processing. The GNSS data were processed with GrafNav from Novatel, and the inertial data with AeroOffice from IGI. The process is almost the same with the one explained in Inertial Explorer. The processing results are also expressed with graphics. In Figure 10 the accuracy obtained on positioning is expressed as a difference between GNSS/INS methods of post-processing navigational data. In Figure 11 are the final processing results as a summary. The post-processed trajectories will be used in the next phase of calibration the LiDAR strips. In the end of the processing the trajectories were exported in the SBET global format.

Noise filtering and primary classification
To proceed to the LiDAR strip calibration, the LiDAR data must be prepared for this phase. The software used in this stage are TerraScan and TerraMatch from TerraSolid. First step is the split of the data in clusters to easy manage the big data amount (more than 100GB). First of all, the LiDAR point clouds must be clear of the noise recorded by the sensor. The noise affecting a LiDAR point cloud can cause errors in classification and therefore a strips calibration mismatch (Figure 12).  In the primary filtration of the noise points these should not be deleted but classified in the noise points class, because some of the points could be wrong interpreted as noise. After the LiDAR strip calibration, the data classification will be more precise. Then the unassigned LiDAR data was classified in the following classes: ground and buildings (and subsequently low, medium and high vegetation). Each strip was classified separately. Now the strips can be calibrated depending on the class corelation in the overlapping areas of adjacent strips.

LiDAR strip calibration
The LiDAR strip calibration is a necessary step to obtain a high inner precision of the scanned data. No matter how performant is a scanning system or how good is calibrated, a strip calibration is a must because it also eliminates the influence of the weather and atmospheric conditions that cannot be changed. The LiDAR flightline (strip) calibration was made with TerraMatch, using the two methods Find Match and Tie Lines.
The methods were used depending on the size of the mismatch identified. We consider that the biggest error cand cause false mismatch on other directions. Therefore, in the beginning we were searching for the biggest mismatch, we applied it, and then verify the matching error. This step is a very important one, but also a delicate one. We apply iteratively correction on North, East, Up, Roll and Pitch. The corrections were first applied for flightlines constant offsets or systematic errors. Then the corrections were applied temporal by surface matching and finding the fluctuations. The process of flightline calibration and partial results are shown in Figure 14.

Figure 14. Strip calibration and partial results using TerraMatch
We start from an average magnitude between strips of 10 cm and in the end of adjustments we reach an average magnitude between strips of 3 cm (Figure 15). After this important step was done we proceed to final classify the LiDAR point clouds.

Final classification and DTM extraction
To extract the DTM (Digital Terrain Model) it is necessary to classify the calibrated LiDAR data. We use TerraScan to complete this step. To obtain a complex and a comprehensive 3D model of the scanned corridor we classified the LiDAR data in the following classes: Low noise, High noise, ground, low vegetation, medium vegetation, high vegetation, buildings and bridges. In Figure 16 is presented an example of the final classification of a LiDAR block on the Union Highway.

Figure 16. Example of a final classified LiDAR block
To obtain good result in ground data classification we used iteratively classification processes with different parameters. The quality of the DTM depends on the ground classification quality. Because of the kneaded relief and the vast forested areas this was not an easy job. For example, for the same LiDAR point cloud presented in Figure 16, the classification of the ground class is presented in Figure 17.

Figure 17. Ground class used to extract DTM grid
All the final results (Figure 16, Figure 17) were presented in the Prosig-Potree web-browser LiDAR application. The web application is based on the web Potree 1 algorithm. The web algorithm is known as Octree structure, a fast web rendering for massive point clouds (Schütz et al., 2020). The web-browser algorithm was conceived by Markus Schuetz in his diploma thesis (Schuetz, 2016). This is an application further developed by the team of Prosig Expert in a national research program.
Because of the large amount of data, for the highway design engineers it is difficult to use LiDAR point clouds on their software solutions. The LiDAR web application, Prosig-Potree, eliminate a part of this inconvenient. However, a DTM extraction is more suitable for the design engineers to work with. More than that, the LiDAR point clouds are processed in the UTM (Universal Transverse Mercator) projection because of the lack of precise transformation to the Romanian national coordinate systems. The only one official application for coordinate transformation from ETRS89 to national coordinate systems is TransDatRo, but it is working just for text files. Also, this application is limited at a maximum number of points of about 1 million (depending on the computer and other technical parameters). In conclusion, the only solution is the extraction of the DTM. In the end of the classification, we extracted the DTM for a grid of 1 meter, but also for a grid of 20 meters. The DTM grid of 20 meters was transformed with TransDatRo, but for the DTM grid of 1 meter we were forced to use another solution.

Applying corrections for the national geoid to the DTM point files
The biggest challenge is the transformation from ellipsoidal heights in the national reference coordinate system (Black Sea 1975, Ed. 1990). This is because the Romanian vertical reference frame is a frame with normal heights. To transform the ellipsoidal heights, we use a two-steps transformation. First we used a Helmert transformation with 7 parameters, based on common points. Then we measured pair of point along the corridor (see Figure 18) to determine the remaining correction to adjust to the national geoid. There were surveyed 70 points (an average of 6 km between pair points) using the same GNSS network used for image georeferencing. The GCP for geoid adjustment were chosen on horizontal hard surfaces (maximum slope 3%). In the Figure 18 it is exemplified the point distribution for a part of the Union Highway on © CNES/Airbus Image Copyright 2020, distributed by © Google Earth 2020. To apply the remaining correction for adjusting the DTM to the national geoid, we used TerraScan. The working modes, the corrections applied, and the results are illustrated in Figure 19. The 1 meter DTM grid was transformed in the national coordinate systems with a good precision. This DTM will be used therefore to obtain enhanced results.

Combined aerotriangulation with classical GCP and vertical control points from LiDAR clouds
To obtain precise orientated images we use the indirect georeferencing method, which consist in aerotriangulation, made with InPho application. Because of the conformation of the highway corridor and the vast forested relief, in some areas was impossible to assure a satisfying number of GCP (Ground Control Points). We measured 191 points in the field (using the same GNSS network, Rompos). Of these 109 GCP was used as horizontal and vertical fix-points, 38 GCP was used as horizontal, 37 GCP was used as horizontal, and 7 as control points. To obtain a better precision on elevation we used another 127 VCP (Vertical Control Points) extracted from the adjusted DTM (1 meter grid). In the end of the aerotriangulation adjustment we obtained a very good precision (Figure 20).

Figure 20. Improved aerotriangulation in high forested areas by using VCP from LiDAR
But most important is that we also obtained an aerotriangulation that follow the national geoid undulations. This leads to more accurate topographical plans realised through stereo-restitution (Figure 21). From a statistically point of view, to obtain these results, about 40% of the GCP were VCP, extracted from the LiDAR data (DTM at 1 meter).

Presentation of the enhanced results
In the end of the article, we will show the enhanced results obtained by combining the Open Skies data with civil LiDAR data in the Union Highway project. The first improved product are the topographical plans. The 1 meter grid of DTM was used to generate precise contour levels for the topographic plans. The 20 meters grid of DTM was used to give detailed information about the elevation on the topographic plans (Figure 21).

Figure 21.
Topographic plans with 20m DTM support and level curves generated on the DTM grid at 1m from LiDAR data The second improved product are the orthophoto plans obtained from the new Open Skies calibrated system. The precise results obtained in the aerotriangulation adjustments influenced the generation of the tie points dense cloud, and therefore the trueortho rectification. The good results can be seen in the Figure  22, especially in the zoomed image. The orthophoto plans was generated with Pix4D Mapper 2 and the GCP verification was made with the help of the Global Mapper.  The use of the web-browser application, Potree-Prosig, transform the results in a very realistic 3D model (Figure 23) of the Union Highway project.

ROBUST CALIBRATION OF THE OPEN-SKIES SYSTEM FOR DIRECT GEOREFERENCING
The results obtained with indirect georeferencing of the Open Skies System was used to obtain a precise boresight calibration. A precise boresight calibration between IMU coordinates system and camera coordinates system will give the opportunity to use Open Skies with direct georeferencing. Compared to indirect georeferencing, which consist in an indirect determination of the exterior orientation parameters trough aerotriangulation, the direct georeferencing method involves straightforward determination of external orientation parameters. This can be made using modern sensors (GNSS, IMU), innovative technologies as SPAN (Synchronized Position Attitude Navigation between GNSS and IMU), and a priori precise systems calibration (Habib, 2012).
In addition to the calibration presented in chapter 3, a complete precise calibration is acquired after the boresight calibration between IMU and camera. The external orientation parameters obtained through aerotriangulation was used to determine the boresight angles. The angles ω, φ, κ, represents the attitude of the camera coordinate system relatively to the planimetric coordinate system, so the datum must be set up accordingly to the input data (Novatel development team, 2022b). The boresight was computed in the frame of the Stereographic 1970 system (Figure 24) using the implemented algorithm in Novatel Inertial Explorer. In this case, the boresight angles were determined using the conventional frame settings.
The implemented algorithm corrects the map convergence between the grid north and the true north and apply it to the boresight angles (Novatel development team, 2022c). This will help in future projects to obtain georeferenced images directly to the national coordinate system. The setup of the precise boresight calibration process is presented in Figure 24 along with the obtained results.
For a higher quality of the direct georeferencing the boresight calibration will be recomputed in another iteration based on a future project where it will be used the actual boresight values but also aerotriangulation for the determination of the external orientation parameters. In this way it will be acquired a precise and a confident calibration of the Open Skies System.

Figure 24. Precise boresight calibration -setup and results
The direct georeferencing method is faster, with economic advantages and is very suitable in areas with poor or spare GCP (Habib, 2012). Those arguments make the direct georeferencing method the most suitable for projects in very forested areas or in any other area with difficult accessibility.

CONCLUSIONS
The very realistic 3D models of an environment, utilised in the web-browser application, conduct to some major consequence.
The design engineers will reduce to the minimum the field displacement. This implicitly leads to a saving of time and money in realising projects like the Union Highway. The design engineers could extract more geo information that was not included in the topographical plans. Another benefit is the management of the big amount of the LiDAR data using the web-browser application. Also, the realistic virtual 3D models are easy to be used by staff without technical specialised skills.
On the other hand, introducing the digital technologies in Open Skies missions it is a big step forward. The biggest gain of this action is the money and time saved due to the fact that some process like film developing and scanning are not anymore required. Also, introducing the digital era in Open Skies needs a new approach on what means digital photography and diplomatic relations between the world states, that can give Open Skies Treaty a completely new relevance.
The collaboration principle between military institutions and civil economical companies brings a lot of benefits for the entire community. In addition to the technical benefits, the military institutions gain more appreciation in the society, in a world that is becoming more and more segregated day by day. Technical research shall not be split from the real life and the social influence in the community.
By the fusion of the data obtained with a military customised digital aerial system designed only for a specific domain, as it is Open Skies, with LiDAR data from a civilian source, conducted to a high level of resulted products. These products can be used with good results in any infrastructure project, and demonstrates once more that with combined efforts, two domains that at first glance seemed very different and almost impossible to be put together, were successful integrated.
In the end of this article, we want to emphasize the scientific and the innovative parts of this work. Those are represented by the integration of military and civilian sensor types using a complex methodology that combine multiple software, methods, algorithms, procedures, and adapted web application to obtain superior photogrammetric outcomes. The integration of the rigorous calibrated LiDAR data in aerotriangulation, corrected with the quasigeoid undulation, led to a new level of high-class accuracy.
This collaboration opens also new future research directions in which the authors foresee the evaluation of optical and SAR imagery complementarity for military surveillance missions and civilian applications. The specific image processing methods will develop new approaches for multi sensor data analysis and exploitation.