GEOVISUALIZATION OF AERIAL PHOTOGRAMMETRIC FLIGHTS FOR DATA QUALITY ASSESSMENT

Spatiotemporal data visualization plays an important role for simulating the changes over time and representing dynamic geospatial phenomena. In aerial photogrammetry, image acquisition is the most important stage for obtaining high-quality products; and can be affected by various factors such as the weather and illumination conditions, imaging geometry, etc. 3D simulation of the aircraft trajectories at the planning stage helps the flight planners to make better decisions especially for unmanned aerial vehicle (UAV) missions in areas with mixed land use land cover, such as rugged topography, water bodies, restricted areas, etc.; since images with poor texture or large differences in scale may deteriorate the quality of the final products. In this study, a geovisualization approach for photogrammetric flights carried out with UAVs or airplane platforms was implemented using CesiumJS Virtual Globe. The measured flight trajectory parameters, such as image perspective centre coordinates and the camera rotations, the time of acquisition, and the interior orientation parameters (IOPs) of the camera were used for spatiotemporal visualization. In the developed approach, the EOPs and IOPs of the images were utilized to reconstruct the flight paths, the camera position, the footprints of the acquired images on the ground, and the rotation of the aircraft; and to present them on a 3D web environment precisely. The approach was demonstrated by using two case studies, one from a UAV flight mission and the other one from an airplane carried out with a largeformat aerial camera. * Corresponding author


INTRODUCTION
In recent years, photogrammetric applications, platforms and use cases have been broadened, and the actively used fields have been diversified such as the production and updating of 3D city models . With the increasing popularity of unmanned aerial vehicles (UAVs) in photogrammetric missions, they can be used for various 3D modelling and mapping studies (Colomina and Molina, 2014;Nex and Remondino, 2014;Hassanalian and Abdelkefi, 2017). Different types of cameras or sensors with various resolutions can be mounted on the UAVs, such as LiDAR, which has relatively low cost and provides better solutions for smaller projects (Alsadik and Remondino, 2020). However, suitable solutions for accurate planning of photogrammetric flight missions are immensely required. Although a number of flight planning software and tools are available (e.g. Pix4Dcapture, 2021;DJI, 2021;eMotion, 2021;UAV Toolbox, 2021;UgCS, 2021) on the market both for the use of airplanes and UAVs, they often do not provide the functionality for visualizing the actual flight paths and image acquisition properties including geometries for existing projects.
For aerial photogrammetric projects, image acquisition is the most important stage for providing high accuracy, and has direct effect on the quality of output products. When compared with airplanes, UAVs and especially quadcopters are in general much more sensitive to the weather conditions at the time of flight due to their smaller sizes and aerodynamic properties. This situation leads to certain level of degradation in the originally planned flight routes and may yield to sub-optimal image acquisition geometries under unfavourable weather conditions. In addition, sudden changes in the flight altitude and large elevation differences on the terrain, particularly in areas with hilly or mountainous topography, may lead to unusable data or other types of flight problems such as accidents. These adverse effects also appear as problems in image geometry, which are eventually transferred to the generated orthophotos or digital surface models (DSMs). A 3D/4D visualization environment for aerial photogrammetric flight missions can help users to analyse potential problems in image geometry and figure out problematic parts in the project area and the datasets.
Nowadays, virtual globes, such as Cesium (2021), Google Earth (2021), etc., have been extensively used by many institutions and researchers for sharing geospatial data in different formats and analysis for diverse projects (e.g. see Goodchild et al., 2012;Liu et al., 2015;Auer and Zipf, 2018;Kocaman, 2018, 2020). In addition, spatiotemporal data visualization on virtual globes has been gaining importance in many different areas (Buschmann et al., 2016;Gonçalves et al., 2015;Pattel et al., 2015;Zhu et al., 2018;Amini et al., 2014;Andrienko et al., 2007). Virtual globes can also be customized according to the application requirements at various levels depending on the flexibility and interoperability of the selected tool or platform.
CesiumJS is an open source JavaScript library capable of visualizing different static geospatial data format types such as 3D Tiles (2021) or KML (2021). It can also visualize timedynamic geospatial models using Cesium Markup Language (CZML) (2021). CZML is a data specification for visualizing dynamic scenes on Cesium virtual globe. Since its release in 2011, it has created a big impact on the web-based visualization of time-dynamic objects and was utilized in many different projects carried out by companies or individual users. Although geovisualization of static data is possible by various software or tools, visualizing time-dynamic objects in large volumes remains a challenging task. CZML allows the visualization of an object or event partitioned into multiple packets for different times. Thus, it allows users to adjust the size of the data to be visualized in single packet, and to visualize large volumes of geospatial data with least performance problems on web.
In this study, a geovisualization approach for photogrammetric flights performed with UAVs and airplanes was implemented by using CesiumJS Virtual Globe. The trajectory elements measured during the flight, such as image perspective centre coordinates and camera rotations, the time of acquisition, and the interior orientation parameters (IOPs) were used for the 4D (XYZ + time) visualization. In the developed approach, the exterior orientation parameters (EOPs) and the IOPs of the images were utilized to reconstruct the flight path in time, the camera position, the footprints of the acquired images on the ground, and the rotation of the aircraft. The flight and image geometry information were presented on the virtual globe on the web. The approach was evaluated by using the data acquired from two different platforms, a UAV and an airplane, as explained in the next sections.

Extraction of Flight Trajectory and Image Footprints
In this study, the flight trajectories (paths) were reconstructed by using the image metadata, i.e. EOPs, and the acquisition times. The temporal interval between two consecutive images was calculated in seconds and used as the flight duration between the two acquisitions. The three camera rotation angles (Omega, Phi, Kappa) were obtained from the post-processing of the inertial measurement unit (IMU) data aboard the aircraft for UAV and bundle block adjustment results for the airplane data. The aircraft rotations (Roll, Pitch, and Yaw angles) required for the visualization were derived from the image EOPs to simulate the platform movement during the flight. The UAV EOP values were already defined in the geodetic coordinate system (GCS) with WGS84 datum. For the airplane dataset, the EOPs were defined in the Universal Transverse Mercator (UTM) projection system and thus were transformed into GCS with WGS84 datum for utilizing in the platform.
A custom FME workbench (Safe Software, 2021) was implemented to project the image footprints on the ground with the help of image EOPs and IOPs (Figure 1) with the assumption of nadir imaging conditions. First, the image perspective centre coordinates were retrieved from the respective CSV (Comma-seperated Value) files. As shown in Figure 1, the coordinate values were used to create point features with the "VertexCreator" transformer. The "Bufferer" transformer was then utilized to generate the initial image footprint polygons (2D) with a dummy size of 1 m x 1 m around the points. In order to calculate the actual image footprint on the ground, the sensor size (width and height) was employed together with the nominal GSD (Ground Sampling Distance) value. For example, a UAV camera with 6000 x 4000 pixels image size with 5 cm GSD has 300 m x 200 m image footprint on the ground. Using the "Scaler" transformer, the initial polygons created with the bufferer function were resized based on the actual image footprints. Figure 1. Generated FME workbench for calculating image footprint corner coordinates on the ground As a last step, the 2D corner coordinates of the image footprints were extracted using the "CoordinateExtractor" transformer to be employed in CZML format. The corner coordinates were stored in a new CSV file together with the photo IDs as an intermediate step for the conversion to the CZML and visualization on the Cesium environment. A view of image footprint visualization is given in Figure 2. Here, again it must be noted that the images were assumed to be taken in the nadir direction; and a constant GSD value was employed for the whole flight.

CZML Preparation for the Flights
CZML is a JSON-based file format that for visualizing lines, points, spatial objects and 3D models in a 4-dimensional (timedynamic) environment. CZML can be used on Cesium Virtual Globe for visualizing various georeferenced data types such as 3D models in the form of 3D Tiles and geometric shapes (rectangles, circles, ellipses, cones, cylinders). The rotations of the aircraft were visualized by using the image EOPs. It was assumed that the EOPs of the nadir-looking camera images have the same rotation angles with the aircraft.
glTF (2021) is the main format used by Cesium for visualizing animated 3D models. Depending on the type of aircraft used for the visualization project, an animated 3D UAV or airplane model stored in glTF format was used here. Since, glTF models uses quaternion (x, y, z, w) to animate rotations; the aircraft rotation angles obtained from the transformed EOPs in the previous step were converted into quaternions for the visualization on Cesium. The Cesium built-in functionality for converting heading, pitch, and roll values from radians into quaternions was employed for the conversion (first "HeadingPitchRoll.fromDegrees" and then "Quaternion.fromHeadingPitchRoll"). A camera icon was also added for each image perspective center location, which can be used to retrieve image previews and the other attributes (i.e. photo ID, coordinates, acquisition time, EOPs) by clicking on the web interface. A view of the aircraft models defined in glTF and used in this study is given in Figure 3. The models are included in the Cesium model library. The flight data (i.e. flight path, aircraft rotations, image footprints and flight speed obtained from the distance and the temporal interval) were converted into a single CZML (Cesium Language) file for visualization in a web browser running CesiumJS virtual web globe. CZML can be used in many different visualization projects; and for each project, a projectspecific CZML template must be developed. A part of the code snippets for defining the employed 3D glTF model and its positions are provided in Figures A1 in Appendix. The code for the image footprint is given in Figure A2 in Appendix.

CZML Visualization
A couple of different methods can be used to visualize generated CZML on Cesium virtual globe. The models can be visualized on a local Cesium installation by loading the generated CZML file. Another solution would be to visualize the CZML on Cesium Sandcastle  platform. The Sandcastle is a geovisualization interface application for viewing Cesium demos and for quickly testing code parts. It is available as both for local and online versions. The CZML file can be visualized by entering the path of the generated file in the Sandcastle interface or by using the CZML code in the interface. The codes developed on Cesium Sandcastle can be shared with the other Cesium users for obtaining help or feedbacks. However, the Sandcastle is not the proper tool for sharing data and projects. The cloud-based hosting and visualization platform solution called Cesium ION (2021) is more suitable for uploading and visualizing large amounts of geospatial datasets.
Cesium ION (2021) is a platform for hosting, optimizing and streaming large volume of 3D geospatial datasets, which is also used in this study for visualizing generated CZML dataset. It is capable of tiling massive 3D geospatial content for efficient streaming on CesiumJS virtual globe. It also provides access to Cesium World Terrain and different imagery layers which can be combined due to requirements of the project. Uploaded data will be automatically processed and visualized on Cesium virtual globe, which can be later shared via generated link. The user data management and configuration interface of Cesium ION in presented in Figure 4. In this study, the Cesium ION platform was preferred for configuring the projects and sharing the information.

RESULTS AND DISCUSSIONS
In Figure 5, a screenshot from the visualization of the the photogrammetric image acquisition with a UAV with 80% forward overlap is given. The UAV data belongs to a small mission over Istanbul, Turkey. In Figure 6, a view of aerial The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B4-2021XXIV ISPRS Congress (2021 photogrammetric data, which belongs to the Bizimsehir Project in Gaziantep, Turkey, as a part of multi-disciplinary smart city development project is presented (Buyukdemircioglu and Kocaman, 2020). The dataset consists a total of 736 images with 80% forward and 60% lateral overlap. Further views on the platform and the semantic information retrieval are presented in Figures 7 and 8.   In the developed geovisualization environment, it is possible to observe the aircraft rotations caused by external effects such as wind, air turbulence or skidding during the image acquisition. In addition, by computing the temporal interval between consecutive acquisitions, it becomes possible to analyze the image overlaps. At the same time, the image footprints projected on the ground allows to assess the actual overlaps for improved processing of the data. The image IDs and previews were also provided on the developed platform as attribute information (Figure 8).
By simulating the trajectories of photogrammetric missions after the mission completion, the users can assess the actual image footprints and the flight conditions accurately. Such visualization efforts are not only useful for photogrammetry professionals, but also for students and non-professionals to comprehend the aerial photogrammetric data acquisition principles and the outcomes relatively easily.

CONCLUSIONS AND FUTURE WORK
In this study, the open-source CZML format was utilized for web-based streaming of photogrammetric flight data in terms of flight trajectories and the image acquisition geometries via Cesium ION platform. The format specifies the time-dependent change of geospatial datasets with their geometric and semantic aspects. The Cesium ION platform and the CZML format were found useful for the 4D data visualization.
4D visualization of the aerial photogrammetric flights will help to identify the error sources during flights, and helps users to prevent the occurrences of similar problems in the planned missions. The developed 4D visualization approach can also be used in photogrammetry courses in universities to make students visually understand photogrammetric flights. Thus, a better understanding of photogrammetric terms such as image The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII- B4-2021XXIV ISPRS Congress (2021 overlap, image resolution, flying height and the flight route can be achieved. Also, effects of weather conditions such as wind, rain, etc. on the aircraft or UAV can be seen in a 4D environment; which eventually helps to analyze the potential problems with the image acquisition geometry.
As future work, a Python-based software tool to perform the conversion between the provided EOPs and the CZML will be developed for the automation of the proposed visualization workflow. In addition, demonstrating the shadows in the project area by using the day and time of imaging will help to analyze the illumination conditions at the given time; and thus allow the assessment of image radiometric quality visually. Further work is also needed to increase the accuracy of image footprints by utilizing the image rotations, which are especially essential for oblique acquisitions. Moreover, the use of average GSDs for each image or the projected corner coordinates with the help of a DSM and the image EOPs and IOPs will ensure obtaining the highest possible accuracy in the footprint extraction process.