UAV MISSION PLANNING FOR AUTOMATIC EXPLORATION AND SEMANTIC MAPPING

: Unmanned Aerial Vehicles (UAVs) are used for the inspection of areas which are otherwise difﬁcult to access. Autonomous monitoring and navigation requires a background knowledge on the surroundings of the vehicle. Most mission planing systems assume collision-free pre-deﬁned paths and do not tolerate a GPS signal outage. Our approach makes weaker assumptions. This paper introduces a mission planing platform allowing for the integration of environmental prior knowledge such as 3D building and terrain models. This prior knowledge is integrated to pre-compute an octomap for collision detection. The semantically rich building models are used to specify semantic user queries such as roof or facade inspection. A reasoning process paves the way for semantic mission planing of hidden and a-priori unknown objects. Subsequent scene interpretation is performed by an incremental parsing process.


INTRODUCTION
Unmanned aerial vehicles are widely used to survey and act in several complex environments and support solving various tasks. They are for instance applied in the context of 3D mapping and urban analysis (Nex and Remondino, 2014;Grubesic and Nelson, 2020). In the broader context of remote sensing, Pajares (2015) presented an overview and current status of UAV applications. He categorised the applications as following: agriculture and forestry, humanitarian localization and rescue, target detection and tracking in surveillance context, environmental monitoring, photogrammetry, vegetation classification, atmospheric observation and analysis, cultural heritage and archaeology, wild life conservation and last but not least the urban environments and infrastructure.
Compared to the classical manned aerial mapping vehicles, UAVs represent a suitable low-cost counterpart. They allow for performing different missions in a cost-effective manner. For example, the engine could survey regions which are not accessible or fully hidden for rescue teams in the case of a rescue management or evacuation scenario. Flying in such circumstances within challenging 3D areas requires a higher degree of autonomy rather than to rely on a human pilot. For an autonomous monitoring and navigation, a background knowledge on the environment of the vehicle is helpful. In the same context of an evacuation, the question whether a person is asking for help from a window or a balcony belonging to a facade which is not accessible for the rescue team demands a semantic mapping of the scene (see. Figure 1). In this context, semantically rich building models in the sense of CityGML (Gröger et al., 2012) or CityJSON (Ledoux et al., 2019) are relevant since they allow to represent the meaning of geometric objects such as roofs or balconies. To this aim, we introduce in this paper a mission planing platform allowing for the integration of environmental prior knowledge, in particular 3D building and terrain models. This prior knowledge is serving two folds. Beside the ability to specify pre-configured paths the system allows for the formulation of semantic queries, such as roof gutter inspection, based on the underlying semantically rich CityGML model. Further, the generation of an octomap modelling poten-Person in balcony in the 2 nd floor. Figure 1: Use case of a user semantic mission planing. Is a person in the second floor at the back of the building? ( c Wolfgang Förstner, adapted) tial obstacles ensures an automatic collision detection during the navigation based on globally pre-defined paths.

UAV
The mission planing engine has been implemented in the frame of a scientific research unit called 'Mapping on Demand' (MoD) 1 (Klingbeil et al., 2014) including the following sub-projects: 1. pose determination of lightweight UAVs 2. local perception for the navigation of lightweight UAVs 3. 3D-navigation and obstacle detection for lightweight UAVs 4. incremental mapping from image sequences 5. 3D surface reconstruction and interpretation 6. efficient representation and generation of consistent 3D and 4D maps 7. an incremental refinement of a semantic building and landscape model

exploration with lightweight UAVs
The goal herewith is the development and testing of procedures and algorithms for the fast three-dimensional identification and mensuration of inaccessible objects on the basis of a semantically specified user inquiry. The sensor platform is a lightweight autonomously flying robot as depicted in Figure 5. The UAV is equipped with eight co-axial rotors and several sensors such as a rotating 3D laser scanner, two stereo camera pairs, ultrasonic sensors and GNSS modules. It uses the visual information for navigation, obstacle detection, exploration and object acquisition. A detailed platform description will be presented in section 3.2. The semantic mission planing draws upon an interactive graphical user interface where user queries can be specified based on the underlying semantic building models and according to the concrete mission objectives.
At the end of a process pipeline, the aim is to achieve an incremental interpretation of the scene based on a surface interpretation and using a scene model. Simultaneously, key information about potential obstacles is provided to facilitate navigation. The mission planing platform has been used as starting step to predict in particular previously unobserved substructures based on symmetry recognition (Loch- Dehbi et al., 2013) as well as in the incremental semantic interpretation of a geometric map consisting mainly of LoD2 building models. The latter is available in different levels of detail and is based on the flight movement of the UAV using attributed grammars and relational learning (Dehbi et al., 2016).
The remainder of this paper is structured as follows: Section 2 gives an overview on related work. The mission planing system and its components are introduced in section 3. The paper is concluded in section 4.

RELATED WORK
Unmanned aerial vehicles are meanwhile widely used to survey and act in several complex environments and support solving various tasks. An overview and current status of remote sensing applications based on unmanned aerial vehicles can be found in Pajares (2015). In the specific context of digital farming, UAVs are playing a prominent role for mapping and monitoring of plants in the last years. For instance, they are used for digital counting of maize plants (Gnädinger and Schmidhalter, 2017). In the context of rescue management, Doherty and Rudol (2007) presented an application of UAVs for an emergency mission which aims at searching and identifying injured persons.
Our mission planning system provides the basis for a global path planing for specific tasks. Path planing is a high dynamic research topic. Nieuwenhuisen and Behnke (2015) presented an approach which comprises a replanning component and allows for the reaction on newly perceived obstacles not occurring in the underlying geometric map. In a later work, Nieuwenhuisen and Behnke (2016) introduced a layered and navigation planning system based on coarse prior knowledge and local maps from omnidirectional on-board obstacle detection. Tokekar et al. (2016) studied methods for path and sensor planning for a symbiotic UAV system for precision agriculture. The interested reader is referred to the literature review of UAV 3D path planing presented by Yang et al. (2014).
According to the project list in section 1, sub-project 8 has dealt with the exploration with UAVs. In this context, Palazzolo and Stachniss (2018) presented a method for the effective exploration to select viewpoints that provide informative measurements based on the expected information gain.
In the 3D building context, Palanirajan et al. (2019) introduced an efficient flight planning for building facade reconstruction. Based on the building footprint from a DSM acquired from an initial nadir flight, a target building is identified and an image sequence is planned. Grenzdörffer et al. (2015) combined UAS photogrammetry and terrestrial laser scanning for surveying and modeling of cultural heritage monument sites. They demonstrated their approach on the Cathedral of St. Nicholas in the city of Greifswald as example.

MISSION PLANNING SYSTEM
This section introduces the mission planing system in which subsection 3.1 describes the base station and its functionalities, whereas subsection 3.2 gives insight into the used UAV platform. In subsection 3.3, two building-related use cases are presented. The mission planning is used to interpret a hidden facade for a rescue scenario on the one hand and applied to survey and interpret an inaccessible rain gutter based on a semantic request on the other hand.
As previously mentioned, one of the most important goals of the 'mapping on demand' project is to enable an autonomous UAV mapping and interpretation of objects such as buildings. The vehicle should be able to plan its trajectory in an autonomous fashion and simultaneously detects and avoids obstacles and also delivers 3D captured data during the flight. This multi-source data, in particular point clouds, is managed and compressed in real-time in sub-project 6 ( Golla and Klein, 2015). To meet these requirements, an appropriate mission planning is necessary.

Base station
For the sake of simplicity, most mission planning systems assume a collision-free environment. In the context of building The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition) mapping and surveying particularly, they rely on manually performed flights. Towards a fully autonomous flying, a geometric background knowledge on the target area is necessary. In this context, the existence of building and terrain models beforehand is of great benefit for every mission planing platform. Geometric models are in this sense an important prior to pre-build an obstacle map. They also enable to specify global paths which the UAV has to follow depending on the specified mission. Beyond the geometry, semantic knowledge is required in order to achieve a higher level of autonomy. Semantic rich city models allow further for the specification of targeted missions such as roof or facade reconstruction or customised requests such as rain gutter inspection. Thus, we designed and implemented a mission planning platform fulfilling these requirements to support an autonomous UAV navigation.
To support mission planning issues, we provide a ground station with a front-end enabling to specify user requests. Semantic queries can be performed based on a graphical user interface allowing to interact with regions of interests. Figure 2 on the top left depicts a test area from Frankenforst Vinxel in Germany visualized by our mission planing system. Figure 4 illustrates the same area in a Google Earth view. The front-end has been implemented as a plug-in of the CityGML's Aristoteles viewer. From a wide range scene, a specific building can be selected as presented on the most right top figure focusing on building information and paving the way for the subsequent query. As the building models support the CityGML standard, the geometric entities are expanded by semantic information such as roof or facade surfaces depending on the level of detail (LoD). The underlying buildings are in LoD2 according to the CityGML specification (Gröger et al., 2012) including roof tops as illustrated in Figure 4.
The provided building models and the digital elevation model represent the basis to generate an Octomap (Hornung et al., 2013) reflecting the occupancy of the environment and hence supporting the reasoning for obstacle detection and collision management. Figure 3 illustrates an octomap of the test area from Frankenforst Vinxel in Germany generated in project 3 (Nieuwenhuisen and Behnke, 2015).
Once a building has been selected by the user, a set of view poses defining the flying path can be specified for a global plan. Depending on the mission objectives, targeted surfaces are selected as will be explained in section 3.3. At the bottom of Figure 2, a facade selection is visualized on the right, whereas a specified path is depicted on the left. For a roof or a gutter inspection, in particular, targeted key points can be sampled drawing upon the 3D semantic models and leading to captured 3D point clouds of the region of interest. This allows for a subsequent interpretation to identify the gutter model based on the hypothesised model from the mission planing. More details will be introduced in section 3.3.
The georeferencing of the UAV plays a prominent role in order to meet the automation goals. It is influencing the UAV navigation and the subsequent surveying results of the environment and their accuracy. In this context, the city model is applied to improve the direct positioning accuracies under challenging GPS measurement circumstances. To this end, a GPS geometry map is generated based on a 3D model of the surrounding buildings in the region of interest. The GPS map is integrated in the flight planning process in order to identify GPS challenging environments, e.g. shadowed building and city areas, and possibly avoid them (Zimmermann et al., 2017).

Platform description
The UAV platform is a custom designed octorotor multicopter with a co-axialarrangement of rotors (see Figure 5). Two stereo The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition) ? ? Figure 6: Generation of preliminary facade hypothesis for the mission planing according to the objectives of the mission from figure 1. A-priori unknown floor numbers and facade parts are estimated.
camera pairs are mounted pointing backward and forward with a pitch angle of 45 • in order to provide information about object and obstacles in the vicinity of the UAV (Schneider et al., 2016b;Nieuwenhuisen et al., 2013). The cameras are sampled with a frequency of 10Hz and the fisheye lenses provide a 185 • field of view. The images are also used to improve the trajectory estimation (Schneider et al., 2016a). The main sources for realtime position and orientation information are a dual-frequency GNSS receiver and a tactical-grade MEMS-based inertial measurement unit (IMU). These are combined on a custom designed georeferencing unit, together with a radio modem for reception of GNSS correction data and real-time processing components. Custom designed GNSS processing and GNSS/IMU integration algorithms allow for a position and orientation determination of the UAV within a centimetre/degree accuracy range (Eling et al., 2015). The main mapping sensor of the system is a 5 MPix RGB camera with a global shutter and the possibility to change the viewing angle between nadir and oblique positions. A high performance processing PC (IntelCore i7, 8GB RAM) enables logging and processing of sensor data.
During the flight, the images taken with the main mapping sensor are accurately tagged with position and orientation information from the georeferencing unit. In a later processing step, which runs on the ground during or closely after the flight, these geo-tagged images are processed to dense point clouds using a structure-from-motion pipeline (Furukawa and Ponce, 2010). These point clouds are georeferenced with an absolute accuracy in the order of centimetres and can be used for monitoring tasks as described in the following.

Building inspection scenarios
This section discusses two possible scenarios where semantics is playing an important role. The first use case is related to Figure  1  Facade inspection A use case where semantics is of high relevance has been depicted in Figure 1. In order to answer the query whether a person is needing assistance in the second floor of the back side of a concrete building requires semantic knowledge on the facade objects such as balconies or windows. The underlying building models are, however, in the second level of detail (LoD2) in the sense of CityGML where no information about facade structures is available. To this aim, preliminary facade hypothesis are needed to acquire the knowledge on floor numbers and the belonging facade objects such as windows and balconies.
To this end, a reasoning process is performed based on weak observations such as the LoD2 model leading to probabilistically qualified hypotheses such as depicted in Figure 6. Starting from an uninterpreted facade surface from the requested possibly hidden side of the building, the a-priori unknown number of floors is estimated together with facade parts such as windows. This is performed by a combination of combinatorial and stochastic reasoning. The reasoning draws upon constraints on well structured man-made objects and a statistical data base handled by probability density functions. In this context, an underling exhaustive prior knowledge on the location and shape parameters of building objects represents an important pillar to restrict the huge search space of hypotheses. The interested reader is referred to (Loch-Dehbi et al., 2013) and (Loch-Dehbi and Plümer, 2015).
The UAV delivers captured data in the form of photogrammetric point clouds of the requested scene such as depicted in Figure 7 and explained in section 3.2. The model hypothesis from the reasoner as part of the base station are incrementally refined in order to upgrade the LoD2 model towards an LoD3 building model as a part of the geometric and semantic scene. The falsification or rejection of hypotheses is supported based on an incremental parsing process by the use of grammar rules for an automatic 3D building reconstruction. The parser enables a model refinement based on new observations with respect to a weighted attribute context-free grammar (WACFG). Available parse trees acquired from previous interpretations or predictions are handled and adapted in an iterative way using transformation rules if needed. More details can be found in Dehbi et al. (2016).
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B1-2020, 2020 XXIV ISPRS Congress (2020 edition)

Rain gutter inspection
In the same context of building inspection and reconstruction, a further use case to emphasize the importance of semantics consists in the interpretation of rain gutter regions. Geometrically, rain gutters are characterising the connection between facade and roof surfaces. Based on the graphical user interface of the base station, the user has the possibility to select roof surfaces. In particular the according surface related to the questioned gutter is selected and visualized in a separate 2D view. Drawing upon this view, view points which define a path that the UAV has to follow are uniformly sampled along the connection edge between the roof and the neighbouring facade surface. Alternatively, the user can specify own defined path points. Figure 8 shows the hypothesised location of a rain gutter in the facade of the main building in Frankenforst's test area in red. The projection on the 3D point cloud of the building in question reveals the precise localization of the gutter range based on the semantic inquiry. Figure 9 visualizes the result of the semantic interpretation of the geometric map acquired by the use of the mission planning system. The user inquiry aimed at identifying the rain gutter of the main building. The rain gutter fits well to a cylinder model which allows for an autonomous inspection of these building parts. Figure 9: The rain gutter (green) fits well to a cylinder model.

CONCLUSION
In this paper we presented a mission planning system to support automatic UAV explorations and semantic mapping. The latter is drawing upon CityGML's buildings and digital elevation model of the area of interest. These models are the basis for the generation of possible obstacles in form of octomaps to deal with collisions later on. A base station consisting of a graphical user interface allows for the expression of semantic queries. This is based on user interaction which enables to specify flight paths according to the particular mission.
The feasibility of the mission planner has been demonstrated on building reconstruction and inspection scenarios. Hypothesised models of rain gutters which have been specified in the base station have been confirmed by model fitting in the acquired point cloud. A reasoning component allows further for the estimation of building objects not available at the mission planning time. An interpretation of the scene is performed by an incremental parsing process based on weighted attribute grammars.