OBLIQUE AERIAL IMAGERY FOR NMA – SOME BEST PRACTICES

Oblique airborne photogrammetry is rapidly maturing and being offered by service providers as a good alternative or replacement of the more traditional vertical imagery and for very different applications (Fig.1). EuroSDR, representing European National Mapping Agencies (NMAs) and research organizations of most EU states, is following the development of oblique aerial cameras since 2013, when an ongoing activity was created to continuously update its members on the developments in this technology. Nowadays most European NMAs still rely on the traditional workflow based on vertical photography but changes are slowly taking place also at production level. Some NMAs have already run some tests internally to understand the potential for their needs whereas other agencies are discussing on the future role of this technology and how to possibly adapt their production pipelines. At the same time, some research institutions and academia demonstrated the potentialities of oblique aerial datasets to generate textured 3D city models or large building block models. The paper provides an overview of tests, best practices and considerations coming from the R&D community and from three European NMAs concerning the use of oblique aerial imagery.


INTRODUCTION
Oblique airborne photogrammetry is quickly growing and entering the workflow of service providers which are trying to complement (or replace) the more traditional pipeline based only on vertical images. Various camera systems exist, with 2 up to 8 (Octoblique MIDAS) slanted views (Fig. 2) based on middle-or large-format cameras . The virtue of oblique imaging lies in its simplicity of interpretation and understanding for inexperienced users and the revealing of building façades and footprints. These qualities, together with high density and accuracy of point clouds that can be generated with matching techniques, endorse the use of oblique imagery in very different applications, such as 3D city modelling, land-cover updating, urban classification, identification of unregistered buildings, damage assessment, building registration, parcel boundary determination, etc. In Fritsch et al. (2012) the name "All-In-One Photogrammetry" was proposed as oblique aerial photogrammetry offers more potential for information extraction to provide -besides 3D point clouds, detailed Digital Surface Model (DSM) and true orthoimages, also 3D roof structures and corresponding 3D building models needed for updating 3D urban databases. The necessity of having rich information in urban areas is growing along with the urbanization of the world, understood as a population shift from rural to urban areas. According to the UN World Urbanization Trend (http://esa.un.org/), in 2050 about 66% of the world population will live in urban areas. By 2030, the world is projected to have 41 mega-cities with more than 10 million inhabitants. It is clear that a better mapping, understanding and management of the urban ecosystem is required including new environmental applications that allow us to characterize our cities for better planning of urban growth. In such a context, the need for a realistic (3D) modelling of cities, including all its volume and morphology for overlaying and exploiting all thematic and administrative information becomes evident. Therefore, 3D city models will play an increasingly important role in our daily lives and will become an essential tool of the modern city information infrastructure which is integrated in the so-called "Smart City" concept. From an economic point of view, Smart Cities are expected to be a 1.5 trillion-dollar market in 2020, with a 25% of growth rate per year (Frost & Sullivan, 2013).  Figure 3: A footprint comparison -over the same area and with the same GSD -among a traditional nadir flight (a -ca 60 images) and an oblique flight coverage (b -ca 700 images) which requires more strips and thus more flying costs .
Hence, all these numbers and economic forecast support the rapidly growing sector of oblique cameras as well as investments in such sector. National Mapping Agencies (NMAs) are still relying on the traditional workflow based on vertical photography but changes and convergence towards oblique imagery are slowly taking place also at production level. Today some mapping agencies have already run some internal tests whereas others are discussing on the future role of this technology and how to possibly adapt their production pipelines. The same applies to photogrammetric companies which are more and more interested in testing and using oblique cameras in order to deliver more innovative products. EuroSDR, the international organization composed of European NMAs and research organizations of most EU member states, is following the development of oblique aerial cameras since 2013, when a research activity was created to continuously update its members on the developments in this technology. As camera systems consolidated, a new momentum was given in order to explore more deeply the potential of oblique aerial cameras for national mapping purposes. At the same time, research institutions and academia demonstrated the potentialities of oblique aerial datasets to generate textured 3D city models or large building block models. NMAs need practical and costeffective methodologies to remap areas with legacy issues or create, update and manage (3D) cadastre information. For NMAs substantial technical and operational benefits become obvious although not many clear statements or best practices have been yet reported. The paper presents R&D experiences with oblique aerial imagery. Inputs from selected NMAs are also given, with their experience, foreseen advantages and benefits.

R&D EXPERIENCES WITH OBLIQUE IMAGERY
Since some years, FBK Trento and ITC/Twente University are both running investigations on aerial oblique imagery (Gerke and Nyaruhuma, 2009;Nyaruhuma et al., 2012;Xiao et al., 2012;Rupnik et al., 2013;Rupnik et al., 2014;Remondino and Gerke, 2015;Rupnik et al., 2015). The studies are driven by the clear problems in processing oblique image blocks and by the numerous applications opened-up by oblique aerial views.

Research issues
Automated aerial triangulation of oblique blocks is the main research topic. Indeed, the much more complex image network geometry and the larger number of images ( Fig. 3) cause problems to traditional triangulation strategies tailored for nadir image blocks. Problems arise also due to scale and radiometric changes, low similarity due to wide baseline configurations as well as few tie points across viewing directions. Strategies go from multi-step orientation methods to sub-block independent adjustment, connectivity maps or relative orientation procedures . Direct geo-referencing provided by GNSS/IMU onboard sensors is still not applicable to all flights and a bundle adjustment refining is still advisable (Moe et al., 2016). Once the nadir and oblique imagery is oriented, dense image matching (DIM) methods are applied in order to retrieve dense 3D point clouds and, successively, (true) orthoimages. DIM methods  are recently less investigated as actual state-of-the-art multi-view pixel-based algorithms are sufficiently reliable and productive. Coupling oblique & nadir images, there is a substantial increase of completeness in comparison to nadir-only datasets (Fritsch and Rothermel, 2013). Indeed due to the additional slanted views, façade points can be extracted, facilitating the automatic derivation of 3D building models (Haala et al., 2015;Toschi and Remondino, 2015 -Fig. 4).

Benchmark
In the research community, a common way to evaluate new platforms, data and algorithms is to prepare and share benchmarks. Inspired by this concept and previous successful results (Rottensteiner et al., 2013;Haala, 2014), within an ISPRS Scientific Initiative and in collaboration with EuroSDR, a multiplatform photogrammetry benchmark  1 was proposed in order to (i) create and manage new image datasets consisting of different kind of images over the same area and (ii) assess the accuracy and reliability of the current methods in the calibration/orientation as well as integration of photogrammetric data for dense point cloud generation and feature extraction. The benchmark consists of airborne oblique images (acquired with an IGI PentaCam camera flown by Aerowest), nadir and oblique UAV images (acquired with a multirotor DIJ S800), convergent and redundant terrestrial images of

Other research activities
First experiences with oblique imagery and automated aerial triangulation were presented in Jacobsen (2008) and Wiedemann, and Moré (2012). Yang et al. (2012) proposed a multi-stage algorithm based on SIFT matching to improve automatic aerial triangulation results of oblique imagery. Xiao et al. (2013) used approximated exterior orientation elements in NAIF algorithm (Nicer Affine Invariant Feature). Karel and Pfeifer (2015) presented an in-house tool (Oriental) for processing oblique datasets. Hu et al. (2015) proposed a reliable feature matching method using various spatial relationships and geometrical information for problems raised by large viewpoint changes, image deformations, blurring, etc. Frommholz et al. (2015) delivered semantically annotated LOD-2.5 CityGML objects from oblique imagery. Moe et al. (2016) present how oblique imagery affect the traditional photogrammetric production pipeline, comparing software and products. Ostrowski and Bakula (2016) evaluate the performances of commercial software in processing image blocks acquired with oblique camera systems.

OBLIQUE IMAGERY AT OS UK
Ordnance Survey (OS) UK, the national mapping agency for Great Britain, is one of the world's largest producers of maps and since April 1 st 2015 it has operated as Ordnance Survey Ltd, a government-owned company.

Needs for using oblique flights/images
OS routinely captures aerial imagery from two Cessna 404 survey aircraft and Vexcel UltraCam Xp cameras. These provide data for the update of topographic mapping, the production of the OS MasterMap Imagery Layer product, the creation of national-wide DSMs/DTMs and the OS MasterMap Topography Layer -Building Height Attribute (derived from the available DSMs/DTMs). Although we have always been interested in the capture of 3D data, at the moment OS UK does not produce 3D city models as a standard product. Having tested several methods of creating 3D models from our standard imagery (with overlaps up to 80% forward and 60% sidelap) we were very interested in investigating what could be produced from oblique aerial imagery at production level. Our main requirement for oblique imagery is to produce 3D models of urban areas, for use in planning, energy monitoring, noise modelling and various "smart city" applications. For these applications, a high resolution dataset is required (e.g. imagery at 10cm GSD and point clouds with a point spacing of around 10cm). The final product of an oblique image dataset is not immediately obvious, as it often depends on the requirements of the end-user. Products could include a 3D point cloud, a 3D mesh, a true-orthoimage, a 3D city model (e.g. in CityGML form) or simply an oblique image viewer.

Advantages
One potential use of oblique imagery is the capture of topographic vector data directly from a 3D point cloud generated by image matching on oblique imagery. This has the advantage that 3D information could be collected within a monoscopic environment, without the need for expensive stereo software and hardware. A second advantage of oblique imagery is the ability to view features which would be occluded in nadir aerial photography. One of the obvious instances of this is the ability to see building facades from multiple viewing angles. In addition, narrow streets in dense urban areas can be completely occluded in areas of nadir images away from the principal point, while they are more likely to be visible in at least some of the oblique images.

Experiences
In November 2014 Leica Geosystems kindly flew an area of Bournemouth on the English south coast, using their RCD30 penta oblique camera. One of the goals for OS UK was to generate 3D building models using the Building Finder and City Modeller modules of the Tridicon/Hexagon software (Fig. 5). City Modeller requires as input building footprints, it generates dense point clouds and then it tries to fit a pre-defined roof types (flat, hipped, gable, etc.). The Building Finder module generates 3D building models without requiring existing building footprints, detecting faces from which 3D building models are assembled. During our testing we noted that both modeling modules use only the nadir images from the dataset to create the geometry, and not the whole set of oblique images. The best results were obtained when a 3D point cloud was generated from the full set of oblique imagery using external software (Acute3D) and was then imported into the Tridicon system. We found that Tridicon was good at finding simple buildings, even when no building footprints were input. However, all but the simplest building models required some manual editing (again within Tridicon) in order for them to be truly representative of the buildings they depicted. We have used Acute3D software to create 3D point clouds from the Bournemouth oblique imagery and found that it does produce cleaner and more representative point clouds than those produced from conventional overlapping nadir imagery. Further testing on the resulting data will be carried out to determine the effectiveness of such data in a non-stereo data collection process.

Limitations
In order to make the most of oblique imagery, the processing flowline at OS UK would have to be changed considerably, especially as the current flowline is strongly linked to the camera system used (e.g. UltraMap software dedicated to the UltraCam sensors). A potentially major limitation of the oblique image acquisition process is the overlap required of the nadir images in order to produce a 3D point cloud and associated 3D models. In a production flowline, an increase in the speed of acquisition can sometimes take precedence over a small increase in the resolution or quality of the final product. For this reason, imagery for topographic map update is often flown with an 80% fore/aft overlap and a 30% sidelap. With oblique imagery a greater sidelap would be required in order to make the most of the imagery and this, together with the smaller footprint of oblique camera systems, will lead to the need for many more flight lines and consequently a more expensive aerial survey process.

Vision
For the future we can see a requirement for oblique imagery to give a more detailed depiction of urban areas and to create high resolution 3D datasets. Our investigations have shown that oblique imagery can be used to produce a more representative 3D point cloud than that produced using multiple-overlap nadir imagery. At the moment, this requirement is not yet compelling enough to result in a commitment to operational use of oblique imagery at OS UK. We will continue to monitor the hardware and software and to develop our use cases for oblique imagery, in the expectation that it will become an operational data collection method in the near future.

OBLIQUE IMAGERY AT OS IRELAND
Ordnance Survey Ireland (OSi) is the NMA of Ireland. It was established in 2002 as a body corporate and successor to the former Ordnance Survey of Ireland, originally formed in 1824.

Needs for using oblique flights/images
Ordnance Survey Ireland (OSi) is currently focusing on its geometry positional accuracy resolution in light of modern technological advances and in particularly in urban areas that require Positional Accuracy Improvement (PAI). To date a number of applications have been developed to assist with PAI. PAI is not the only option that OSi is considering to deal with these issues. Our preferred option would be to address these accuracy issues by re-capturing the landscape features. Having just completed the implementation of our new data model and data management system that is designed to store and manage 3D data, we have embarked on testing oblique imagery to develop a methodology to automate/semi-automate a process to survey and digitize the landscape in 3D as solid objects, to accuracies that are related to image resolution.

Advantages
There are various advantages of using oblique imagery for this purpose, particularly: (i) reduced occlusion in city landscapes due to increased angled views, (ii) better modelling of 3D surfaces such as sheers, overhangs, canopies and underpasses. The ability to apply the imagery automatically for texturing is a major advantage. A particular advantage is if buildings do not change shape but do change business occupancy. It thus becomes possible to re-texture buildings reflecting their current function automatically. OSi is currently in the process of merging with the Property Registration of Ireland and the Valuation Office of Ireland to form a new organisation called Táilte Éireann. Work carried out by the other two entities of this merger will benefit greatly in their ability to improve identification and interpretation from an office environment thus facilitating a much more efficient evidence-based decision making process.

Experiences
OSi has commenced the design of flowlines to integrate into its existing infrastructure. An area in the north east of the country was flown with Hexagons Pentapod Camera System (Leica RCD30). The newly captured imagery was triangulated based on the national active GNSS network and a point cloud was extracted from the imagery. The intention was to then automatically extract 3D buildings from the generated point clouds using Tridicon/Hexagon applications (City Modeller and Building Finder). We then compared results in terms of positional accuracy and percentage success in building identification (Fig. 6). Our conclusions when comparing both outcomes was that the City Modeller produced a higher percentage of success in identifying buildings. Where buildings were of more uniform shapes, typically in sub-urban areas, the success rate was higher than areas such as city centre locations where building shapes were more unique.

Limitations
How will the fusion of the point cloud data extracted from nadir and oblique images be managed and will there be issues with relative accuracies? Target identification and pointing accuracies in the geo-referencing process can be more challenging. The volume of images in a project increases dramatically affecting image management, processing times, bundle adjustments, point cloud generation, storage and archive operations. Camera calibration is also much more complex with respect to nadir-only cameras. A further limitation during the tests we conducted was that the software application used to process the aerial data for the construction of the 3D buildings, did not consider façade points. This may have had an effect on the outcome.

Vision
As we complete the merging process into Táilte Éireann, we will be in a position to provide the definitive source of all required geospatial information for land management in Ireland from a single authoritative source. OSi has a vision to utilise oblique imagery initially in urban areas to assist in improving positional accuracies. As the National Mapping Agency we will move our data offering into the 21 st century by enhancing our Prime2 data model to store, manage and provide textured 3D real world objects.

OBLIQUE IMAGERY AT ICGC SPAIN
The Institut Cartogràfic i Geològic de Catalunya (ICGC) is the regional mapping agency of the Government of Catalonia (Spain). From its creation in late 1982, ICGC activities have been focused on providing valuable geoinformation to territorial planners and decision makers. ICGC produces topographic databases in urban areas at 1:1.000 scale that results in LOD1 building models (2.5D) in vector format as a by-product (Fig. 7). The roadmap of ICGC includes the 1:1.000 urban map upgrade to the LOD2 standard. For modelling urban areas, ICGC uses also LIDAR-based solutions for obtaining point cloud data that will be used to develop environmental applications (such as analysis of solar potential) or obtain LOD2 building models.

Needs for using oblique flights/images
To improve the realistic appearance of its products, ICGC decided to add real texture to existing models. Moreover, oblique images are tested to generate 3D textured meshes of urban environments for enhancing the benefits of the very high resolution orthoimages and the 3D building models in terms of geoinformation contents and user's experience. As a consequence, oblique imagery has become an important part of ICGC's strategy for urban mapping.

Advantages
ICGC believes that implementing a production pipeline based on oblique imagery would help to improve the quality of its products, primarily in terms of visual quality, in the near future also in terms of geometric quality. The combination of dense image matching and multiple view reconstruction algorithms enables the collection of façade information, becoming the natural evolution of twodimensional orthophoto layers and allowing the creation of 3D city models or 3D orthoimages.

Experiences
According to ICGC's experience, the new oblique camera systems offer a better performance when approaching the 80x80 overlap configuration with a GSD in the order of 6-7cm. Reducing overlap to the usual 60x60 or 60x40 will result in noisier correlation and higher occlusions. At the same time, due to the fact that most of the available systems rely on medium format cameras (using Bayer filter for multiband capture), the apparent resolution is smaller than the nominal flight GSD and, therefore increasing flight altitude is not recommended. Using synthetic targets and images acquired with the Leica RCD30, ICGC has measured effective resolutions of 1.98xGSD, 1.32xGSD and 2.21xGSD for the Red, Green and Blue bands, respectively. Considering the typical sizes of oblique aerial blocks in a real production environment, ICGC has switched to parallelizing processes and established infrastructures, policies and mechanisms for handling large datasets with the level of quality assessment and quality control expected in a standard photogrammetric workflow. From the executed tests ICGC could produce photo-realistic polygonal city models (Fig. 8) to improve the realistic appearance of building models and with the possibility to add semantics and share such info over the web.

Limitations
The high overlap at low altitude and the multi-capture system challenge the flight execution and processing capabilities. Covering large areas requires high amount of flight hours so illumination conditions should be carefully considered. These are more complicated in oblique imagery due to the very different illumination situations that affect the different viewing cameras for a single shot. Processing oblique flights with high overlap involves dealing with very large datasets (Table 1) and requires adapting data management and processing as well as data preservation policies for product scalability prior to move from prototyping to production phase. Current off-the-shelf workflows for image processing, aerial triangulation (AT), dense point cloud and 3D data generation have important room for improvement. Software providers are trying to catch up with these problems which result in failure of AT or manual intervention for subdividing blocks that will require further actions for joining the resulting products. Finally, 3D data visualization and exploitation is limited by few systems with performant 3D capabilities (nor with vector data and even worse with 3D meshes) and it is required to go further in format standardization and system interoperability.

Vision
Further improvements in oblique data processing and management are quickly foreseen. Similarly to the improvements in DSM generation thanks to the Semi Global Matching approach, oblique imagery will continue pushing developments towards the automatization of LOD2 building extraction based on multi-view algorithms. This will be achieved not only due to the future improvements of processing methods, but also because a reduced time-to-market will change user's expectation with respect to the current and classical approach. A similar roadmap should be expected for 3D textured meshes -with an improvement on the quality of the 3D reconstructions using new algorithms and combining different data sources (planar constraints, LiDAR data, etc.) -and a growth of new applications combining 3D meshes and external information to derive structured information that goes beyond the 3D ortho by adding also a semantic interpretation.

CONCLUSIONS
The paper presented the growing interest and availability of oblique imagery and data, in particular for NMAs. Oblique airborne photogrammetry is indeed quickly developing and entering the processing / production pipeline of companies and agencies which are trying to complement (and in some areas replace) the more traditional workflow based only on vertical images. Many applications embrace the advantages of airborne slanted viewing geometry which comes close to human perception of scenes. Most of European NMAs still rely on the traditional workflow based on vertical photography but changes are slowly taking place also at production level. Experiences and considerations from 3 NMAs belonging to EuroSDR were reported. Although weaknesses and problems are still present in data processing and management, the rise of airborne oblique systems will foster a variety of urban applications and force the geospatial sector to face new challenges like: more flight lines to have similar coverage as nadir-only flights, improve actual bundle adjustment approaches, smooth integration and visualization of 3D geo-data, automated generation of LOD2/3 for large urban areas, common acceptance of 3D textured and segmented meshes, semantic 3D city models, etc.