RESEARCH AND APPLICATION OF HISTORICAL 3D MODELING BASED ON ARCHIVED NON-OBLIQUE AERIAL PHOTOS

Nowadays, the oblique and multi-view, large-overlap aerial photography and airborne LiDAR are the main sources to build the 3D scene model. However, most of our archived aerial photos are acquired by non-oblique, normal photography. Because of low resolution, low overlay and poor model texture, there were less relative research and application. With the development of pixellevel matching technology, especially the application of Semi-Global Matching (SGM) and Multi-View Stereo (MVS) algorithm, the normal (non-oblique, non-large overlap) aerial photos could also be explored to restore the dense Digital Surface Model (DSM) and 3D scene model. In this paper, the method of the 3D scene modelling with the non-oblique aerial photos are summarized into 4 steps consisting of Data preprocessing, Ground Control Points (GCPs) collection and aerial triangulation (AT), DSM extraction and editing, 3D modelling and visualization. For the archived non-oblique aerial photos, including the aerial photographic films, digital frame photos and push-broom aerial data, the key steps of the 3D modelling method with these non-oblique aerial photos are discussed. Based on the experiments, the method can effectively explore the archived normal aerial data for large range restoration, 3D restoration, time series change detection and etc., providing new valuable spatio-temporal data for the urban historical research. * Corresponding author


INTRODUCTION
Three dimensional (3D) scene model, as one kind of real-world 3D virtual display model, is normally generated with a panoramic view of surface object from multiple angles, and is usually composed of Digital Surface Model (DSM) in the form of Triangle Irregular Network (TIN) and the corresponding texture information (Chen and Feng et al., 2018). Due to the fact that the three-dimensional real-world model truly represents the real world, with the 3D scene model, viewers can enlarge, shrink, move and view the miniature real-world model from multiple angles, which can be used for simulation, planning and analysis of the real world, such as construction of digital city, disaster analysis, road design and etc. (Xie and Lv, 2020;Guan and Wang et al., 2017;Du, 2020). Nowadays, the tilt and multiview, large-overlap aerial photography and airborne Light Detection and Ranging (LiDAR) to obtain the 3D scene model is the research hotspot and development trend. Most of the researches focus on the aerial triangulation, texture reconstruction of the 3D scene model with tilt photogrammetry, LiDAR both airborne and ground based, as well as combining the two methods (Zhang, 2017;Li, 2016;Lu, 2019).
However, as we know, most of our accumulated archived aerial data, the main objective data sources for restoring historical scenes, are acquired by normal (non-oblique, non-large overlap) photogrammetry with low resolution, low-overlay and poor model texture. There are less relative research and application on the method of 3D scene modelling with this traditional photography. As the development of pixel-level matching technology, especially the application of Semi-Global Matching (SGM) (Shao, 2019) and Multi-View Stereo (MVS) algorithm, the normal aerial photos could also be explored to restore the dense DSM and 3D scene model.
In this paper, with three kind traditional archived non-oblique aerial photos, including the aerial photographic films, digital frame photos and push-broom aerial data, the key steps of the 3D scene modelling method with these non-oblique aerial photos are discussed.

MEHODOLOGY
The work flow is shown in Figure 1 and the yellow box labelled procedures can be improved for the archived aerial photos.
From the Figure 1, the method of the 3D scene modelling with the archived non-oblique aerial photos are summarized into 4 parts in this paper: Data preprocessing, Ground Control Points (GCPs) collection and aerial triangulation (AT), DSM extraction and editing, 3D modelling and visualization.  Figure 1. Work flow of 3D scene modelling with archived non-oblique aerial photos 1) Data preprocessing. The purpose of data preprocessing is to initially establish the relationship both among the flight strips and between the adjacent photos within the same flight. For the aerial photographic films, image except aerial routes sorting, data preprocessing contains digitized scanning of films and internal orientation. For the archived photos, the route sorting based on existing reference documents is very important. As in some cases, there are no reference materials for rout sorting, fast matching algorithms based on image features considering the size and overlap of photos are helpful (Lv and Li, et al., 2019).

2)
GCPs collection and AT. Tie point matching, GCPs measurement and aerial triangulation are the core to restore aerial block. For the tie point matching, free network adjustment is carried out. Due to the problem of rotation in some archived films, the judgment of rotation angle (0, 90, 180 and 270) could be added. Considering the scanning errors of films, the threshold value of matching windows should be increased compared with digital photos. For the GCPs collection, GCPs could be automatically extracted from archived documents. Because most of the historical image GCPs are paperbased preservation, a small number (3-5), evenly distributed of GCPs can be collected for original bundle block adjustment and aerial triangulation, and then optimized by adding more GCPs gradually.
3) DSM extraction and editing. Accurate and dense DSM is the base of the 3D scene model. SGM matching is applied to extract pixel-based DSM. For SGM matching, the Census Transform (CT) was selected as matching cost value without considering setting the initial parallax value. Considering there are flying points and holes in the DSM automatically matched, the DSM needs to be edited before 3D modelling step. Based in the experimental, the dense matching points greater than 5-10 times the average elevation and negative values are set as noises and filtered through threshold setting. The holes on DSM caused by insufficient overlap and shadows are filled with the lowest value to ensure the perpendicular characteristics of sides of buildings.

4)
3D modelling and visualization. Based on the matched dense DSM and the oriented aerial photo data, the 3D scene model can be generated. Based on the 3D platform such as Cesium platform, the archived 3D data can be visualized.
There are some problems caused by the normal photogrammetry with the archived data. For the different problems, the solutions for the model editing and method optimization are suggested as follows: 1) Model edge errors and range optimization. Different from digital aerial photos, film data is usually equipped with mechanical and physical frame marks with internal orientation (Figure 2). The frame marks have an impact on tie point matching, DSM extraction by SGM, and 3D model texture reconstruction, which will lead to model DSM gross error and model wrong texture. Therefore, a method of model range optimization by excluding the edge containing frame marks is proposed to solve this problem. The same method should be applied to removal the background edge for the ADS80, the push-broom aerial data after aerial triangulation.
2) Mismatching and elevation anomaly elimination. It is found that on the final 3D scene model, there are some frequent mismatching cases: at the edge of block with less overlap, in the some building shadow area with poor features and on the dense traffic road with moving vehicles. Especially the last case, due to time difference, the cars mismatches lead to elevation anomaly. The reason for the error area of 3D model is that there are still error areas in dense DSM. This problem generally has a small influence area, so it is necessary to edit DSM and then correct the 3D model. Setting the maximum noise threshold, the slope threshold is also set with a layer mask of road, building shadow area and for the final model, the edge of the block was abandoned (Figure 3).

EXPERIMENTS AND DISCUSSION
With the proposed work flow, the experiments with three main archived aerial sensors including aerial photographic films, digital frame photos and push-broom aerial data are carried out to verify the method.

3D modelling with aerial photographic films
With the method, the historical aerial panchromatic films of RC-10 obtained in 1989 are explored for 3D reconstruction of Forbidden City in Beijing. The flight scale is about 1:3000 with Side overlap 30% and heading overlap 60%. The resolution of final DSM is pixel-based with the resolution of 0.07m. The original films are preprocessed by film scanning, aerial routes sorting and internal orientation. 5 GCPs are used for original bundle block adjustment and aerial triangulation. More GCPs are collected by importing control point list based on the original block. Range optimization is adopted by excluding 10% of edge area to reduce the influence of the frame marks.
Although the panchromatic film with no colour texture and the 3D scene model is like hand drawn sketch, the details of the building such as the steps of the Hall of Supreme Harmony in the Imperial Palace can be identified and even the single tree can be accurately extracted, which is of historical value and could provide special and useful data for the research of Forbidden city in 1980s.

3D modelling with digital frame photos
An additional experiment is carried out with frame digital aerial camera data. The professional cameras, such as Digital Mapping Camera (DMC), UltraCam (UC), SWDC-4 are widely used because of the high imaging quality, wide coverage of single image and high accuracy to meet the requirements of large-scale mapping with the same overlap as the films. In Beijing, before 2015, the archived digital frame camera was used in Beijing every year.
By using the method, the experiment with UC digital frame aerial photos is carried out. Image data processing includes route sorting and tie point matching. Different from the film， the routes can be directly sorted according to POS data acquired with the aerial photos. For the tie point matching, the search radius and correlation threshold depending on the terrain types. The matching search radius of plain and mountain are 1800 pixels and 2500 pixels, while the correlation threshold of plain and mountain is 0.92 and 0.85.
GCPs are extracted from GCP database and collected in images by image matching. Different aerial survey areas could be merged depending on the machine performance to improve the processing efficiency and reduce the edge differences. The plane error and elevation error threshold values increase to 2.0mm and 4.0mm. With the optimized parameters of plain and mountain area, the pixel-level DSM is extracted, interpolated and denoised.
For the plain area, The DSM holes are filled by the lowest elevation filling method, and the minimum height of the building is set as 2.0. While the smooth interpolation method is applied in the mountainous area with the interpolation window size of 7 × 7 pixel to keep the terrain continuous and smooth.
As most of the frame digital aerial photos include red, green, blue, near infrared 4 bands, the 3D scene model with true or false colour texture can be finally obtained. Finally, the 3D scene models with resolution of 0.2m for the plain and 0.5m for the mountain area were generated for the whole Beijing with total area of 16410 km 2 .
The Cesium platform is used for the visualization of 3D models, the integration and comparison of 3D models in different periods.
Compared with the 3D scene models obtained by tilt photography of RC-30 in 2017 with resolution of 0.05m, it obtains satisfactory results in large range, abundant texture details of buildings, road, vegetation, etc. and the irregular buildings could be restored with complete shape and detailed texture in the 3D scene model ( Figure 5). Compared with the elevation point of topographic map, the elevation accuracy of plain and mountain area is 0.68 m and 1.79 m respectively. For the plane precision, 5% models are evaluated by 40-50 checking points collected manually. Compared with the Digital Ortho Map (DOM), the plane accuracy of plain and mountain area is 0.97 m and 2.15 m respectively. In Figure 6, it is also shown that the final model could be effectively integrated with the tilt model at the same position. Figure 6. Integration of the both 3D scene on Cesium platform (the upper part is 3D scene model of UC and the lower part is that of RC-30).
Through the large-scale model construction of plain and mountain areas, it is found that there are two disadvantages in the 3D modelling based on non-oblique frame aerial photos. On one hand, compared with the tilt photography model, the side texture is blur and sometimes missing, especially for the highrise buildings with shadow area (Figure 7 (a)); On the other hand, because of the low overlap, and it tends to cause mismatching and DSM holes (Figure 7 (b)).

3D modelling with push-broom Data
Leica ADS80 is a push-broom airborne digital aerial photogrammetry system. It integrates high-precision inertial navigation and orientation system and global positioning system. With 12000 pixels three-line array CCD scanning and professional single large aperture lens, in one flight it can obtain forward, nadir, backward images with 100% overlaps, to form continuous seamless panchromatic stereo images, RGB and infrared images (Wang et al., 2009). Since 2016, Beijing has launched an annual flight for DOM and DLG production with ADS80.
With the ADS80 with side overlap of 15% acquired in 2016, 3D scene model was restored. The whole modelling process took 2.5 times longer than that of the other two sensors with worse model results compared with that of frame photos: loss of some highrise buildings, more matching errors, more texture holes, and vaguer edges (Figure 8).
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2020, 2020 XXIV ISPRS Congress (2020 edition) However, the pixel based dense DSM data are still useful for urban surface change detection. By using DSM matched with ADS80 in 2016 and 2017 in southern 6 th area of Beijing, simple elevation difference can effectively extract urban demolition and construction area (Zhu, 2019). It is worthy noting that the work flow can be further extended to the 3D reconstruction with pushbroom satellite stereo images of the same period or different periods.

CONCLUSION AND PERSPECTIVES
Based on the study and experiments above, the method can effectively explore a large number of normal archived historical aerial data for large range restoration, time series change detection and etc., providing new valuable spatio-temporal data for the urban historical research. It could be concluded that: 1) Although the historical aerial photos of vertical photography have low overlap and low resolution, they can still be applied to construct 3D scene models through SGM same as the tilt photography. The advantages of vertical photography lie in low cost, making full use of existing data, obtaining 3D scene models with full coverage, uniform accuracy and time sequence, and integrating with high-precision models. One of the disadvantages is the poor or even missing side texture, affected by the low overlap and the time difference of moving vehicle acquisition. The other one is mismatching at the edge of survey area, shadow area and dense traffic roads. DSM elevation and slope threshold and minimum value filling principles proposed are helpful to eliminate mismatches, and filling the DSM holes.
2) The accuracy of the model is related to the resolution of the image and the higher the resolution is, the higher the accuracy of the model is. From three data sources, the effect of film and digital frame photos is better than that of the push-broom data. Because of the panchromatic effect, in addition to the shortcomings mentioned above, 3D scene model built by push-broom data also has the obvious defect of high-rise mismatching and vague edges of the buildings.
3) At present, Context Capture that can produce 3D scene models same as the tilt photographic images can be used for the process the traditional non oblique frame aerial photos, while some photogrammetric software can process the push-broom ADS80. In view of the historical scanned aerial photos, it is suggested to increase the strategy of fast strip connection and range optimization to reduce the influence of edge background and inner orientation marks on matching and model texture. In addition, due to the long history of some historical scanned aerial photos, it is recommended to collect and construct a historical image control point database, make full use of the existing passing point data, to assist construction of 3D scene models by automatic matching of image GCPs.