SAPC – APPLICATION FOR ADAPTING SCANNED ANALOGUE PHOTOGRAPHS TO USE THEM IN STRUCTURE FROM MOTION TECHNOLOGY

The documentary value of analogue scanned photographs is invaluable. A large and rich collection of archival photographs is often the only source of information about past of the selected area. This paper presents a method of adaptation of scanned, analogue photographs to suitable form allowing to use them in Structure from Motion technology. For this purpose, an automatic algorithm, implemented in the application called SAPC (Scanned Aerial Photographs Correction), which transforms scans to a form, which characteristic similar to the images captured by a digital camera, was invented. Images, which are created in the applied program as output data, are characterized by the same principal point position in each photo and the same resolution through cutting out the black photo frame. Additionally, SAPC generates a binary image file, which can mask areas of fiducial marks. In the experimental section, scanned, analogue photographs of Warsaw, which had been captured in 1986, were used in two variants: unprocessed and processed in SAPC application. An insightful analysis was conducted on the influence of transformation in SAPC on quality of spatial orientation of photographs. Block adjustment through aerial triangulation was calculated using two SfM software products: Agisoft PhotoScan and Pix4d and their results were compared with results obtained from professional photogrammetric software – Trimble Inpho. The author concluded that pre-processing in SAPC application had a positive impact on a quality of block orientation of scanned, analogue photographs, using SfM technology.


INTRODUCTION
Within the past decade, digital cameras introduced to the market have totally driven analogue photographs out of photogrammetric production.However, the documentary value of archival photographs is irrefutable.Vast collections of analogue aerial photographs are often the only source of information about a state of an object in the past.These datasets are used more and more frequently by specialists in various fields, e.g., archaeology, forestry or geography (Nocerino et al., 2012;Kunz et al., 2012).Scanned analogue photographs are usually processed into geo-referenced products: orthomosaics, digital surface models (DSM) or point clouds.In environmental and geographical applications, archival orthomosaics allow change detection of the landscape over the years.Measurement of terrain deformation or forest canopy growth is possible with the use of historical DSM (Clery et al., 2014).The fact that the above-mentioned products are more and more often found in GIS databases is worth mentioning here.Nonetheless, a standard approach to scanned analogue photographs processing requires complex, professional photogrammetric software (e.g., Trimble Inhpo, Hexagon ImageStation) and expert knowledge.This fact excludes the possibility of using the full potential of archival photographs by users, who are not associated with photogrammetry in their daily works.Photogrammetric software used for the aerotriangulation process requires the knowledge of camera specification and approximate values of exterior orientation parameters.In the case of archival datasets, this information is hardly ever provided together with scanned analogue photographs.An alternative approach to archival photographs orientation which does not require information from camera calibration report is the Direct Linear Transformation (Ma, Buchwald, 2012).Unfortunately, the disadvantage of this approach is a need of too many GCPs.Clery, et al. (2014) suggest a method where scanned photographs are registered one by one with a topographic database as reference, however, according to the authors, expected quality requirements have not been obtained yet.Software products, based on Structure from Motion (SfM) technology and which popularity is still growing (e.g., Agisoft PhotoScan, Pix4D), are alternative for more expensive, professional photogrammetric systems.Most SfM platforms are now fully automated and userfriendly.SfM software producers are agreed that their software products are not dedicated for scanned analogue photographs processing, On the other hand, they claim that in some cases it is possible to achieve satisfactory processing results.SfM holds great promises for the quick 3D reconstruction from archival data, but its accuracy is very much limited by the quality of the scanned photographs (Gomez, 2012).Therefore, this approach cannot be considered as a plug and play methodology (as in the case of working with digital photographs) with regard to high unpredictability of obtained results (Bakker, Lane, 2017).Working with archival datasets requires the thorough understanding and careful application of these software (Bakker, Lane, 2016).Different principal point position in each photo, not the same resolution and a photo frame with fiducial marks are main problems in case of using scanned, aerial photographs in SfM technology.The solution to the aforementioned problems caused by the specific character of scanned aerial photographs is data pre-processing.A two-stage approach to archival photographs orientation offered by Goncalves (2016) consists in taking advantage of SfM algorithm after all the scanned photographs have been transformed into the same size and the same position with respect to the photographic coordinate system, with the use of affine transformation.In this method, measurement of fiducial marks in the scanned photographs was manual which considerably extends the processing time.The author himself, however, sees potential for application in the offered approach.Integration of photogrammetric methods and algorithms known from Computer Vision (Schindler, 2012) makes automation of the whole process of adapting scanned aerial photographs to software based on SfM technology, in order to use their full potential of these data, possible.

SAPC APPLICATION DESCRIPTION
The author's open-source SAPC software (Scanned Aerial Photographs Correction) (Figure 1A) is designed to transform scanned aerial analogue photographs to the form with specificity similar to digital images.Its main task is data pre-processing thanks to which, through the process of self-calibration, a set of scanned photographs is treated as being taken with the same camera (as it has happened in reality).SAPC does not require any a priori information from a camera calibration report, such as photo-coordinates for fiducial marks or a principal point.Application does not determine interior orientation of camera.Python 3.5 programming language and OpenCV 3.1 library were used to develop processing that includes i.e. template matching and affine transformation of scanned images.Images, which are created in the applied program as output data, are characterized by the same principal point position in each photo (that is, its pixel coordinates are constant) and the same resolution through cutting out the black photo frame.Additionally, SAPC generates a binary image file, which can mask areas of fiducial marks.Content of each photography remains unaffected.The first step of the technological process of archival photographs processing in SfM software products is using SAPC applicationdata pre-processing before camera self-calibration and aerotriangulation processes are applied (Figure 1B).

Workflow
Figure In the first stage, the application user's task is to find and indicate a fiducial mark, which is clearly seen on a black frame within the scanned photograph.The indicated part of the image will be treated as a template at further stages.SAPC software can only detect optical fiducial marks correctly (not mechanical fiducial marks) (Figure 3).(1 ) where T(x',y') = pixel value in the template image I(x+x',y+y') = pixel value in the source image (in which we expect to find a fiducial mark) R(x,y) = pixel value in the result image (which contains the match metric for each location (x,y)) For each location, there is a value of the match metric concerning the images determined and stored in the result image (R).This image can be treated as the so-called matching quality map.
Template matching takes place where the global maximum of a function R(x,y) occurs.
The next stage involves the accuracy analysis of the conducted detection of fiducial marks in photographs and image geometric transformation.These require certain theoretical assumptions (Figure 4): -all the calculations are done in a pixel coordinates system; -fiducial marks are numbered clockwise; -distances dij and diagonals pij are calculated on the basis of the pixel coordinates determined in the template matching process; -corresponding distances and diagonals in the subsequent photographs are of the same length (e.g.expected value of d12 distance is arithmetic mean of the calculated d12 values in all photographs); -fiducial marks coordinates determined in the template matching process are treated as coordinates in the primary system (different for each photograph); -coordinates corresponding with subsequent fiducial marks in the secondary system (constant for each photograph) are determined on the basis of linear section based on dij and pij mean values.The main goal of SAPC software is transforming input scanned analogue photographs into the form in which the principle point in each image occurs in the same place; that is, its pixel coordinates are constant.Therefore, transforming all the rasters to the same position with respect to the photographic coordinate system is necessary.With regard to possible nonperpendicularity of scanner system axes or occurrence of errors caused by a aerial film shrinkage, SAPC software uses affine transformation taking the effects of these phenomena into account.
After the successful transformation, there are fragments of the black frame left in the result images, including the range of fiducial marks.Using the editor, the user can specify the mask range and export it to a binary image file.In SfM software, exported file can be applying on each photo in a single, automatic process.

Output data
The output data, that is, the cut scanned analogue photographs transformed to the same position with respect to the photographic coordinate system are automatically saved in the catalogue after the application ends the process.The result format of rasters is TIFF with LZW compression.Additionally, the software generates a binary image (mask) in PNG format in the directory path.The exported dataset is fully compatible with SfM software.Exemplary samples of input and output data were shown in Figure 5:

EXPERIMENTAL SECTION
The main goal of the experiment was to investigate the influence of SAPC pre-processing of archival aerial photographs on the accuracy of processing in software products based on SfM technology.In order to achieve that, there were 22 scanned at 14 µm pixel size, analogue photographs at the scale 1:5000 of Warsaw, which had been captured in 1986, were used in two variants: unprocessed and pre-processed in SAPC application.
The process of orientation and generation of point clouds was carried out using two SfM software products: Agisoft PhotoScan and Pix4D.Additionally, the obtained results of aerotriangulation were compared with results obtained from professional photogrammetric software -Trimble Inpho, which is the standard approach to scanned, analogue photographs processing.There were no additional information on the camera specification introduced in the process of aerotriangulation since, in most of the cases of working with archival photographs, the calibration report is not available.Only the value of the focal length taken from photo's frame was used as a reference value in the analysis of self-calibration results.
In case of archival photographs orientation; essentially.there is a problem with GCPs.Coordinates of these points are hardly ever provided together with scanned, aerial photographs.Therefore, three variants of control/check points distribution (Figure 6) were The measurement of photo points in photographs was done once using AgiSoft software, and, subsequently, the measurements were exported to a XML file.In Pix4D, the XML file was imported so that the influence of the differences of photo point indications on aerotriangulation was avoided.
Apart from the comparison of deviations at Check and Control points, the self-calibration results and exterior orientation elements of the photographs were also analysed.Additionally, geometric accuracy of point clouds generated in several variants was investigated.

The influence of GCPs distribution on results of aerotriangulation with unprocessed and pre-processed data
In all the used software: Agisoft PhotoScan, Pix4D and Trimble Inpho, in aerotriangulation process, the same statistical weights were assigned to GCPs: X/Y accuracy -0.1 m, Z accuracy -0.2 m.Thanks to assuming higher accuracy values than accuracy of measurement with the use of the RTK technique the impact of the following phenomenon was reduced.GCPs were the objects that have not been changed in the past 30 years.However, in reality, these might have been modernised during this period but it could not be verified correctly on the basis of aerial photographs solely.In 1st and 2nd variant of GCPs, the were no considerable differences in RMS Errors at Control and Check points between unprocessed and pre-processed in SAPC photographs.Only in the case of a limited number and unfavorable distribution of GCPs (3rd variant), vertical accuracy shows clear difference to pre-processed photographs' advantage.
In case of pre-processed photographs, the obtained results of aerotriangulation in all variants of GCPs were comparable to Trimble Inpho reference results.When comparing the results from Table 1 and Table 2, one can notice that Pix4D manages the case of a limited number of Control points, which are used to georeference a spatial model to the ground, better.Both for unprocessed and pre-processed photographs, in the third GCPs variant, vertical accuracy for Pix4D is twice higher than for Agisoft.However, using SAPC prior to taking advantage of both software products has a positive influence on the final results of aerotriangulation.

Camera self-calibration
The clearly visible Z-accuracy difference between a few variants of aerotriangulation is caused by results of self-calibration.The below table (Table 3) shows determined ck value for a block of photographs unprocessed and pre-processed in SAPC in the 3rd variant of GCPs, for which the difference of aerotriangulation accuracy was the most visible.Reference value ckref = 213.75mm was taken from the photo's frame.Value Δcki was determined according to Equation 2: where ckref = ck reference value taken from the photo's frame cki = ck value from i-th set of parameters determined in the process of self-calibration

Agisoft PhotoScan Pix4D
unprocessed unprocessed  Regarding unprocessed photographs, several sets of interior orientation were calculated by self-calibration algorithm, which assumes that these photos hadn't been captured by the same camera (not the same resolution of each photo).On the other hand, only one set of camera calibration parameters was calculated for the images pre-processed in SAPC.The same phenomenon was observed in both programmes: Agisoft PhotoScan and Pix4d.
The number of the sets of parameters determined in the process of self-calibration would not be so alarming if their results were similar.Unfortunately, for the unprocessed photographs, determined cki values do not fluctuate around ckref reference value.For Agisoft Δcki it is even up to -15 mm, while for Pix4D it is up to 1.6 mm.Discrepancy of the self-calibration results within the block of photographs taken with the same camera has a direct impact on the results of aerotriangulation.Essentially, Δcki is lower by an order of magnitude for both software products for pre-processed photographs.Better selfcalibration results were obtained in Pix4D than in Agisoft PhotoScan.

Comparison of exterior orientation
Additionally, apart from the direct analysis of RMSE at Control/Check points, the results of the photographs' exterior orientation elements determined in SfM software for 1st and 3rd variant of GPCs distribution were compared with the reference ones obtained in Trimble Inhpo.RMS Error for each of the 6 exterior orientation elements was calculated on the basis of the When analysing the aforementioned table, one can state that a high value of RMS Error Y for the unprocessed photographs in the 1st variant of GCPs is puzzling.In the analysed block, Y-axis was related to forward overlap of the photographs.RMSE values for the rest of the positional elements of exterior orientation are similar to the values obtained on the basis of the photographs preprocessed in SAPC.In the case of angular values, the fact that RMS Error for Kappa angle is lower by an order of magnitude for pre-processed data is worth mentioning.
In the case of the 3rd variant of GCPs, RMS Error Z was as many as 37 metres for the unprocessed photographs.Therefore, one can state that SfM algorithm determined the location of the perspective centres at a completely wrong height.The reason for such results was, most of all, the aforementioned camera's selfcalibration error.For photographs pre-processed in SAPC, RMS Error for all the positional elements of exterior orientation is lower, and RMS Error Kappa is lower by an order of magnitude compared to the unprocessed.
Analogously, an analysis was conducted for the results of exterior orientation obtained in Pix4D (Table 5).In the case of results from Pix4D, RMS Error for all the positional elements of exterior orientation is lower for pre-processed photographs in both GCPs variants.One should especially focus here on RMSE Z, which reaches several (1st variant), or even several dozen (3rd variant) lower value for pre-processed data.What is more, RMSE for angular values also look promising.Data pre-processing in SAPC resulted in a clear improvement of results of spatial orientation of the photographs.The scale of the described problem with the unprocessed photograph exterior orientation is well presented by the below visualisation from AgiSoft (Figure 5).The perspective centres of the unprocessed photos (Figure 5A) are not at similar flying height as opposed to the photographs pre-processed in SAPC (Figure 5B).Knowing the rules of photogrammetric project planning, the results of exterior orientation for the unprocessed photographs cannot be considered correct.  5.The results of assessment of exterior orientation accuracy determined in Pix4D for the block of photographs unprocessed and pre-processed in SAPC A) B) Figure 5.An example of an adjusted single strip from block of scanned, analogue photographs: A) unprocessed; B) pre-processed in SAPC application.

Geometric accuracy of point clouds
In most of the cases of working with archival photographs, aerotriangulation is not the final stage of processing.Photographs alignment is only necessary to generate derivative products, that is, a point cloud, orthophotomap or DSM.Below, there is the analysis of the influence of an increase in accuracy of aerotriangulation through pre-processing in SAPC application on geometric accuracy of point clouds generated from the block of the scanned analogue photographs.The analysis used point cloud generated by image matching: Additionally, there is much noise visible on the edges of the block.The point cloud created from the photographs preprocessed in SAPC (Figure 6B) does not have clear data holes and seems to be complete.
In case of 3rd variant of GCPs, numerous data holes are visible for both point clouds (Figure 6C and 6D); however, there are far too many for the cloud of points generated from the unprocessed photographs.
When comparing point clouds from Figure 6A and Figure 6D one may draw a conclusion that data pre-processing in SAPC has a greater impact on the quality of the resulting model than increasing GCPs density and generating point clouds from unprocessed photographs.

DISCUSSION
SAPC application seems to be a useful tool when working with archival aerial photographs, as this was confirmed by the results of the conducted experiment.In the case of limited access to GCPs, obtaining acceptable results of aerotriangulation in SfM software products on the basis of unprocessed data is very difficult.Thanks to using pre-processing in SAPC, the problems with camera self-calibration and photograph spatial orientation do not occur.This has a direct influence on the obtained level of processing accuracy -comparable with reference results obtained in the professional photogrammetric system.Additionally, a close relationship between accuracy of photographs' block's orientation and data completeness in the resulting derivative product is visible.Better results of aerotriangulation determine a higher quality of the generated model.On the other hand, apart from accuracy aspects, working with the use of SfM software on archival data earlier processed in SAPC, is much nicer and less time-consuming.Unprocessed photos are characterized by varied resolution, so applying masks on each photo manually is necessary.In case of SAPC images, this awkward action can be substituted by a single, automatic process.Moreover, results of initial alignment (without GCPs) for preprocessed photos are much better than for unprocessed ones.Thereby, when GCP was placed in two photos, SfM software suggests its location in images far more accurately.

CONCLUSION
SfM software producers are not planning to implement the modules designed for archival data in the near future due to relatively low percentage of users interested in these add-ons.
Treating scanned analogue photographs as photographs taken digitally clearly facilitates and speeds up the process of orientation.The presented SAPC application automatically provides proper processing of archival data therefore enabling one to take advantage of the full potential of photographs in photogrammetric processing with the use of software based on SfM.Described two-stage approach: pre-proccessing in SAPC and aerotriangulation in SfM software, may be an alternative to professional photogrammetric systems, essentially in case of lack of required data about a camera and approximate values of exterior orientation parameters.Apart from the economic issues, this approach seems to be more friendly and easier for users, who are not associated with photogrammetry in their daily works, and who are using archival photographs more and more willingly.
Figure 1.A) GUI window of SAPC application B) Diagram of the technological process for the two-stage approach to scanned, analogue photographs processing in SfM software products.

Figure 3 .
Figure 3. Examples of optical fiducial marks, which can be automatically detected in SAPC SAPC software limits itself to measuring four corner fiducial marks, regardless of their number in the image.Determined pixel coordinates of the four fiducial marks provide redundancy in the target image affine transformation.Template matching in SAPC software is done with the normalized cross-correlation method (OpenCV, 2011).A match metric between template image (fiducial mark area) (T) and the source image (scanned photograph) (I) is calculated according to Equation 1 based on the correlation coefficient:

Figure 4 .
Figure 4. Illustration showing the theoretical assumptions of SAPC software

Figure 5 .
Figure 5. Example of input and output SAPC data

-
Figure 6.Top view of point cloud generated by image matching in Pix4D using: A) unprocessed archival photographs in 1st variant of GCPs; B) pre-processed in SAPC archival photographs in 1st variant of GCPs; C) unprocessed archival photographs in 3rd variant of GCPs; D) pre-processed in SAPC archival photographs in 3rd variant of GCPs.The above figure shows a clear difference in completeness of data between point clouds in Figure 6A and Figure 6B.Problems with spatial orientation of photographs and wrong results of selfcalibration caused holes in the point cloud of Figure 6A in areas where the actual overlap between photos in the block does occur.Additionally, there is much noise visible on the edges of the block.The point cloud created from the photographs preprocessed in SAPC (Figure6B) does not have clear data holes and seems to be complete.

Table 1
below shows the results of aerotriangulation of photographs carried out with the use of Agisoft software, assuming two variants of photographs: unprocessed and preprocessed in SAPC, and three variants of GCPs distribution.Additionally, the table lists reference results obtained in Trimble Inpho.When analysing the aforementioned table, one can notice that in every case, RMS Errors at Check points are higher for the unprocessed photographs.Together with a decrease in a number of Control points (3rd variant of GCPs), horizontal accuracy is twice higher and vertical accuracy even three times higher for pre-processed in SAPC photographs.Over one metre value of RMS Z for unprocessed photographs is unacceptable.A clear relation between GCPs distribution and RMSE values at Check points was observed.Additionally, in the first two GCPs variants, the results obtained from pre-processed data are comparable with the reference results from Inpho.Table2lists the results of aerotriangulation in Pix4D, carried out in the variants analogous to the previous case: 0.147 0.105 0.010 0.086 0.052 0.008 0.022 0.041 0.011 Check Check Check 0.308 0.406 1.210 0.166 0.097 0.358 0.115 0.090 0.237 Table 1.Aerotriangulation results in Agisoft PhotoScanFigure 6. Visualization of three variants of Control/Check points distribution

Table 3 .
Camera self-calibration results in both SfM software products: Agisoft PhotoScan and Pix4D.

Table 4 .
Results of the assessment of exterior orientation accuracy determined in Agisoft for the block of photographs unprocessed and pre-processed in SAPC