DATA FUSION OF HISTORICAL PHOTOGRAPHS WITH MODERN 3D DATA FOR AN ARCHAEOLOGICAL EXCAVATION – CONCEPT AND FIRST RESULTS

Through the destruction of war, most of the documents of an archaeological excavation from 1934 1939 of a megalithic tomb in north-west Germany have been destroyed irretrievably. Fortunately, more than 500 historical pictures have been preserved, which visually document the excavation situation at that time. Parts of the image collection are preserved on fragile glass plates that are difficult to preserve and have to be digitised urgendly. A method for digitising these glass plates will be presented first. With the help of the digitised historical images, the excavation situation at that time shall be reconstructed. Since a reconstruction based only on the historical images is not possible, the current state of the megalithic tombs has been recorded with modern measuring technology and a 3D model has been calculated. The aim is to fuse the historical images with the modern 3D model. For this purpose, different possibilities of linking the data are presented. As first results, point clouds calculated by Structure from Motion and the orientation of historical images in relation to the modern 3D model using direct linear transformation are shown. The hybrid model of historical and modern data will be used for archaeological interpretations of the excavation.


Motivation
For 3D digitalisation and reconstruction of historical monuments or cultural objects, varieties of well-accepted methods are available (Stylianidis & Remondino, 2016). As examples, there are automatic solutions for smaller objects (Santos et al., 2014) and case studies for monuments (Kersten et al., 2018). However, in many tasks adapted or new solutions have to be developed in order to meet specific requirements. This became also true for the current research project "MoDi". In this project, 3D information is to be extracted from historical images that are available as historical photographic glass plates. The objective is to reconstruct the earlier excavation process by combining historical image information with modern 3D data of the site (Both et al., 2020).
The key research object of the project is given by two megalithic tombs (graves) "Kleinenknetener Steine I and II", which date from around 3600 -2800 BC (Figure 1). Several of these can be found in north-western Germany. The Kleinenknetener Steine (both about 35 x 15 m) were excavated from 1934-1939. Besides important touristic aspects, they are of great interest from an archaeological point of view, as they are almost completely preserved. According to that period of time, it is not sure that the restoration has been conducted objectively, so that a comparison between the original and the restored situation is required. Unfortunately, many of the excavation documents were destroyed irretrievably by a bomb hit during the Second World War. However, historical pictures have been preserved for visual documentation of the excavation situation at that time. In total, more than 400 photo negatives, ca. 100 glass plate negatives, seven aerial photographs and one analogue film recording have * Corresponding author been preserved. Since the glass plates are difficult to preserve permanently, digitisation is urgently needed. With the help of these pictures a (3D) reconstruction of the original excavation situation is desired. In general, the megalithic tombs can be described as a very complex object due to arbitrary free-form surfaces and a high degree of shadowing and occlusions.

Related work
Historical photos have already been used in various applications for photogrammetric reconstruction. A well-known example is the reconstruction of the Buddha statues in Afghanistan, which were destroyed by the destruction of war (Grün et al., 2004). Another example using historical images from a similar time period as in this work is the reconstruction of dinosaur tracks. Here, the excavation of dinosaur tracks were reconstructed with the help of 17 photographs from 1940 (Falkingham et al., 2014). The MoDi project has the benefit that the object is basically still available. The results can be compared directly with the situation in the field. Furthermore, especially in the urban and architectural field, many examples are given in which historical images and modern 3D data are linked together. The results of the research group of the "UrbanHistory4D" project (Maiwald et al., 2019) is a very comprehensive example. In this case, historical images are oriented and visually represented in relation to a 3D city model in a browser application so the development of the city can be traced over time. (Schindler and Dellaert, 2012) also link historical images with a 3D city model, whereby the facades of the buildings can be textured with historical image information. In both cases, Structure from Motion (SfM) approaches are used to get the orientation of the historical images. The advantage of using architectural images is that they usually contain geometric primitives such as corners or lines, which allow better feature detection to orientate the images (Maiwald et al., 2018).

Historical database
Before the concept is depicted in chapter 1.4, the historical data base and the general challenges regarding historical images are presented. As already mentioned, almost all records and documents of the excavation have been lost. Fortunately, the preserved historical image data shown in Figure 2 still exist. The aerial photographs, along with some drawn plans, provide a good overview of the excavation. However, the greatest potential is in the photographs and the glass plates. The quality of those is much higher than in the film and in the aerial photographs.
Of course, the historical terrestrial photographs were not taken according to photogrammetric requirements at that time. Mostly, the photographer rotated around his own axis or took individual pictures of certain areas. Only in a few cases overlapping images can be found that might be useful for triangulation. Furthermore, an archaeological excavation is considered as a temporal process. Photographs depicting the same scene show different progress of the excavation (Figure 3). Furthermore, some stones, mostly capstones, have been moved during the excavation to expose underlying areas.
The biggest challenges in this case are the conditions of the historical images due to: No information about the historical camera (IOR/EOR) • Insufficient overlap across the entire scene Despite these problems, the fact that such a high number of good quality images are available despite their great age, is rather unusual. Therefore, the potential of these images is high, even though the excavation and the tomb complex itself can be described as very complex.

Concept for data fusion
Not all areas of the megalithic tomb are adequately covered with historical images. There are only a few areas where overlapping images are present, which are necessary for a triangulation of 3D points. Therefore, the current situation of the tombs was measured with modern methods and a new 3D model was calculated (chapter 3). This serves as the basis for further investigations. It is expected that the challenges and limitations with the historical images can be solved by the combination with up-to-date 3D data.
The idea is to realise a hybrid model from historical images, 3D point clouds calculated with historical images and a current 3D model. If possible, overlapping images and SfM are used to calculate a 3D point cloud. If not, the images are spatially oriented with respect to the 3D model.
There are different options to orient the historical images. An easy and automated determination of the orientation is possible with SfM approaches if the historical images fulfil the relevant requirements. For some areas in our database, it works quite well (chapter 4).
If there is insufficient overlap, it can be helpful to integrate historical and current images in the evaluation process. (Maiwald et al., 2017) succeeded in this way to support the external orientation of historical images in the SfM process, which cannot be oriented in advance. In our case, no results could be achieved when combining historical and current images in a joint SfM project. This might be due to environmental changing over time.
For further investigations, the use of masked images will be tested in the future. Furthermore, there is the possibility to determine the orientation of the images to a 3D point cloud using contour cues (Pujol-Miro et al., 2017), which will be the subject of future investigations.
Another possibility is to determine the interior and exterior orientations of the historical images based on corresponding points of the 3D reconstruction. In this case, the direct linear transformation (DLT) seems to be suitable (Luhmann et al., 2019b). The advantage is that information on the image coordinate system is not required, since there is no technical information about the historical camera. The parameters of the interior and exterior orientation can be derived from the determined parameters of the DLT if at least six control points can be identified. The computed orientations can be used to integrate the historical photos into the recent data sets. An overview of the concept to combine historical and modern data is shown in Figure 4. In order to realise the concept, the existing historical images must be digitised first. There are no commercial solutions for the digitisation of the glass plates. A developed method for this is introduced in chapter 2. Subsequently, the modern data basis is presented in chapter 3. Chapter 4 presents the first investigations and results. Finally, chapter 5 summarises the work, draws a conclusion and provides an outlook.

DIGITISATION OF THE HISTORICAL DATA
When digitising museum artefacts, it is important to consider the purpose of the digitisation. However, in this case it is not the photo negative or glass plate and its condition as a cultural object.
Here the primary aim is to record the image content as accurate as possible, both in radiometric and geometric context.
In case of the photo negatives, aerial photographs and the 16 mm analogue film recording there are standardised and commercial procedures for digitising. They include automated correction of radiometric and geometric effects with a scan resolution of 10 µm. Due to fragility of the glass plates (Figure 2, d) a scan with a flatbed scanner seemed too risky. The digitisation had to be done manually, which is described in the following.
The presented glass plates (quarter plate 83 x 108 x 1 mm) are used as a carrier material for the photo emulsion. Compared to photo films, they have the advantage of higher dimensional stability. Nevertheless, the risk of breakage is not negligible and the photo emulsion fades over time. The actual condition of the glass plates varies. Some are well preserved, others show single or multiple cracks. In addition, there are parts of the image which are noisy or which contain no information. This may be due to the fact that no photo emulsion was present at the spot when the picture was taken, so that the exposure was incorrect or that the "pixels" were subsequently eroded. Furthermore, the area of the photo plates with image information also varies in size and is slightly rotated. Consequently, a uniform image coordinate system cannot be assumed.
For scanning the glass plates a workflow was developed which is similar to the digitisation strategy that were used for the famous stellar glass plate negatives of the Harvard College Observatory (Simcoe et al., 2006).  To eliminate influences of the scan camera from the entire reconstruction process, it is highly recommended to calibrate the actual recording camera before the digitisation process. Possible influences due to lens distortions thus can be excluded in advance. Furthermore, the digitised data is also prepared for future applications in the museum context, so that no further expensive and time-consuming digitisation needs to be carried out. The aim of the calibration is to determine principal distance, principal point and lens distortions. It should be taken into account that the camera constant and the principal point may be correlated during calibration. Due to the low depth of field of the camera (Table 1) and because the determination of the camera constant is not required to be highly accurate, a flat test field with coded measurement marks is used (Figure 5, right). The coded measurement marks are spatially well distributed so that the distortion parameters can be determined in the best possible way. In addition, tilted images are taken in order to be able to determine the principal point position appropriately. The parameters of the interior orientation are determined by bundle adjustment. The correlations are shown in Table 2. It becomes clear that the correlations are within an ordinary magnitude.
Finally, the entire image material is digitised and ready for further processing.

MODERN DATA
Since the historical data do not show the megalithic tombs completely, the current situation has to be recorded. For the 3D reconstruction, terrestrial laser scanning, close-range photogrammetry and UAV were combined, as done by (Luhmann et al., 2019a) for complex churches. The data fusion, evaluation and modelling was done with the SfM software Reality Capture (Version 1.1.1).
For initial investigations, the open tomb "Kleinknetener Steine II" was first recorded. Due to the high object complexity, the recorded data volume is correspondingly high (Table 3). In this case the complete recording of the tomb structure is more important here than accuracy and resolution.  were used for the measurement. The BLK360 was mainly used in the inner area of the tombs because of is smaller minimum distance range. In addition, a low recording height was chosen in order to be able to record data of the undersides of the capstones. With the RTC360, the tomb was recorded from the outside, and some viewpoints on the capstones were chosen to overlook the scene. Usual checkerboard targets were placed around the megalithic tomb, which were previously recorded by GNSS (RTK) for georeferencing. Due to shadowing, the targets could only be recorded by the RTC360. The registration of the recorded scans is done separately for each device using the software Cyclone Register. The accuracy of the BLK360 by means of ICP is 0.014 m. With the RTC360, an accuracy of 0.004 m via the targets and 0.009 m via ICP was achieved. This was to be expected due to the higher measurement accuracy of the RTC360.
The terrestrial images were taken with two DSLR cameras (2x Nikon D850), equipped with a 24 mm lens each. In order to realise a high completeness and to avoid shadowing, a small acquisition distance of 1-2 m was chosen. Furthermore, a high overlap was aimed at for the best possible determination of tie points, and rotated images were taken for a good camera calibration.
The drone images were taken with a DJI Mavic Mini and a flight altitude of approx. 10 m. The aim is to be able to record the tops of the tombs.  Figure 6 shows the 3D model calculated by Reality Capture, including all camera orientations and laser scan viewpoints. The mean reprojection error (Sigma 0) is 0.6 pixels. For further processing, the model was reduced from 663 to 24 million triangles and a resolution of 5 mm in order to achieve faster processing times.

INVESTIGATIONS AND FIRST RESULTS
Now that the historical images and the 3D model are available, the aim is to merge the data. The modern 3D model forms the basis coordinate system in our approach. For the sake of simplicity, the investigations are carried out using tomb chamber 2 as an example.
Unfortunately, the images are initially available in a completely unsorted and arbitrary order. If no prominent places, such as special stones, are depicted, it is difficult to roughly assign the historical images in the excavation. Therefore, an archaeologist manually oriented the images in advance in an excavation plan. Through the manual pre-sorting, it was possible to find some areas where there was potential overlap and which could thus be processed using the SfM Software Agisoft (Version 1.7.2).  Figure 7 shows an example of an object area where the historical images have sufficient overlap for SfM. A total of 12 images could be oriented. It is noticeable that the ground between the stones could not be reconstructed. This is due to the fact that soil was removed in the course of the photographs. The calculated model can be transformed into the coordinate system of the modern model by control points including scaling and fineregistered by ICP. Figure 8 shows the deviations between the models. The average deviation is 1 cm. In some places deviations in the range of 25 cm occur. Due to the fact that the deviations are mostly systematically in the range of 1 cm and only in some areas at 25 cm, it can be assumed that the digitisation and reconstruction process does not contain any undetected errors.
The 25 cm deviation can be explained by the subsidence of individual stones over time or during excavation ( Figure 9). Based on the comparison, the absence of a stone corner has been detected, which is shown in Figure 8 and was previously difficult to recognise visually in the pictures. Overall, it is remarkable that the position of most of the stones has hardly changed, even relative to each other. In order to determine the orientation of the historical images that do not have sufficient overlap for SfM approaches, a manual orientation is carried out using DLT. This requires a minimum of 6 control points, such as those depicted in Figure 10. Despite the radiometric differences, some corresponding points can be identified in both the image and the 3D model.  The deviation between the calculated orientations is 1.2 m for image 02-043. This is due to the fact that there are high radiometric differences between the point cloud and the images. Furthermore, an incorrect perspective, which is more difficult to find with such an object than with a facade, can lead to incorrect point assignments. The weakness of the DLT also becomes clear that outliers can hardly be found in the adjustment. Nevertheless, in this case it is sufficient as a rough orientation of the images in relation to the 3D model. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2021 XXIV ISPRS Congress (2021 edition) Figure 12 shows a first attempt to display the orientation of historical photo in relation to the actual 3D model. For the visualisation of the historical images, the historical point clouds and the actual 3D model, different applications will be tested in the future. Interdisciplinary dialogue with archaeologists will play a role in finding the best possible solution. Figure 12: First idea of a visualisation of an historical image in relation to a 3D model using the software Blender.

SUMMARY AND OUTLOOK
The aim of our work is the fusion of historical photographs and modern data in order to be able to reconstruct a historical excavation situation. For this purpose, the historical images first have to be digitised. A method for the digitisation of historical glass plates was presented. Due to pre-calibration of the scanner camera, the scanned historical images only contain the distortion effects of the unknown historical camera. With a limited subset of historical images, a point cloud could be calculated via SfM. A comparison with today's situation showed average deviations in the range of 1 cm, which can be considered as an impressive result. The high potential of historical images, in this case 80 years old, thus became clear. Furthermore, areas that have changed over time could also be identified.
With the help of DLT historical images were successfully oriented with respect to the new 3D model. This has to be done for most images, as often there is not enough overlap for SfM. The historical point cloud and the calculated orientations (DLT or SfM) could be transformed in the modern 3D model, which defines the common coordinate system. Finally, the modern and actual data could be fused to a hybrid 3D model. Since finding corresponding points for the DLT has been done manually so far and took a lot of time, an automated procedure would be desirable. Various methods of feature matching can be investigated here.
In the future, a suitable form of representation of the hybrid 3D models must be developed. Visualisation techniques such as AR and VR might give additional opportunities to visualise newly composed historical data.