CULTURAL HERITAGE RECONSTRUCTION FROM HISTORICAL PHOTOGRAPHS AND VIDEOS

Historical archives save invaluable treasures and play a critical role in the conservation of Cultural Heritage. Old photographs and videos, which have survived over time and stored in these archives, preserve traces of architecture and urban transformation and, in many cases, are the only evidence of buildings that no longer exist. They are a precious source of enormous informative potential in Cultural Heritage documentation and save invaluable treasures. Thanks to photogrammetric techniques it is possible to extract metric information from these sources useful for 3D virtual reconstructions of monuments and historic buildings. This paper explores the ways to search for, classify and group historical data by considering their possible use in metric documentation and aims to provide an overview of criticality and open issues of the methodologies that could be used to process these data. A practical example is described and presented as a case study. The video “Torino 1928”, an old movie dating from the 1930s, was processed for reconstructing the temporary pavilions of the “Exposition” held in Turin in 1928. Despite the initial concerns relating to processing this kind of data, the experimental methodology used in this research has allowed to reach a quality of results of acceptable standard.


INTRODUCTION
The importance of Photogrammetry in the field of Cultural Heritage is by now recognised, as in the past and without any doubt, as having a key role in the documentation of Cultural Heritage assets.Metric documentation is the collection of all the useful data needed to form the geometric knowledge necessary to act on Cultural Heritage assets; such as deepening assessing the cultural values of the assets, forming a correct base for restoration and conservation actions, and planning monitoring strategies for their management.The implementation of photogrammetric techniques to recover the shape and dimensions of Cultural Heritage resources has reached characteristics of a mature and stable system with a welldefined pipeline.This has led to an increase in the use of these techniques for 3D surveying and modelling.Recent developments in digital photogrammetry have heightened the need for data acquisition and processing to become more and more fast and automatic and to use low-cost devices, with evident and numerous advantages.But the current trend to extend photogrammetric methods also to unskilled users, involve the risk to not put in evidence the assessing of quality and accuracy of metric data.If this need of quality and precision assessment is important for new surveys, the same need is necessary for the data extracted from historical sources.Metric data without a certified quality cannot give the correct information and can result in the misuse of them.Researchers have shown an increased interest in this direction and a proliferation of studies have compared different acquisition instruments and processing software in different situations and case studies.This continuous evolution further offers the possibility to update well-defined approaches of extracting metric information from images.One of the most intriguing challenges is the utilization not of new data, but resources stored in historical archives (i.e images, paintings, surveys, etc.).
Archives are powerful platforms for saving invaluable treasures of enormous informative potential and play a critical role in the conservation of Cultural Heritage.Besides written documents, old photographs and videos which have survived over time are, in many cases, the unique witnesses of architecture and urban transformation.Monuments, historic buildings, and urban landscapes, that have been transformed or destroyed over time, appear in them, and they become the only way to document changes of currently existing objects and parts that are no longer visible.In extreme situations the archive documents can allow the recovery of shapes, dimensions, and locations of assets that no more exist.This is an opportunity that could help historical studies and drive, in some way, restoration and conservation decisions.A well-known example of the importance of historical images is represented by the images acquired by Albrecht Meydenbauer, a young German architect.In 1858, he had the idea to use photographic images for the documentation of buildings.He was aware of the upcoming risks for cultural objects and was convinced that the most important cultural heritage objects should be recorded in a Cultural Heritage Archive, so that they could even be reconstructed in the case of destruction.Photogrammetric images were the most effective way to reach this goal and he fought against many obstacles and criticisms for its acceptance as a scientific documentation method.He became the pioneer of cultural heritage valorization by means of photogrammetry (Albertz, 2001).Those images were widely used during the reconstruction of the city of Berlin at the end of the last century.All the international charters on Cultural Heritage conservation identify photography as one of the best ways to document Cultural Heritage assets.These recommendations have always been interpreted as the requirement of photographic documentation without considering the metric potentialities of the photographic images and the benefits that these properties could give to the effective documentation actions which are required before any kind of restoration.
In recent years, due to the digitalization efforts of many Archives, interest has increased in historical photographs and videos as precious sources for heritage studies, and historical development reconstruction.The major problem with historical images is the availability of material of different types with low image quality, a total lack of camera parameter knowledge, the presence of deformations of the original dimensions, and damage due to improper storage.However, recent developments in the field of image processing and computer vision have led to a renewed interest in processing data with a lack of essential properties for 3D metric information extraction.These improvements have resulted in an increase in the already demonstrated metric power of old images.The specific objective of this paper is to offer a short excursion through the metric potentialities of different data available in historical archives, by considering the essential role of photogrammetry.The aim is to explore how metric information about buildings which no longer exist could be extracted from old photographs and videos of different quality, for 3D virtual reconstruction.The first section of this paper will analyse the different data stored in historical archives and the ways to classify and group them by considering their possible use in metric documentation.The second part will provide an overview of criticality of the methodologies that could be used to process them.The third part focuses on the methodology used to process historic data and a case study is presented.The last part presents the first findings of the investigation, focusing on the limits and open issues.

RESEARCH STRATEGY OF CULTURAL HERITAGE "BIG DATA"
To look for ad hoc historical images to recover metric information could be a "big data" management problem.
Images are published on books, reviews, and can be found in some ways (negatives, positive copies, etc.) inside public or private archives.Unfortunately, not all the archives have a digital database to ease the research of interesting images.The era of digital distribution and "Information and Communication Technologies have introduced new exciting avenues for producing, accessing and consuming images and information.Simple traditional Information Systems supporting the management of cultural artefacts have left the place to complex systems that expose rich information extracted from heterogeneous data sources (e.g.Sensor Networks, Social Networks, Digital Libraries, Multimedia Collections, Web Data Service, etc.) by means of sophisticated applications that enhance the user's experience" (Amato et al., 2017).Within this realm, "access to Cultural Heritage information has been significantly benefited by digital technologies, facilitating new ways of engaging with heritage and broadening public participation by facilitating interconnections among pieces of Cultural Heritage information, public view points and physical venues.A key contribution to this endeavour is the creation of a semantic knowledge base capable of interrelating a wide set of (existing and future) disparate digital cultural heritage resources.One of the fundamental problems in dealing with Cultural Heritage data is to make those contents mutually interoperable, so that they can be searched, linked, and presented in a harmonised way across the boundaries of the datasets.In the sphere of contemporary information science, there is abundance of instruments for managing and modelling any kind of information including Cultural Heritage data" (Vlachidis et al., 2017).As more and more cultural institutions publish their data using the web-of-data semantic level, "there is a need for innovative applications for exploring, analysing, mining and visualizing such data" (Markhoff et al., 2017).For this reason, the methodology used to collect the data for this study proceeded in two ways: classical research in historical archives and the experimental use of I-Media-Cities web platform (available: https://imediacities.eu/, last access 06/04/2018).I-Media-Cities is an ambitious and innovative research project led by 9 European cultural institutions (film and audio-visual archives from 8 different Countries) to share, provide access to, and use their digital content, turning it into the lever of new approaches to multidisciplinary research, to business innovation, and to the overall accessibility of the European Cultural Heritage.As an experimental innovation action, the project focuses on the digital content that refers to cities. I-Media-Cities strives to be a cross-border, cross-language platform for the study of the history and urban development of large EU cities through large media collections previously non-accessible, and the study of the history of Media through the way they depicted urban spaces.
The project is still under a development and experimentation phase, but it will deliver a digital content access platform (interoperable and multilingual), made available to a growing community of researchers and creatives Europe-wide.The research of photographs and videos is possible because they are automatically annotated by machine learning approach that allows to make the shots searchable via dynamic map/timelines, and through semantic searches.

ANALYSIS, CLASSIFICATION AND STATE OF THE ART
Previous studies highlighted the restrictions in the use of historical images for automatic reconstruction of buildings such as image quality, availability of camera parameters and ineffective geometry of image acquisition.These constrains are very hard to solve and it is difficult to discover good datasets in the case of terrestrial close-range photogrammetry (Adami, 2015).Actually often they are affected by: i) inaccuracy or total lack of meta-information about inner orientation (focal length and coordinates of principal point and fiducial marks when present) and additional (i.e., distortion) parameters; ii) poor radiometric image quality (haze, image darkness, non-uniform luminosity); iii) improper transport or storage procedures of the film (humidity, temperature, etc.) and iv) inaccurate processing of original films or hardcopies in field laboratories (Zawieska et al., 2017).
The new potentialities offered by the last research results in the domain of photogrammetry (both by using nonlinear approach and the one based on projective geometry) open the door to a new and more complete analysis about the metric content extraction strategies from any kind of images.
The proposed approach splits historical images into five main categories: (1) single images; (2) photographs for photogrammetric purposes; (3) random images; (4) stereoscopic views; (5) videos.In the following paragraphs short state-of-art and potentiality of these kind of primary data are presented.

Single Images
3D reconstruction of historic buildings from single images is possible.Some studies have been carried out for the rectification of historical images that can efficiently offer measurements from a single image using a-priori object geometric properties like linearity, parallelism, perpendicularity, symmetry etc. (van den Heuvel, 1998).The methodology involves the determination of segments and vanishing points and, if possible, intrinsic camera calibration (Bräuer-Burchardt and Voss, 2002).The knowledge of a reference geometry allows the scale reference estimation.In case of a destruction of the object, the reference information sometimes is given by still existing buildings in the neighbourhood (Hemmleb, 1999).If parameters of the camera model, like the focal length, are unknown, it is possible to estimate them by using geometric object information.

Photographs for Photogrammetry
If rules of photogrammetric acquisition were kept and existing calibration data of the camera or information about the viewpoint still exist, a scaled evaluation under certain conditions is possible.This is the case, for instance, with many photographs of the archive of metric images of Meydenbauer.The success of a photogrammetric evaluation also depends on the number of pictures of an object, ideally taken with the same camera (Hemmleb, 1999).In the past, data processing was carried out on digital stereo plotters or workstations; it was necessary that all analogue photos were converted to digital form using a photogrammetric scanner.Nowadays the photogrammetric materials still exist, but the technology and processing are completely different.A recent study (Pavelka et al., 2017) tried to process different kind of photogrammetric images with different results.A good database of overlapped photographs was possible to process in Agisoft Photoscan.For another set of photographs without any calibration Photomodeler software was chosen as appropriate for processing and for reverse calibration computing.The were not so good because images were with poor geometry, no quality calibration, and significantly different cameras were used.

Random Images ("big data")
Most researchers investigated the case in which images are acquired from different kind of source like internet or books.The main problems with these images are their differences in size and scale, the unknown pixel size and camera focal length and most of all the different times of acquisition; therefore, some parts visible in one image are missing in others.The illumination conditions (shadows) are usually very different and this can create problems with automatic matching procedures (Gruen et al., 2014).
To solve these problems different photogrammetric techniques and algorithms are used.Undoubtedly the bundle adjustment presents a lot of advantages (Gruen et al., 2014), but the most popular are structure-from-motion and image-based rendering algorithms that can compute the camera information from the images themselves using computer vision techniques (Snavely et al., 2008;Schindler and Dallaert, 2012).Data from these studies suggest that the integration in the process with current measurements data and images of the same object (Maiwald et al., 2017) allows to obtain many reliable control points for the photogrammetric determination of the interior and exterior orientation parameters of the historical photographs (Hanke et al., 2015).

Stereoscopic Views
While concerned with processing of pair of stereoscopic images, structure-from-motion techniques, suitable in the case of many convergent images, fail, as demonstrated by a recent study (Miranda and Melón, 2017) that experimented the processing of old collections of analogue negatives coming from works of terrestrial stereoscopic photogrammetry of historic buildings.
There was hardly any overlap that allows computing the relative orientation and in such a situation, the algorithms of SfM are unable to automatically solve the orientations of the photographs.
Traditional photogrammetry can fill the gap and obtain adequate results.

Videos
When images are not available, videos are indeed an interesting source for a 3D measurement technique.Existing research defined 'videogrammetry' as a measurement technique based on the principles of 'photogrammetry' (Gruen, 1997).The accuracy of a videogrammetric system and the obtained results from different studies, certified that is high and like laser scanner acquisition (Gruen, 1997;Herráez et al., 2016).Another advantage is the reduction of computation load and so a higher efficiency (Sung et al., 2017).In case of historical video, the advantage is the possibility of frame extraction from the hundreds of each video sequence and of selecting stereopairs (Herráez et al., 2016) that can be processed with photogrammetric techniques.
Figure 1.Classification of historical images in five categories

CASE STUDY DATA PROCESSING AND RESULTS
The application presented below aims to combine different typologies of primary data, with the research strategy previously described, for the three-dimensional photogrammetric reconstruction of buildings that no longer exist.The case study is the "Exposition" held in Turin, in the park of Valentino, in 1928 (May 1 st -November 11 th ), within which temporary pavilions were built.The exhibition is part of a series of civil, military and economic events promoted to celebrate the 10 th anniversary of the end of the First World War and the fourth centenary of the birth of the king Emanuele Filiberto.In the design of the pavilions, the architects had the opportunity to experiment and offer to the public new architectural languages in real buildings even if temporary.The sources that have preserved traces of this event are the Dezzutti and Melis collections of the historical archive of architecture library of Politecnico di Torino, and the magazine "Domus" (No. 9, September 1928) from which it was possible to recover descriptive texts of the pavilions, measurements, surveys, photos and technical drawings of the projects.

Frame video processing
The frames in which the pavilion appeared were extracted from video sequence.Initially they were 239, but only 23 were selected with high degree of overlapping to remove redundant information and improve the speed of the process.
Figure 5.A selection of the frames extracted from the video "Torino 1928" in which the pavilion of "Mines and Ceramics" appeared Once extracted, the film frames were processed with MicMac, the photogrammetric opensource software developed by IGN (France).After a first experimentation with commercial software, it was clear that the they are not suitable for processing images extract from an old video.In fact, although some commercial software can manage and process videos without frame extraction (3DF Zephyr for example), they allow to obtain a 3D model only by pushing a button without any control and knowledge of problems such as object distortions and deformations, scaling and obtained metric accuracy.
In case of images with low convergence and overlap, only grey level information, bad resolution and a lack of metadata, a proper understanding of the theoretical background of algorithms running in software applications is necessary.The advantage of using MicMac is that in each step of the photogrammetric pipeline the process algorithm can be chosen, and the errors controlled to reach reliable results and metric products.Since the lack of calibration certificate of the images, the first problem for the processing is how to calibrate the optics used.
Usually MicMac use EXIF metadata to determine the approximate values of the calibration parameters.However, the selected images don't have such metadata but from the provided information of the video camera it was possible recover image format and focal length and manually insert them.This was possible because the conversion procedure into digital format of the film did not conditionate, in appreciable way, his quality and characteristics.The specifications are summarized in Table 1: in, each implying an aspect ratio of 1.34:1.Double-perf 16 mm film, the original format, has a perforation at both sides of every frame line.Single-perf is perforated at one side only, making room for an optical or magnetic soundtrack along the other side.The suitable camera calibration model adopted for the case study is the one that consider a radial distortion model with limited degrees of freedom, to limit the risk of divergence.The five degrees of freedom of the searched calibration model are: focal length, fiducial coordinates of the principal point, and the K1 and K2 coefficient for radial distortion modelling.The obtained results and the check of achieved precision are here presented.
After the external orientation, the errors reprojection of tie points are examined and it was highlighted that the residual average is overall of about 1.2 pixel.By considering the low quality of images, the numerous problematic concerning the process workflow and that with high quality images the average is about 0.2-0.3pixel this result is acceptable.It means, in fact, that the model of exterior orientation estimated is adaptable with the matching of tie points.Finally, the process was completed with the creation of point cloud.The obtained differences show a good quality of the survey also by considering that the reference distances are the one extracted from a design drawing, and it is acceptable that some of them were not correctly realized on the site.Finally, the results, within tolerances described just now, could be considered adequate.

CONCLUSIONS
The aim of the present research was to outline the intrinsic potentialities of data stored in historical archives for cultural heritage documentation.This paper is a first effort to analyse the research strategies of cultural heritage big data and create a systematic classification of them.From the study of the state of the art some important open questions still emerge: the implementation of commercial software of structure-frommotion, recently widely used, presents limits such as the impossibility to process a single image and the bad results when only two photographs or images with few overlapping areas are available.Another critical problem is the lack of camera calibration information and the measurements of specific control points and consequently the control of metric accuracy.The importance and originality of this study lies in the experiment of a metric reconstruction of a building that no more exist starting from an old film.The results of this investigation show that combining different information from historical archives such as old surveys from which measurements can be extracted and inserted during the photogrammetric process, a threedimensional reconstruction is possible, with acceptable range of accuracy.In spite of its limitations, the study certainly adds awareness in cultural heritage data exploitation and should be repeated using different image sets and conditions.Further research should be carried out to establish a general method for processing historical archival material and to give assessed accuracies to 3D virtual reconstructions.The findings of this study have important implications for future practice and could be used as basic information by researchers, public administration, and tourists for documentation and valorization purposes.

Figure 2 .
Figure 2. Planimetry "Exposition" held in Turin in 1928 -In red the "Mines and Ceramics" pavilion.In: Library of Politecnico di Torino "Roberto Gabetti"

Figure 3 .
Figure 3. Project drawing of "Mines and Ceramics" pavilion.Plan, Section and architectural details.In:Pagano et al., 1930

Film
Specifications of video framesMicMac provide in a unique step the camera calibration and relative orientation of the images after a matching step that allows the tie-point extraction.The visualisation of the matching for a pair of images and the control of the correct extraction of tie points is possible.

Figure 6 .
Figure 6.Visualization of matching result: tie points between two images, the first one coloured in blue and the second in orange

Figure 7 .
Figure 7. Measure extracted from the old project in façade

Table 2 .
Comparison of distance and residuals calculation