RECOVERING OLD STEREOSCOPIC NEGATIVES AND PRODUCING DIGITAL 3D MODELS OF FORMER APPEARANCES OF HISTORIC BUILDINGS

Three-dimensional models with photographic textures have become a usual product for the study and dissemination of elements of heritage. The interest for cultural heritage also includes evolution along time; therefore, apart from the 3D models of the current state, it is interesting to be able to generate models representing how they were in the past. To that end, it is necessary to resort to archive information corresponding to the moments that we want to visualize. This text analyses the possibilities of generating 3D models of surfaces with photographic textures from old collections of analog negatives coming from works of terrestrial stereoscopic photogrammetry of historic buildings. The case studies presented refer to the geometric documentation of a small hermitage (done in 1996) and two sections of a wall (year 2000). The procedure starts with the digitization of the film negatives and the processing of the images generated, after which a combination of different methods for 3D reconstruction and texture wrapping are applied: techniques working simultaneously with several images (such as the algorithms of Structure from Motion – SfM) and single image techniques (such as the reconstruction based on vanishing points). Then, the features of the obtained models are described according to the geometric accuracy, completeness and aesthetic quality. In this way, it is possible to establish the real applicability of the models in order to be useful for the aforementioned historical studies and dissemination purposes. The text also wants to draw attention to the importance of preserving the documentary heritage available in the collections of negatives in archival custody and to the increasing difficulty of using them due to: (1) problems of access and physical conservation, (2) obsolescence of the equipment for scanning and stereoplotting and (3) the fact that the software for processing digitized photographs is discontinued. * Corresponding author


INTRODUCTION
Today, three-dimensional models with photographic textures are a widespread product for the visualization of the state and appearance of elements of heritage.They are also useful for conducting studies, taking measurements and as departure point for the generation of cartographic derivatives such as orthoimages, plan views, elevations or cross-sections.The current development of the documentation techniques, with rising performances with regard to recording methods (e.g.improvement of the resolution of the photographic images, weight reduction of the laser scanners...), as well as the availability of increasingly powerful software for processing and visualizing geometric data, permit offering results even more detailed and accurate.On the other hand, both costs and complexity of use are being significantly reduced, which makes the generation of three-dimensional models more affordable.In any case, we must not forget that virtual models are still cartographic products; hence, they must comply with various quality criteria relating to traceability, precision, completeness, etc.
One of the frequent needs concerning elements of heritage is its study with a time perspective so the evolution can be examined (Stefani et al., 2011).In this respect, it is necessary to dispose of comparable products from different moments, for instance, a sequence of three-dimensional models.Nevertheless, generating models from archive information is not always simple since the features of old data regarding resolution, coverage and overlap between images, points of view, geometry of the cameras, colour... are very different to nowadays' datasets and, as a consequence, the procedures and software that are in force today can be useless when it comes to process that data.
In this text, different possibilities of reprocessing old collections of film negatives coming from terrestrial photogrammetric works originally aimed at stereoplotting are analyzed in order to generate new three-dimensional models of surfaces with photographic textures.Film negatives were a common way of documenting during the last decades of the 20 th century and the first years of the 21 st , until the wide adoption of digital photography for photogrammetric works.By way of example, the following chart (figure 1) shows the experience in our laboratory regarding the application of different techniques for 3D documentation based on photographs and laser scanning from 1996 up to now.There are important collections of negatives stored in archives and institutions related with cultural heritage (only our laboratory counts on around 5000 film photographs corresponding to stereopairs).They constitute an impressive pool of information with great potential for re-use, especially because their original aim was the line drawing of the defining contours of the recorded objects -in particular buildings and archaeological sites-which means that the information about textures has not been extracted yet, neither has the information about the surrounding elements and everything else not directly related with the original subject of study.Nowadays, both hardware and software for working with analog photographs are obsolete, besides, it is not easy to find operators who know how to use them (figure 2).In addition, the negatives themselves are fragile and suffer from several conservation problems (Boadas et al., 2001;Luirete & Escandar, 2008).
Figure 2. Display of a stereoplotter for analog photographs, Adam MSP-2 (at present, out of order), together with several folders with 6x6 cm negatives of some projects, plans, sketches with the distribution of the control points, complementary photographs, etc.The description of the analog equipment (e.g.cameras or stereoplotters) can be found in reference texts of that time (such as Atkinson, 1989).
Recently (Rodríguez et al, 2015), we carried out some tests with digital terrestrial stereopairs (which have been common from the mid 2000's until now, although at the moment their use is limited being more frequent the documentation by means of collection of convergent photographs).The study showed that, often, there were enough photographs to reprocess the projects satisfactorily with current software options (aimed at convergent photogrammetry) and to obtain new productsvirtual models with photographic textures and orthoimageswith accuracies similar to the original line drawings; therefore, these new products serve as an excellent complement to the line drawings generated in the past.
Nevertheless, in some other cases, the configuration of the photographic shots was unsuitable for 3D modelling and it was suggested that this situation could even happen more frequently with analog stereopairs due to several factors like the reduced availability of photographs in comparison with digital projects (as a consequence of the cost of generation and processing), the amount of error caused by the need to scan and process the images before generating the 3D models or the noisy presence of fiducial marks over the images.

OBJECTIVES
This paper focus on the use of analog negatives corresponding to works of stereoscopic terrestrial photogrammetry and the combination of different techniques for the generation of 3D models of surfaces with photographic textures.Ultimately, it aims to set out a methodology that allows obtaining outcomes of evaluable characteristics (precision, completeness, aesthetic...) so they can be used in an interoperable way with other 3D models from different times.
We must consider that, when re-using this information, not only we can count on the negatives themselves but also with all the auxiliary information of the original projects, such as the control points, the reports with the orientations and the line drawing that was originally obtained by stereoplotting.
In particular, the different tools for reprocessing the negatives that are examined are: a. Techniques of 3D reconstruction from convergent sets of images (Fathi et al., 2015), notably, the ones based on algorithms of Structure from Motion (SfM).b.Dimensional analysis through perspective of individual images.c. 3D modelling in a CAD environment and application of the textures by means of monoscopic orientation of the photographs.

Scanning of the negatives
As first step, the negatives were scanned, for this task a device Umax PowerLook 2100 XL was used with a special holder for the negatives of 6x6 cm (figure 3).The selected scanning resolution was 10 microns.The device is a conventional desktop flatbed scanner, therefore, no parameters for the geometric calibration are available.One problem of this kind of devices is that they are hardly used nowadays; as a consequence, it is not easy to find them operational.Actually, this particular piece of equipment had not been used for several years prior to this work; hence, it was necessary to spend two whole days finding a computer with a suitable port to which connect the scanner, searching the drivers, reinstalling the software, etc.

Structure from Motion (SfM)
Once the negatives were scanned, we moved on the modeling stage.As first option, computer programs of convergent photogrammetry -based on techniques of Structure from Motion (SfM)-were considered since, for current works, they provide accurate and detailed results with a high level of automation.Nevertheless, some authors (such as Adami, 2015) had already noticed that archive collections of architectonic photogrammetry are not usually adequate for this kind of algorithms and, indeed, this fact was quickly confirmed since, most times, the geometric configuration of the shots did not permit a proper reconstruction of the scene.For instance, the following image (figure 4) shows the configuration of the shots taken for the documentation of the hermitage of Andra Mari in the town of Ullibarri Arana (Álava, Spain), done in 1996.Under this configuration, most of the documented surfaces appear only in two photographs that form a pair and there is hardly any overlap that allows computing the relative orientation with any other photograph but the previously mentioned own pair.In such a situation, the algorithms of SfM are unable to automatically solve the orientations of the photographs, so it is necessary to turn to other methods for generating the surfaces of the building, as will be described below.
Notwithstanding the above, some cases in which SfM can be used do exist.Let us have a look to the following example, corresponding to the documentation done in 2000 of the walls of the church of San Miguel in the fortified town of Labraza (Álava), these walls are now part of the boundary of the local graveyard.As can be seen (figure 5), the overlap between images is greater than in the previous case of the hermitage and the fact that there are some buttresses resulted in some oblique stereopairs that provide better intersection angles.
Figure 5. Sketch of the shots (left) and one of the photographs (right) of the graveyard of Labraza.Although not specifically thought for that purpose, the configuration of the shots is rather appropriate for being processed by means of SfM.
The software is able to solve automatically the relative orientation of the cameras in 3D, as well as to generate a first reconstruction of the scene in the form of a dense point cloud (figure 6).The image files were included as they were generated by the scanner.Under these circumstances, the software also needs to compute the internal geometry of each camera separately.They all are diverse since so were the position and the orientation in which each negative was placed on the glass during the scanning and because each image was subsequently trimmed off differently.Now, looking to the point cloud, three main problems can be noted: -Firstly, there are some missing parts where no photographic data exists, such as for the roofs.Of course, this is not a problem due to the modelling procedure but, in any case, the consequence is that the 3D model looks strange and is difficult to understand.
-In addition, more gaps are noticeable, this time in areas that did have photographic coverage.The problem here is that the radiometric quality of some areas of the negatives were poor (too saturated) for texture matching.
-Finally, the point cloud over the walls shows some degree of ruggedness that should not be since the surfaces are mainly even.
The second and the last kinds of errors (gaps in photographed areas and the ruggedness of the walls) can be solved, at least to some extent, during the meshing phase.In any case, it is clear that the mesh generation will need validation by an operator and that the outcomes will probably require some edition.As for the remaining missing parts, it will be necessary to complete the model with other sources of data in order to obtain a representative view of this part of the building (see below).
Concerning the geometric accuracy, the coordinates of 16 control points distributed all throughout the model were computed by intersection from a minimum of 4 photographs, the error obtained were in the order of 4.5 cm.
An alternative processing of the same group of photographs was done.This time, the interior orientation of the images were previously performed, to that end, a computer program was created in which the user can mark on an image the position of several of the réseau crosses and identify them with their coordinates in the certificate of calibration, the image is then resampled to a predefined resolution after having applied an affine transformation and, afterwards, trimmed off to a fixed size of 5x5 cm with the center coincident with the principal point of the photograph.In addition, with the cutting, the frame of the negatives is removed, which avoids the problems that this frame might produce when projecting the textures on the surfaces of the 3D model.In a following step, the Exif metadata of the images is edited in order to write the focal length and the pixel size.The visual appearance of the result (dense point cloud) was similar to the one obtained with the images without interior orientation.However, significant differences were observed with regard to the geometric accuracy, in particular, over the same group of control points that was used previously the average error was around 3 cm.
The estimation of the geometric accuracy of the model is essential since it allows us to decide about the possible uses, for instance, in which range of scales it can be merged with the line drawing formerly obtained by stereoplotting.
As to whether the obtained error value (3 cm) is reasonable or not, we must consider that the values of the exterior orientation that were performed with the analytical stereoplotter (Adam MSP-2) in the year 2000 were in the order of 1.5 cm.For the current 3D models we are adding the cumulative effects of the deformation of the negatives, their flatness during the scanning, the quality of the scanner (both geometric and radiometric), the residuals of the interior orientation (from 1.2 to 1.6 pixels of mean square error for the affine transformation) and the fact that the distortion function of the lens has been ignored.If it is to obtain better results, the aforementioned factors should be reduced, however, this is not always doable.
Moreover, these photographs were also processed with the software Autodesk Remake®.Unlike Photoscan, this computer program does not give intermediate results to work on but shows directly the final model of surfaces with the textures already mapped.As the following image presents (figure 8), the resulting model is fairly comprehensive, although some areas with erroneous meshes would require further edition.
Figure 8. Virtual model from the collection of images of the graveyard, using the software Autodesk Remake®.
Besides, from the photogrammetric stereoplotting there is a wireframe model of this same area (figure 9).This wireframe model can be used as a guide for the edition of the model of surfaces recently created, as well as the basis for the generation of new surfaces that complement it (figure 10).
Figure 10.Model of surfaces with photographic textures of the graveyard, complemented with meshes obtained from the old wireframe model.

Dimensional analysis through perspective
We now turn to the groups of photographs that cannot be processed through the programs of SfM.One possible alternative is the monoscopic modeling from the vanishing points (Williamson & Brill, 1990, Van den Heuvel, 1998;Criminisi et al., 2000;Furno, 2013;Tsai & Chang, 2014, etc.).Nevertheless, it should be stressed that this situation will not be very common because, as a general rule, the photogrammetric stereopairs are obtained frontally, being exceptional the oblique stereopairs in which the directions of the vanishing points are clearly marked.As an example, the following image (figure 11) shows another part of the defensive wall of Labraza that was documented with oblique stereopairs due to the impossibility to take photographs straight in front of the tower reproduced.
Figure 11.Photograph that is part of an oblique stereopair of one of the towers of the wall of Labraza.
For the processing of this type of pictures there are many options concerning modelling software.In particular, the following image (figure 12) shows an intermediate stage and the final model generated with SketchUp®.In this case, the geometry of the visible volumes of the tower and adjacent wall has been geometrically modelled from the perspective.Then, the textures have been mapped from the image but, as in the previous case, there are no textures for all the modelled surfaces.In consequence, the outcome combines surfaces with photographic textures and others with generic colours.The image also illustrates a situation that arises with some stereopairs: the fact that there are elements -such as the thick cable that crosses horizontally this picture from one side of to the other-placed ahead the main element.Although for the stereoplotting they do not pose a problem since the operator could ignore them while he or she is drawing, they do cause distortions when applying the textures.This problem is especially meaningful if, as usual, the edition of the content of the photographs is not advisable so as not to induce any deceptive representation of the element (Valle & Rodriguez, 2011).In such cases, the solution is twofold: either deleting this part leaving the area without texture or assuming the distortion considering that the user will be able to identify it as such and, consequently, that it will not affect the interpretation of the three-dimensional model.At any rate, the aesthetic quality of the product suffers.
Obviously, a better solution would be to have at our disposal more photographs from other points of view that allow completing the real texture of the covered parts.However, as told above, when dealing with photogrammetric stereopairs the redundancy tends to be scanty so multiple points of view for the same area are infrequent.An example of this is shown in the following image (figure 13) corresponding to the documentation done in the year 2002 of the tower of Martiartu in Erandio (Bizkaia, Spain) by means of analog stereopairs.For this work, a virtual model with photographic textures was also generated but the textures themselves were not taken from the photographs of the stereopairs but from a specific set of digital photographs taken for this purpose and that covered the walls from different angles in order to reduce the hidden parts and the influence of the obstacles.

CAD modelling and monoscopic orientation
Finally, for the cases in which the previous methodologies are not applicable (neither SfM nor the dimensional analysis through perspective) it is still possible to resort to a general method in which the surfaces are generated in the CAD environment from the line drawing, then, the textures are mapped after orientating the photographs that are considered more appropriate for each area of the building.Actually, this is one of the methods used to generate some of the earliest virtual models with photographic textures at the beginning of the first decade of the new millennium -such as the example of the tower of Martiartu-(Rodríguez et al., 2004).Although with some modifications, this approach can be found over time in other works such as the virtual reconstruction of the city of Bam after the earthquake that devastated it in 2003 (Futragoon et al., 2011).
Let us see this procedure applied to the case of the hermitage of Andra Mari that was presented at the beginning of the paper.As indicated, surfaces are first generated from the wireframe model (figure 14).Then, the images from which textures will be obtained are selected and the meshed corresponding to each image separated.
In the following step it is necessary to orientate individually each image that will be used for mapping the textures over the surfaces, in case the photograph have the interior orientation done and the geometry of the camera is known the exterior orientation can be performed with mathematical models like collinearity, otherwise, algorithms that avoid the interior orientation such as the Direct Linear Transformation (DLT) (McGlone, 1989) might be employed.
The following image (figure 15) shows a computer program developed by our laboratory for performing this procedure of texture mapping (LDGP, 2003).The program shows the photograph and the control points that have been marked on the image with the aim of calculating the exterior orientation through the DLT algorithm.On the other hand, the red dots over the image show the position of the meshes after projecting them over the image and indicate which part of the image will be use for extracting the textures.
Figure 15.Computer program for mapping the textures from the photographs to the selected surfaces.
As occurs with the textures of the tower of Labraza, the applied textures over the surfaces of the hermitage show some foreign elements, in particular, a tree and the array of réseau crosses.
The procedure is repeated with the rest of the selected photographs until all the surfaces that can be given textures have been processed.The areas without texture will be modelled in the CAD and given generic colours in such a way that, in the final model, both parts can be clearly differentiated (figure 16).
Figure 16.Model with photographic textures of the hermitage of Andra Mari.

RESULTS
As seen in the analyzed cases, although with different techniques, it was always possible to obtain some sort of 3D model of surfaces with photographic textures from the old projects of terrestrial stereoscopic analog photogrammetry.This is an important result since many of the monuments documented this way might no longer exist or have been altered to a greater or lesser extent.
In any case, it should be taken into account that these modelslike the ones that are done from new data-have some specific features concerning their geometric accuracy, resolution, completeness and the presence of effects that can affect the aesthetic quality of the product.Features that has to be evaluated in order to assess the adaptation of the virtual models to the new proposed uses (dissemination, metric analysis, traceability of the evolution, etc.).
In addition, the models can be also a basis for the generation of derivatives such as orthoimages, which can be combined with the original line drawing, improving the expressiveness and the richness of the outcomes (figure 17).

CONCLUSIONS
Along the previous sections we have seen that the negatives from old photogrammetric projects have a huge potential for reuse, being a clear example the possibility of generating 3D models with photographic textures that show the former appearances of these elements of heritage at the time of the documentation.This is a strong reason to pay attention to their proper preservation, digitalization and public availability in order to obtain these and other products that allow new uses from this data.
With regard to the generated products, the following characteristics can be underlined: -The quality of the textures will be conditioned by the original pictures and the scanning procedure.Obviously, in case negatives are used, only grey level information will be available (full colour if diapositives are used); on the other hand, different kind of superimposed/fiducial marks and external elements that were not limiting for stereoplotting but that affect adversely the quality of the projected textures should be considered.
-Usually, the geometric configuration of the shots does not permit the use of the algorithms for automatic processing of convergent images (like SfM).Even when these algorithms can be used, the quality of the result (at least as far as geometric accuracy is concerned) improves if the images are preprocessed with the interior orientation.
-As alternatives to convergent algorithms we can resort to other methods for three-dimensional modelling and texture mapping, for instance working photograph by photograph by means of the perspective analysis.However, the standard practice for stereoscopic recording mainly involve frontal views, therefore, pictures suitable for perspective analyses will be scarce.
-Perhaps the most general method that can be used in more situations consists of generating the threedimensional model of surfaces from the line drawing in the CAD environment and, afterwards, projecting the textures from manually oriented images.
-At any rate, it is expected that not all the model can be wrapped with textures.Since the model has to be complete in order to give a representative view of the monument, parts with and without photographic textures need to be combined.
Finally, it should be pointed out that, in contrast to new projects in which we can decide the data to be generated, when we want to re-use previous information we must do the best with what exists.In this regard, we need to adapt our working procedures and even to recover old practices fallen into disuse.

Figure 1 .
Figure 1.Timeline showing the use of different techniques for 3D documentation based on photographs and laser scanning by the Laboratory for Geometric Documentation of Heritage of the University of the Basque Country from 1996 until now.Film negatives refer to the second line: "stereoscopic pairs (analog photography)".

Figure 4 .
Figure 4. Geometric configuration of the shots (left) and example of photograph (right) from the geometric documentation of the hermitage of Andra Mari.

Figure 6 .
Figure 6.Relative orientation of the cameras and dense point cloud of the graveyard of Labraza, processed with the software Agisoft Photoscan®.

Figure 7 .
Figure 7. Interior orientation of the images.Original picture after scanning (left) and clipped image (right), green lines show the 5x5 cm area centred in the principal point of the photograph.

Figure 9 .
Figure 9. Wireframe model of the graveyard, drawn in the year 2000 by means of photogrammetric stereoplotting.

Figure 12 .
Figure 12. 3D modeling from perspective.On the left it is shown the photograph with the identification of the vanishing directions, on the right, a view of the 3D model with textures.

Figure 13 .
Figure 13.Tower of Martiartu, line drawing from the analog stereopairs (left) and 3D model with photographic textures (right).

Figure 14 .
Figure 14.Ermitage of Andra Mari, original line drawing generated in 1996 (left) and model of surfaces derived of this drawing (left).

Figure 17 .
Figure 17.Orthoimage of the southern elevation of the hermitage of Andra Mari -generated from the 3D model with photographic textures-with the original line drawing superimposed.