RECONSTRUCTION OF LOST ARCHITECTURAL VOLUMES BY INTEGRATION OF PHOTOGRAMMETRY FROM ARCHIVE IMAGERY WITH 3-D MODELS OF THE STATUS QUO

Cultural heritage includes several cases of missing architectural element or entire buildings, due to destruction, replacement or radical changes caused over time by other structures. The investigation of these lost elements aimed at their virtual reconstruction, for both scientific and cultural-leisure applications, is therefore a topic of great interest. To this purpose, methodologies for surveying and photogrammetric processing provide a very powerful tool, extracting descriptive and geometric information, both 2-and 3-D, using diverse archive images. This paper presents the issues related to the use of archive images in photogrammetry, pointing out the need for an integrated approach to operations of virtual reconstruction of lost volumes. This approach provides a multidisciplinary effort, in order to evaluate all iconographic sources, of which images processed by geomatics techniques are a component. The paper also presents the early results of a reconstruction project of the Palazzo di Cosimo de’ Medici, in the Fortezza Vecchia site (Livorno, Italy), heavily damaged by World War II bombings and subsequently razed.


INTRODUCTION
Photogrammetry is currently regarded as one of the most effective and cost-efficient methods to perform 3-D surveys in the architectural field.On the other hand, although photogrammetric methodologies have long been known, they did not achieve their current popularity until the last few years.This revival is due to several reasons, but surely a key element is the availability of software tools enabling the automation of the photogrammetric processing, hence making it more affordable also for less specialized users (Adami, 2015).The huge potential of photogrammetry applied to architecture is linked to the ability to check, over time, both geometric and visual changes.By the mid-18 th century, Albrecht Meydenbauer pioneered development of cultural heritage by means of photogrammetry, accurately detailing the ability of photographic images to hold object information.In addition, in foresight of the related upcoming risks, Meydenbauer developed the idea of a Cultural Heritage Archive, where the most important cultural heritage object should have been recorded so that they could be -given the case -reconstructed after catastrophic events (Albertz, 2001).The availability of new algorithms and software and the de facto "democratization" of photogrammetric processing has brought Meydenbauer's envisioning back into topicality, especially as regards the ability to use archive images, independent of their sources, to reconstruct lost architectural volumes, details or textures.The use of archive images, which lack native features suitable for photogrammetric processing, cannot ensure homogeneous geometric accuracy at any moment.The reconstruction of lost volumes, therefore, cannot follow merely geomatics guidelines, instead requiring an integrated approach which involves specific skills in the historic-architectural field, for a deeper and fuller understanding of all available sources.This paper presents the case study of the Fortezza Vecchia (Livorno, Italy) (Fig. 1) as an application of this methodological approach.

Figure 1: 3D model of the Fortezza Vecchia
The Fortezza Vecchia in of Livorno, Tuscany represents the latest in a series of fortifications designed by Giuliano and Antonio da San Gallo in the years from 1488 to 1519, with which the two architects experimented and improved the new modern profile for fortifications with angular bastions.The construction of the Fortezza Vecchia included an existing fortification, known as Quadratura dei Pisani, built in the second half of the 14th century to strengthen an existing medieval keep.The pre-existing structures, as well as the orographic condition of the site -surrounded by the sea -deeply influenced the construction of the fortress, which entailed several irregularities and dissimilarities from the ideal regular shape as theorized by Renaissance military architectural theories (Ulivieri 2014).
It is revised and adapted to meet two kinds of landscape constraints, i.e. site completely surrounded by the sea and medieval remnants.In addition to starting a detailed reading of architectural typicalities and anomalies of the Livorno fortress, virtual reality navigation and interactive multimedia exploration are also necessary to conjure the presence of those long disappeared architectural volumes standing on the platforms, which, seriously damaged during the Second World War, in the post-war restoration works have been heavily and indiscriminately torn down.In this light, 3-D architectural survey is essential for a detailed reading of architectural features and anomalies of the Livorno citadel.Through these innovative investigative methodologies, Fortezza Vecchia can be placed in a wider historical context as well as analysed for its technical building aspects.Performed testing on Fortezza Vecchia provides collection and use of archival terrestrial images for metric 3-D reconstruction of the 16 th -century Palazzo di Cosimo de' Medici which stood at the centre of the Quadratura dei Pisani (Fig. 2), incorporating the medieval keep.

Photogrammetry from archive imagery
The idea of using archive images for 3-D reconstruction of objects is not new.By the early 2000s, the issues inherent in the use of traditional photogrammetry software to process archive images, lacking any photogrammetry-related feature, led to the development of the first "hybrid" photogrammetry/computer vision software.Alongside the availability of this software, photogrammetric requirements for good quality results were also defined.Several studies (e.g.Fraser, 2001;El-Hakim et al., 2003;Zawieska et al., 2017) pointed out that: -Accuracy increases accordingly to base-to-depth ratio and using convergent images.
-Increasing the number of measured points per image improves the accuracy, although not significantly if the geometric configuration is strong and the points being measured are well defined (like resolved targets) and well distributed in the image.
-Accuracy improves significantly as the number of images where the same point appears increases.However, measuring the given point in more than four images yields less significant improvement.
-Self-calibration is reliable only with suitable geometric configurations.
On the other hand, historical archive image sets usually do not meet such requirements.Even if basic workflows are independent of the age of the photograph or creation process -i.e.digital or analogue -older photographs are more challenging to process due to archival quality, digitization issues, technical equipment and film media (Mainwald et al., 2017).The major problems with historical images include the availability of material from different sources and of unknown origin with low image quality, a total lack of information about camera parameters and the presence of deformations of the original (Condorelli & Rinaudo, 2018).The use of historical images to generate 3-D models of lost historic-architectural heritage is widely documented in literature.A prominent, high-impact example is provided by the reconstruction of the Bamiyan Buddha statue, in Afghanistan, which was destroyed in the early 2000s (Gruen et al., 2004); anyway, there are several other cases of use of historical images, both early and more recent, such as (Bitelli et al. 2007;Gouveia et al., 2015;Henze et al., 2009;Falkingham et al., 2014).In a diverse context as regards possible iconographic sources, the acquisition of data for the reconstruction of historical buildings requires the combined application of a wide variety of photogrammetric techniques (Wiedemann et al., 2000).In a best-case scenario, available images allow the automation of the 3-D geometric reconstruction process, fully exploiting current software methodologies based on Structure from Motion (SfM) and Multi-View Stereo (MVS) algorithms (Remondino et al., 2014).However, this is not the case for the vast majority of the available sources, which require different photogrammetry techniques, such as rectification, suitable to generate façade plans.Besides, other drawings (e.g.cross-sections) have additional requirements, such as depth information for each façade.Even though point cloud reconstruction could not be automated, the adjusted tie points resulting from the bundle block adjustment provide the geometric reference for further processing.Additional points may be measured in two or more separate images and calculated afterwards by ray intersection, using the orientation results of the bundle adjustment.Despite its seeming simplicity, this task actually presents some practical problems.Due to differences in image scales, geometric and radiometric image resolution, points of view, illumination conditions etc, it is often very difficult to identify the same spot in two different images.The accuracy of the resulting point coordinates depends on the worse accuracy of the used pairs of image coordinates and the ray intersection conditions (Wiedemann et al., 2000).

The Fortezza Vecchia in Livorno: an ancient construction site of the art of fortification
Fortezza Vecchia is a proper architectural palimpsest, showing different historical and structural paths.Pisans and, later, Florentines designed this site as directional centre for port activities and, over time, built an array of fortifications, still embedded within its perimeter.Its peculiar position, on a dominant rocky point directly above the sea in Livorno east end, along with its function in the complex system for sightings and signalling, planned by Pisa since 12 th century for coastline defence, confirms the importance of the Fortezza Vecchia site (Vaccari, 2009).At a time when the new "military architecture" followed the ongoing evolution of firearms in war strategies, Livornocuddled by Florence -opted for a "modern" building (Fara, 1993).In 1506 Antonio da Sangallo the Elder drafted the citadel, under the supervision of his brother Giuliano; anyway, building took place between 1519 and 1533, with the contribution of an array of experienced engineers.The original design of the Fortezza Vecchia almost certainly provided a fourth bastion on the northernmost vertex facing the sea.In fact, only three bastions were built, with the Quadratura dei Pisani (the only case of a real provision of the still rudimentary canon observed in Tuscany during the second half of the 14 th century, apart from the Santa Maria in Castello stronghold, is in the Quadratura dei Pisani) (Severini, 1970;Bevilacqua Salotti, 2011) -substituting the fourth one, thanks to its circular tower (late 13 th century), acting as a keep, overlooking the sea from West to North.At a closer look, this very front acts in fact as favoured observation post and headquarters, guarded by the mighty towers of the ancient Porto Pisano (Pisa port), which shared lines of sight with the fort's keep.Among the three built bastions, the one at the northern corner of the curtain, called 'verso terra' (landward) or, according to modern acceptation, 'la Capitana' (the Captain), features a very elongated shape and takes the place of a prominent portion of the outer walls of the old hamlet of Livorno as well as a sizeable conglomeration of medieval dwellings; some assays executed in the area have uncovered the relics of ancient walls.

Palazzo della Fortezza vecchia di Livorno (Palace of Old Fortress, Livorno)
In 1543, Cosimo I de' Medici managed -thanks to a substantial amount of money -to turn the Spanish away from Livorno.A few years later, between 1546 and 1547 the building is "finished ...to wall up" (Frattarelli Fischer, 2018), he decided to build one palace for his family right inside the fortress along with another one on the mainland nearby.The massive, stern-looking Medici building, attributed to the architect Bernardo Buontalenti [Piancastelli Politi, 1995], was built on three levels partly leaning, partly inscribed in the perimeter of the Quadratura dei Pisani.The main body stood over the central fortress platform and leant over the façade of the Quadratura, while the grandducal apartments were placed around the central Quadratura court -later known as "Cortile del Castel Vecchio" (Old Castle Court) -at a lower elevation than the external platform.Very little is known about this Medici residence in Livorno, due to the scarcity of documents either written or iconographical.The Palace suffered severe damage as a consequence of WWII bombings and was later deliberately and completely razed.

Iconographical documents and archive material
To date, the only records regarding the inner structure and layout of both residential and public spaces are provided by a set of plans (Fig. 3) dating from 1669 to 1676 (AAVV, 1980) and two inventories "di tutte le masserizie" (of any furniture and fittings) carried out in the first decades of the 16 th century (Baldini, 2009;Baldini, 2012).Besides, another set of plans dating to the 18th and 19th centuries tracks multiple changes of the intended use of the building after its termination as military stronghold; these included military Academy, bathhouse for inmates and public housing district (Piancastelli Politi, 1995).Photographs shot before WWII show the size of the building and the massive barracks standing on the large squares of the Fortezza.Images dating to the early 20 th century almost exclusively provide overviews of the complex, while the occasional pictures of the interiors refer to the upper rounds and the lower ramp floors.The portion of the building standing on the 14 th century structure (Quadratura dei Pisani) rose for two more levels above the walls of the balcony.On the other hand, both the 18 th -19 th century plans and the photographs show an ongoing building jam due to the scarcity of the space on which all these structures stand.

Status quo geometric survey
The Authors worked on data collected via a Terrestrial Laser Scanner (TLS) survey of the entire fortress (inside and outside), made available by the managing body (Autorità di Sistema Portuale del Mar Tirreno Settentrionale -Port System Authority for the North Tyrrhenian Sea).The survey, available as a point cloud, features a density of about 3 points per square centimetre, certainly suitable for drafting graphical documents at a scale lower than 1:50.TLS data are supplemented with UAV-borne imagery, which allowed in turn 3-D reconstruction of the fortress' exteriors and model texturing.Besides, UAV-borne imagery allows to detect homologous points for orientation of archive images based on unchanged architectural details.The images were acquired by a DJI FC330 drone, which is equipped with a camera with a CMOS 1/2.3"sensor (effective pixels: 12.4 M, FOV 94° 20mm lens -35 mm format equivalent), image size 4000 × 3000 pixels).The 110 images are divided into two series acquired respectively 40 and 60 meters from the top of the walls.

Photogrammetric processing
The photogrammetric project included the set of UAV-borne images of the status quo and five archive images, dating back to the 1930s (Fig. 4-8).
The low quality of the archive images and their shooting geometry, along with relevant scene changes, prevented automatic tie points detection, which therefore have been manually detected.This operation provided several steps, as follows: -identification, by means of historical researches, of regions unaffected by changes and major restoration works since the 1930s; -within the regions previously identified, detection of architectural elements whose definition allows to restrain image collimation errors to a minimum; -selection of the points providing the most homogeneous layout, on both images and 3-D models.
In this case, evaluation of the available historical images allowed to detect the top of the cylindrical tower, the sequence of vertical courses of stone along the brick scarp wall, the cannon mouths on the upper section of the walls and the building at the edge of the southwest bastion.A total of 52 points, 26 of which present on the images collected along the south-to-north direction and 26 in those collected along west-to-east, have been identified and collimated (Fig. 9).These points have been labelled as "ECP_nnn" (Existing Control Point_number), and their coordinates have been extracted from the TLS point cloud.Such a high amount of points as bundle adjustment input was required to minimize collimation errors due to the poor resolution of archive images.Besides, 20 additional check points have been selected for alignment verification.After alignment process a mean 1.3 pixel and 12cm error was calculated.

Integration with archival drawings and historical sources
The points collimated on the historical archive photos and subsequently referred to the UAV model allow to describe some of the main edges of the Palazzo, even if belonging only to its toward the sea (southern and western façades), and partially to its eastern façade towards the internal square of the Fortress.The points refer in fact to the eaves, to some vertical edges and to the windowsills of the Palazzo façades (fig.10).Based on the results of the historical archival sources analysis and the geometric data coming from the UAV model, it was possible to understand, albeit partially, the state of the building before the war events.Based on the critical analysis of this information, some hypotheses about the original structure of the building could be proposed.

RESULTS AND DISCUSSION
The comparative analysis of the TLS survey model of the status quo fortress and the historical plans of the 17 th century, allowed to verify that most of the Palazzo's spaces located under the upper square level of the Fortress are still existing.
As already mentioned, the same cannot be stated for a large part of the Palazzo, demolished during the restoration works following the war damages.The in situ analysis of the fortress and of its 3D model from UAV survey, highlights the presence of traces -more or less apparent -of some of the bearing walls of the Renaissance building.The planimetric layout of the courtyard of the Palace ("Cortile del Castello Vecchio") is in fact still recognizable, as well as the bases of the walls of the buildings which delimited the courtyard itself.On the other hand, the parts of the Palace, built outside the Quadratura on both the west and on the south side, are less identifiable.
The determination of the points deduced from the orientation of historical archive photos in the UAV 3D model, allows to highlight the external alignments of the perimeter walls of the building along the south and west fronts towards the sea.These points, moreover, allow to obtain, with a certain precision, the position of the building eaves and of the sills of some of the second order windows along the above-mentioned façades.Based on this information -either derived from the integration of historical images or deduced from the traces still present on the site -it was possible to trace the planimetric print of the entire building.This represents the first step for a plausible and dimensionally reliable three-dimensional reconstruction of the building.The quota of the points deduced from the orientation of the archive photos, in fact, will provide useful metric information to propose a plausible hypothesis of reconstruction of the inner spaces, in addition to some information about the geometry of the vaulted surfaces and the heights -expressed in braccia fiorentine -of the inner spaces coming from the 17 th century drawings.Other information about the geometry of the covering structures of the inner spaces will be hypothesized on the base of some descriptions in the two inventories "di tutte le masserizie".The 3D reconstruction of the building will also be based on constructive considerations, as for example those deriving from the alignments of the walls of the still-existing spaces under the level of the fortress upper square.
Figure 11 summarizes the first results of the reconstructive study are.The figure compares the ortho-photos of the UAV model and one of the 17 th century archive plans; in both images, the planimetric print of the walls currently identified is colour-coded: in red, the lines deduced from the orientation of the historical photos in the 3D model; in blue, those representing the traces of still visible walls.In both images, the ortho-projections of some the points obtained from the orientation of the historical archive photos in the UAV model are described.

CONCLUSIONS
It has been shown that historic archive images can represent a major information source for 3-D reconstruction of lost architectural volumes.Most of the time, the photogrammetric processing of these images, particularly with low image quality and number and poor shooting geometry, does not automatically allow to obtain point clouds for the relevant volumes.This situation requires therefore to use the orientation results of bundle adjustment for ray intersection.Although available geometric information is not usually sufficient for a complete 3-D reconstruction, it does provide some constraints for subsequent integration of other iconographic sources.This is what also occurred in this case study, for which further research is scheduled for a likely reconstruction of the interiors and the modelling of the resulting geometry hypothesis.The final model of the reconstructed volumes shall eventually be integrated, by means of the photogrammetry points, in the status quo model, for fruition through virtual and/or augmented reality platforms.

Figure 10 :
Figure 10: ECP on archive image At the same time, a critical reading of the archive drawings -and in particular the set of plans dating from 1669 to 1676 -was carried out, in order to detect any traces of the Palazzo walls still visible in the site and, consequently, on the 3D survey model.Based on the results of the historical archival sources analysis and the geometric data coming from the UAV model, it was possible to understand, albeit partially, the state of the building before the war events.Based on the critical analysis of this information, some hypotheses about the original structure of the building could be proposed.

Figure 11 :
Figure 11: Comparison between the ortho-photo of the UAV model and a 17 th archive plan (fig.3); in evidence, the planimetric print of Palace.