VIRTUAL AND PHYSICAL RECOMPOSITION OF FRAGMENTED ECCLESIASTICAL FRESCOES USING A PHOTOGRAMMETRIC APPROACH

The octagonal domed church of the Christ Antiphonitis in the district of Kyrenia (Cyprus) was originally completely decorated with frescoes along its interior walls. Two of these are exceptional for their artistic and historic value: the story of the Tree of Jesse (a pictorial genealogy of the Virgin) located on the southern wall of the octagon, and the Last Judgment, on the northern wall. Following the invasion of Cyprus by Turkish military forces in 1974, looters stripped many of the region churches, removing an estimated 15,000 to 20,000 icons and several dozen major frescoes and mosaics, intending to sell them illegally on the antique market. The Church of Antiphonitis was among them. The walls with the two biblical episodes suffered major damages and big portions of their pictorial decoration were removed. Since the end of the 1990s, due to efforts of Cypriot authorities, more than 70 fragments of its frescoes returned from USA and Europe to the Byzantine Museum of Nicosia, where they are currently under conservation and restoration for future display. These were digitally documented through high-resolution ortophotos. The inner space of the church was documented by similar means, in order to virtually re-position the frescoes in their original locations. The virtual re-composition of the frescoes along the looted walls helped quantifying the missing parts, correctly re-locate virtually each fragment at its original position, obtain accurate colour information and prepare a digital musealisation product, to be included in the permanent exhibition display at the museum which will feature a 1:1 scale reproduction of the church walls.


INTRODUCTION 1.1 The Church of the Christ Antiphonitis and Cypriot Heritage in the Occupied Area
The Church of Christ Antiphonitis (literally the "Christ who Responds") near the village of Kalogrea, (Kyrenia district, Cyprus), is all that remains today from a monastery complex founded most probably in the 12 th century.It is built in the octagonal 'insular' style with a dome, and it is the only one of its type to have survived in good condition in Cyprus (Figure 1).

Figure 1. The Christ Antiphonitis Church, Cyprus
The church was decorated with Comnenian style murals immediately after its construction.Few of the original paintings have survived, the church being re-painted in the late 15 thearly 16 th century with new biblical scenes (Stylianou, 1997).
Among these, our research focused on the representation of the Last Judjment (north wall) and the Tree of Jesse (south wall).The 1974 Turkish invasion of Cyprus and the subsequent occupation of its north part has negatively affected its cultural heritage, and, despite internationally treaties, its protection.More than 580 churches of various Christian doctrines were looted, vandalised or damaged, often irreversibly (Hadjisavvas, 2004;Jansen, 2005;Chotzakoglou, 2008); stolen ecclesiastical icons and vessels, church frescoes or mosaics ended up in the black market of Europe's illegal antiquities and in auctions around the world.The church of Antiphonitis made no exception, its main frescoes being smuggled out of the island, only to recently return, following their identification in various collections around the globe and international court decisions.The returned frescoes are currently stored and exhibited at the Byzantine Museum in Nicosia, Cyprus (Figure 2).

Related Works
Digital fragments recomposition is a topic still under development according to different approaches proposed in the literature.In the domain of virtual restoration of frescoes, most examples focus on the automatic matching of scattered bidimensional datasets, based on shape properties of pictorial content of decoration (Papaodysseus et al., 2002;Renna et al., 2005;Fornasier et al., 2005;Riccio et al., 2015) and tridimensional datasets (Huang 2006;Brown 2010).Although advanced image processing techniques are being incorporated in most recent systems, semi-automatic solutions where cultural heritage professional are involved still represent the most reliable approach (Scopigno at al., 2011).An interesting approach to virtual recomposition of fragmented frescoes exploiting image and range based techniques has been presented in (Chiabrando et al., 2014) for the church of San Pietro in Tuscany (Italy) severely damaged by an earthquake.In (Caroti et al., 2015), authors highlighted the advantages of a multidisciplinary approach, coupling active and passive sensors, for frescoes documentation and restoration of the case study of the Ronconi palace in Pisa (Italy).

Musealisation
The repatriated frescoes are currently being under study for restoration and exhibited in a permanent display at the Byzantine Museum.A digital installation, together with the physical reproduction of the two original walls (scale 1:1), will show the interior of the church with its re-positioned frescoes.
Visitors will be able to see high-level details of each fresco, the entire composition and take a virtual tour inside the church.
Information to be included regard the church itself and its architecture, narratives of each fresco theme but also their recent story on how they were smuggled out of the country and finally repatriated.In this sense, the 3D model of the church serves as a digital platform for storytelling, where the narrative starts by users choosing which frescoes to visualise.Since historic documentation of the church is limited to a few colour and black & white pictures, the material used for the musealisation consists mainly of recently acquired ortophotos of each repatriated fresco fragment and a set of digital photographs documenting the interior of the church, later used to create its 3D model, as a support for virtually re-position the frescoes fragments and analyse architectonic details.
Although range-based sensors such as laser scanners are able to provide a large amount of accurate 3D data in a short time span, they show limitations in terms of cost and portability.Imagebased modelling techniques instead are portable and the sensors are often low-cost.From this perspective, orthophotos represent a powerful and useful solution for Cultural Heritage since they couple metric and radiometric information allowing a complete representation of the analysed object (Boccardo et al., 2001;Balletti et al., 2003;Chiabrando et al., 2015).

Frescoes Fragments Photogrammetric Survey
Due to geo-political constraints in the northern occupied area, photogrammetry has been chosen for the field data acquisition, due to its speed, reliability and non-invasive approach (Remondino et al., 2014a).Two photogrammetry surveys have been set up at the Byzantine Museum of Nicosia and at the Antiphonitis church respectively.Currently more than 70 fragments were repatriated and now stored at the Byzantine Museum.Some are displayed on shelves, while others are still covered with a protective tissue waiting for restoration.
The photogrammetric documentation of the frescoes fragments followed the CIPA 3x3 rules in order to ensure a high accuracy of results (Waldhäusl et al., 1994).Its main purpose was to create orthophotos with accurate colour reproduction.Two scale bars in horizontal and vertical positions were placed at the bottom and aside of each fragment.The frescoes were properly lighted using two professional photography lamps, and a colour calibration board has been used (Figure 3).The camera used is Nikon D3X with a 24 megapixel full-frame CMOS sensor (6 µm pixel size).Before the survey the camera was calibrated using a test field to insure maximum accuracy and internal orientation parameters fixed.The lens used was a Nikon ED AF-S VR-Nikkor 70-200 mm 1:2.8 G.

Figure 3. Fragments photogrammetric survey
Due to the variety in fragments' dimensions, they were divided in two groups.The expected ground sample distance (GSD) was calculated at 0.2 mm for all of them.The camera's autofocus and image stabilizer were disabled in order to avoid unwanted changes of interior orientation parameters during the photogrammetric survey and insure its rigidity.All acquisition protocols were performed with the same photographic parameter setup (Toschi et al., 2014).For the large fragments (~ 75 cm h x 40 cm w) a 80 mm focal length was selected and kept fixed with an insulating tape, yielding a camera-to-object distance of about 260 cm (Figure 4).For small size fragments (~ 55 cm h x 20 cm w) a 135 mm focal length was selected yielding to a camera-to-object distance of about 450 cm (Figure 5).
Figure 5. Small fresco fragment from Tree of Jesse wall

Antiphonitis Church Photogrammetric Survey
Due to logistic difficulties in setting-up a documentation campaign involving terrestrial laser scanners, total station and GPS, it has been decided to 3D document the Antiphonitis church by means of terrestrial photogrammetry, a portable and reliable solution that can achieve accurate results for systematic 3D surveys (Wenzel et al., 2013;Luhmann et al., 2014;Remondino et al., 2014b).Although declared a monument, the church is largely neglected and un-maintained, therefore vegetation along its exterior walls obscure large parts of its façade.Therefore, it has been decided to focus on the interior, while the documentation of the external walls is under study.
The main goal of the survey was to obtain accurate architectural details of northern and southern walls of the octagon, to later serve as reference frameworks for the recomposition of the frescoes fragments.The entire inner part of the building was therefore documented, for further conservation and valorisation studies, using the Structure from Motion (SfM) approach.The survey of the inner part of the church took place during the opening hours of the site (10 am -2 pm).(Figure 6).The camera used was a Canon 600D, 18-megapixel CMOS sensor (4.3 µm pixel size), equipped with a Canon EF 20 mm 1:2.8 prime lens mounted on a tripod.For the northern and southern walls the ground sample distance (GSD) at an average camera-object distance of 5 m was calculated in ~ 1 mm, with an image scale of 1/250.Ground truth measurements were taken to scale the model.The image acquisition was performed without a proper light setup, in an environment with changing luminosity.A standardised colour chart was used for each image sequence.

Orthophotos Creation
The digitized frescoes fragments at the Byzantine Museum and the image dataset of the Antiphonitis church were processed using Agisoft Photoscan software.Each photo was preprocessed to equalize its colours according with to a colour checkerboard, and the background has been masked to facilitate and improve the quality of the alignment process.The extracted tie points were filtered out in order to limit their image reprojection errors below 0.5 pixel and to keep only well distributed and reliable points.A dense point cloud, a mesh 3D model and orthophotos with an average pixel dimension of 0.4 mm were finally created for each fresco fragment (Figure 7).The same procedure was adopted to create ortho-rectified images of the northern and southern walls of the church with a pixel dimension of ~ 3 mm (Figure 8).The masking procedure of the wall dataset was a particularly time-consuming process, due to the presence of light spots that occurred during the photos shooting, that appeared because of the uncontrolled light environment mentioned above.The final number of orthophotos amounts to 72 frescoes fragments and 2 walls (Table 9).
Prior the virtual repositioning of frescoes, some orthophotos of fragments still covered by tissues for conservation purposes were pre-processed with a photo editing software in order to improve their colour information (i.e.white balance, colours enhancing, etc.) and consequently their readability.9. Orthophotos dataset details

Orthophotos Mosaicking
Before the final alignment of all fragments in a single reference system, the current situations of both walls, originally fully covered by frescoes, was assessed.The dense point clouds of the two walls obtained with image-based modelling technique have been analysed for a preliminary evaluation of the current conditions using the CloudCompare open source software (Cloud to Mesh distance tool).For each point cloud, a best fitting plane was first extracted and used as reference, resulting in a mean plane fitting RMS of ~ 0.015.The walls portions showing a greater positive distance from the best fitting-plane are protruding with respect to the rest of it.
This thickness difference is apparently due to presence of surviving fragments.
After the calculation of the distances between each point cloud and its best-fitted plane, the resulted scalar field visualization tool highlighted the parts of the wall where frescoes survived (red areas) and the damaged surface of the wall from where frescoes were ripped off (green areas) (Figure 10).All orthophotos were imported into AutoCAD environment for a correct alignment within a unique reference system.After the photogrammetric process all images were properly scaled and no further adjustment was necessary.Sixty-eight of the repatriated fragments were virtually recomposed to their correct location: the southern wall (Tree of Jesse) was virtually recomposed from 32 fragments, while 36 fragments were repositioned along the northern wall (Last Judgment) (Figure 11).The remaining four small missing fragments, which represent plant decorations and probably belong to the scene of the Tree of Jesse, have an uncertain collocation.Two more rectangular frescoes with saints in sitting position were identified as coming from the adjacent columns that flank the northern wall of the church.The proposed reassembly of both walls was finally validated by CH professionals and will serve as main reference for the physical reproduction inside the museum halls.
Figure 11.Last Judgment fresco virtual recomposition

Frescoes Area Analysis
Starting from the best fitting plane results, a quantitative analysis of the walls area, calculated in square meters (m²), was performed in AutoCAD.The assessment was made on orthophotos with and without the fragments' virtual disposition.
The Tree of Jesse wall extends over ~ 15.7 m².The looted area is ~ 7.20 m², or ca.45% of the original wall.The recovered fragments represent an average surface of ~ 3.65 m², thus reducing the missing frescoes to ca. 23% (Figure 12).The Last Judgment scene extends over an area of ~ 17 m².The larger size is due to the absence of a window, not built or closed when the church was re-painted in the late 15 th century.This area showed a percentage of missing frescoes close to ca. 75% (~ 12.8 m²).After the repositioning of the 36 fragments, the missing surface decreased to ca. 28%, equivalent of ~ 4.75 m² (Table 13).In addition to this, it should be considered that ca.3% to 5% of the frescoes on each wall were totally destroyed during their looting.
Figure 12.Another advantage of the virtual re-position of the fragments was to clearly identify the missing portion of frescoes.This was possible thanks to the comparison with historical images.The latter have been imported into AutoCAD and processed in order to highlight lost areas.Graphic pattern has been created on historical images to help restores to visually identify missing surfaces (Figure 14).-Hakim et al., 2003;Barazzetti et al., 2010;Doneus et al., 2011;Verhoeven et al., 2011;Verhoeven et al., 2012;Lo Brutto et al., 2012;De Reu et al., 2013;Manfredini et al., 2013).Although these procedures may suffer from reduced accuracy when applied to large structures and buildings without the support of a proper topographic survey, they resulted suitable for the purpose of the present study, i.e. the 3D reconstruction of the Antiphonitis church interior architecture.Photogrammetric rules have been followed as much as possible according to the environment and operating working conditions.The building was subdivided in blocks; for each one a sequence of images was taken.The image acquisition was performed without a proper light setup but in the natural light changing condition of the church.The only possible shrewdness was to turn off the yellowish lights emitted by the chandelier hanging from the vault ceilings.The camera model and setup was the same used for the looted walls.In order to digitize the entire structure some 700 pictures were acquired from the ground level, in raw image format.The extracted dense point cloud consists of ~ 255 mil points, with an average spatial resolution of 0.6 mm.This was obtained by applying a high redundancy and image overlapping.The final textured mesh model will be further used in the musealisation process (Figure 15).It is also a comprehensive documentation outcome, preserving with a high accuracy the geometric details of the church's internal architecture.Thus plans, sections or various views can be extracted from the 3D model and further used for scientific, administrative or dissemination purposes.

Figure 2 .
Figure 2. The frescoes fragments examined in the storerooms of the Byzantine Museum, Nicosia

Figure 4 .
Figure 4. Large fresco fragment from Last Judgment wall

Figure 6 .
Figure 6.Antiphonitis Church, Interior, view of the dome

Figure 7 .
Figure 7. Fragment Orthophoto from Tree of Jesse wall

Figure 10 .
Figure 10.Last Judgment fresco, cloud to mesh distance Cultural Heritage experts contributed to the final recomposition pipeline, by identifying correct matches and positions.Historical images showing the status quo before the looting were used as visual reference.For a better comprehension of the visualisation outcome, it has been decided to represent the recovered frescoes in true RGB values, while the reference background walls appear in grey scale, in order to facilitate the comprehension of the in situ situation today and the virtually repositioned fragments.Sixty-eight of the repatriated fragments were virtually recomposed to their correct location: the southern wall (Tree of Jesse) was virtually recomposed from 32 fragments, while 36 fragments were repositioned along the northern wall (Last Judgment) (Figure11).The remaining four small missing fragments, which represent plant decorations and probably belong to the scene of the Tree of Jesse, have an uncertain collocation.Two more rectangular frescoes with saints in sitting position were identified as coming from the adjacent columns that flank the northern wall of the church.The proposed

Figure 14 .
Figure 14.Tree of Jesse fresco, unrecovered area assessment

Figure 15 .
Figure 15.3D mesh model of the Antiphonitis church Tree of Jesse fresco virtual recomposition, surface analysis

Table 13 .
Frescoes area analysis