MAXIMIZING THE VALUE OF PHOTOGRAMMETRIC SURVEYS IN THE CONSERVATION OF WALL PAINTINGS

In the field of wall painting conservation, photogrammetric surveys are an increasingly common documentation tool. The outputs from these surveys, including captured images, orthophotos, and three-dimensional models, are used by conservators to monitor and record condition phenomena and conservation treatments. To serve these conservation needs, the outputs should have visual fidelity, meaning they are a close representation of the subject based on their spatial resolution, overall focus, even lighting and color accuracy. However, wall paintings present unique technical challenges for photogrammetry considering their often large scale, irregular topography, range of surface attributes, and access issues. As a documentation technique more commonly applied to capture the geometry of buildings and objects, using photogrammetry to record both the topography and fine surface details of a wall painting requires thorough planning. This paper examines specific challenges in documenting wall paintings and how collaboration between conservators and heritagerecording specialists has improved practice and maximized the value of photogrammetric surveys for conservation.


INTRODUCTION
In the past decade, with the advent of powerful algorithms in computer vision delivered through user-friendly software, photogrammetry has become an increasingly utilized documentation technique in the conservation of wall paintings. 1 By generating comprehensive sets of images, orthophotos, and three-dimensional models, photogrammetric survey data enables improvements to conservation activities such as condition recording and monitoring.
For conservation purposes, the most important aspect of the photogrammetric survey data is the 'visual fidelity' or faithfulness to the original object (MacDonald et al., 2006). In this paper, visual fidelity refers to how accurately an image reproduces a wall painting in terms of its spatial resolution, overall focus, even lighting, and color accuracy. As these surveys are often undertaken by heritage-recording specialists for use by conservators, achieving satisfactory results requires close collaboration and mutual understanding between these two professions.
Photogrammetry aims to capture the geometry of an object using a workflow that is often tailored for documenting buildings and structures. However, when applied to wall paintings, which have qualities that are both two-dimensional and three-dimensional, the resulting images may not adequately reproduce their surface attributes to meet specific conservation needs. This paper 1 Reflectance Transformation Imaging (RTI) is another common documentation tool, however, it is used primarily for the rendering of surface topography, and unlike photogrammetry, explores the challenges encountered in the photogrammetric documentation of wall paintings, identified through the ongoing collaborative work between the Getty Conservation Institute (GCI) and Carleton University's Immersive Media Studio (CIMS), and highlights how conservators and heritage-recording specialists are developing solutions to achieve optimal survey results to enhance condition recording and monitoring in conservation projects.

TECHNICAL CONSIDERATIONS FOR ORTHOPHOTOS
The outputs generated in a photogrammetric survey of wall paintings, usually captured and processed images, orthophotos, and three-dimensional models, provide a metric record of the paintings at a single point in time. The most utilized output are orthophotos, which are rectified images that are corrected for most distortions (Granshaw, 2020). Orthophotos are typically generated at the beginning of a conservation project to serve as basemaps for recording and graphically documenting condition and intervention, and as a visual reference for condition monitoring. An orthophoto also enables a conservator to review an entire wall painting in a single undistorted image, which can help to visualize spatial patterns of deterioration and aid conservators in diagnosing their causes.
lacks the ability to capture large areas of wall paintings at high resolution, which is necessary for site-wide condition monitoring.
To support these activities, the orthophoto should have maximum visual fidelity in the following aspects: • Spatial resolution • Image focus • Even lighting • Color accuracy As visual fidelity is dependent on the way the images are captured, discussion between the conservator and the heritagerecording specialist on each of these aspects is vital prior to undertaking the survey.

Spatial resolution
Determining the appropriate spatial resolution of capture for wall paintings is challenging, but it is a critical decision as it will have a direct impact on a conservator's ability to identify change to the paintings in the future.
In 2017, under a partnership between the GCI and Egypt's Ministry of Tourism and Antiquities, CIMS undertook a photogrammetric survey of the wall paintings in the Tomb of Nefertari, Valley of the Queens, Luxor, Egypt. The aim of the survey was to create a comprehensive three-dimensional record and to generate orthophotos, which were used to monitor and graphically document any change that occurred after the reopening of the tomb to visitors in 2016.
With 520 square meters of painting and a complex condition history (namely due to salts), the tomb presented a challenge in predetermining the desired level of detail in the images, and therefore in selecting an ideal spatial resolution for future monitoring. Based on the size of the tomb, the amount of painting, and constraints (narrow spaces, limited time on-site, and available equipment), the tomb was captured with an average spatial resolution of 0.35mm/pixel with a Nikon D800 (Figure 1).
A condition assessment of the wall paintings undertaken in 2020 found that the resolution of the captured images and orthophotos was not high enough to clearly distinguish small losses and cracks. This prompted a deeper exploration of the relationship between the spatial resolution of capture and the level of visible detail in the captured and processed images. 2 The ideal spatial resolution for the photogrammetric survey of wall paintings can be identified with on-site tests, where an area of painting with a detail of interest is captured at a maximum possible GSD (based on available equipment) and Spatial resolution for photogrammetry can be defined as the Ground Sample Distance (GSD), given as a real-world measurement of the distance between two pixel centers in an image, which is determined by a combination of the sensor size, pixel size, focal length, and the distance of the camera from the surface (  GSD cannot be equated to the size of the smallest visible detail in the image, as two pixels do not provide enough information to be understood by the human eye. The smallest feature of interest should be captured with at least four pixels across its width (i.e. the GSD of a 1mm wide crack, should be at least one-fourth the width of the crack, or 0.25mm/pixel) (Verhoeven, 2016). 2 Other factors, in addition to GSD, affect the level of visible detail in an image. While a small pixel size can increase the number of pixels on a sensor and result in a finer GSD, this can produce a noisier image which obscures detail. Similarly, if a high ISO setting is used the image can exhibit noise.
While it is possible to achieve a predetermined level of detail by balancing GSD, pixel size, and ISO, identifying the smallest feature of interest is not straightforward. As shown in the Tomb of Nefertari, it is difficult to anticipate the scale of change for future monitoring at the start of a conservation project. In practice, the resolution of photogrammetric surveys is often determined by the available equipment, degree of access to the painting, time on-site, and data capacity constraints (for both processing and storage). The GSD has a significant impact on the survey time and volume of data produced; for example, increasing the resolution by a factor of two will quadruple the number of images, requiring longer capture and processing.
To understand the range of resolutions used in the documentation of wall paintings, publications describing relevant photogrammetric surveys were reviewed, with results shown in Table 1.
then the image is subsampled to progressively lower resolutions to identify the GSD at which the detail can no longer be resolved. This will provide a range of acceptable GSDs. 5cm The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August- There is a wide range of spatial resolutions for the photogrammetric capture of wall paintings, which are selected based on the needs of each project (primarily to create basemaps for graphic documentation) and resource constraints. For example, Lucet 2013 arrived at a resolution of 0.25mm/pixel based on their limited time, equipment, and data capacity, capturing 964 images, but indicated that it was sufficient for the conservation needs of the paintings. Chiabrando et al. 2015 captured the lowest resolution of the reviewed articles with 4.4mm/pixel which was determined by the distance of scaffolding to the ceiling paintings. The Cultural Heritage Imaging (CHI) survey of the Diego Rivera Pan American Unity wall painting noted that their 0.12mm/pixel resolution enabled brushstrokes to be visible in the final orthophoto, but this necessitated capturing over 2000 images with significant processing time and data management infrastructure (CHI, 2021).
In general, it is preferable to undertake a survey at the highest resolution possible, however, if there are constraints due to time, access and data storage, the overall capture can be carried out at a lower resolution with select areas captured at a higher resolution for detailed recording or condition monitoring.

Image focus
To utilize the captured level of detail, the images must be in focus. However, both a wall painting's size (necessitating a large number of images) and irregular surfaces make it difficult to ensure that each image is in focus and within the depth of field. Zhang et al. 2011, Lucet 2013, and Vidman et al. 2011 were challenged by the topography of wall paintings, noting that areas of the paintings fell beyond their depth of field.
In focus capture is more difficult when recording the geometry of wall paintings using convergent imagery as it can increase the occurrence of only partially focused images. However, if two separate capture workflows are used, one for geometry and one for surface attributes, this can ensure ideal results for both purposes. The CHI Diego Rivera project captured the painting's geometry with a wide-angle lens using convergent images, then recaptured the painting's surface attributes using a longer focal length with parallel images. With this system, the details of the wall painting were documented with a lens providing reduced distortion, a higher spatial resolution, and fewer depth of field concerns. Using a scissor lift to keep a consistent distance from the wall, they had the access and mobility to achieve optimal results (CHI, 2021). In some archaeological contexts, this level of access and mobility are not achievable.
Increasing image overlap can ensure that an orthophoto is in focus. In the past, a minimum image overlap of 60% was suggested, but today 80% overlap is recommended to increase image redundancy, providing more options for the software to use in orthophoto generation (3DFlow, 2019; Agisoft LLC, 2021). For example, if seven images are taken with 80% overlap, the first, third, fifth, and seventh images have 60% overlap and the first and fifth images are still overlapped by 20%. However, out of focus regions can still appear in the final orthophoto due to errors in image selection made by the software or poor mesh generation, necessitating a final review of the orthophoto and image re-assignment (Agisoft LLC, 2021, 98).
The camera aperture can also influence an image's overall focus.
In general, small and large apertures should be avoided to prevent diffraction or a narrow depth of field respectively; for most lenses the sweet spot of the lens is generally three stops from the largest aperture.

Even lighting
Survey images and orthophotos with uneven lighting make it difficult for conservators to interpret areas that appear brighter or darker as either inherent to the painting or an artifact from the amount of light hitting the surface. However, a wall painting's large scale, varied topography, and reflective surface textures (e.g. gloss, gilding etc.) also present obstacles in generating images with even lighting. Achieving even lighting using metered continuous light sources, such as LEDs (Historic England, 2017), is not always a practical approach. The low brightness of continuous light sources necessitates a lengthy shutter speed, risking motion blur even when using a tripod during capture. On-site ambient lighting, either from natural or artificial sources, is not always possible to control and may preclude the use of continuous light sources, which are often not powerful enough to compensate for it.
Speedlights with diffusion are frequently used in the documentation of wall paintings as they can be compact and are bright enough to support handheld capture. However, speedlights are not recommended by Agisoft Metashape as they can result in uneven lighting from improper placement or insufficient diffusion, hampering the software's ability to generate an accurate three-dimensional model (Agisoft LLC, 2021). Uneven lighting can easily occur on wall paintings while using speedlights due to changes in surface topography, the distance of the devices to the wall, and paint color or surface gloss.
In the Tomb of Nefertari, speedlights were the best lighting option due to the limited space. The speedlights were set-up to limit repositioning during capture and their light was diffused with reflective sheets (Figure 3). However, areas of uneven lighting were present in some of the orthophotos (Figure 4). This was due to on-site obstacles: railings close to the paintings cast shadows, localized areas of painting were glossy and reflective, and fragile architectural features limited access to large areas of painting.

Color accuracy
Color is an important attribute of a wall painting and images need to be color corrected to serve as a useful record for future condition monitoring. Due to a wall painting's large scale, lighting conditions can change during the survey and color charts and gray cards designed for conservation imaging should be regularly included in the capture. In the Tomb of Nefertari, color charts and gray cards were used successfully to color correct the images ( Figure 5). Maintaining color accuracy during photogrammetric imaging and processing has been investigated on a small canvas painting in a lab by CIMS in Dhanda et al. 2019. In this research, Agisoft Metashape was found to process images without manipulating the color values up to 16-bits, whereas RealityCapture altered color values past 8-bits. This indicates that some photogrammetric software can affect the color accuracy at high bit depths. Additionally, Dhanda et al. 2019 recommended two separate capture workflows to achieve both geometric accuracy and color fidelity with glossy painted surfaces. A geometric capture was carried out at a closer distance with a 50mm lens, followed by a color capture with a 300mm lens placed outside of the range of specular reflection. The color and exposure were corrected on the RAW images using both a gray card and a color chart and converted into TIFFs. These workflows can be applied on a larger scale to achieve color fidelity in wall painting surveys.

THE VALUE OF CAPTURED IMAGES AND THREE-DIMENSIONAL MODELS
While orthophotos are the most common output from a photogrammetric survey of wall paintings, the captured and processed images and three-dimensional models are also valuable resources for conservation activities. The survey images are useful in condition monitoring as all areas of the painting will be visible in at least one image. These image files tend to be smaller than orthophotos and are therefore easier to open and use on-site. In the 2020 condition assessment of the Tomb of Nefertari, the conservators used the images from the 2017 survey in tandem with the generated orthophotos to identify change. In using the captured images as a monitoring reference, level, parallel images of the paintings were the most useful to the conservators. However, when building a photogrammetric model, tilting the camera slightly to the right or left during capture does not affect the end results, as long as there is sufficient overlap, therefore a heritage-recording specialist may not consider this an issue. The future use of these images for monitoring should be accounted for in the survey protocol.
The three-dimensional model is another important tool for conservators in monitoring wall paintings. Changes to surface topography, such as losses and lifting, are easier to analyze on a three-dimensional model than on a two-dimensional image. Painted details and condition phenomena can be measured from the model, making off-site analysis easier and more accurate. The model can also be segmented into individual sections, providing The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China the visualization benefits of orthophotos while preserving the three-dimensional geometry. The GCI is developing methods to use photogrammetric survey data as a diagnostic tool to objectively detect and quantify condition changes in wall paintings (Rose et al., 2021).
Conservators may not always be aware of the utility of the data produced by a photogrammetric survey and how it can improve conservation practice. Collaboration between heritage-recording specialists and conservators can help to identify potential uses for the data to maximize the value of the outputs in serving the needs of a site.

CONSIDERATIONS FOR ON-SITE PRACTICES
As outlined by Santana et al. 2019, during any survey the values and material integrity of the site must be protected. While photogrammetry can be thought of as non-contact, the placement of temporary physical targets as Ground Control Points (GCPs) around the site is common practice. Most heritage-recording specialists are conscientious and know that GCPs should not be placed on painted surfaces, but other locations, such as architectural features, unpainted plaster, or original masonry, can also be fragile. 3 The placement of equipment and scaffolding should also be carefully considered. It is recommended that heritage-recording specialists and conservators review these considerations together on-site prior to undertaking the survey.

DOCUMENTATION AT MYIN-PYA-GU
Myin-pya-gu in Bagan, Myanmar is an 11 th century Buddhist temple with a square circumambulatory corridor. Wall paintings and painted niches containing Buddha statues decorate the interior of the temple. Multiple photogrammetric surveys have been undertaken in Myin-pya-gu for different purposes and they demonstrate how variations in capture approach affect the visual fidelity achieved. In 2018, as part of professional training to the Department of Archaeology and National Museum of Myanmar, orthophotos were generated of the interior walls of the corridor using a Nikon D750 at a spatial resolution of 0.5mm/pixel and speedlights with diffusion directed towards the subject (  In 2020, the temple was surveyed again using laser scanning and photogrammetry to create floor plans and elevations as part of the Getty Conservation Institute and Department of Archaeology and National Museum's Bagan Conservation Project. The entire circumambulatory corridor was documented at 0.2mm/pixel spatial resolution using a Sony a7R IV, capturing 6700 images with speedlights reflected off a white sheet ( Figure 6, Figure 7).  Another survey is required to record the decorated elements in Myin-pya-gu with higher visual fidelity. To plan the documentation strategy, the two previous sets of photogrammetric data were analyzed. The resolution of the 2018 orthophotos was not high enough to discern areas of conservation interest including cracks and details on the painting. The images were not completely in focus and intermittent bright and dark spots from speedlights without enough diffusion made legibility of the paintings difficult. The 2020 survey images had a high enough resolution to discern areas of interest, and the evenness of lighting was improved. However, in both sets of orthophotos, the niches and Buddha statues cast shadows onto the painted surfaces behind them, resulting in a loss of information in those areas.
The previous two sets of orthophotos identified many of the challenges in recording complex decorated elements, particularly in capturing three-dimensional niches and Buddha statues, which require careful lighting and a deep depth of field. The ongoing collaborations between the GCI and CIMS, the existing surveys of the temple, and recent research into wall painting capture have provided an opportunity to develop a fully collaborative approach to achieve optimal results in the documentation of decorated elements at Myin-pya-gu.

CONCLUSIONS
This paper reflects on how conservators use photogrammetric survey outputs and the ways in which increased communication and collaboration between conservators and heritage-recording specialists can meet the needs of conservation projects. Due to the complexity of wall paintings and the important role that photogrammetry now plays in their conservation, it is critical that the conservator undertake an assessment of the paintings prior to a survey to identify the spatial resolution required. Together with a heritage-recording specialist, an appropriate capture methodology can be developed to meet the needs of the site, while being mindful of constraints. In highlighting the importance of visual fidelity in the documentation of wall paintings, the aim is to encourage greater consideration and strategic planning at every stage of a survey. Wall paintings present unique challenges and there is no single solution or workflow, instead approaches must be tailored to meet the needs of each site.