OPTO-TECHNICAL MONITORING-A STANDARDIZED METHODOLOGY TO ASSESS THE TREATMENT OF HISTORICAL STONE SURFACES

Monitoring is an essential requirement for the planning, assessment and evaluation of conservation measures. It should be based on a standardized and reproducible observation of the historical surface. For many areas and materials suitable methods for long-term monitoring already exist. But hardly any non-destructive testing methods have been used to test new materials for conservation of damaged stone surfaces. The Nano-Cathedral project, funded by the European Union's Horizon 2020 research and innovation program, is developing new materials and technologies for preserving damaged stone surfaces of built heritage. The prototypes developed are adjusted to the needs and problems of a total of six major cultural monuments in Europe. In addition to the testing of the materials under controlled laboratory conditions, the products have been applied to trial areas on the original stone surfaces. For a location-independent standardized assessment of surface changes of the entire trial areas a monitoring method based on optotechnical, non-contact and non-destructive testing methods has been developed. This method involves a three-dimensional measurement of the surface topography using Structured-Light-Scanning and the analysis of the surfaces in different light ranges using high resolution VIS photography, as well as UV-A-fluorescence photography and reflected near-field IR photography. The paper will show the workflow of this methodology, including a detailed description of the equipment used data processing and the advantages for monitoring highly valuable stone surfaces. Alongside the theoretical discussion, the results of two measuring campaigns on trial areas of the Nano-Cathedral project will be shown.


INTRODUCTION
Monitoring is needed for the planning and assessment of a successful implementation of restoration measures.It has to be a standardized and reproducible observation, acquisition and controlling of processes or changes of historical objects.Depending on the object and the leading questions, there are a variety of different techniques and procedures used.These are, for example, combinations of visual anamnesis combined with technical and scientific examination methods (Nappi, Côte, 1997 andVan Balen, 1997).Within the Nano-Cathedral project (www.nanocathedral.eu),funded by the European Union's Horizon 2020 research and innovation program, new materials and technologies for preserving damaged stone surfaces of built heritage were developed and tested on original stone surfaces of the Cathedral of Pisa (IT), the Cathedral of St. Peter and Mary in Cologne (DE), St. Stephen´s Cathedral in Vienna (AT), Sint Baafs Cathedral of Ghent (BE), Santa Maria's Cathedral of Vitoria (ES) and the Oslo Opera House (NO).On average these trial areas were around 2m² in size.At present, location-independent assessment of preservation treatments still requires destructive methods such as drilling resistance measurement or the extraction of sample material for cross sections and scientific analysis (Auras, Meinhardt, Snethlage, 2010).As a result, only very small areas of the treated surfaces can be examined and so statements on the effects of the materials on larger areas are merely hypothetical.Therefore a suitable workflow using nondestructive testing methods (NDTs) to monitor the entire treated areas was developed.

FUNCTIONALITY OF THE METHOD
Four opto-technical analytical methods regarded as NDTs were used for the monitoring.All technologies used result in images, and by using photogrammetry the different outputs are superimposed distortion-free and combined in a layer-based CAD-plan.In addition this methodology allows the overlaying of the different monitoring campaigns for a better visualization of changes.The NDTs used are:  3D-Scanning: three-dimensional measurement of the surface topography  VIS photography: high-resolution photography in the visible light spectrum  UV-A-fluorescence photography: ultra violet fluorescence photography  Reflected IR photography: reflected near-field infrared photography

3D-Scanning
3D scanning is needed for the visualization of changes in the surface topography caused by continuous damage processes or a growth of the surface e.g.biogenic colonization or flaking and blistering.Furthermore the 3D measurement is used as the basis for rectifying the photographs of the other measurement techniques used.More precisely, Structured-Light-Scanning is used for the sub-millimetre-accurate, high-resolution documentation of the surface topography in 3D.Originally developed for quality control in manufacturing industry, this technology has been used in the field of heritage preservation and restoration for several years.The technique uses a projector which transmits a defined structured light pattern onto the surface of the object at different frequencies.With one or more cameras oriented at a defined angle to the projector, individual three-dimensional points on the surface are calculated using triangulation.The result is a single scan consisting of a point cloud, and by matching several individual scans, the complete object can be recorded.The individual scans are combined into an overall point cloud, which after further processing will result in a 3D surface model.Depending on the resolution of the digital image sensor and the camera lenses used, the resolution of the 3D measurement may vary within a range of approximately 1 mm to 15 μm.In order to ensure the desired resolution and accuracy of the measurement, it is necessary to calibrate the system before starting the measurement (Gühring, 2002, Akce, 2007, Vloth,Breuer, 2011).
For a three-dimensional measurement of the surfaces of trial areas in Nano-Cathedral a Comet L3D 5M from Zeiss Optotechnik was used (Figure 1).In this scanning system both the camera and the projector are fixed in the sensor head.By replacing the lenses, the measuring field of the scanner can be adapted to the required surface resolution.For capturing the trial areas in this project a measuring field with a resolution of 0.3 mm was used.In this case a single scan has a width of about 800 mm.The system uses a blue LED as the light source, so it is able to capture reflective surfaces such as polished marble.
The scanner can be operated on a simple photo tripod and a laptop for controlling.The system can easily be operated on tight scaffoldings.In general, the data was recorded during evening and nighttime to prevent over-irradiation of the structured-light pattern by sunlight.Approximately 10 to 15 individual scans were captured from different angles and positions in order to document the entire trial area.The individual scans were registered to each other as they were acquired.
After acquisition the scan data was processed further on high performance computers in the office.To reduce and compensate for possible errors due to the rough matching carried out on site, a so-called best-fit matching was carried out.Subsequently, the point cloud of the single scans was combined and processed in to a coherent surface model by triangulation.In the case of the Nano-Cathedral trial areas the surface models were created with a resolution of up to 0.3 mm.All these steps are carried out in the scanner's own software, CometPLUS v.9.96.The surface model was then imported to Geomagic Wrap 2017 to remove triangulation errors, i.e. intersections of triangles and spikes.Intentionally no holes were closed in the 3D model so that the data set consists only of measured data with no interpreted or reconstructed areas.Afterwards the surface model was exported as an STL-file and a distortion-free orthoimage was created by using Aspect3D v.16 software.This surface image shows the entire geometry of the trial area (Figure 2).This is used for observing the surface roughness, which is usually more difficult on textured stone surfaces.The generated view of the 3D model was imported to CAD software, providing the basis for rectification of the other images.
Figure 2. Orthoimage of the scan data of trial area 1 at Sint Baafs Cathedral in Ghent Furthermore the surface models are used for a mathematical 3D comparison.By using 3D inspection software, two data sets can be examined revealing differences in the surface topography.With this workflow, changes up to the μm-range that have occurred between different monitoring campaigns can be recorded and visualized easily.In the case of Nano-Cathedral it was possible to capture the initial conditions of the trial areas in 2016 and again in 2017, after cleaning and treatment with newly developed consolidants and protectives.
The 2016 and 2017 data sets were imported into the software and aligned using best-fit matching.It should be noted that the matching was based on stable reference areas where no changes had occurred between the campaigns.In the case of trial area 1 at Sint Baafs Cathedral in Ghent (Figure 3), the matching was processed via the cement mortar joints between the stones.Subsequently, differences between the 3D models can be displayed in a color grid.The green parts show the areas where changes are less than the surface resolution and inside the tolerance (+/-0.15mm).Yellow to red shows a growth of the surface in steps from +0.15 mm up to +15 mm and turquoise to dark blue show a loss of material in steps up to -15 mm.In this way sanding or spalling of the surface, as well as the growth of crusts and lifting of flakes can be measured and visualized in submillimeter accuracy.Processing the 3D comparison of the Nano-Cathedral trial areas was done by using InspectPRO v.5.22 software.

VIS photography
VIS photography is equivalent to well-known, classical color photography where only that part of the electromagnetic spectrum visible to the human eye is captured during image acquisition.The visible light spectrum includes electromagnetic waves in the range of about 400 to 700 nm (Figure 4).Shorter wavelengths in the ultraviolet and longer waves in the infrared are blocked using UV-and IR-filters, often preinstalled directly in the digital camera or its lenses (Verhoven, 2012, Verhoven, 2016).).For a standardized and repeatable VIS documentation of the monitoring areas, a complete documentation of these factors is essential.In addition the use of color management tools, such as "Kodak Color Control Patches" or the "X-Rite ColorChecker" with calibrated color patches and gray scales is necessary to ensure comparable color settings across all the surfaces.All images must be recorded in RAW format to avoid format-related, automated image processing (ibid.and Verhoven, 2010).
In the Nano-Cathedral project a Hasselblad H2 camera with a digital iXpress CFH back (39 megapixel image sensor) and an HC 3.5/50mm II lens was used for the VIS-photography of the trial areas.To ensure uniform lighting conditions and consistent color settings across the campaigns, the images were recorded at night.A BronColor Grafit A4 Power Pack with two Pulso G 3200J flashbulbs and softboxes was used as a light source (Figure 5).The images for the second campaign were recorded using the same set up and settings to ensure color consistency.In some cases the trial areas of Nano-Cathedral were covered with dark crusts and therefore cleaned by micro abrasive blasting before treatment.In this case images of the second monitoring campaign might tend to be over-exposed compared to the first campaign.Because of this, stable reference areas of the stone surface where no cleaning and treatment had occurred were required.The color between campaigns was aligned using these areas and the color patches (Figure 6).

UV-A-fluorescence photography
The electromagnetic wavelengths below nm to about 1 nm include the ultraviolet (UV) spectrum (Figure 4).It is further divided into three to four categories but only two of them are relevant for photography.These are Near UV or UV-A with wavelengths between 320 and 400 nm and Middle UV or UV-B in the range of 280 to 320 nm.Normal camera lenses start transmitting at around 320 nm, conventional image sensors can acquire this wavelength too, so it is possible to record UV-A photographs without too much additional technical equipment.UV-A photography differs, however, from direct UV-A photography and UV-A-fluorescence photography (Dorell, 1994).
For direct UV-A photography a UV-pass filter in front of the lens will restrict the transmitted light to the UV spectrum.The method is also called UV-reflectance-photography because only the UV radiation reflected by the object will be captured.Therefore, whilst the light source used must emit a huge amount of UV light, it can also emit additional wavelengths, since all other interfering wavelengths are blocked by the UV-pass filter.
UV-reflectance-photography can be used for localization of organic material such as biological colonization, but until recently it has rarely been used in the field of heritage preservation.
With UV-fluorescence photography the fluorescent behavior of materials is used for image acquisition.During excitation with ultraviolet light, the object might reflect the energy as radiation in longer wavelengths in the visible light spectrum.This may result in distinct color differences which depend on the fluorescent behavior of different materials.For this method it is necessary that the light source emits only UV light and scattered light of other wavelengths has to be avoided.In addition a UV blocking filter is needed in front of the lens to capture only the UV fluorescence reflection in the visible spectrum.For heritage conservation UV-fluorescence photography is used to detect worn decorations, overwritten inscriptions on paper and certain pigments, as well as organic materials and the growth of gypsum crusts which is important for the observation of the stone surfaces in Nano-Cathedral (ibid. and Warda, 2011, Isacco, Darrah, 1993, Bravo Pereira, 2010).Within the project, UV-A-fluorescence photography was carried out using the same equipment as the VIS photography.In addition, a UV-cut filter was placed in front of the camera lens.
To generate pure UV light, the UV cut filters were removed from the Pulso G bulbs and replaced by Broncolor UV adapters.UV-A-reflectance photography is only used when dealing with highly fluorescent materials, which would lead to overexposure in UV-A-fluorescence photography.For example, lime incrustations and calcium carbonate exhibit strong UV fluorescence resulting in a very bright and shiny white (Dorell, 1994).In this case additional UV-reflectance images may provide more detailed information in those previously overexposed areas.Until now UV-A-reflectance photography has not been needed in the project.
The UV images were recorded in RAW-format.To avoid contamination by scattered light and to guarantee reproducible conditions all images were taken during the nighttime.Based on the same color control patches and reference areas, color adjustment was carried out analogously to the VIS photography and according to the AIC specifications for UV photography (Warda, 2011).These patches might also fluoresce under UV energy but affordable color stable control patches are rare (Warda, 2011).The "Target UV and UV Gray Gard" (UV Innovations, 2018) might guarantee more accurate color adjustment.Nevertheless, getting comparable and reproducible image series using identical control patches between the campaigns is essential.
During image processing there is a difference between UV-Afluorescence and the UV-A-reflectance photography which should be mentioned.Images from reflectance photography are artificially created to display the UV range normally invisible to the human eye.Therefore these images should be displayed in gray scale to avoid ambiguities.Fluorescence photography, however, shows reflection behaviors in the range of the visible light spectrum.Therefore these must be shown in color otherwise characteristic material differences will be lost (Figure 7; Warda, 2011, Isacco, Darrah, 1993, Bravo Pereira, 2010, Cosentino, 2014).

Reflected IR photography
The infrared spectrum is located above the VIS up to a wavelength of about 1 mm (Figure 4).It is separated in to five parts, beginning with near infrared (NIR: 700-1000nm) via short-, mid-and long-wavelength infrared (SWIR: 1.0-3.0µm,MWIR: 3.0-5.0µm,LWIR: 5.0-14µm) up to far infrared (FIR: 14µm-1mm).Photography of the different types of IR is used for thermography and in aerial photography to detect crops and soil marks or for differentiation of vegetation.Already in the NIR spectrum many materials and pigments have a different reflection behavior than in the VIS.Through this it is possible to distinguish between materials that look identical in visual light but differ in IR.With only a little effort it is possible to capture NIR using an ordinary digital camera.The camera has to be modified by removing the IR-cut filter in front of the image sensor, so that it is able to detect electromagnetic waves in a spectrum from around 320 up to 1100 nm.For IRreflectance photography the light source used has to provide IR radiation; though, sunlight, and also many conventional flash lamps, emits enough IR.IR-reflectance photography also requires an IR pass filter to ensure that only light in the IR wavelength range reaches the image sensor (Dorell, 1994, Cosentino, 2014, Sfarra, et al., 2015, Bloisi, et al., 2010, Falco, 2009).
For the reflected IR photography of the trial areas (Figure 8) a modified Canon EOS 5d Mark II with 21 megapixels and a Sigma Art 35 mm f1.4 DG lens in combination with the Broncolor flash system was used.To ensure that only electromagnetic radiation in the IR spectrum reaches the image sensor, a B + W 092 IR pass filter was used in front of the lens.This filter is similar to the often recommended Kodak Wratten 89B (Warda, 2011).
Image processing was done analogously to VIS and UV photography.Color adjustment via the gray patch on the control patches worked well and delivered stable results across the campaigns.IR images will not contain any surface information within the visible light spectrum.To avoid color ambiguity, the image has to be transferred in to gray scale.

CAD plans
After capturing and processing the images, stable, comparable and reproducible color conditions are available.But depending on the variety of cameras and lenses used for the different types of documentation, the single shots will not show exactly the same image section and depending on the camera angle and lens distortion the images are not right in scale.To guarantee an accurate, comparable analysis of the surfaces cross-technology and cross-campaign the images need to be superimposed distortion-free on each other.Therefore the images are further processed in CAD plans using photogrammetry.Usually this method needs targets or control points fixed to the surface in the images and which have been measured by total station for scaling (Wiedemann, 2004).However, scan data make these targets unnecessary.As described above, the scans are true in scale and it is possible to easily render orthoimages from them.These images were imported to AutoCAD 2016 and used as reference.In the next step a UV, VIS or IR image will be imported via the add-on Faro PhoToPlan 2017.By using natural control points such as edges, cracks or holes which are visible on the surface in both, the scan and the image, the images can be rectified.The individual image rectification was carried out with approximately 10 to 15 natural control points per photo, so a submillimeter-accurate rectification and localization of the surfaces was possible.
During the rectification process PhoToPlan creates a new image file in Tiff or Jpeg format.Unfortunately an automatic color manipulation was observed when the image was resampled.This color falsification needs to be compensated for.All images are linked as external references to AutoCAD, so it is possible to adjust the color of the rectified image to the correct values using Photoshop.During this color adjustment, the geometry and number of pixels of the image must not be changed otherwise the orientation and scaling in AutoCAD will be lost.After all of the images had been rectified, readjusted and superimposed in CAD, they were uniformly cropped to show exactly the same area (Figure 9).Per campaign, each technology used is arranged on a separate layer.For easier analysis a 10 cm cross grid and a scale bar are included in the plan layout.This simplifies analysis, measurements and mapping of findings in the plan.The results are available as an AutoCAD DWG-file and layer based PDF.

Report
To guarantee cross-campaign comparable results, an accurate documentation is essential.Therefore the report must contain all information regarding the devices and settings used for each of the different techniques.For 3D-scanning it must include model designation, lenses, resolution and calibration procedures as well as the processing steps and settings.For all methods of photography and for each image the color control patches, camera model and lenses used including all settings i.e. exposure time, aperture, ISO, white balance, color temperature, camera profile (color values) as well as the flash system with its settings, lamps and size of soft boxes need to be noted.As well as the technological components, the surrounding conditions must be documented in the report too.This includes date and time of documentation, weather conditions and temperature as well as the surface conditions.These involve the actual conditions (dry or wet surface) and information about possible conservation measures carried out between the monitoring campaigns.

Figure 1 .
Figure 1.Capturing a trial area at the Oslo Opera House by using the Structured-Light-Scanner Comet L3D Camera and flashbulbs were mounted on tripods.To ensure color consistency as well as balance and exposure adjustment, all photos were recorded twice.In the first shot both the "Kodak Color Control Patch" and "QPcard 101 v3" were manually placed in the center of the image.The second shot used the same settings but without the control patches.Placing the patches in the image center ensures the most uniform illumination.If the corner or sides of the image area are used, external scattered light or possible color changes in the flashbulbs could negatively influence the color processing.Subsequent processing of the RAW images was carried out in Adobe Lightroom software using a calibrated and profiled monitor.During image processing, initial color adjustments were made based on the gray fields of the control patches.The use of two different patches in the same picture compensates potential errors.Afterwards the same settings were applied to the photo without patches to ensure color consistency.Finally the lens distortion was removed and the photos exported as high-resolution Tiffs using Adobe RGB color space.

Figure 3 .
Figure 3. Trial area 1 at Sint Baafs Cathedral: 3D comparison of campaign 2016 and 2017.Stable areas are shown in green, red to yellow shows a growth and turquoise to blue a loss of the surface

Figure 5 .
Figure 5. VIS-documentation of a trial area at Sint Baafs Cathedral in Ghent

Figure 8 .
Figure 8. IR image of trial area 1 at Sint Baafs Cathedral in Ghent

Figure 11 .
Figure 11.Trial area 3 at Santa Maria de Vitoria.3D-Comparison of 2016 and 2017.Green: stable surface +/-0.15mm; yellow to red: growth up to +1 mm; turquoise to blue: loss up to -25 mm