MULTISPECTRAL PHOTOGRAMMETRIC DATA ACQUISITION AND PROCESSING FOR WALL PAINTINGS STUDIES

In the field of wall paintings studies different imaging techniques are commonly used for the documentation and the decision making in term of conservation and restoration. There is nowadays some challenging issues to merge scientific imaging techniques in a multimodal context (i.e. multi-sensors, multi-dimensions, multi-spectral and multi-temporal approaches). For decades those CH objects has been widely documented with Technical Photography (TP) which gives precious information to understand or retrieve the painting layouts and history. More recently there is an increasing demand of the use of digital photogrammetry in order to provide, as one of the possible output, an orthophotomosaic which brings a possibility for metrical quantification of conservators/restorators observations and actions planning. This paper presents some ongoing experimentations of the LabCom MAP-CICRP relying on the assumption that those techniques can be merged through a common pipeline to share their own benefits and create a more complete documentation.


A short review of image-based multimodality for CH studies
The documentation and study framework of Cultural Heritage (CH) objects requires nowadays a wide range of skills, knowledge and experts coming from different domains and research fields (Boochs et al., 2014). The CH community is currently trying to redefine such interdisciplinar approaches into a complex entanglement of multimodality and interoperability issues. Similarly the documentation process is also led by the use of different and complementary scientific imaging techniques which benefit nowadays of the technological advancements in both hardware or software. In the field of wall paintings studies several imaging techniques are commonly used firstly to provide documentation supports for diagnosis and analysis purposes and subsequently used for decision making and action planning in term of conservation and restoration. Regardless of the complete 2D/3D sensing technologies the CH community is currently benefiting (NIR, SWIR, TIR, HSI, infrared reflectography, X-ray, RTI, microscopy, tomography...) this paper will focus only two photographic-based techniques ; Technical Photography (TP) and Close-Range Photogrammetry (CRP). Digital CRP has been used widely by CH community to document geometrical features providing accurate footprints of the visible surface of an artifact (i.e. the appearance of this object with a definite spatial resolution and at a given time). Indeed recent development of algorithms in term of automation enables quite straightforwardly to provide decent results as point clouds or orthophotos. The latters are more and more requested for wall paintings studies because it enables metrical quantification of conservators/restorators observations on a user-friendly 2D supports. However, those CH objects also have been broadly documented for decades with TP imaging, formerly as a film-based technique which turned into digital-based thanks * Corresponding author : anthony.pamart@map.cnrs.fr to the development of digital still cameras (DSCs). It consists of capturing modified spectral pictures among different part of the Electromagnetic Spectrum (EM). Indeed even common DSCs offer possibilities to take pictures under different radiations in the range of the full spectrum sensor's sensitivity (i.e. in a range of 360-1000 nm). Each techniques of the TP imaging collection enables to reveal specific and precious information to understand or retrieve the painting layouts and history : • Visible (VIS); is the common image in visible spectrum and enable to restitute fine details thanks to the high resolution of digital sensors.
• Raking-light (RAK); is obtained by a light source obliquely set (almost parallel) to highlight surface features like retouches paint losses and brushed strokes providing layering or painting techniques information and their executions.
• Ultraviolet induced fluorescence (UVf); is captured by emitting UV radiation on the surface and using a filter to restraint some (UV) wavelength to detect chemical material with a UV fluorescence response in the visible range (e.g. inpaints, varnishes...).
• Reflected ultraviolet (UVr); reproduces the UVf set-up with different filter and require a full spectrum modified sensor to capture non-visible UV reflectance on top of painting surfaces or to detect specific pigments.
• Infrared (IR); is captured by emitting VIS and IR radiation passing through some surface layers and using a filter to block visible wavelength to reveal under-drawings. It requires a modified sensor to acquire some information in the Near-Infrared (NIR).
• Infrared fluorescence (IRf); is captured by emitting VIS light on the surface and using an IR filter to restraint some wave-length to detect chemical material with a IR fluorescence response.
• False color (IRfc/UVfc...); are composite images digitally computed by swapping RGB layers in between IR/UV and VIS to link non-visible areas of interest with a visible falsecolor relevancy but could be also used for pigment mapping. An overview of the literature in this field permits to find all required technical set-up specifications for IR (Bridgman and Lou-Gibson, 1963, Verhoeven, 2008, Cosentino, 2016, UV (Cosentino, 2015) and false-color (Aldrovandi et al., 1993, Aldrovandi et al., 2005 practices. Commonly, CH examination surveys are often done by simultaneous applications of TP techniques and at different time range (see Fig.1) according to object's specificity and context, sometimes including experimental or hybrid techniques (e.g RAK/IR (Cosentino et al., 2014). TP techniques have to be seen as complementary tools enabling by comparison to reveal similarities and differences in materials or pigments. Most of the time their application is required on different areas of a painted surface necessitating multiples handled acquisition which are hardly reproducible. However a registration issue remains concerning the combined multispectral aspect of TP documentation due to incoherent spatial overlapping (Cosentino, 2016) also known as short focusing and technically explained by longitudinal chromatic aberration phenomenon (Verhoeven, 2008, Hackforth, 1960. Moreover this registration issue could be extended to most of 2D/3D multimodal approaches dealing with cumulated multi-sensors, multi-dimensions, multi-spectral and multi-temporal studies, currently explored by several CH oriented research projects; COSCH (Boochs et al., 2014), Scan4Reco (Dimitriou et al., 2016), FIATLUX (Pamart et al., 2016). Hence other interesting works transposed the state of the art in image fusion from other domains like medical imaging and remote-sensing , Lanaras et al., 2014. However it remains an important gap in high-resolution/accuracy registration of 2D/3D data in a multispectral context where some recent works achieved promising results but still using constraining hardware/software requirement (Simon et al., 2010, Chane et al., 2013. Regarding the registration of multimodal images within a photogrammetric process some related works explore possible combination of SIFT features and Mutual Information (MI) also popularly used in medical imaging fusion (Corsini et al., 2013, Shen et al., 2014. Finally, most of multispectral photogrammetric approaches are re-lated to aerial or airborne surveys rather than close-range ones (Gehrke and Greiwe, 2013). Nevertheless some recent works explored 2D/3D photogrammetric registration for CH objects on Reflected Infrared (Webb, 2015) or Transmitted Infrared (Bennett, 2015). Nevertheless while those hybrid photogrammetric approaches focus on a single technique, our current methodology enable to merge several of them.
As far as common photographic equipment is actually used for the data acquisition our ongoing experimentation presented in this paper relies on the assumption that those techniques can be merged through a common pipeline to share their own benefits and create a consolidated documentation. This hypothesis is that a direct multiband photogrammetric pipeline could be reached by combining both methodologies and requirements of TP and CRP techniques, see Fig.2. Therefore, the idea presented in this paper is to combine them through a multiband data acquisition protocol followed by an optimized data processing to obtain multiband photogrammetric results benefiting of a certain degree of automation, versatility and reproducibility. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W3, 2017 3D Virtual Reconstruction and Visualization of Complex Architectures, 1-3 March 2017, Nafplio, Greece

Mechanical structure design
After some trials with existing data sets, a solution emerge to improve the promising results by creating an ad-hoc data acquisition methodology and facilitate the multimodal data registration, as shown in Fig.2. The necessity of such a system was guided by previous experiments (see Fig.3) showing the relevancy of dedicated hardware development to ensure best accuracy possible for the specific purposes of data acquisition campaign . Wall paintings can cover extended surfaces or encompass a whole inner space thus TP acquisition are done on representative but restricted samples pre-selected for the purposes of study while interesting information could potentially be revealed everywhere else. Indeed TP documentation need a high reproducibility and a rigorous acquisition process thus an handled hybrid CRP/TP survey would have been too constraining and time consuming. To fit this goal, a versatile mechanical structure has been conceived and realized. This also helped to fasten the acquisition stage and prepare the data for a multisensor automatic registration. This system is similar to an XY table adapted for vertical surface and tiltable along the X axis: a plate can support two cameras (or more) and freely move vertically from 50 centimeters above the ground up to 2.75 meters. The main structure is mounted on 5 meters horizontal traveling rail (divided in 5 elements of 1 meter each and possibly extendable) parallely set to the surface to acquire. Micrometric plates allows to set the baseline in between cameras for a precise overlapping while operating a translation along the X axis. The same system has been added on Z axis to correct the position of each sensor to facilitate the short-focusing effect (enounced in 1.1). Finally the structure can be tilted up to fifteen degrees past upright to align the focal plane with the slope of the painting surfaces and ensure a correct depth of field setting on the whole surface, even with long focal length or macro photographic equipment. Therefore, the structure can cover a motion area of 11,25m 2 in one set-up in all overlapping and GSD prerequisites. It has been designed to be folded flat so as to fit in car trunk. In a near future it will be motorized and robotized for a fully automatic data acquisition. Equipped with two similar cameras and lenses remotely controlled, one for visible spectrum and one modified on for IR photography, this system notably allows simultaneous multiband photogrammetric survey in a very short time. This multiband photogrammetric data acquisition RIG ensures the automatic registration process of different sensors thanks to a reproductible protocol. It terms of processing, multispectral acquisition are merged using an optimized script for each spectral domains so as to obtain multiband 2D/3D data continuum.

Data acquisition
It's still necessary to underline that good photogrammetry starts with good photography. Besides TP surveys also require a rigorous and controlled camera settings and data acquisition protocol, though special care is given to the preparation of this primordial stage regarding the equipment, setup prerequisites and shooting environment. The structure have been tested on different sites and acquisition contexts, using different techniques and had so far answered our expectations in term of versatility.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W3, 2017 3D Virtual Reconstruction and Visualization of Complex Architectures, 1-3 March 2017, Nafplio, Greece On one hand, best camera settings are chosen and fixed for each spectral acquisition sometimes compromising with both TP and CRP requirements according to the context of the survey (i.e. object specificity, lighting conditions, environment...). All technical specifications of the equipment (camera and lens) used need to be taken in consideration to guaranty the optimal image quality in terms of signal to noise ratio, aperture/shutter speed, focus, white balance in order to obtain sharp and correctly exposed pictures minimizing post-processing steps (Verhoeven, 2016). Obviously data acquisition are performed in RAW file format which keeps color information for later processing to a specific color space. The images will be post-processed to correct colors and tonal responses for archiving and afterward enabling comparisons within the CICRP documentation database. Furthermore, in a RAW file, the color space is just a meta data from the shooting, and does not limit its potential in any way. However, archiving RAW must be thought with a concern of recording the process and the tools used for RAW processing as applied algorithms change during time and software, the comparison with older acquisition campaigns may be difficult (Verhoeven, 2005). Furthermore, archiving RAW files open up to future processing possibilities with new and more efficient tools for better photogrammetric results. On the other hand, every CRP data acquisition needs to answer to a targeted spatial resolution and accuracy (i.e point cloud density, orthophoto definition). Similarly to UAV flight planning, the Ground Sample Distance (GSD) computation is used to prepare and define off-site the optimum data acquisition protocol and photographic equipment mobilized. It helps to foresee and check the distance to the object, focal length, overlap, XY baselines, numbers of pictures that could be transposed in-situ to the structure set-up to obtain the requested or expected results in term spatial resolution and accuracy.

Setup :
The data acquisition is operated as follows, the structure is set according the GSD planning using laser meter for the parallaxes of the traveling rail and digital level for the coplanarity of the structure to the painted surface. For VIS/RAK and IR acquisition, a lighting system composed of studio flash light and umbrellas (ELINCHROM kit Ranger RX Speed AS) is disposed to obtain a diffuse and homogeneous render on the area to acquire. Flashes are radio controlled and their power is set to discard any effect of the ambient light. The system is powered by batteries to be fully autonomous according electric supply conditions of data acquisition sites. Colorimetric calibration is done using a color checker in a preliminary picture for each spectral acquisition. Contrarily to VIS and IR there is not official standardized reference exist yet for UV calibration even if an aid for color balancing can be done using the AIC PhD target completed with known samples of emitters activated under UV radiation (Cosentino, 2015). Even tough it remains an open-issue concerning the time life of those UV charts, a short-term reliable solution could be the use of Target-UV TM and UV-Gray TM provided by UV Innovation TM despite of its cost. Once the cameras are properly installed their settings are manually fixed and will be re-used to acquire all necessary data set for geometric camera calibration. A wireless remote control system (CamRanger) is used to control, preview and shoot all along the acquisition process.

Shooting :
Multi-spectral acquisitions could be done simultaneously (VIS/IR) or sequentially (VIS,RAK,UVf) following the protocol fastly and straightforwardly displacing the structure along the millimetered traveling rail defining the X axis, then rising the cameras plate along the Y axis. This acquisition system allows to reproduce quite precisely, with a different technique, the matrix protocol extracted from the GSD calculation according to a predefined photogrammetric overlap and baseline. It also has the advantage that a protocol could be repeated at a different time range to create multi-temporal data set for comparison and assessment of degradation or restoration process (Peteler et al., 2015).
The visible light (VIS) from 380nm to 780nm is acquired with a non-modified camera with the same lighting equipment used for RAK and IR. Near-infrared (NIR) acquisitions are ranged from 780nm to 1000nm, the limits depending of the transmission of the filter used (see Fig.7) and the characteristics of the camera. The camera needs to be modified in order to be sensitive to wavelengths beyond the visible. The hot filter on top of the sensor is removed and may be replaced with a clear glass preventing dust. NIR creates a response in the red channel only, therefore pictures will be desaturated in post-production. Afterwards, UV lamps are installed after the interior space has been occluded from sunlight. UVf acquisitions are done in the visible spectrum under ultraviolet sources (16 tubes type Philips TL-D 36W BLB, distributed in 2 ramps of 8 tubes). The wavelengths emitted by the surface under this light are photographed and the colors of the different fluorescence can lead to assumptions about the materials used and the history of modifications. It is important, as under visible light, to keep the post-production of the pictures as identical as possible between waves of acquisition in order to compare them properly. This technique require long time exposure (around 30s) with a classic DSLR but high sensitivity sensor such as Fuji X-Trans will be used soon to slightly decrease the time required for the acquisition and gain in image quality but also because UV exposure could damage the surface which are not protected by varnish unlike other painting supports (e.g. canvas, wood...).

Data Processing
Multispectral photography implies getting information in different part of the EM thus some pre-processing stages are necessary but not detailed below. Some of the TP acquisitions have been implemented so far on different case-studies: VIS, RAK, IR, UVf. Other like IRf and UVr will be added further to complete this multiband photogrammetric documentation using the work flow described in Fig.8 below. From a technical point of view, the whole pipeline is operated thanks to the flexible and open-source photogrammetric tool chain Pastis-Apero-MicMac (Deseilligny and Clery, 2011). This software solution appears to be the most relevant choice to fit and optimize data processing with the peculiar hardware used for data acquisition. Moreover it is giving enough degrees of freedom and advanced parametrization to make possible this multispectral CRP-based data registration in an automated way. Data are semi-automatically renamed (Fig.8[A]) in-situ by modifying naming and numbering setting directly through the camera or off-site by a metadata-aware scripting method, to embed the technique used (i.e. VIS, IR, UV added as a prefix). This is helpful for file management all along the pipeline but also compulsory to be injected into an automated data processing. Additionally, images could be renamed according to the data acquisition protocol to embed their respective position in gridded system where rows are alphabetical and columns numerical (e.g. VIS-A-06.tif) in order to optimize the processing step such as 2D image matching.
A special care is given to color calibration (Fig.8[B1]) to ensure colorimetric accuracy along different sensors, lenses, lightning conditions and spectral sensitivities. Raw data are batched to fit precisely color space, white balance and exposure with minimal modifications and exported in Tiff file format without compression for an optimum image quality. As data have been acquired using a strict parallel axis acquisition protocol (i.e. without oblique images) self-calibration using those data should be avoided. The geometrical calibration of cameras is done using an initial calibration data set taken in-situ for each sensor and fixed settings (aperture, focus, distance). The demanding Fraser photogrammetric camera calibration model is used to ensure the lowest error of reprojection (i.e. below the 1 pixel or less) (Fig.8[B2-B3-B4]).

Image matching :
At this step data has been properly pre-processed to be integrated into an incremental photogrammetric process. At the current stage of the development, tie-points are derivated from the classical SIFT features, but other options (Local Contrast Enhancement, Wallis filter) or descriptors could be implemented to gain in velocity (ASIFT, SURF) or robustness (Maximal Self-Dissimilarities).
Thanks to the controlled data acquisition image matching is straightforward using an incremental approach ; sequentially in between a spectral acquisition (Fig.8[C1-C2-C4]) and then across several of them (Fig.8[C3-C5]). While all image blocks have been matched, a tie-points filtering step is added to the pipeline (Fig.8[C6]) to discard inter-spectral false matches and to decrease uncertainty in calibration and alignment.

Cameras orientation :
Images are spatialized using a two-steps approach (Fig.8[D1]). Firstly, data are roughly oriented using an SfM method based on triplets and tri-focal constraint to provide orientation files for each pictures. Secondly, the global orientation is refined using the classical bundle-adjustment method and giving as input both initial camera calibration and orientation calculated previously. A sparse point cloud is generally computed to verify and validate camera positions (see Fig.2(a)). Moreover optional and manual steps could be integrated to create a 3D mask from the sparse point cloud or to define scaling and coordinates system ( Fig.8[D2]).

Geometric registration :
An automatic dense matching algorithm is used on all pictures to obtain a first raw and temporary point cloud to a given density (e.g. 1 point for 4 pixels) and according to the GSD requirement ( Fig.8[E1]). This temporary result is used for a perpicture orthorectification intermediary process, preparing data for orthophoto production (Fig.8[F1]). Once every pictures have been orthorectified according to a reference plane, automatically defined by RANSAC plane detection for clean and planar surface or user-defined for more complex geometry (thus [D2] manual input becomes mandatory), they are implicitly stored all together in the same space coordinates reference system.

Multiband 2D/3D output generation :
An orthophotomosaic is then computed for every spectral acquisition to have a perfect multispectral per-pixel matching (Fig.8[F2 -F3-F4]). A final point cloud is extracted from the orthophoto to create a continuum in between 2D pixel coordinates and 3D points (Fig.8[E2]). This point cloud is colored by each spectral orthophomosaics (as is Fig.2(b)) to obtain a multiband point cloud (Fig.8[E3 -E4-E5]).
This pipeline creates a multispectral continuum of information in between 2D and 3D. It has been implemented on several case studies whereas each point of the studied wall painting can be described with both colorimetric and geometric information: in xy pixels locations (locally in every pictures or globally in the resulted orthophoto); in XYZ points coordinates (in the pointcloud); in NxNyNy (for normal point); and colored by RGB value for each spectral domain. Our goal is to compile all those information and possibly completed them with other useful features calculated from different descriptors (e.g. ambient occlusion, roughness, curvatures, shaded maps, hypsometric values...), as shown in Fig.11 but also with other attributes coming from automatic, semi-automatic or manual segmentation and their intercorrelation with all relevant layers, as detailed in Fig.12. Figure 9: Example of multi-spectral orthophoto (VIS/IR/UVF/Composite), Anamorphosis case-study 3. CONCLUSIONS

Results and future works
At the current stage of development and regarding to the first results obtained, we proved the benefits of a relevant methodology from a dedicated data acquisition system to an optimized data processing providing innovative and efficient solution for the documentation, studies and analysis of wall-paintings. Nevertheless, this pipeline will continue to be improved toward an expected dissemination into conservation/restoration routines.
The use of orthophotographies in multispectral analysis helps to superimpose precisely multispectral information for easy reading and comparison asked by conservators and restorators. It also makes further correction easier on a whole assembled image instead of a collection of unstitched images. If CRP processing is done on RAW files instead of Tiff it opens possibilities to modify or update outputs without reprocessing. For example, applying a global contrast curve can be used as a tool for easier reading of underlying elements. An other interesting application is the possibility to compute straightforwardly false-color IR and UV from the orthophoto, instead of the previous manual technique which include inaccurate pixel shifting in between RGB layers (Aldrovandi et al., 2005, Fent andMeldrum, 2016). From the assumption that only the red channel is used in NIR. The visible fluorescence under ultraviolet light will generally be captured under the green or blue channel, we imagined a synthetic multispectral false-color image with the red channel of the NIR image, green channel of the UV image, and a third channel from the VIS image or inversely (Fig.10). Infrared false-color images are made assembling blue and green channel from the visible and recombining them with the red channel of the NIR image while swapping channel order.
Therefore, this multi-layered orthophotomosaic offers wide possibilities in term of false-color image generation while 18 RGB It could also open new lead to Computational Photography techniques and more precisely in the field of Image Processing by explorating multiband mixing and correlations for features extractions similarly as it could be made on hyperspectral pixels (Doneus et al., 2014). As a basic example, composite images have been processed using Mean or Median (possibly ponderated by a polynomial approach) among each pixel of different RGB layers and orthoimages assuming that those images as no scientific values it could nevertheless enhance or highlight some specific features to help restorators (Fig.9, 10 & 13).
Besides all of the technical considerations in term of computational exploration cited above, one final step will be to discuss possibilities for an operative and user-friendly visualisation system so as to enable intercorrelation and transfer of features among this complex multi-layered data set. The goal of this future work could be reachable by creating a multilayered data visualisation and annotation system or using existing solutions such as; Spatial Image analysis and Viewing Tool aka SIVT developed by i3Mainz (Wefers et al., 2015), Toolbox for Archaeological Image Fusion aka TAIFU developed by Ludwig Boltzmann Institute (Verhoeven, 2015) or the Image-based 3D annotation cloud service aka AIOLI in development by MAP-CNRS (Manuel et al., 2016). Indeed as bottom-up intercorelation approaches appears to be exponential our idea is to involve CH experts to experiment and reveal by themselves what could be the most relevant combinations for their studies purposes. Hence the idea to offer an interactive tool for analysis guided by cumulated expert observations (even though it implies a trial and error process) rather than a blind and unsupervised learning only computer-driven approach. While the semantic enrichment of CH 2D/3D documentation remains a challenge in the near future, we rely on the idea that integrating expert's knowledge could be the most relevant way to explore all 2D/2,5D image processing possibilities (Verhoeven et al., 2016) or 3D multispectral feature extractions (Del Pozo et al., 2016, Lerma, 2005, Bianco et al., 2011.

Perspectives
We believe that the continuum of information created by this multiband photogrammetric data acquisition and processing could be a new step in the field of investigation of multimodal data correlation (Fig.12) for CH objects studies. Even tough our methodology is limited to geometrically simple morphologies (i.e. wallpaintings, frescoes...), it succeed to provide so far a complete, traversal and interoperable data set for conservation/restoration works. Finally one could foresee some possible and interesting applications in term of automatic semantic annotations propagation, to link in-situ or in-lab observations and action planning supported by mobile devices development for real-time sharing of qualitative and quantitative aspects useful for the follow-up of transformation dynamics of our heritage.