DOCUMENTING PAINTINGS USING GIGAPIXEL SfM PHOTOGRAMMETRY

Capturing paintings with gigapixel resolution (resolution around 1000 megapixels or greater) is an innovative technique that is starting to be used by some important international museums for documenting, analysing, and disseminating their masterpieces. This line of research is extremely interesting, not only for art curators and scholars, but also for the general public. The results can be disseminated through online virtual tours, offering a detailed interactive visualization. These virtual tours allow the viewer to delve into the artwork, in such a way, that it is possible to zoom in and observe those details, which would be negligible to the naked eye in a real visit. Therefore, this kind of virtual visualization using gigapixel images becomes an essential tool to enhance this cultural heritage and to make it accessible to everyone. This article will describe an affordable methodology, based on SfM photogrammetry techniques, with which it will be possible to achieve a very high level of detail and chromatic fidelity, when documenting and disseminating pictorial artworks. As a practical example, there will be shown a case study of the altarpiece, from the Museo de Bellas Artes de Valencia (Spain), entitled Virgen de las fiebres, painted around 1500 by Bernardino di Benedetto di Biagio, nicknamed ‘Il Pinturicchio' (Perugia, ca. 1454 Siena, 1513)


INTRODUCTION
Digital photographic capture with gigapixel resolution is not an easy task and serious difficulties may arise due to physical problems as the diffraction of light, which represents a barrier that limits the sharpness that can be obtained with an optical device and a digital sensor. In fact, there have been few attempts to develop prototype cameras providing gigapixel images on the fly, see (Flint, 2005) and (Brady et al., 2012), and those special prototypes are far from becoming an accessible option due to their high complexity and cost.
Since today's professional digital cameras provide images of around 40 megapixels, obtaining gigapixel images requires some special capture and editing techniques. Many important international museums are using innovative technologies in collaboration with the multinational Google, which has developed its own high-resolution digital capture system for paintings. These artworks are exhibited on the Arts & Culture Project website, which contains many examples that can be displayed with a high level of detail (Proctor, 2011).
Apart from Google, there are very few companies specialized in capturing gigapixel images of artworks, due to the technical complexity and the specialized equipment. There are some examples such as the French state organization Centre de Recherche et de Restauration des Musées de France CR2MF, the Italian company Haltadefinizione or the Spanish Madpixel. All these companies use a gigapixel capture methodology that is based on panoramic stitching techniques, which requires a set of pictures, taken from a single viewpoint with the help of a panoramic head and a telephoto lens (Kopf, 2007). This methodology can be great for documenting moderate size paintings, but there are some drawbacks that can reduce the image quality. The narrow depth of field provided by long focal lenses may produce blurred zones in the off-centred shots. This fact is due to the oblique angle between the canvas and the camera sensor, which increases as the camara is rotated towards the boundaries of the painting. This problem would be overcome when using a multi-viewpoint capture technique, as it is usual in SfM (Structure from Motion) photogrammetry, so the camera sensor can be parallel to the canvas when capturing the set of pictures. This method has an additional advantage, which consist in obtaining a total 3D restitution of the painting, while the usual stitching method provides only a rectified image of the canvas. This advantage is especially interesting when capturing threedimensional works such as altarpieces in which the carved frame is as valuable as the painting itself, so it also would be documented properly.

CASE STUDY
This study is part of the research project entitled Gigapixel photographic capture for documenting and disseminating pictorial heritage, whose main objective is the documentation and the virtual musealization of some of the main masterpieces of the Museo de Bellas Artes of Valencia, one of the main art museums in Spain, with ultra-high resolution also known as gigapixel resolution.
Firstly, a selection of the artworks to be digitized was carried out. This selection was focused on highlighting the main pieces of each historical period in the collection. This list of art pieces provided an interesting formal diversity that put the capture and reconstruction methodologies to the test, so there are largeformat paintings, small ones, some of them are protected with a glass and there are altarpieces in which the carved frame is part of the masterpiece. This last case may cause important issues when obtaining the gigapixel orthophoto image by means of stitching methods, due to its three-dimensionality, since the resulting image would not be a real orthophoto, but only a rectified image. For that reason, SfM photogrammetry techniques were chosen to document the altarpiece entitled Virgen de las fiebres, painted around 1500 by Bernardino di Benedetto di Biagio, nicknamed 'Il Pinturicchio' (Perugia, ca. 1454-Siena, 1513. This artwork is exposed at the Museo de Bellas Artes de Valencia (Spain). The painting shows the scene in which the Virgin, represented as Mater Sapientae, teaches the Christ Child to read, while the bishop (patron of the artist) stays in a praying attitude (Fig. 1). Museo de Bellas Artes de Valencia (Spain).

METHODOLOGY
There were used different techniques in the whole research project to obtain the final orthographic image. The most used were mainly based on 2D stitching techniques, which were applied to document flat canvases. However, this case study consists in a volumetric piece, so the SfM reconstruction technique was chosen for documenting this three-dimensional artwork.
The SfM photogrammetry provide very accurate results when following a proper methodology, and its high accuracy has been already demonstrated (Rodríguez-Navarro, 2012). Nevertheless, it is important to remark that image quality is crucial for SfM reconstruction (Stanco, Battiato, Gallo, 2017).
Despite the fact that SfM techniques were used in this case, it must be considered that the main aim was not obtaining a threedimensional model itself, but an orthoimage of the altarpiece, which is part of the rest of the gigapixel images that were created within this project.

Equipment
The photographic equipment consisted of a Canon EOS 90D digital camera, equipped with a Canon 17-40mm f4L lens, a Manfrotto 28B tripod and a Manfrotto 303 SPH panoramic head.
The Canon EOS 90D has a 32.5 Mp APS-C sensor (22.3x14.9mm), with a multiplication factor of 1.6, compared to a full frame sensor. This camera produces a quite low level of noise, especially at low ISO settings.
An X-Rite ColorChecker Colour Card was used as a reference target to adjust the proper white balance and to generate a specific colour profile for the lighting conditions. Creating this profile is essential to achieve an accurate colour reproduction of the paintings (Fig.2).
A specific aluminium lighting mount was designed and constructed to allow a uniform illumination of the painting. This special mount system was attached to the tripod to be moved along with the camera. This mount supported two 28-inch light box providing a 5500K colour light with a CRI (Colour Rendition Index) greater than 90 (Fig.3).

Figure 2.
ColorChecker card used as a reference blank for achieving an optimal colour accuracy. The light sources were strategically placed to avoid specular reflections and in a symmetrical position with respect to the camera to provide a uniform illumination of the picture. This fact was especially critical in this artwork since it was protected by a glass. To avoid reflections, the lights were placed outside the volume defined by the family of angles that would produce specular reflections ( fig. 4). A Topcon Image IS robotic total station was used for taking the measurement of the altarpiece accurately and without touching the artwork. This total station has a reflector-less measurement capability between 1.5 and 250 m, providing a short-range precision of ± 5 mm (MSE), and an angular precision of 0.3 mgon.

Shooting techniques
One of the most important decisions in this work was choosing the proper lens, so it had to provide a good optical definition and sharpness, while its focal length had to fulfil the requirements and limits suggested by most of the SfM software developers. It is advised to use lens around 24 up to 70mm. In this case, a 40 mm lens was chosen. Thus, applying the multiplication sensor factor (1,6), results a focal length of 64 mm for this camera.
The camera was attached to a panoramic head and a tripod, so shooting at low speed was not a problem. All the camera adjustments, such as focus, diaphragm aperture, speed, and ISO were remotely controlled via Wi-Fi by a laptop computer, using EOS Remote Utility software by Canon.
The exposure settings were locked during the capture process in order to maintain the same brightness levels. The ISO setting was set to the lowest value (ISO 100) to minimize the noise. The aperture was set to 8 to maximize the lens sharpness, thus avoiding the negative effects of the diffraction of light that arises when using too small diaphragm apertures.
Another important goal was determining the proper distance between the camera and the canvas in order to achieve a good final image resolution, while maintaining a relative low number of shots, so the time in the museum was limited. Ground Sampling Distance (GSD) is the factor used in photogrammetry to express the distance between two consecutive pixel centres measured on the ground (on the canvas in this case). The lower the value of the image GSD, the bigger the spatial resolution of the image and the more visible details.
A premise of this project was to obtain full-scale images of the canvases at a resolution greater than 300 ppi. Therefore, by doing the conversion of this lower resolution limit, there was stablished a maximum GSD of 0.083 mm/pixel. Moreover, in order to stay at a safe distance from the canvas, the photographic equipment was always placed at least 75 cm away from the artwork to prevent any accidental damage.
Taking into account all the previous considerations, the camera distance was calculated to achieve an optimal GSD of 0.07 mm/pixel, while maintaining a safe distance from the canvas.
GSD is defined in the equation below: Consequently, the optical centre of the camera was placed at 0.88m from the artwork. Then the camera was moved parallel to the canvas, horizontally and vertically, to cover the entire altarpiece (Fig. 5).
There was shot a mosaic of the canvas area containing 143 pictures (11 rows and 13 columns), which was completed with additional shots to cover the volumetric elements of the frame. A total of 170 pictures where shot.
In order to ensure a good SfM reconstruction, the adjacent pictures were overlapped about 60 and 70% (Fig.6). Although all the shots served for the orientation and generation of the 3D model, only a subset of them were used for generating the orthoimage as will be discussed later.  The pictures were saved using the raw format to maximize the image quality and the dynamic range. The X-Rite ColorChecker The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVI-M-1-2021 28th CIPA Symposium "Great Learning & Digital Emotion", 28 August-1 September 2021, Beijing, China target was used to evaluate the exposure and served as a reference blank during the raw development process, which was carried out using Adobe Camera Raw.
A specific colour profile was created by ColorChecker Camera Calibration software from a picture of the ColorChecker target, which was taken under the same scene lights. The specific colour profile was applied during the development process to ensure the maximum colour accuracy.
Moreover, the proper white balance adjustment was set with the help of the ColorChecker target, by using the medium grey colour patch as a reference to neutralize any colour dominance.
In order to maintain the highest colour accuracy during the whole process, the developed pictures were saved using a 16 bits per channel Tiff format and the ProPhoto colour profile.

SfM Workflow
Agisoft Metashape was used for the photogrammetric reconstruction of the model. The process was performed according to the following steps.

Checking the Image Quality:
Once the images were loaded, their quality could be checked by using the option 'estimate quality of the cameras'. In this process, Metashape assign a value between 0 and 1 for each image, based on its sharpness and all the images under 0.5 are discarded. In this case, all the images were over this limit, providing values between 0.53 and 0.99. All the pictures that were taken frontally to the canvas, achieved a high level of quality. Only those oblique to the altarpiece, shot to reinforce the 3D reconstruction of the frame, reached lower scores due to the effect of the deep of field.

Determining the Image Orientation:
In this process, the software computed the internal orientation of the pictures (Fig.7) and built a sparse 3D point cloud (Fig. 8a) by analysing the correspondences between images. Highest accuracy profile was selected at this moment, so it is decisive for the subsequent projection of the textures over the 3d model to obtain the final orthoimage. When using this setting, Metashape uses the full resolution of the image ('Medium' profile would use 50% of the resolution, while 'Low' profile, only the 25%). Then, the software calculates the orientation automatically by matching homologous points, which allows solving a self-calibrating algorithm and a beam adjustment to obtain the image positions, the point coordinates, and the camera parameters.

Building the Dense Point Cloud:
At this point, the software generated the dense point cloud. Medium quality profile was chosen since the main goal was the orthoimage and not the 3D model itself. Using a higher profile would have been much more time consuming and the improvement would have been negligible. The result was a dense point cloud that included the colour information and revealed the morphology of the object clearly (Fig. 8b).

Creating the Polygonal Mesh:
At this step, a mesh of triangles interpolated from the dense point cloud was generated to fit the surface of the model. The greater the number of polygons, the greater the similarity between the digital and the real model. The software allows defining the number of polygons and it is possible to obtain the densest possible mesh by setting the value to zero. Nevertheless, the average number of polygons proposed by the software (3,709,047 polygons) was selected in this case. This value is more than enough for such a model, which is composed mostly by flat surfaces (Fig. 8c).

Applying the Texture:
Although it is not a strictly necessary step for this particular case, it is advisable to compute the texture over the model for a better identification of the reference points for the scale, so this process was carried out using the default settings (Fig. 8d). It is important to highlight that this texture will not be used by the software to generate the orthomosaic. Therefore, the resolution of this texture is irrelevant, as long as the reference points are clearly visible in the model.

Scaling the model:
A topographic instrument was used to obtain the required measurements for scaling the model. The total station was placed in front of the painting, and, with the help of the telescopic sight, there were identified the corners of the canvas. The coordinates of these points were obtained using TopSurv, a software that comes with the total station. Then, these coordinates were assigned in Metashape to their corresponding points of the model in order to match the real scale.

Building the Orthomosaic:
Finally, the generation of the orthomosaic was the essential step of the whole process. This image was composed from the oriented images, which were taken with a significant overlap, greater than 60%. This fact facilitated the reconstruction of the model by SfM algorithms, but this overabundance allowed also being even more demanding at this stage. Hence, the quality value of the images was checked to use only the best ones when building the orthomosaic. The final orthomosaic was rendered with a pixel size of 0.07mm to match the GSD provided by the pictures. This resulted in an image with a total resolution of 18249x30181px, more than 550 Mp.
For further information about the SfM reconstruction process, see (Nyimbili, P. et al, 2016).

CONCLUSIONS AND RESULTS
The orthoimage fulfilled all the requirements; its dimensions, its resolution, and the accuracy of the composition met the initial objectives perfectly (Fig 9). The strengths and weaknesses of the use of SfM photogrammetry for this kind of gigapixel images can be analysed now. On the one hand, the shooting methodology can be similar to the one used when using stitching software. Nevertheless, the SfM algorithm requires taking the images from different viewpoints, and a greater overlap on adjacent shots. Consequently, this methodology requires taking more pictures and needs much more time-consuming processes. On the other hand, the SfM method is the one that can generate a real orthophoto of the whole altarpiece, since using stitching techniques only would provide rectified perspective projections and not orthographic ones. Stitching methods can be useful when the objective is taking only the canvas, since it is a plane surface, but not in this case, so the carved frame would not be represented in true size and shape but distorted by the effect of perspective.
Additionally, the SfM method can provide a greater image sharpness and quality, so the mosaic was shot moving the camera parallel to the canvas. When doing so, the camera sensor remains parallel to the canvas and, consequently, there is no loss of sharpness due to the deep of field, which is especially narrow when using long focal lenses and when shooting at short distances.
It should be mentioned that SfM technique requires a high-level computer, especially with a powerful graphics card and a lot of RAM memory, to resolve the whole project at once. Otherwise, the work must be divided into chunks that can be joined later.
The orthomosaic of each of these chunks could be composed with any image editing software such as Photoshop.
Fortunately, the dissemination of such as heavy gigapixel images through the Internet can be carried out by decomposing the image in a multiresolution mosaic, which can be also called pyramidal image. The process consists in applying an algorithm that generates different tiles from the gigapixel image with different sizes and levels of resolution. Then, these tiles can be loaded progressively in real time by using a specific viewer. In this way, when the user demands more definition by zooming up, the viewer loads only the specific tiles for the visualization area.
For visualizing the resulting gigapixel through the Internet, there is no need of using a powerful equipment, since even a conventional smartphone can manage this kind of pyramidal images. This technique is used in very well-known applications like Google Maps or Google Earth.
It is important to highlight that, due to the different kinds of displays that the spectators can use for the visualization of the images via Internet, it is advisable to generate the multiresolution mosaic using the standard colour profile sRGB, which is suitable for the majority of the devices. Nevertheless, the accuracy of the colour reproduction will depend on the gamut of the display and its proper calibration.
Zoomify is one of the software that can provide multiresolution mosaics and has its own viewers, including a free version. There are another powerful opensource viewers, such as OpenSeadragon, which is the one that has been used for this project.
Using pyramidal images or multiresolution mosaics allowed to develop a dedicated website www.gpix.upv.es for disseminating the results of this research project. The main aim of this site was to satisfy the needs not only of the general public but the specialized collectives, such as art curators and restorers.
The website offers a virtual musealization of the captured artworks, which are shown with gigapixel resolution. The virtual visit offers also some extra information about the art pieces. In addition, the gigapixel visualization allows using some advanced features, such as the ruler, which allows taking real measurements directly from the gigapixel images. This feature would be very useful for the experts to analyse the state of conservation of the artwork and to control some effects such as the craquelures in the painting produced by the thermal expansions of the support.
The gigapixel visualization of the altarpiece can be carried out by following this link: https://gpix.webs.upv.es/gpix/273.html or by using the QR code below (Fig. 10). Figure 10. QR code to get access to the virtual visualization of the gigapixel image using any mobile device.
The virtual gigapixel visualization allows the user to dive into the artwork progressively, from the general views to the deepest level of zooming. As a result, it is possible to observe those details, which would be negligible to the naked eye in a real visit.
There can be seen the general scenes in which the Virgin, represented as Mater Sapientae, teaches the Christ Child to read (Figs. 11 & 12). This altarpiece was commissioned by the Spanish Cardinal Francisco de Borja, relative of the Pope Alexander VI, who is represented in a praying attitude (Fig. 13).  In a more detailed view, there can be seen some elements. Such as the Borja family coat of arms, in which an ox is represented (Fig.14). It is remarkable the meticulous treatment of the mitre, which is not a simple one, but a sacred garment loaded with ornaments and precious gemstones (Fig, 15). The artwork has a wide chromatic spectrum, where the red tones, the carmine of the clothes and the green of the vegetation stand out. The botanical elements are also carried out with a great rigorousness (Fig 16). It is noticeable the coexistence of traditional elements, such as the gold leaf work (Fig. 17), with the Flemish influence, which is especially evidenced by the meticulous and detailed technical treatment.  To conclude, this kind of virtual visualization using gigapixel images become an essential tool to enhance pictorial heritage and to make it accessible to everyone by means online virtual musealizations. The SfM methodology can be used successfully to generate these gigapixel orthoimages, especially in the case of volumetric artworks. The results can acquire a double functionality; the dissemination task, which is oriented to the general public, and the documentary one, which is dedicated to experts, such as art curators and restorers, who would appreciate the accurate results.