REPRESENTING WITH LIGHT. VIDEO PROJECTION MAPPING FOR CULTURAL HERITAGE

: In this paper, we describe a cross-disciplinary process that uses photogrammetric surveys as a precise basis for video projection mapping techniques. Beginning with a solid basis that uses geoinformatics technologies, such as laser scanning and photogrammetric survey, the method sets, as a first step, the physical and geometrical acquisition of the object. Precision and accuracy are the basics that allow the analysis of the artwork, both at a small or large scale, to evaluate details and correspondences. Testing contents at different scales of the object, using 3D printed replicas or real architectures is the second step of the investigation. The core of the process is the use of equations of collinearity into an interactive system such as Max 7, a visual programming language for music and multimedia, in order to facilitate operators to have a fast image correction, directly inside the interactive software. Interactivity gives also the opportunity to easily configure a set of actions to let the spectators to directly change and control the animation content. The paper goes through the different phases of the research, analysing the results and the progress through a series of events on real architecture and experiments on 3d printed models to test the level of involvement of the audience and the flexibility of the system in terms of content. The idea of using the collinearity equation inside da software Max 7 was developed for the M.Arch final Thesis by Massimo Visonà and Tommaso Pasini of the University of Venice (IUAV) in collaboration with the Digital Exhibit Postgraduate Master Course (MDE Iuav).


INTRODUCTION
Video projection mapping techniques are usually intended for ludic purposes and the relation between the object and the animation projected on it, is functional to entertain the audience with creative and artistic contents.Once the purpose of video mapping changes towards a communication that is intended to be more specific or even used as an educational tool, the system needs to be calibrated with a different approach.The investigation reported is part of a wider research about possible uses of video projection mapping techniques for Cultural Heritage, that involve the Photogrammetric Lab of Iuav University in Venice.The collaboration permitted to use the survey of many facades through laser scanning and UAV photogrammetry.Many of them have been already subject of research and were used to generate replicas with various 3D printing methods, as additive 3d prints, SLS technology or SLA technology.At first, the existing models were used to experiment different techniques of video mapping, using various software like Madmapper, VPT Video Projection Tool and Resolume.The point cloud of the Redentore Church Façade in Venice was selected as a case study for prototypation and 3D printing in order to have available the architectural object in real scale and in its 3D printed replica.The second step of the method consists of merging historical and technical data, based on a critical selection, to build a narrative visual language for a collaborative learning through an interactive storytelling.History of art and architecture, archaeology, restoration, diagnostic or philology gives resources and materials to create potentially infinite forms of communication.All these resources need to be processed in collaboration with experts and based on a teamwork that very often brings the research to discover details that is possible to understand only thanks to the use of digital instruments.The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2, 2018 ISPRS TC II Mid-term Symposium "Towards Photogrammetry 2020", 4-7 June 2018, Riva del Garda, Italy The last step of the process is to add interactivity, that together with video projection mapping gives the opportunity to the spectator to be active and involved in the process of acquiring knowledge about the artefact.Video projection mapping for CU represents a new field of exploration in order of making museums more attractive to audiences of all ages and level of expertise, preserving the science-based and the emotional value of information.Also, public institutions could create new informal learning scenes, such as the urban context where the acquisition of knowledge becomes a form of experience that combines learning with entertainment.

PROCEDURE
One of the main factors to keep under control in the video mapping is linked to the close relationship that must be present between the digital environment and the object on which to project: the adaptation between these two systems is often accomplished through the warping method, ie the deformation of the projected image, in such a way as to perfectly match the digital 3D model to be projected with the real object on which it is projected.This is why all the video mapping software has a module for image distortion.Usually this operation takes place manually: moving 4 control points (cornerpin) located at the corners of the image tries to position the projection coherently with the physical object; often, however, this operation is tedious and not very precise.So we tried to create a method to facilitate the warping by identifying in digital image and on the real object 4 homologous points: using a homographic transformation between object coordinates and image coordinates it is possible to automatically calculate the exact coordinates of the 4 control corners of warping.
The equations able to describe the relationship between the real object and the image to be projected, then between object points and image points, are the following: where X and Y are the coordinates of the points on the real object, x and y are the coordinates on the projected image and the remaining 8 parameters (a, b, c, a ', b', c ', p, q) describe the orientation in the projector space and its internal orientation parameters.We chose to write the code using the software Max7, a programming environment oriented to visual and multimedia arts, that is often used in the field of video mapping.
The procedure defined by the software requires the user to select with the mouse 4 homologous points on the projected image (with the movement of the cursor in a digital environment) and on the real object (with the projected image): the calculator records the coordinates of these 8 points and thus constructs a system with 8 equations (2 for each homologous point) and 8 unknowns (the parameters a, b, c, a ', b', c ', p, q previously described).The writing of this software exploits the laws of photogrammetry and adapts it to the environment of the mapped video projection.Once you have obtained these parameters, you can calculate the coordinates of any other object point from any image point.Analysing the object coordinates of the cornerpin, the image will automatically position itself on the real object to perfectly match the 4 homologous points between the real object and the digital image.It will then be the responsibility of the user to manually refine the coordinates of the cornerpin, just as you would do with a traditional warping.To increase the accuracy of this method, the number of homologous points to be identified and the internal and external orientation parameters could be increased with the Ordinary Least Squares.However, it was considered in the programming phase more than enough the use of only 4 homologous points, both to not complicate the writing of the software and for a faster use of the system during the installation of video mapping, considering that the final warping however, it must in any case go through a manual intervention.

PROGRAMMING AND INTERACTIVITY
The software Max7 has also been used to program the part of interactivity between video mapping and the user/observer.On the occasion of some events realised using the University's buildings owned by the University, some forms of interactivity were experimented using different types of sensors such as the Kinect® infrared system, Arduino® with motion sensors or the Leap motion sensor to detect the movement of the hands in the case of use of interactive video mapping on scale models such as that of the Redentore Church.Surely the most effective was the Microsoft Kinect®, which was used to track the position of the skeleton of the user and determine over time the X, Y, Z coordinates of the position of the hands.With this information, it was possible to animate circular areas on the façade, which, following in real time the movement of the user's hands, gave the effect of "scraping" the architectural façade.The user has the impression of removing the plaster to discover the underneath masonry wall or to reveal some fantasy wall coverings or simulating new architecture facades.
Figure 5. Interactive event in the University cloister, using the infrared Kinect® sensor.
In the second phase of the research the interactive system was applied on the 3D printed model of the Redentore Church with the purpose of building a narrative storytelling of a dynamic range of information.Thanks to the interaction the spectator is able to control the fruition of the information.All the information used for the animations were composed under the guidance of architectural historians and restorers.The relation between the stone cladding and the structure, between the inside structure and the façade decorations, the possibility to zoom in details and observe small elements, otherwise invisible, thanks to the digital precise acquisition of the model projected on the physical replica, are just some examples of the infinite possibilities of contents to comunicate.The interactive video projection mapping designed is also scalable to the real building as the model comes directly from the scanner survey.The distance of the point of view of the observer in that case cover the need of more detailed images.
Video projection mapping has various fields of application: in architecture, it allows you to use alternative educational methods and solutions through the direct projection on the facade or architectural surface or its scale reproduction.Some of the possible applications can easily highlight a particular architectural element, proving the relationship with others or illustrating the implementation process and demonstrating the relationship, or lack thereof, between the external and internal spaces.In museums video mapping allows to approach the observer to the object exposed, enhancing knowledge, favouring a more simple and immediate approach to the acquisition and the transmission of information.Both in museums and in public events, it is possible to divulgate sophisticated details and technical information otherwise destined only to experts, if transmitted through a written description.Interactivity gives a direct relationship between the observer and the object.

CONCLUSIONS
The main purpose of this paper is to define a field of investigation that can develop new lines of research starting from the principle that the intersections between disciplines and the transfer of methods and practices from one disciplinary area to another can widen and enrich the results of the itself.
Methods and tools such as video projection mapping, which are part of the entertainment world, can be used with a different approach when they intercept the practices related to the precision instrumental survey and give rise to a type of products suitable for scientific communication, as well as the creation of diversified contents based on the type of user.

Figure 1 .
Figure 1.3D printed model of Redentore Church in Venice realised with Selective Laser Sintering

Figure 2 .
Figure 2.For each pair of homologous points it is possible to define 2 equations

Figure 6 .
Figure 6.Screenshot of the simultaneous sections controlled by Leap Motion sensor.The red line on the left shows the cutting position.The image on the right is showing the changing section.

Figure 7 .
Figure 7. Screenshot of the superimposition of the inner space on the facade.

Figure 8 .
Figure 8. Hand control of the spot