THE EUROPEAN RESEARCH NIGHT : NEW WAYS FOR COMMUNICATING SCIENCE WITH ICT AND VIDEOMAPPING

Thrilling the society toward science and research is not trivial. Albeit the “academic industry” lavishes many efforts to spread its results not only among the insiders but to the whole mankind, the importance of sharing the knowledge of research is seldom a priority. The European Researcher Night is probably the most important EU action trying to overcome this limitation, putting altogether researcher from different disciplines to show their findings through stands, short communications and events. Within this framework, the event able to attracts citizens is the video mapping projection. In this article is described a multi-disciplinary process that makes use of a photogrammetric survey as an accurate source for video projection mapping. While well-established geomatics technologies (e.g. laser scanning and photogrammetry) paves the way for the virtual reconstruction of the architecture, they are even essential to perform analysis and studies which enables visual artist or art historians to tell the story of a building in a new and fascinating way. Besides the realization of the visual mapping and a critical discussion over the procedure that has been used to translate a 3D model in a visual storytelling of the building, the article also describes an innovative way that has been set up for the management of the whole SHARPER event. The system is app-based and was designed to allow the visitors to interact with the event directly from their smartphones; several active sensors have been displaced among the city, asking the user to search for virtual owls and to catch them by answering some questions, engaging the people by exploiting the gamification paradigm. This latter has been stressed further, since the video projection was conceived as a competition between the students of the Master degree course in Engineering-Architecture. Through the application, the attendant to the visual mapping where thus enabled to vote his/her favourite video in real-time.


INTRODUCTION
Thrilling the society toward science and research is not trivial.Albeit the "academic industry" lavishes many efforts to spread its results not only among the insiders but to the whole mankind, the importance of sharing the knowledge of research is seldom a priority.There is a lack of open forums or events that might serve as a showcase and that might give to the science the visibility that it deserves.When it happens that an event attracts visitors to discover the work that every day is performed by myriads of researchers in their laboratory, instead, the response of the wide public is conforming, giving new hints for undertaking the impervious way of cultural dissemination.The European Researcher Night is probably the most important EU action in this sense, putting altogether researcher from different disciplines to show their findings through stands, short communications and events.From five years now, the Universitá Politecnica delle Marche organizes in Ancona (Italy) the SHARPER (Sharing Passion for Evidences and Resilience) event, with the purpose of bringing the knowledge along the street to make the city acquainted about the University research activities.Within this framework, the event able to attracts citizens is the video mapping projection.This latter, even known as 3D architectural mapping (Barbiani et al., 2018) (Neuman, 2018) or spatial augmented reality or SAR, is a video projection technique that can be achieved by realizing a perfect correspondence between the shape of a building and the images (or videos) to be projected.This new kind of visual art, becoming very popular for big audiences, is usually intended for playful purposes and functional to entertain the audience with creative and artistic contents (Catanese, 2013).However, regardless the purpose for which the entertainment is designed for, its realization requires a rigorous pipeline of work which spans from the acquisition of a faithful survey of the fac ¸ade to the adaptation of the contents upon it.
In this paper is described a multi-disciplinary process that makes use of a photogrammetric survey as an accurate source for video projection mapping.While well-established geomatics technologies (namely terrestrial laser scanning (TLS) and photogrammetry) paves the way for the virtual reconstruction of the architecture, they are even essential to perform analysis and studies which enables visual artist or art historians to tell the story of a building in a new and fascinating way (Maniello, 2016) (Empler, 2017).The adaptation between these two systems (the real object and the virtual one) is accomplished through the warping method, to perfectly match the digital 3D model to be projected with the real object.The visual mapping described in these pages has been developed for the San Domenico Church; the Church dates back its construction to the end of the 1700 and is designed in neoclassical style, so that it makes it the ideal stage to test such experience.The basic image used for the creation of the video mapping is an ortho-image realized by combining two kind of surveys: photogrammetry and TLS.Besides the realization of the visual mapping and a critical discussion over the procedure that has been used to translate a 3D model in a visual storytelling of the building, the article also describes an innovative way that has been set up for the management of the whole SHARPER event.
The system is app-based and was designed to allow the visitors to interact with the event directly from their smartphones; sev-eral active sensors have been displaced among the city, asking the user to search for virtual owls and to catch them by answering some questions, engaging the people by exploiting the gamification paradigm.This latter has been stressed further, since the video projection was conceived as a competition between the students of the Faculty of Architecture.Through the application, the attendant to the visual mapping where thus enabled to vote his/her favourite video in real-time.The main contribution of the article can be summarized as follows: i) definition of a pipeline of work spanning from the acquisition and processing of the architectural heritage toward its interpretation for realizing the SAR experience; ii) creation of an app-based system which enables both visitors and organizer to manage a complex city event; iii) evaluation of the performances of the whole event by analysing the digital footprints left by the individuals when experiencing SHARPER.

RELATED WORKS
Photogrammetry has long been used as a tool for collecting threedimensional (3D) information of cultural heritage objects as well as texture information.The 3D coordinates of points on an object surface are determined thanks to overlapping images with camera position and orientation information known as exterior orientation.The parameters of interior orientation that describe the principal point location and calibrated focal length of the camera have to be known.The exterior orientation can be determined if at least three control points are available in the overlapping image area.The 3D coordinate of all measured points in the overlapping image area can be determined by collinearity equation that defines the relationship between object and image coordinates (Cabrelles et al., 2010).Close range photogrammetry has the benefits of being relatively cheap and easy to set up; the portability of taking mainly the cameras (and tripods) to the different sites is an advantage that makes it appropriate for recording a large number of objects.The direct acquisition of digital images has a number of advantages in photogrammetric applications such as direct data flow and quality control, high potential for automation, and good geometric characteristic.Digital ortho-image is one of a method used in digital photogrammetry and its product is equivalent to the photogrammetric line drawing of the cultural heritage object.The effects of tilt, relief displacement, and lens distortions are removed with the help of the digital surface model.This product has the same geometric accuracy with the digital photogrammetric line drawing, but photographic information is also presented.The complete photogrammetric workflow to derive metric and reliable information from imagery consists of: (i) design, i.e. sensor and network geometry, (ii) 3D measurements, (iii) structuring and modelling, (iv) texture mapping and visualization.Nowadays, different commercial packages are able to perform these tasks.Even if photogrammetry can cover a wide spectrum of architectural applications (Remondino and El-Hakim, 2006), research application are moving toward the combination of techniques (Pierdicca et al., 2016), (Manferdini et al., 2016).One of these is the TLS, which enables a large quantity of three-dimensional measurements to be collected in a short time.It generates a point cloud in a local coordinate system with intensity values; additional information such as RGB values is usually provided by internal or external digital cameras.While the point cloud generated by TLS may be useful on its own, it is usually only a means to an end.It is generally used to record surface's information in order to generate 2D sections, profiles and plans, and 3D models1 .The best approach for the survey consist in fusion of different modeling techniques and instruments.This typology of three-dimensional data, originated from image-based and range-based systems, can be integrated with other metric information deriving from survey or maps, to obtain a correct georeferencing and adding metadata (Agudo et al., 2016).In fact, the only use of one of these technologies of three-dimensional survey doesn't allow to reach a satisfying result in terms of geometrical accuracy, portability, automatism, photo-realism and low cost, at same efficiency and flexibility.In most published works the concept of "integration" or "combination" of 3D models derived from photogrammetry and TLS is regarded basically in two ways: as texture mapping of the meshed TLS model or, conversely, as production of orthophotos using the aligned and triangulated point clouds.Here data fusion is aimed at creating a unique 3D virtual representation where photogrammetric and TLS data are merged together so that the final product can be seamlessly explored.With this approach 3D modeling applied to Cultural Heritage can benefits of the use of both surveying techniques (Clini et al., 2017), (Guarnieri et al., 2017), (Pepe et al., 2016).The videomapping technique, even known as 3D architectural mapping, is a video projection technique that can be achieved by realizing a perfect correspondence between the shape of a building and the images (or videos) to be projected.This new kind of visual art, becoming very popular for big audiences, is usually intended for ludic purposes and functional to entertain the audience with creative and artistic contents.A very interesting definition of this kind of visual art performances is the Spatial Augmented Reality (SAR), very diffused in the Italian panorama thanks to the work of Studio Glowarp throughout a 4 year festival and an intensive and very qualified publication activity (Maniello, 2014) (Maniello, 2018b) (Maniello, 2018c).Since from the first appearance of this locution (Bimber and Raskar, 2005), it was clear that technological variation of SAR displays should be able to overcome technological and ergonomic limitations of conventional AR systems.Due to the decrease in cost and availability of projection technology, personal computers, and graphics hardware, there has been a considerable interest in exploiting SAR systems.Parallels to the development of virtual environments from head-attached displays to spatial projection screens can be clearly drawn.Analogue evolutions of augmented reality has the potential to yield a similar successful factor in many application domains.Thereby, SAR and body-attached AR are still not competitive, but complementary.In SAR there is a transfiguration of the building, that involves some responsibilities.A building is not a simple screen where images are projected: it becomes a space of transformation, the place where the architectural idea arises an unusual level of attention, while, at the same time, architecture is temporarily liberated from the building's dependence.The form becomes the instrument of abstraction, metaphor, concept.Art is transformation of perceptive reality by means of transgressive optical contradictions towards evasion and abstraction (Antonelli and Mordenti, 2011).Those concepts are very useful in activities that aim to engage a large public in Cultural Heritage (CH).Recent studies and applications deal with how to stimulate the active involvement in CH knowledge thanks to a technological and social process.A specific passage, especially in gamification, is connected to the desire to complete and collecting: the involvement inherent in the forms of progress and discovery, the role of trophies and collections.It is clear that the challenge of involvement, even before technology and design, is in the mind (Viola and Cassone, 2017).The use of SAR technique offers new communication ways for archaeology and museum fields.It helps to show fragile or destroyed goods, sometimes far away from the exhibition set, for the listed reasons considered as fragmented and hard to be reached by a bigger audience, become interesting case studies of digital anastylosis.The aim of this kind of studies is to show from a different point of view SAR method that now can be considered not only as an "artistic" technique but also as a technique serving the cultural heritage, and to recommend it as one of the possible means of the future restoration and of the augmented heritage field (Maniello, 2018a).A quick resume about digital storytelling is also mandatory in this context: digital storytelling is generally a creative process of telling stories and sharing information with multimedia tools and resources.According to several educational studies, digital storytelling entered the academic mainstream due to the technological explosion and teachers are encouraged to engage the students with the help of technologies in creating their own stories (Ohler, 2013).Digital storytelling is quickly capturing the hearts and imaginations of educators because it combines traditional storytelling with modern-day pop culture and technology.As a result, digital storytelling offers educators a new and exciting way to captivate students interests like never before (Lowenthal, 2009).According to those studies, a digital storytelling at an architectural/urban scale seemed very compliant with the target of the European Night of Research, which is the need to educate and better inform about all research activities the large group of all citizens.Even if the data processing and editing the core part of the paper, it is important to discuss a bit more in detail the systems that are nowadays in use for the management of city events.In fact, the ICT driven approach should be a flywheel for sharing the knowledge of research and of CH.Events stakeholders and urban planners in general can have at disposal a powerful tool to collect information about the performances (Frontoni et al., 2016).Location-based services represent innovative models of interaction with stands and recreational areas by adding tremendous value to the visitors experience and extending the possibility to affect trajectories with way-finding services.Creating systems able to estimate the volume of people in a dynamic environment is a very important task for several applications.Knowing the volume and the movement of people in major public events offers interesting business possibilities for the organizers (Paolanti et al., 2018) (Liciotti et al., 2017).

WORKFLOW
The paper deals with different cultural and technical issues regarding: • acquisition and processing phase; • videomapping and spatial augmentation of reality with contest among students; • App based system for the management of the event and for the interaction by the public with the event itself.
Technical choices, visual results and feedbacks are able to depict a quite novel technological framework mainly based on knowledge and awareness of architectural heritage values as well as on communicative needs and expectation in research dissemination.Similar works contribute to offer a reusable experience and a scalable exploitation of in-deep architectural studies and cutting edge ICTs, with the aim to make built heritage alive and to engage wide public in its knowledge and conservation.

Architectural survey and data processing
The basic image used to realize the video mapping is an orthoimage realized by combining two kind of surveys: photogrammetry and TLS.The TLS survey was realized within a wider thesis work concerning the entire Plebiscito Square, where Church of San Domenico stands.Between acquisition point of the instrument Faro Focus 3D and the fac ¸ades objects (as railings or statues) were interposed.These objects produced shadow areas on the fac ¸ades and for this reason the point cloud is missing some portions.So it was decided to make a photogrammetric survey on filed: photos were made with a Sony camera model ALPHA A6500 mounted on a tripod.Photos were taken by twelve capture points, from each position were made 3 or 4 shots (with different angles of the camera axis: from 0 to 30 degrees) for a total of 44 shots.In order to be properly combine in one product, the two models had to be firstly related into a common reference frame, which in this case was defined by the TLS survey.To this end the globally aligned and reduced point cloud was georeferenced, before the merging step.Basically the transformation parameters were estimated through quaternion representation applied to a dataset composed by the 10 GCPs measured with the total station and corresponding points identified on the TLS-based 3D model.Photos obtained were then processed by means of Agisoft Photoscan software.The workflow within the software consists of the following steps: photo alignment, dense cloud construction, cloud point orientation and scaling, mesh construction, ortho-image export.The first phase allows to detect points in common between the photos and consequently allows to estimate camera position and orientation of each photo (as depicted in the image sequence reported in Figure 1).Then the software allows to generate and visualize a dense point cloud model.Based on the estimated camera positions the program calculates depth information for each camera to be combined into a single dense point cloud.The final point cloud was composed of about 18 million points.Next step concerns the orientation and scaling: given the lack of Ground Control Points, 10 markers were placed in well-identifiable points of the Church's fac ¸ade (mainly intersection of edges, because they are uniquely identifiable).Subsequently x,y,z coordinates of homologous points of TLS cloud point were assigned to these 10 markers.In this way, the point cloud, obtained from photogrammetric survey, has been oriented and scaled (Figure 1), with a final RMSE of about 4 mm, as visible in Tables 1 and 2, for both control points and From the point cloud the mesh was generated, counting about 3,5 millions of faces and about 1,5 millions of vertices.Finally texturized ortho-image was extracted (Figure 2), suitable to be managed for the next phase of the work, here following described.
Figure 2. Ortho-rectified image of the Church.

The spatial augmented reality contest
Each year, during the first steps of SAR contest, our team's commitment is to push students to leave their all day occupations in order to animate the final show of the SHARPER NIGHT.The main motivation for them come from the possibility to face and experiment a new form of expression.They already know this type of video experiences that are usually based on the projection of 3D motion graphics in real-time onto the surface of a structure or on a fac ¸ade of architecture.The result is the exterior of the building appears to come alive with animated forms, transforming into a dynamic construction.The projections are meticulously mapped to fit the specifications of the building and sound usually accompanies the display, but sometimes this kind of events are affected by a lack of sense regarding the framework in which are inserted or, in the worst cases, are not respectful regard to the architecture behind the projection.We are used to say that sometimes the unique goal is the "wow" effect.On the contrary, we motivate the student teams to convey their message in a new way, always aware that they will write and draw in a historical built heritage fac ¸ade.An effective sentence in their motivation is "Our University give you the chance to draw on a big screen!Take this opportunity!".In all previous editions of Sharper Videomapping projection the students who participated were left free about theme of their proposal.So in the four past of the event, from 2015 to 2018, were realized around 15 videos.These videos are all different from each other because, as said, students were totally free to express their ideas without any restrictions (only condition they had was on the duration that had to be at least of one minute and not more of two minutes).Despite the obvious differences, results of student's work can be clustered in three typologies according to the object they represented and the relationship between projected objects and church's fac ¸ades.These categories can be summarised as follows: architecture, narration and abstraction.Video projections belonging to first category are joined by the dominant role of architecture.This type of video enhances the architecture on which the video itself is projected.Movements, rotations, deformations, distortions, collapses and reconstructions of architectural elements (as pillars, columns, capitals, entablature etc.) allow a new reading of the architecture both from a semantic point of view and also from a more imaginative point of view.In particular, this technique was adopted by two groups.First group with his video projection called "Back to the future" transformed San Domenico fac ¸ades's from his original Neoclassical style into the different ones depicting a journey in the history of architecture.So, the small openings of the Romanic style are transformed into larger and bright glass windows of Gothic style; straight lines of Neoclassical fac ¸ade are inflected in curved lines of the Baroque style.Another group instead proposed a representation of the fac ¸ades's architecture according to the different styles of painting: from Impressionism to Pop Art, through Cubism: the video was entitled "Art attack!".In this case the authors message could be summarized as a suggestion about changes that art can imprint in reality, sometimes viewed as an attack or a violent modification.On the other hand, in the title there is joke lent from a TV series for kids (Figure 3).
Another approach can be described as narration.In this case the developing of video projection tells a sequence of images, linked to each other by a common thread.In this context, architecture no longer plays the role of absolute protagonist, rather becomes the setting of the narrated story.If in the previous approach there was an interaction "architecture-architecture" (between the physical and the projected one), now the interaction is between characters of the narration and architecture.This way of thinking video projection can produce outcomes belonging to different worlds: two of these are certainly original and are entitled "To the moon" and "Super Mario Ancona" (Figure 4)."To the moon" tells lyrics of the song "Rocket man" while in the other video the famous plumber Super Mario (protagonist of the homonymous videogame) faces his adventures in the unusual scenery of the San Domenico faade.The title is Super Mario Ancona considering that also some monumental heritage of the town is included in the video sequence.
In the present analysis, the third way is defined as "abstraction".
Here architecture plays a secondary role: it becomes a theatrical backdrop or a white canvas.On it are reproduced abstract shapes, lights, colours that do not represent physical objects but produce suggestions.A successful outcome of this kind is the video projection entitled "Paint my ark".Also Dorian, the winner of 2017 edition, despite the name that seams to tell a story, showed an abstract style (Figure 5).In fact the idea of death and decay are narrated manipulating the architecture.
In addition, we can took in consideration two main different approaches carried out during the concept phase of the videos: the completely digital approach and the more traditional "drawn by The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W9, 2019 8th Intl.Workshop 3D-ARCH "3D Virtual Reconstruction and Visualization of Complex Architectures", 6-8 February 2019, Bergamo, Italy hand" storytelling.The samples of these two quite different methods should be argued in the videos "To the moon" and "Ice and fire", both from the 2018 edition (Figure 6).
The traditional by hand approach has always been viewed as one of the best teaching approaches and always, in the concept phase of architectural design, we encourage the students to first use pencil.The same tactic is very rare in this kind of context: in all editions, only a group tested a complete storytelling in a traditional way and then transferred it in a video flow.It is interesting to show a comparison between the traditional and the digital drawing (Figure 7), highlighting that the steps to make sense of the complex world based on the storyline are not easy.Basically it summarises how important storytelling is in enhancing students' reading skills by engaging with the information they want give  with sound track.this digital approach is based on a sort of "action painting" on the faade: it requires to divide each time frame in an ordered sequence of layers, that are then animated (Figure 8).Those considerations show that the complexity of these narratives and their success derive from all their components: music, images, synchronization and consistency with the fac ¸ades.An easy recipe could be an incessant rhythm and sometimes a sort of "epilepsy" effect.It is possible to conclude that the traditional approach of storytelling is seen as an approach that brings lot of benefits to the students but they need more engagement with other sources.Therefore, digital storytelling was introduced in order to ensure a guaranteed impact on the public, especially the youngest.

Event management system
Analysing spatio-temporal dynamics of visitor movements and interacting with them at the same time is valuable.Planning events in fact should be on quantitative data (the crowdedness of a certain attraction, patterns of visitors among the festival, duration of their visit) that permits public authorities and planners to arrange the festival with more awareness.On the other side, also visitors of a public event could benefit from these technologies.In fact, it is possible to offer them real time information (i.e., timing, position, etc.), improving the fruition of the festival and even engaging them to interact with specific events within the festival.The proposed system, designed to manage the overall event, is composed of a cloud repository to store information and collect user-generated data and of a mobile application.This latter was designed to guide the user among the attractions, giving contextual information about the event.Its main functionality is the contextual notification of the attractions and the possibility to vote for the best videomapping.Notwithstanding, other functions increase the user's experience.The localization service allows finding attractions in a virtual map.The list of attractions is useful to reach the one of interest for the visitor; the program was continuously updated to permit a more complete visit of the event.As stated before, entrusting the management of the event on an ICT oriented system, has the twofold advantage of providing information to the users and at the same time to infer useful statistics from them.In the next section a brief overview of the performances of the event are showed.

RESULTS
The number of visitors tracked by our system is around 500.Considering that the event attract about 5000 visitors, the sample is not so hight to be considered meaningful at all w.r.t. the population.The reason is that, up to now, not all the users are inclined to use the application during the festival or were not present the day of the event.Notwithstanding, we had the possibility to extract some features and patterns from the visitors.For instance, in some points people were only passing through, while other were crowded for long time, from which we can deduce that some events are considered more interesting than others by the public.Even if the number of samples is low respect to the total number of visitors, it is possible to notice natural trends of people moving around and inside the urban event, allowing for studying spatiotemporal crowd density variations for predefined areas.The distribution of probability of the time spent at the event (Figure 10a) reports the range of time in minutes (x axis) and percentage of persons who spent time at the event (y axis).Almost a half of the sample seems to spend no more than 3 minutes at the event, meaning that a more effective strategy of engagement is required.The hourly distribution instead (Figure 9b) describes the percent- age of people present at the event (y axis) w.r.t. the time of the day (x axis).This information is very important to infer the peak of the event, which is clearly concentrated when the videomapping event occurred.This information can be even visualized in Figure 9.The graph describes the spatial distribution.The three subfigures represent the same area but with different resolution.The first one highlights the peak in red, again demonstrating that the peak was reached during the voting of the videomapping.

DISCUSSION AND CONCLUSION
The paper was aimed at describing a methodology of work, comprising heterogeneous visions, but with the common thread of creating a system able to put at the center two components: the knowledge and the users.Albeit recent advances in acquisition and data processing techniques enable insiders and researchers to collect huge amount of data, the channels of communications seems to be not enough to spread the Digital Cultural Heritage (DCH).The combination of different techniques represents a state of art methodology to achieve precise and effective output.Nowadays, digital methods like AR, VR, the WEB permit to spread the digital replicas all over the world in an easy way, but their are not probably exploited enough to achieve a complete democratization of CH.The SHARPER event described in the paper, even if localized in a city and for a short time frame, is a very good example of knowledge sharing.The ICT architecture developed facilitates the management of the entire event but, at the same time, enables the visitors to take part to it thanks to the voting tool.This way, he/she become protagonist, being engaged with a gamification approach.And more, it has been demonstrated how, by engaging the users to be active in the event, it is possible to collect useful information from them.As expected, the crowd grouped during the videomapping challenge.This latter deserve some further comments.First of all, SAR is a visual art that enable to tell the story of a building from different perspectives.Besides, it requires an accurate knowledge of the architecture, that can be achieved just with photogrammetric acquisition.This loop of work, combining the survey, a new form of communication and a new method of management should be adopted as a best practice, thanks to whom public administrators, researchers and the community could take part together to the common objective of communicating science in a more aware and fascinating way.

Figure 1 .
Figure 1.Sequence of steps performed to achieve the orientation and scaling of the 3D model.

Figure 3 .
Figure 3. Two examples of storytelling that put in the center architecture itself: Back to the future Sharper 2016 (a), Art Attack Sharper 2017 (b)

Figure 6 .Figure 7 .
Figure 6.Two examples of different approaches: To the moon (a), Ice and fire (b) Sharper 2018

Figure 8 .
Figure 8. Two screenshots of the sw environment for Ice and fire video: the digital painting splitting the sequence into layers, the digital animation of video frames.

Figure 10 .
Figure 10.Statistics about the event.Probability of distribution of time spent during the event, hourly distribution.
check points and a GSD of about 3mm/pix.