360o FILM BRINGS BOMBED CHURCH TO LIFE

This paper explores how a computer-generated reconstruction of a church can be adapted to create a panoramic film that is presented in a panoramic viewer and also on a wrap-around projection system. It focuses on the fundamental principles of creating 360o films, not only in 3D modelling software, but also presents how to record 360o video using panoramic cameras inside the heritage site. These issues are explored in a case study of Charles Church in Plymouth, UK that was bombed in 1941 and has never been rebuilt. The generation of a 3D model of the bombed church started from the creation of five spherical panoramas and through the use of Autodesk ImageModeler software. The processed files were imported and merged together in Autodesk 3ds Max where a visualisation of the ruin was produced. A number of historical images were found and this collection enabled the process of a virtual reconstruction of the site. The aspect of merging two still or two video panoramas (one from 3D modelling software, the other one recorded on the site) from the same locations or with the same trajectories is also discussed. The prototype of 360o non-linear film tells a narrative of a wartime wedding that occurred in this church. The film was presented on two 360o screens where members of the audience could make decisions on whether to continue the ceremony or whether to run away when the bombing of the church starts. 3D modelling software made this possible to render a number of different alternatives (360o images and 360o video). Immersive environments empower the visitor to imagine the building before it was destroyed.


INTRODUCTION 1.1 Panoramic photography and cinematography
In the last decade there has been a growth in the use of panoramas for visualisation and documentation of complex architectures (Fangi, 2007(Fangi, , 2009) ) (Pisa et al., 2010), but these projects were prepared for scientific analysis, rather than for cinematography.(Gruen et al., 2005) computer-reconstructed the Great Buddha which was used in a 90-minute movie "The Giant Buddhas" (Frei, 2006).This 3D reconstruction was a part of the film, whereas the researcher proposes a method where the computer-reconstructed heritage site is presented using still and video panoramas with a 360º field of view, not with a limited field of view as in the cinema or television.Panoramas have been used in cinematography and in 3D modelling (Bienias, 2008), but this project focuses on 360º imagery and 360º video as a crucial method of presenting not only the interior of buildings from the past, but mainly narratives.Autodesk Image Modeler enables the generation of 3D models of buildings from panoramas (Downing, 2008) without the knowledge of closerange photogrammetry which is not necessary for artists and filmmakers.This paper presents a new approach for the documentation of heritage sites via 360º narratives that presents an artistic visualisation of the site which is necessary for the creation of 360° films.This visualisation was based on historical photographs, but due to the fact that there are no panoramas from the past, only limited areas of the ruin were 3D reconstructed (Figure 1).The views for the purpose of a 360º film were rendered in Autodesk 3ds Max and also recorded using spherical video cameras that generate 360º video.

360º video and 360º camera as the basis for 360º film
Panoramas improved the visualisation of interiors of the world and they became a part of our lives (Huang et al., 2008), however the advances in computer technology, larger broadband and the improvements in digital camera technology led to the creation of panoramic video (360º video) which is a sequence of 360º images displayed as a film (for example 30 panoramas per second).They are becoming widely accessible through the use of spherical and panoramic video cameras: Ladybug (Point Grey Research Inc, 2008), Kogeto (Glasse, 2011), Totavision's rig (Garlot, 2010) or Immersive Media Figure 1 A montage of 360º panorama of the ruined church and computer-reconstruction created on the basis of historical images.
(Immersive Media, 2009).This research paper examines the use of Ladybug cameras (Figure 2) within a heritage site that enable the creation of 360º narratives.

360º stories
The concept of 360º stories (Kwiatek, 2011) relates to the early stage of creation of feature 360º film stories by the application of the same or similar techniques of presenting narratives as in the early cinema.They characterise the following features: the application of commentary of a lecturer; creation of travel genre movies; a lack of advanced video editing; a lack of actors; a lack of dialogues.360º narratives may not only become a technological innovation, but also a new form of art, similarly to what had happened with the early cinema at the beginning of the 20 th century by the inventions of Georges Méliès who changed the way of thinking about the cinema.Méliès enabled the creation of first stories by using a number of visual effects in order to present first narratives (Popple and Kember, 2004).No longer was the documentary film the main attraction in the cinema.Correspondingly, 360º narratives attempt to find a method of presenting narratives using 360º imagery and video.

Motivation
Finding an appropriate method of telling narratives via the use of panoramas was of the challenge that the researcher faced when creating panoramic publications on CDs and DVDs. 360cities.net (360cities, 2009) or Google StreetView (Google, 2009) have collected thousands of spherical panoramas that are presented in interactive form in Internet browsers, but they still do not tell narratives.The invention of panorama paintings at the end of 18 th century and a method of presenting allsurrounding images in rotundas provided an inspiration for the researcher to create an illusion of reality using today's digital 360º panoramic environments.360º video is a new form of visualisation of the world and when displayed on a 360º screen can be a powerful method for presenting narratives, especially when merged with a panoramic video of the same site, but reconstructed in 3D modelling software.360º film discussed in this paper brings a bombed church to life in a "cinema 360º" (Michaux, 1999).Sarah Kenderdine states that such environment "can help people to better appreciate these often fragile heritage sites" (Gaffney, 2006).The researcher's aim is to provide a new type of experience for viewers within 360º screen where the audience would feel totally immersed and engaged with the narrative projected on a wrap-around screen.

Overview of the paper
The next section discusses the history of Charles Church and describes the location of the building.The third part of this paper focuses on a 3D reconstruction of the ruined church, whereas the forth part illustrates the creation of still and video panorama on the site and in 3D modelling software.The process of creation of 360° films is discussed in Section V. Section VI is focussed on the displaying of 360º films to a large audience gathered within panoramic environments.Finally, the last part (Section VII) summarises the project and presents the potential development of 360° films in the future.

The ruined church
Charles Church in Plymouth (Figure 3) is the object of interest in this paper.The researcher's photographic panoramic experiments were conveyed inside the ruined church.It also became the location where the story of a wedding from 1941 is virtually embedded.

The history of Charles Church
The building of Charles Church was finished in 1657 and it was consecrated by the Bishop of Exeter in 1665.Charles Church was destroyed by incendiary bombs and burnt out on the night of 20 th and 21 st March 1941.(Moseley, 2010) suggests that this church was regarded as "one of the last Gothic churches to be built, before the style disappeared".Today, the church is situated in the middle of a busy roundabout (Figure 4).There is no access for visitors because it remains a memorial ruin after the Blitz of 1941.The church is locked and there is no crossing available on the roundabout which could help visitors to access the site.The church was dedicated to King Charles I and not to be confused with Charles the Martyr (Robinson, 1991).

Panoramas of the church
This site was chosen by the researcher to attempt the development of an interactive 360º story.A number of photographic experiments are described in more detail on the researcher's website (Kwiatek, 2009) where interactive panoramas (Figure 5) from the interior are also published.The church was visualised using panoramas and then reconstructed in 3D modelling software.Panoramas created in 3ds Max (Figure 6) present the possible views of the interior as it looked before the bombing in 1941.They are located in the same spots as the 360º images recorded using a DSLR camera and fisheye lens.The next section describes the process of a 3D reconstruction of the ruined church.
Figure 5 Panorama created in the middle of the ruin.
Figure 6 Reconstruction of the church presented from the same location as the panorama in Figure 5.

3D reconstruction for 360º film
The site discussed in this paper is a ruin in the city centre and has the potential for panoramic applications related to cultural heritage.The pertinent church enables the researcher to present narratives about the event that took place in the past.To create the story-based panoramic interactive narrative based on Charles Church, the researcher needed to reconstruct the site.This type of narrative was defined in (Kwiatek and Woolner, 2010).

Approach to 3D reconstruction
The 3D reconstruction was not based on photogrammetrical methods (scientific approach) described by D'Annibale and Fangi (2009) as the researcher had planned at the beginning of the project, but a photorealistic visualisation was performed instead.The aim of the project was to create a 360º film based on a 3D reconstructed site and the existing ruin.A measurable 3D model of the building was not necessary for this purpose.A number of historical images were applied in this process, but due to the limited number of images of the interior being available, some parts of the church are visualised according to other resources.The purpose of this 3D reconstruction was the generation of 360º films from a 3D modelling application in the way that the rendered files are matched to recordings with panoramic cameras on the site in order to move the viewer to the past during 360º story.The next paragraph presents the method of 3D reconstruction from panoramas.

Procedure of 3D reconstruction
This project attempts to present a method of rendering content of photographic quality in 3D modelling software in order to apply it to storytelling in 360º environments.The 3D reconstruction was performed according to the following procedure: Pictures were taken using Nikon D90 (DSLR camera), Nikkor 10.5mm fisheye lens and Manfrotto panoramic head to create five spherical panoramas of the interior of the church (Figure 7); PTGUI software was used to stitch images.Each panorama was imported to ImageModeler to generate 3D models of the locations presented on each panorama; this process is divided as follows: o defining a coordinate system; o defining straight lines and flat planes; o constructing more difficult shapes (columns, arcs etc.) (Figure 8) Each of the five spaces were exported and merged in 3ds Max.3D reconstruction based on historical images was performed in 3ds Max using V-ray render engine (Figure 9, Figure 10, Figure 11 and Figure 12).

Historical images of the church
A number of historical images were found in various Plymouth archives (Local and Naval Studies Librarydepartment of Public Library, South West Image Bank and SWFTA) and this collection enabled the process of virtually reconstructing Charles Church.The reconstruction started by combining the historical pictures with the contemporary views, and SPi-V panoramic viewer (Hoeben, 2009) enabled the fitting of historical images to the right location in the panorama (Figure 11) using XML programming.3D reconstruction started from importing historical images to 3ds Max environment (Figure 12).Once the 3D reconstruction was ready (Figure 13), the generation of single panoramas within 3ds Max was performed in the same locations as these 5 spherical panoramas (defined in Figure 7).The accurate position of these 360º images was necessary in the process of creating still and video panoramas that are described in the next section.

Still and video panoramas in 3D modelling software
The generation of panoramas in 3D modelling software in the same locations as they were created on the site is not an straighforward task.The method of trials and errors was executed in this case because there is no preview with 360º field of view in 3ds Max that allows an immediate comparison of the position between the currently rendered panorama and 360º image created on the site.The application of SPi-V panoramic viewer made this task possible, because of the use of two layers and additional programming in XML.The process helped in the generation of five spherical panoramas in 3D modelling software in the same location as the starting panoramas.These five locations were the start and end points of 360º video rendered within 3ds Max (Figure 14).In order to view the results before generating the content for an immersive screen, panoramic viewers are necessary.They are discussed in the following subsections.
Figure 12 Historical images are the basis for 3D reconstruction of the church.
Figure 13 A completed 3D reconstruction of Charles Church.
Figure 14 The generation of 360º video in 3ds Max environment.

Panoramic viewers
There were, at the time of writing this paper, many sophisticated panoramic viewers for presenting high resolution panoramas on the Internet.The most recent being: krpano (Krpano, 2009) and Pano2VR (Rauscher, 2009) all of which are based on Flash technology and enable the display of gigapixel files using streaming technology.The researcher will employ Lucid Viewer (Villmer, 2011), which is the most recent panoramic viewer and provides the opportunity for presenting video panoramas in a creative and flexible manner by using XML coding.The limitations of panoramic viewers (limited field of view; necessity of interaction e.g.via a mouse to see other parts of a 360° video; the lack of spatial audio effect) also have an impact on the creation of 360º narratives.These limitations can be eliminated in immersive environments based on 360° screens which are discussed in the sixth section of this paper.

Panoramic viewers for 360º video
Only a few panoramic viewers are capable of displaying spherical and panoramic video at the time of writing this paper.These are: Lucid Viewer; KrPano; Pano2VR; Yellow Bird's viewer (YellowBird, 2010).Software for viewing wrap-around images is necessary in the process of creating 360º films because such content needs to be tested first on a computer screen and then if successfully implemented, in an immersive environment.

Recordings of 360° video on the site
Recordings on the site were performed using spherical video cameras -Ladybug2 and Ladybug3 (Figure 15), manufactured by Point Grey Research.These cameras have 6 lenses, where one of them is looking up to record the zenith area.They generate a large amount of data (approx.2-3GB per minute; 15-30fps).Table 1 includes a few parameters that differentiate these two cameras.360° images and 360° video are the basic elements for the panoramic film presented in this paper.360° footage helps to visualise the site only if presented on an immersive screen which provides the illusion of being in that location.
Figure 15 Ladybug 3 -spherical video camera in action.The process of recording on the site was not a straightforward task due to the fragility of the equipment that was produced for the office use, rather than in an environment such as a ruined church.A powerful laptop with Firewire 800 output and hard drive with RAID0 system has to be constantly connected to the camera.There are also a number of cables connecting a laptop, a spherical video camera and a battery and this set up makes the process of recording spherical video from point A to point B rather difficult.The researcher developed a motorised wheelchair for the recording of such 360° films, but this solution cannot be applied to this heritage site.Instead a trolley (Kwiatek and Woolner, 2010) or a Manfrotto Dolly was used for the recordings inside the ruin.

Merging 360° footage
In order to achieve the effect of changing the environment from the current ruin to the reconstructed site which was necessary in the 360° film (to present the dream of a bride), the researcher had to render similar trajectories within a 3D reconstructed church (Kwiatek and Woolner, 2009).To do this an animation of panoramic cameras was generated and resulted in thousands of panoramas which were then used to create 360° video.The time of rendering only one panorama (2000x1000px) was about 5-10 minutes, so the process of rendering animation from spherical camera is considerably longer.The method for creating video panoramas is based on creating a number of individual panoramas, while the environment around the cameras is animated or alternatively the set of cameras is moving through the space.Video panoramas in 3D modelling software were rendered between these 5 locations (Figure 7) where the first still panoramas were taken.The animations were set up between these points: 1-5, 1-2, 1-3, 1-4.The movement in the opposite direction was also used in the generation of 360° film and such video panoramas were rendered in Adobe After Effects CS5 using the reverse function.The frame rate of recordings in this project is 15fps which is enough for the projection on a large wrap-around screen.The sequence of spherical images was imported to QuickTime Pro and exported as MOV files using H.264 codec that is compatible with Lucid Viewer or without compression for display on a 360° screen.One of the advantages of using 3D modelling software is that it facilitates the rendering of files with a resolution adequate for the 360° screen (max.resolution of 9600x1080 pixels, whereas 360° video from Ladybug2 is limited to the resolution of 3500x1750 pixelsand Ladybug3 to 5400x2700px).These resolutions from Ladybug cameras are in equirectangular (spherical) format, but 360° screens accept content which is in cylindrical format.

Cylindrical and spherical video
The footage from 360° rig (Figure 16) provides the resolution of about 8000x1080 which is the resolution needed for a cylindrical screen.Ladybug cameras do not provide enough resolution for such screens, but were necessary to prove the concept of creation 360° films.They create spherical video which is acceptable by panoramic viewers which needs to be cropped (from an equirectangular to a cylindrical format) when used on 360° screens that are discussed in Section 6.
360° film discussed in this paper also consists of actors that are playing roles of guests, parents or a groom and a bride within this panoramic environment.The next section describes techniques that were used in order to transport real actors to 1941.
Figure 16 Six HD cameras create a 360° rig which generates cylindrical video panoramas.

Process of creating 360º film
The process of the creation of 360° film in Charles Church consists of these elements: 3D reconstruction of the site; definition of a storyboard; recording 360° images and 360° video, first on the site, then in 3D modelling software; recording actors in a green screen studio; editing and testing of the final film in a panoramic viewer; presentation of the 360° film in an immersive environment.This project uses a 360° field of view to tell a narrative of a wedding that happened in Charles Church in 1941.In order to follow the tragic moments of that day, the researcher decided to use 360° imaging.It also helps the viewer to be transported to the past and by making decisions, hopefully changes the sequence of the narrative.

The narrative
The researcher chose a story of a wedding of Ken Beer and Phyllis Corry who married in the ruins of Charles Church on 22 nd March 1941.The marriage was the last one that happened on this site (Rees, 2010).The Blitz began on the night of 20 th and 21 st March 1941.As a result, incendiary bombs left the church in a ruin.The vicar of Charles Church decided to have the wedding in the church the morning after the bombing, even though the site was destroyed.In fact, the original wedding happened on the day following the bombing of Plymouth and they had their wedding in the ruined church, but the researcher decided to move this event in time to create a more dramatic atmosphere.The mentioned couple are in their 90's now and they still live in Plymouth.In 2011, they are going to celebrate their 70 th wedding anniversary.

Locating actors within the 360º film
The researcher decided to use video recordings of actors who are used as characters in 360° narrative whereas the coevolutionary narrative developed by Kenderdine (2008) presented computer-generated (CG) characters which were embedded with this type of storytelling.The concept of adding actors to the 360° film was one of the tools which helped in the conversion of viewers gathered within a 360° screen to participants of 360° stories.Chroma keying techniques enabled the researcher to use the same recording of a person walking in a number of stages throughout the narrative.On one occasion the actors are displayed on a background of the reconstructed site, another time they have a ruined site behind them.The process of the actors walking was recorded using Ladybug 2 camera (Figure 17) in a green screen studio (Figure 18) to achieve the same type of distortion that appears in spherical panoramas.Figure 19 presents a spherical panorama that was created by merging footage recorded in the green screen studio and the spherical panorama rendered in 3ds Max.
Figure 17 Recording actors with Ladybug 2 camera in a green screen studio.
Figure 18 A spherical view of a green screen studio, created using Ladybug 2.
Figure 19 The wedding takes place in a historical (3D-reconstructed) Charles Church.

The idea of 360º cinema
In the past, early panorama paintings presented in a wraparound form on a cylindrical screen, transported viewers to remote locations or historical or biblical events (Wilcox, 1996).Today, seamless multi-projector 360º displays maximise the realism of such travels by displaying high resolution 360° images and 360º video.Panoramic films discussed in this paper might be the next step of the development of the cinema.One of the best known movie directors, Steven Spielberg, has already seen the potential of 360° cinema.Spielberg, in the interview in The Times in 2006 (Philadelphia, 2006), states that: "in the not [too] distant future you'll be able to go to a movie and the movie will be all around you.The movie will be over your head, it will be 360 degrees around you".It seems that this time has come and this type of experience is now available due to the use of projectors and proper computer software.Some of these 360º screens focus on the presentation of scientific research (Elbe Dome in Magdeburg (Schoor et al., 2008), Cyclorama in Montreal (Chapdelaine-Couture, 2009)), whereas others (ICCI360 (Innovation for the Creative and Cultural Industries, 2010) or Totavision's screens (Garlot, 2010)) are good examples of telling 360º digital narratives that are presented on 360º screens.

360º screens
360° screen provides an opportunity for spatially situating the audience within the site where the narrative is located; here they would feel personally involved.This project supports the visitor in imagining the church before it was destroyed by triggering emotional responses to the tragedies from the past.Additionally, the 360° arrangement of screens offers the potential for the user to feel as though they were truly there and feel personally present on the site.

Screenings of the 360º film
The panoramic film about the wedding in Charles Church was presented three times on immersive screens.First in August 2009 in Vision3D lab in Montreal (Canada) (Figure 20), then in February 2010 in Plymouth during the event called Arena360 (Figure 21), and then in September 2010 in Plymouth (ICCI360 Festival) (Figure 22).The construction of the screen was provided by IglooVision (Igloo Vision, 2010).
Figure 20 The screening of the 360º film on a 360º screen in Vision3D lab at the University of Montreal (Canada).
Figure 21 The interior of an immersive space (Arena360) where the audience is watching 360º film about Charles Church.Photo: D. Hotchkiss.
Figure 22 360º view of the interior of ICCI360 screen with the presentation of the 360º film about Charles Church.

360º documentary of these events
The recordings from these events are available on the researcher's website: http://www.360stories.net(accessed 3.02.2011).

CONCLUSIONS
This paper presented a method of generation of 360º film based on the example of the bombed church in Plymouth.3D reconstruction of this heritage site (built on historical images) and recordings in a green screen studio were necessary in order to present the interactive narrative on immersive screens.The application of spherical video cameras (Ladybug2 and Ladybug 3) and a camera with a 360º field of view in 3D modelling software, provided a number of alternatives which are necessary for the generation of interactive films.The audience gathered inside 360º image spaces had a chance to decide about further developments of the story.The future development of the project could progress in the direction of presenting stories, not only within cylindrical screens, but also on a fulldome projection system or even within the existing ruins of buildings, where 360º films can be projected on walls of buildings.

Figure 3 A
Figure 3 A ruin of Charles Church in Plymouth.

Figure 4 Figure 2
Figure 4 Aerial pictures of Charles Church located in the middle of a roundabout.Source of images: http://www.bing.com/maps(accessed19.11.2010)

Figure 7
Figure 7 Locations of five spherical panoramas.

Figure 8
Figure 8 Modelling of windows of the ruin in ImageModeler.

Figure 9
Figure 9 3D model overlaid with panoramic picture.

Figure 10 A
Figure 10 A wireframe model of the reconstructed church.

Figure 11
Figure 11 Historical image merged with the panoramic image is the beginning of 3D reconstruction.
Differences between two spherical video cameras.