URBAN HISTORY IN 4 DIMENSIONS – SUPPORTING RESEARCH AND EDUCATION

The new research group on the four-dimensional research and communication of urban history (Urban History 4D) aims to investigate and develop methods and technologies to access extensive repositories of historical media and their contextual information in a spatial model, with an additional temporal component. This will make content accessible to different target groups, researchers and the public, via a 4D browser. A location-dependent augmented-reality representation can be used as an information base, research tool, and means of communicating historical knowledge. The data resources for this research include extensive holdings of historical photographs of Dresden, which have documented the city over the decades, and digitized map collections from the Deutsche Fotothek (German photographic collection) platform. These will lay the foundation for a prototype model which will give users a virtual experience of historic parts of Dresden.


INTRODUCTION
Imagine you're exploring the historic center of a city with its impressive town houses, churches and monuments.What if you could just use your mobile to find out about the historic buildings around you, with detailed visual information about how they were built and the story behind them, making history come alive before your eyes?
Photographs and plans are an essential source for historical research (Burke, 2003;Paul, 2006;Pérez-Gómez and Pelletier, 1997;Wohlfeil, 1986) and key objects in eHumanities (Kwastek, 2014).Numerous digital image archives, containing vast numbers of photographs, have been set up within digitization projects (Hermon and Vassallo, 2011;May et al., 2011).Information and image retrieval with these extensive repositories is still challenging.It is not easy to identify, analyze and contextualize the relevant sources, or compare them with the historical original.The eHumanities research group HistStadt4D, funded by the German Federal Ministry of Education and Research (BMBF), is investigating and developing methods and technologies for this.Historical media and their contextual information are to be transferred into a 4D -spatially and temporally scaled -model to support research and education on urban history.Content will be made accessible in two ways; via a 4D browser and a locationdependent augmented-reality representation.The prototype database consists of about 200,000 digitized historical photographs and plans of Dresden from the Deutsche Fotothek (German photographic collection).

Usage scenarios and research values
Digital image repositories meet a wide range of needs, from research in humanities and information technologies, through museum contexts and library studies, to tourist applications (Münster, 2011).Architectural historians have developed various methods of analyzing both preserved and never-built or destroyed structures (Brassat and Kohle, 2003).Style analysis, iconographic approaches and art sociological methods all address art historical questions.The trend in recent research is towards a comprehensive view of art which can be supported by image repositories: The advantage of the new digitality consists in a vast fundus of pictures available, but repositories need to be specially equipped to support art and architectural history research (Bauer, 2015).This raises two issues for urban history: 1) the analyzed objects and the spatial frame are large, making gathering, cataloguing and locating the sources more difficult.
2) Each source forms part of a specific timeframe, which means that often the scholarly examination will extend to several stages of urban development.The sources thus need to be organized on both a spatial and a temporal basis.Computerbased 3D and 4D models can help with this, enabling the combination of both components in one tool (Kohle, 2013).Nevertheless, sources from urban history are still challenging to work with.The view of the city they represent is mostly internal and thereby biased.Some sources are more objective, construction plans or mapping photographs for example.But written descriptions of a city, engravings or artistic photographs show a much more personal attitude towards the cityscape.Scientific researchers are therefore responsible for contextualizing them, taking these biases into account.Art historians are aware of this problem and objectivize resultsstill predominantly a manual process (Schneider, 2002).As a result, these more laborious studies consider only narrow time spans or locations.Computerized tools enable easier access to the sources and wider research foci, creating more opportunities for utilization and making a multi-focus analysis of the urban tissue possible.This leads to new questions: How do buildings and cities change over time?In which contexts, such as political or formal developments, does a historical cityscape evolve?What similarities can be found between objects in terms of construction standards and requirements, building codes, regional, temporal or personal tastes and styles?Furthermore, what connotations does a building have?Does the number of pictures taken of one specific building change over time?Are there relationships to other buildings or urban spaces?Which interactions of architecture with other artistic genres, inscriptions or infrastructural facilities can be found?Which buildings are likely to form the architectonic backdrop for social events, demonstrations, or celebrations?Digital libraries and computer-based analysis tools can help urban historians to overcome existing methodological barriers in both contentrelated and quantitative research.Reliable results can be drawn from different approaches, combining several methods and even disciplines.

Creating targeted tools for image repositories
The vast increase in digitization of historic images, objects and information makes more content available for investigation.More cross-analyses are possible; more knowledge is collected, structured and shared (Schuller, 2009).A systematic survey of the needs of users of image repositories is needed.Sweetnam et al. (2012) nicely break down users of digital collections into (a) professional researchers, (b) apprentice investigators, (c) informed users and (d) the general public.The actual user benefit of applications greatly depends on the usability, suitability and efficiency of the technical solutions (Dudek et al., 2015).The target group depends on the objective, whether (a) scientific research, (b) pedagogical application, or (c) study of historical sites (Barreau et al., 2014).The new scale of research and information retrieval creates various new challenges.Many scholars note that online searching for images and information is "counter-productive" due to the amount of irrelevant data retrieved or their limited technical abilities (Beaudoin and Brady, 2011).The degree of search expertise certainly influences research progress and results (Kemman et al., 2012), as does the implementation of suitable filtering and handling tools.A lot of the existing research platforms and applications stem from computer science and do not necessarily meet the needs of scholars from other disciplines like the humanities (Dudek et al., 2015).To improve the technological support options for researching visual media, the demands of the stakeholders need to be identified and distinguished.A thorough survey through focus groups provides insights into research abilities, habits, needs and motivation, that is research questions and what researchers find useful (Krueger and Casey, 2014).Scholars emphasize the importance of adhering to scientific standards, thorough documentation through the supply of metadata and the potential for collaboration and interdisciplinary work (Bentkowska-Kafel et al., 2012;Maina and Suleman, 2015).Using focus groups is a method from marketing research and can easily be adapted to researchers proposes.As Morgan (1996) notes, focus groups have various uses as a source of preliminary information that can be exploited for further studies: (a) to classify research approaches and the relevant information, (b) to emphasize findings and generate suitable survey questionnaires, (c) to develop test scenarios, and (d) to systematize methods to evaluate the suitability of applications (Lew et al., 2010).Students and scholars of art and architectural history were chosen for initial focus groups.The students are obliged to uphold the academic standards of their home institution.Their research approaches may not be limited to or driven by established disciplinary research processes.In contrast, more advanced scholars may have identified recurring issues.3D visualizations correspond with human viewing habits (Petrovič and Mašera, 2005) and allow intuitive presentation of further information like the location, appearance and surroundings of an object, display of photos at camera position, showing the time when they were taken.Moreover, they support depiction and validation of hypotheses and communicate connections and coherences (López-Romero, 2014).Interaction, manipulation and analysis of the visual data is essential for research (Hecht et al., 2015;Webb and O'Carroll, 2013).Image rectification, scaling and combination provide an insight into changes.Statistical analyses of acquisition habits over time may yield information on the relevance of certain buildings.The simplest way to link and contextualize visual information is to use highlighted keywords as hyperlinks in texts and captions.

Photogrammetric methods
A possible technological basis for creating access to large scale image repositories is the spatial and temporal aggregation of data, in this case historical photographs in a 4D model.The potential of photographic images ranges from pure documentation in archaeology and monument preservation, through image interpretation -for example damage documentation -to the production of complex 3D models for archaeological investigations (Bührer et al., 2001).Geometrical reconstruction from historical photographs is based on photogrammetry.Information from multiple historical 2D images has frequently been used to acquire 3D object geometries (cf.Bräuer-Burchardt and Voss, 2001;Henze et al., 2009;Siedler et al., 2011;Wiedemann et al., 2000).For some decades, traditional analytical photogrammetry has increasingly been complemented by digital image processing and analysis.The elaborate process of manual image analysis can be largely automated, creating large image collections from which geometric information can be generated automatically (Pomaska, 2011).Unstructured digital online archives like flickr.com,Google image search or Instagram provide a huge number of hits when searching for architectural keywords.Models have been automatically generated from commonly photographed objects like the Trevi Fountain in Rome, the mountain Half Dome in Yosemite National Park or the Great Wall of China (Snavely et al., 2006).The "Build Rome in a Day" project even focuses on rebuilding the entire city of Rome using structure-from-motion (SfM) technology (Agarwal et al., 2009).Today, automated photogrammetric methods are generally used primarily to evaluate current, mostly digital images.This has rarely been done for historical images, because scanned analogue records usually have unknown camera metrics, missing or minimal object information and low radiometric and geometric resolution.Especially for reconstruction using SfM, a high resolution is required to find enough feature points in overlapping images (Lowe, 2004).To combat this, existing historical photography can be rescanned with a higher dpi.Photographs could also be re-evaluated in photographic software, for example after sharpening.To ensure that enough images are available for reconstruction, recent photographs or 3D models can be combined with historical data of a specific building.Current models of the observed buildings are absolutely essential for this.Existing models can be used, like the virtual 3D city model of Dresden in different levels of detail (Landeshauptstadt Dresden, 2017).Individual buildings could also be recorded with a laser scanner or a camera and modeled using SfM and 3D software (Stojakovic and Tepavcevica, 2009).Our aim is to develop application-oriented tools for photogrammetric analysis of historical photographs, to integrate them into the process of historical image analysis (Fig. 1), and to create a spatial relationship to today's situation.These approaches will be combined using photogrammetric methods based on the photo library of the SLUB (Saxon State and University Library Dresden).Firstly, the photographs will be filtered in different categories, for example contrast or pixel count.Secondly, these images will be processed to generate a 3D model using different techniques like spatial resection, SfM or texture mapping.These photogrammetric methods will then be applied to a test object and checked for feasibility.The prototype 4D model, and the 4D historical photographs, drawings, plans, and information within it, will be made accessible via location or context related information access as augmented reality (AR) (Münster and Niebling, 2016).This technology has gained importance in the last few years and undergone extensive testing (Chang et al., 2015;Chung et al., 2015;Livingston et al., 2008;Walczak et al., 2011;Zöllner et al., 2010).AR is the enrichment of the real world through virtual data, which can include 3D models, texts, pictures, film or audio data.Enriching the reality or replacing parts of reality can help to bridge the cognitive gap between our daily life experience of a cityscape and its depiction in historical photographs (Niebling et al., 2008).In the historical context, the viewer is able to interactively capture visual and textual information about objects in their historical spatial reference system (Ridel et al., 2014).Our investigation will focus on the accessibility of historical data.How can interaction with virtual buildings be designed?Which metaphors can be used?How can augmented reality support educational and research settings?

Augmented reality
We surveyed the current state of the art in Mobile Augmented Reality Systems (MARS) (Höllerer, 1999), and Handheld Augmented Reality (HAR) (Pintaric, 2005) for cultural heritage applications.Stationary AR experiences as spatial AR (Basballe, 2010), or stationary HMD-based AR (Hall, 2001), used in museums and other installations, have enabled firstperson exploration of local and remote cultural heritage (CH) sites and content.They also allow for an active inspection of and interaction with digitized artefacts, annotated with rich multimedia elements conforming to interactive pedagogical approaches.This offers an enhanced experience to visitors and advanced working paradigms to researchers.AR systems can improve learning and interaction by providing self-guiding context-aware media for viewing and interpreting CH resources.
With the emergence of affordable mobile computing systems, HAR has made its way into CH.ARCHEOGUIDE (Vlahakis, 2001) is one of the first such applications for archaeological research, education, multimedia publishing, and cultural tourism, employing mobile AR for on-site visualization of reconstructed 3D virtual models of artefacts and buildings.Mobile computing devices -whether handhelds or wearableshave additional advantages.Content and experiences can be personalized to the users, and collaboration facilitated.As studies of technology acceptance have shown, pragmatic relevance of information (Olsson, 2009(Olsson, , 2012)), perceived usefulness, and enjoyment (Haugstvedt, 2012), have a direct impact on intentions to use mobile AR.Different approaches have been taken to using handheld AR in CH projects in the last 15 years, depending on the usage scenario, types of data to be visualized, and desired mode of interaction with both real and virtual objects.To support design choices in CH applications, we aim to provide a taxonomy of approaches to visualizing and interacting with CH data by analyzing and categorizing existing projects.

4D Browser
One alternative is to make the 4D model accessible via a 4D web browser interface for spatially and temporally located searches in media repositories.Such online applications can be equipped with useful features that are tailored to the research needs of art and architectural historians, but are also useful for locals and tourists.It is important to involve both target groups in the design and conceptualization of such applications, to ensure that the application is used and appreciated (Warwick, 2012).In 3D space, the user is not restricted to an orthogonal top view of a map, but can also navigate in custom angles and take up the position and orientation of the camera.The user obtains a more detailed understanding of photograph and the photographer's intention.A 3D model of the area of interest can improve the visualization an object.In addition, a time-varying 3D model (i.e.4D model), which shows the historic states of the construction, can enhance the understanding of the site and point from which the photograph was taken (Schindler and Dellaert, 2012).Although the usage and underlying technology of web 3D has evolved significantly in recent years and client computer resources are constantly improving, some challenges remain.3D models of whole cities tend to be large.To guarantee smooth user interaction, data needs to be efficiently transferred from the server to the client, where it has to be handled intelligently.To minimize the data load, 3D models are usually compressed to transmission formats like glTF or CTM (Geelnard, 2010;Limper et al., 2013).
Progressive loading and multi-resolution rendering algorithms help to avoid latency while loading huge 3D models (Ponchio and Dellepiane, 2016).Additionally, the model is processed server-side to multiple levels of detail (LOD) and segmented into tiles (Cozzi, 2015;Gaillard et al., 2015).In this approach, only 3D objects close to the camera are loaded and displayed in detail, while objects further away are smaller and less detailed.
Objects outside the viewport are not even loaded.Navigation in three dimensions is another challenge, as more degrees of freedom are possible.User studies showed that new-to-3D users "can get into navigation trouble very quickly […] and once in trouble, have a hard time getting back to a known, comfortable viewing state" (Fitzmaurice et al., 2008).This can result in frustration and rejection of the application.Advanced navigation tools that support new-to-3D users should be adopted to ensure a good usability for everyone.Another usability concern is the perception of the 3D models.When buildings are only represented by blocks (i.e.low LOD), they are difficult to recognize as such, while highly-detailed objects are visually more appealing.Nevertheless, users accept compromises, if those 3D models are aesthetically integrated into the visualization (Weller, 2013).Finally, concepts need to be evaluated in terms of how whole image collections and their metadata can be automatically retrieved from the online archives for further processing.The data also needs to be stored in appropriately so that the application can work.This requires a sustainable database concept that needs to cover the scope of CH and geospatial data.The CIDOC Conceptual Reference Model (ISO, 2014) with appropriate extensions (Doerr and Hiebel, 2013) is a standard ontology in this field, and is adopted by many projects.It is also suitable for semantic web technologies, so newly-generated data can be queried by other institutions.As the 3D models and their real-world objects become part of the database and are interlinked with the images, the application becomes more than a search tool for historical images.It is also a documentation of urban development (Samuel et al., 2016).

SUMMARY
As a result of huge and concerted digitization efforts, extensive digital repositories of historical photographs have been created in the past few decades.This volume of data presents a major challenge to support search, access and information enrichment for users.In August 2016, the HistStadt4D research group started examining scientific methodological requirements and intuitive user interfaces for dealing with massive media repositories from a multidisciplinary perspective.

Figure 1 .
Figure 1.3D model based on current and historical photographs (proof-of-concept)