A synchronized approach for integrating photos and 3D GIS in the applications of disaster management

Photos are an effective tool for recording ongoing disaster situations or damages after hazards. The emergence of smartphones has revolutionary influences on the use of photos. Even the general public can take photos, report any disaster situations they observe via mobile network, and share photos in social media because nearly everyone has a smartphone. These photos may provide valuable information about the disaster in reality. In addition to visual inspection, effectively and efficiently taking advantage of the available photos remains an interesting challenge. From a 3D perspective, we propose a synchronized approach for integrating the visualization and manipulation of photos and 3D geographic information system (GIS) data. The photos provide newly acquired disaster situations, while the 3D GIS data provide a realistic and comprehensive reference and links to other domain data to analyze damages from different viewpoints effectively. The key idea is to restore the position and orientation when the photo is taken in the system and synchronize the visualization of both types of data with a dual-window interface. If common feature points between the photos and 3D GIS data can be found, the damage assessment can be even more accurate. This approach will be extremely useful when before-and-after situations are compared. With the awareness of environment changes and willingness of participation from citizens increase, the proposed approach presents a new direction by highlighting the use of photos for improved decision making in disaster management.


INTRODUCTION
Climate change impacts global warming in two ways. One is the change in extreme climate, and the other is the change in average weather state. Climate change is now well established as a global issue of scientific and political importance. ENSO, which is an important source of natural climatic variability (Collins et al., 2010), causes the average distribution of precipitation to shift to higher or lower values. This condition indicates that thresholds to flooding or drought are often crossed.
The Intergovernmental Panel on Climate Change (IPCC) released its latest report on March 31, 2014. This report pointed out that each of the last three decades has been successively warmer on the Earth's surface than any preceding decade since 1850. The globally averaged combined land and ocean surface temperature data as calculated by a linear trend showed a warming of 0.85 °C in the period of 1880-2012. Nearly the entire globe has experienced surface warming according to the calculation of regional trends in a long period   (IPCC, 2014).
Since 1950, the temperature and climate have changed dramatically. These changes include atmospheric and ocean warming, snow and ice cover reduction, sea level rise, drought, and heavy rain. The extreme climate change and the environmental vulnerability have resulted in disasters that occur anytime and anywhere and made them a global normal trend. Effectively and rapidly recording the detailed information of disasters and describing their impact in numerical records have become a popular issue in academic institutions and government departments.
Photos are an effective tool for recording ongoing disaster situations or damages after hazards and are a type of static description of real-world in a short time. Users can obtain considerable information from photos. The increasing popularity rate of using smartphones and the Internet in recent years has facilitated the two revolutionary impacts of the use of photos as follows.
1. Taking photos with a smartphone attaches metadata to photos. Metadata add value to photos because they offer useful information, such as date, location, and height. 2. People who take pictures with a smartphone are members of mobile sensors, and the masses can send the observation information of a disaster with cellular data.
Photos captured from the disaster area can directly pass the information of the disaster to people. In addition to checking and analyzing photos in visual method, effectively using these photos is an important issue. Crowdsourcing has become a popular and indispensable component of many problem-solving pipelines in disaster management. Crowdsourcing can offer the department of disaster management real-time and exact data by combining smartphones and photos. For example, crowdsourcing played a significant role in the Haiti earthquake in 2010 and the spread of wildfires in Colorado in 2012 and 2013 (Zhong, Duckham, Chong, & Tolhurst, 2016;Zook, Graham, Shelton, & Gorman, 2010).
3D data can provide people with a more realistic feeling than 2D data. The digitization of the 3D shape of real objects is a rapidly expanding field and has been applied in various domains. Riding the tide of global expansion and the development of 3D technology, interest in 3D city models has increased significantly in the past years. 3D city models are usually composed of a description of the terrain, streets, buildings, and vegetation in build-up areas. 3D city models show the geospatial phenomenon of a city and thus not only express the plane relationship between objects but also describe the vertical relationship between them.
Using the 3D city model in the 3D geographic information system (GIS) can overcome the problems that the traditional 2D GISs cannot solve, such as analysis of the field of view and sunshine rights. This approach can provide considerable information for decision makers in operating disaster management systems.
This study is described from the perspective of 3D geographic information. We propose a synchronized method that combines the collection of photos obtained by the masses with the 3D GIS. When people provide latest photos from the disaster, the 3D GIS provides practical and total reference data to optimize the analysis of damage assessments from different perspectives.
The key idea is to restore the position and orientation when the photo is taken in the system and synchronize the visualization of both types of data with a dual-window interface. The conceptual model of the dual-window system is shown in Figure 1. The system has two major components: photos and 3D city model. The photos are taken by the residents, the crowd, or the disaster relief agency assigned commissioner. These people went to the disaster-struck area to take photos during or after the disaster. The 3D city model is provided by the government agencies or academic institutions, and the model is established before the disaster.
Figure 1. Conceptual model of the dual-window system As the awareness of environment changes and willingness of participation of citizens increase, the proposed approach presents a new direction by highlighting the use of photos for improved decision making in disaster management. In the future work, we will demonstrate that certain operations, such as measurement (e.g., that of the depth of floods or damaged area), can be readily implemented when the common feature points between the photos and 3D GIS data are precisely matched. This approach will be extremely useful when before-and-after situations are compared.
The rest of this article is organized as follows. Section 2 reviews the 3D GIS, photos, and crowdsourcing techniques for disaster management. Section 3 presents a solution for integrating the key elements. Section 4 is a system implementation case of flooding. Section 5 summarizes the study and proposes the direction of future extension.

RELATED WORKS
Our study builds upon the prior art in 3D city model, Web 3D city model, photos, and crowdsourcing. The studies relevant to our key components are reviewed as follows.

3D City Model
3D descriptions of urban areas are increasingly required for various applications, such as town planning, microclimate investigation, or transmitter placement in telecommunication (Weidner & Förstner, 1995). 3D city models provide considerable important information for different aspects of disaster management. First, they memorize the shape and configuration of a city. In case of severe destruction of infrastructure, government departments can immediately access the reference data to check the extent of the damage and rebuild the damaged sites. Second, 3D city models enable 3D visualizations and facilitate localization in indoor and outdoor navigation. Third, 3D escape routes inside and outside of buildings can be determined with an appropriate city model. Fourth, 3D city models identify affected building stories in flooding scenarios (Kolbe, Gröger, & Plümer, 2005). Thus, 3D city models can provide excellent auxiliary information in all stages of disaster management.
Two methods can be used to build 3D city models. One is procedural modeling, and the other is urban reconstruction (Zhu, Shen, Hu, & Hu, 2018). Procedural modeling encompasses various generative techniques that can automatically produce a specific type of content based on a set of input parameters (Furukawa, Curless, Seitz, & Szeliski, 2009;Vanegas, Aliaga, & Beneš, 2010). Urban reconstruction is a method that utilizes computer graphics, computer vision, and photogrammetry and remote sensing techniques to reconstruct individual buildings and façades of models of urban areas (Musialski et al., 2013). Both methods can generate a large number of 3D city models to provide GIS value-added applications. The cost of building 3D city models is directly proportional to the level of detail of the model. Zhang et al. achieved a cost reduction of 80% in building 3D city models by using a self-developed 3D laser scanner. Ftáčnik proposed the low-cost input data, such as DTM and terrestrial images taken with ordinary digital cameras with special focus on the cadastral data, to provide building footprints that can be used for reconstruction. Neitzel et al. used computer vision to generate 3D point clouds and 3D city models from photos taken by lowcost UAVs (Ftáčnik, Borovský, & Samuelčík, 2004;Neitzel, Klonowski, Siebert, & Deshbach, 2011;Zhang et al., 2008). These works show that the trend of building 3D city models moves toward a low-cost, high-precision direction.

Web 3D City Model
3D city models cannot be applied in the Internet at early stages due to the limitations in hardware, data size, and algorithm. The development of the Internet and image processing technique has enabled the 3D city models to be browsed and operated in the Internet with web browsers of installed plugin or relevant applications. Different web browser plugins introduce compatibility issues and cause difficulty in using and developing such applications. Plugins can also be a security problem. Thus, instead of relying on plugins, developers have regarded JavaScript as a means to create and manipulate 3D objects directly in the browser. WebGL, which is a JavaScript 3D rendering package that has been developed in recent years (Web Graphics Library), does not require installing plugin 3D applications and can be run directly from the browser (Nazarov & Galletly, 2013;Prandi, Devigili, Soave, DiStaso, & DeAmicis, 2015). The technique of WebGL provides large benefits for the application of Web 3D GIS in visual display and spatial analysis aspects. A Web-based 3D GIS can rapidly change the viewpoint to allow users to understand the relationships between objects (Ware et al., 2001). Many 3D file formats for storing information about 3D city models are available, such as OBJ, DAE, KMZ, I3S, and FBX. The distinct specification of each model increases the difficulty in applying 3D city models.

GL transmission format (glTF) has been designed by Khronos
Group. It supports the basic elements of 3D scenes (node hierarchy, materials, textures, animation, cameras, and lights). glTF separates the scene hierarchy and a binary block containing vertex, index, and animation key frame data as raw byte array. glTF adopts the JSON format to describe the node hierarchy, meshes, and materials and access the binary block. glTF is also a common 3D tile format and can transmit 3D city model smoothly. glTF has many advantages, such as compact transmission, fast loading, full scene description, neutral runtime, open, and extensible (Miao, Song, & Zhu, 2017;Schilling, Bolling, & Nagel, 2016). Most 3D formats can be saved to glTF using a conversion program. Data saved as glTF can increase the chances of sharing and reduce the cost of largescale restructuring.
The traditional 3D city model used in disaster management has two critical issues. One is that the size of the model is too big to operate in the Internet. The other is that the cost of model construction is too high, which is not conducive to updating large-scale data. The corresponding solution is to use glTF. This study uses a synchronized approach based on existing 3D city model and WebGL technology to establish a dual-window system for enhancing the application of 3D city model in disaster management.

Photos taken with Smartphones
Smartphones with built-in high-quality cameras, GPS, gyroscope, and other sensors have become more popular than before. With the availability of integrated GPS and gyroscope, the 3D location information ( ,Y,Z) and three-axis attitude information (ω, φ, and k) can be obtained directly. Figure 4a shows the relationship of pitch, roll, and azimuth (or called heading) to the initial reference frame. Pitch, roll, and azimuth represent rotations around the x, y, and z axes of the device, respectively. Figure 4b shows that the range of the value of pitch is from −180° to 180° when the phone rotates around the x axis. Figure 4c shows that the range of the value of roll is from −90° to 90° when the phone rotates around the y axis. Figure 4d shows that the range of the value of azimuth is from 0°-360° when the phone rotates around the z axis (Novakova & Pavlis, 2017  Digital photos can be streamed instantly by smartphones with wireless network. These photos can be used in many applications, such as news reporting, disaster reporting, and phenomenon display. Photos as raster data can record the real situation instantly. However, if photos have no information explaining the time, place, and event theme, then the user can only subjectively analyze the information of these photos and cannot faithfully explain the theme that the photographer wants to present at the time. This situation greatly reduces the effectiveness of photos. When the photo is edited by someone, the original credibility of the photo will be destroyed, and its property of evidence will be lost. Photos with uniform standards can promote the ability of photo exchange widely and improve the availability of photos (Huang &Fang, 2010). Using Exif offered by the Camera and Imaging Product Association as the metadata of photos is a popular method of scholars and the industries. A camera has many essential parameters, such as date, position, height, date of manufacture, type, and resolution of width and height, and record data in Exif (Weiqi, Zhenhua, Feng, & Jiwu, 2007 (Chun, Hwang, & Kim, 2012).

Crowdsourcing
Crowdsourcing is an online, distributed, problem-solving, and production model that uses the collective intelligence of networked communities for specific purposes (Brabham, Ribisl, Kirchner, & Bernhardt, 2014). The popularity of smartphones has created a large opportunity for crowdsourcing. In this process, someone sends out requests for certain information, whereas others help collect, process, and return them to the requester. An important application of crowdsourcing is to collect photos during emergency (Karaguchi, Sakai, & Fukumoto, 2018). Government departments will establish the Emergency Operation Center (EOC) when disasters occur. The disaster report must be returned to the EOC for the initial decision making to determine the damage situation as soon as possible. However, the government departments often have difficulty dispatching appropriate personnel to arrive at the disaster site due to lack of manpower or blockade in the disaster area. At this time, the local volunteers can immediately report the damage within the scope through smartphones (Ishida et al., 2014) and achieve the effect of crowdsourcing immediately. In this case, the response center can make the most immediate judgment and response on the basis of the photos provided by the volunteers and a few textual descriptions. The EOC may also need certain objective information (e.g., landmark or buildings). If the photo provided by the crowd contains the message (e.g., the situation of the flooding-struck buildings), then it will help the EOC make critical assignments to manpower, equipment, and supplies (Wu, Wang, Hu, Zhang, & Cao, 2016). However, the judgment of immediate disaster rescue will be affected due to the long time it takes to for the government to publish correct and reliable news. At this point, the photos provided by crowdsourcing are the best way to obtain instant information.

ARCHITECTURE
The disaster management process established by Federal Emergency Management Agency (FEMA) has become a general flow. The process of FEMA can be divided into four stages: mitigation, preparedness, response, and recovery. Before the disaster, the management plan emphasizes on mitigation and preparedness. When the disaster occurs, the plan emphasizes on rapid response. After the disaster, surveying the damage is an important part of the plan. Currently, government agencies face the following problems in disaster management. First, disaster management is limited by factors, such as too far location, bad weather, and insufficient staff, during the disaster. Assigning an officer to record the disaster is also difficult. Second, photos taken by smartphones are convenient. Photos with no key information can only describe the disaster situation visually but cannot be used for spatial analysis. Third, regionality is a key influencing factor of the occurrence of disasters. Previous information of disasters should be recorded in 3D GIS system. The development of 3D GIS has enabled the function of spatial analysis to 3D city models to provide more information than that to 2D objects. Fourth, if the government departments have not fully obtained the disaster information (e.g., water depth in flooding area) after its occurrence, then the difficulty in compensating the victims increases.
This study refers to the research results and theoretical basis of 3D city model construction, crowdsourcing, and 3D GIS-related fields and proposes a dual-window synchronized system for solving the key problems of government agencies in disaster management. First, previous disaster events can be recorded for disaster management in the next phase. Second, we can use crowdsourcing to instantly determine the current situation of the disaster area and related data through the cooperation of the crowd and the government departments. Third, this approach can immediately record and return the disaster-struck area information to the decision makers and present the disaster situation in specific data to provide a reference for policy makers. Fourth, 3D GIS can integrate the application data of the disaster area and provide historical and current data to assist in analyzing the situation of the area.
On the basis of the architecture of 3D GIS, synchronized dualwindow system can analyze photos and parameters uploaded by the crowd in the Internet and restore the scene in the moment of disaster by 3D city model. However, problems in the following aspects must be overcome: (1) the use of the existing parameters in the photos and additional parameters needed by the system to restore the view angle in the 3D city model, (2) the relationship of the view angle in the 3D scene to the photo, (3) the format the system reproduces and provides to the user as reference in the collection of historical data, and (4) the number of save formats the 3D city model has and the integration of this heterogeneous model into Web 3D GIS.
The external orientation parameters are key elements of the dual-window interface. The traditional photogrammetry uses aerial triangular adjustment to solve these parameters. With the advancement of computer vision technology, Nadeem et al.
proposed to obtain the external orientation parameters directly from feature points by 3D point cloud data matching (Nadeem, Jalwana, Bennamoun, Togneri, & Sohel, 2019). The external orientation parameters are also the main elements in the system to restore the position and attitude angle of the photographer in the 3D city model. Using these models can achieve the synchronization of 3D city model and photos. However, they cannot sufficiently restore the view angle of the photographer in the 3D city model. They also need parameters of azimuth (or heading), pitch, and roll data. Apart from obtaining the position parameters from Exif of photo, we also use the application to automatically record the information of pitch, roll, and azimuth in the text file format and then upload the photos and the text files synchronously in our system. We use these location and orientation parameters to synchronize the 3D city model and the photos. In addition to utilizing parameters, the dual-window synchronization system aligns the feature points in the photo with the feature points in the 3D city model to provide the user with the same viewing angle in both windows for performing spatial operations. The architectural diagram of the research system is shown in Figure 5. The components and their functions are described below. 2. The photo is another main component of the dualwindow synchronization system. It is photographed by the people or government agencies at the disaster area. Photo is an instant disaster description record. The metadata are description files that describe the instantaneous position, posture angle, time, date, and other information of the photo. These data can provide a dual-window system to restore the view angle of the photographer. 3. Dual-window synchronization system can connect to the 3D city model and the photos uploaded and processed by the public and provide decision makers with information without arriving in the disaster area.
We design two processing modules called "Format Transformation Service" and "Photo Transformation Service" to solve the problem of system operation in implementing the system architecture. Processing module 1 is a format transformation service and can integrate the heterogeneous 3D city model into the same format to support the dual-window system. Processing module 2 is a program to extract parameters from metadata and Exif of photos. These parameters are the input values that show a synchronized view in the dual windows. Photos obtained by crowdsourcing have problems of alteration and misinformation. Processing module 2 provides the function to filter composite photos by analyzing the difference in the product date of photos and the date the photos are captured. If the two types of data are different, then these photos are modified.
This study has two key breakthroughs. One is to restore the position and orientation when the photo is taken in the system and synchronize the visualization of both types of data with a dual-window interface. The other is to demonstrate that the common feature points between the photos and 3D GIS data can be marked. The advantage of this function is that the system can implement measurement in the 3D city model in the future. In this study, we use a single photo and 3D city model to present the effect of marking co-points in the system. The external orientation parameters of the photo are utilized to restore the view angle of the photographer in the 3D city model. The photo is taken as the basis for our operations. When we click on a specific point on a photo, we can display the common point in the correct position of the 3D city model. When these successfully corresponding point data between the photo and the 3D model can be recorded in the system, the analysts can use this system to understand the disaster situation before and after without arriving at the disaster-struck area. In this study, we use the Cesium as a base platform and develop a self-developed 3D GIS, which can load photos and metadata, 3D city models, and historical disaster data from different periods.
We also adopt glTF to solve the problem of heterogeneous data. We can also solve the network speed of transmission due to the advantages of glTF. When the 3D data are converted to glTF, they will carry the information, such as position and height of each feature point. Spatial operations, such as measurement and loop analysis, can be performed on the basis of these feature points. The system can provide space operation functions, such as measurement and spatial analysis, in the future by analyzing these co-point data. The flowchart of the dual-window system in the real world is shown in Figure 6.
1. Before the disaster occurs, the government departments, academia, or industry use various sensors or software to establish procedural 3D city model or urban reconstruction city model and collect historical disaster data. These data will be stored in a national or regional spatial database for people who need these data for download and use. 2. When the disaster occurs, the system will connect and download the proper 3D city model and historical data of the disaster-struck area. 3. Rescuers assigned by the government departments to the disaster area can use smartphones to take photos of the disaster area and upload photos and metadata to the dual-window system. 4. The camera of the 3D city model in Cesium is defined by position, orientation, and view frustum. After receiving the photos, the dual-window system parses the Exif and metadata of photos and sets the camera position and orientation at the same view angle as the photographer. 5. The 3D city model in the dual-window system can be moved synchronously with the photo and the copoints in the photo, and the 3D city model can be displayed at the same time. 6. These point data will be used for spatial analysis, such as measurement, for the decision makers.

CASE STUDY
Flooding is a common natural disaster in Asian countries. The statistics of the United Nations Emergency Events Database (EM-DAT) reported that 127 flooding incidents accounted for 40.3% of the total annual disaster rate; thus, flooding is the most widespread disaster to the public among 315 large-scale natural disasters that occurred in 2018 (EM-DAT, 2019). This study will be applied to flooding management as an example to verify the implementation of the mechanism. The system is mainly divided into two parts to manage and store historical disaster data and current disaster photos. One is the photo classification and processing subsystem, and the other is the dual-window subsystem. The intelligent photo management subsystem can provide photos and metadata uploaded by the crowd. This subsystem can also record historical disaster photos in the past and provide functions, such as filtering and excluding synthetic and changed photos. The synchronization subsystem can import the 3D city model, historical disaster data, and photos uploaded by the crowd. By using the parameters extracted from the photo and metadata, the system will restore the view angle of the photographer and synchronize the view angle of the 3D city model with photos uploaded by the crowd. The system can provide the function of marking in the dual windows. The flowchart of using the system to manage flooding is shown in Figure 6. The steps are explained below. The first step is to collect and use a 3D city model. The 3D city model used in this study is a large-scale urban reconstruction model, which is loaded on the government department website (e.g., TGOS) after confirming that the model is updated. The obtained obj format can be converted to glTF through the program conversion service module and provided to the dualwindow system. The second step is to process the photos uploaded by the crowd. Multiple photos and metadata of the simulated flooding area are collected and uploaded to the photo management system. The photo management system has four functions. First, photos and metadata are synchronously uploaded to the end of the system. Second, the occurrence is taken as a point on the 2D map through Geotag. Third, the photo date label is checked, the correctness of the photo is verified, and a thumbnail image is created and displayed in the photo upload area. Fourth, the uploaded photos and metadata are managed and recorded as historical data in the system. The third step is to integrate photos and metadata into 3D city models and synchronize the dual windows. The three components of this system are shown in Figure 10. The first part is the 2D map that shows the overview of the city. The second part is the photo that will be uploaded by the crowd and will show the current situation of the disaster-struck area. The third part is the 3D city model that can present at the same view angle as the selected photo. Figure 11. Synchronized dual-window system The fourth step is to synchronize the system in dual windows. After the crowd upload photos and metadata, the system will save these data as historical data and pass them to the dualwindow system to achieve synchronization. When we click the photo on the left window, the 3D city model on the right window can be automatically marked to achieve the effect of dual-window interaction. Figure 12. mark co-point in dual window

CONCLUSIONS
3D city models can be established with professional techniques and tools. Meanwhile, photographers need to capture the disaster situation immediately to pass the information of impact or damage to the rescue team. As the awareness of civic participation increases, the photos of disasters provided by the crowd will lead to improved decision making for disaster management operations and point to new directions. The corresponding government departments have many challenges before, during, and after every disaster. Before the disaster occurs, the government departments need to collect spatial data to make appropriate judgments. During the disaster, the government departments need to collect the latest disaster information at any time to facilitate immediate disaster reduction. After the disaster, the government departments need to handle post-disaster reconstruction and asset compensation properly.
Using traditional 2D GIS brings difficulty in assessing financial losses, human injury, and death during disasters. The current information of disasters is also difficult to obtain due to bad weather and terrain. Gathering information in bad weather is also challenging for the safety of workers. The contributions of this research are discussed as follows. First, we describe the key elements and operational methods required for disaster management in a dual-window synchronization system and provide reference for disaster prevention and management unit construction. Second, we reuse the existing 3D city model established by aircraft or satellite to enhance the maximum value of data. Third, we use the system to collect and record historical data. The historical data of the disaster area can be used to find the vulnerability of the region.
Further research will be conducted on the following topics to improve the performance of the dual-window synchronization system. First, sensor chips in smartphones with high stability and accuracy need to be explored in the future given that the parameters of the photo affect the perspective of the 3D city model. Using these chips can reduce the difference in view angle between the photo and the 3D city model. Second, the 3D city model in the study is built by the urban reconstruction method from aerial drone images. The model elevation accuracy is approximately 30 cm, and the plane accuracy is nearly 20 cm. Therefore, a highly elaborate 3D city model will provide a highly accurate model of dual-window system in the future. Third, the specification of Exif added a composite label called CompositeImage to the synthetic image in version 2.32 of May 2019 to distinguish composite photos. Thus, an image can be verified as composite by parsing the label.