HIERARCHICAL DATA MODEL FOR STORAGE AND INDEXING OF MASSIVE STREET VIEW

Abstract. Maintaining an up-to-date inventory of urban infrastructure such as fire hydrant is critical to urban management. Street view database such as Google Street View and Baidu Street View contain street-level images, their potential for urban management has not been fully explored. For the massive image, data model for storage and indexing is an important research issue. Considering multiple cameras and GPS device in the image capturing platform, a hierarchical data model named 3D-Grid is proposed. Massive street view images were stored according to grid ID, GPS time and camera ID. An efficient time indexing algorithm is brought forth to replace the spatial indexing. Real test experiments are conducted in a project, and the validation and feasibility of 3D-Grid including time indexing algorithm were validated.



INTRODUCTION
In recent years, under the rapid development of urban structure and boundary, the level and pace of urbanization has greatly improved. Government has put more attention to the urban infrastructure, such as traffic signs, light poles. Maintaining an up-to-date record of the inventory for urban infrastructure is critical and difficulty. The field survey is the most widely used method to collect exact location of infrastructure for management (Bean et al., 2007;Tao, 2000). This method has workload and safety influence on the project. Some online image services can provide the novel solutions. On-line street-level panoramic images presents a truly massive information of street supporting the exact location extraction, exploring (Creusen et al., 2012;Du et al., 2013). The Google Street View(GSV), Microsoft street side are the most famous dataset (Balali et al., 2015). A research work was published for extracting the locations of traffic signs for the construction of road database inventory from GSV (Yan et al., 2013). It demonstrates the error of extracting location is less than 1.0m by increasing the number of GSV images and GCPs. For urban green space environment, urban greenness was extracted and assessed from profile view of streetscape images by Google Street View (GSV) in conjunction with deep learning techniques (Lu, 2018). This study explored a unique research opportunity arising in a citywide residential reallocation scheme of Hong Kong. Data model for storage and indexing such massive street view image has been an increasing hot research topic. It is not a novel issue about image data model. But, how to organize and retrieve image data from big database has been an important area. For remote sensing image and aerial image, pyramid is the normal algorithm for image data model. Reviewing literatures, many new methods were proposed. Akcay proposed Markov random field method (Akçay and Aksoy, 2013). In this method, the statistical modelling of compound structures enables high-level and largescale retrieval applications. Kang proposed a general framework for semantic classifying the functionality of individual buildings based on Convolutional Neural Networks (CNNs) (Kang et al., 2018). Although it is a good method for generating building classification maps, it only supports the panoramic image instead of the street view images with multiple cameras. For the massive panoramic indexing model, non-consistency (Li et al., 2010) and big indexing file data model (Yang, 2013) are introduced. For the image indexing, spatial-based indexing is widely used, for example R-tree, QUAD-tree. A dynamic pyramid R-tree index for the massive geospatial files management was presented (Cui et al., 2014). This method developed inspiring pyramid spatial index and R-tree index, which can be adjusted with the data distributed dynamically. Particularly, it has higher efficiency than the Hilbert R-tree index and Quad R-tree index in range query. A new full-time index structure referred as GQ-tree was proposed with the one-to-one match between Quad-tree nodes and GeoHash codes (Yu et al., 2018). This method has advantages in index update, spatiotemporal query. A novel image indexing algorithm based on transform concept with time indexing was proposed in this paper. In the algorithm, spatial indexing was converted to time indexing according to the only corresponding relation between images and capture time. A hierarchical data model called 3D-Grid with time indexing was introduced. The validation was conducted in a project with 800GB data size, about 400,000 images. The street view data is collected by Mobile Mapping System (MMS), which is equipped with high resolution cameras and GPS (Global Positioning System). The key benefit of such data is that infrastructure's geographic coordinates can be derived and visualized from street view. The rest of our paper is organized as follows. Section 2 introduces our objectives of the 3D-Grid logical data model. In section 3, we discuss the principle of 3D-Grid model and introduce the image indexing structure. Section 4 concludes that how to design the 3D-Grid model and the implement in the project. Section 5 mainly introduces the main functions of the system, including image browsing and fast positioning and retrieval. Finally, section 6 summarizes the article and proposes that this new method can improve image retrieval better.

OBJECTIVES
The objective of this paper is to provide a hierarchical data model design and implementation that support urban level street view image storage and management. This goal is achieved based on MMS physical components and through using 3D-Grid logical data model. Finally, by designing a platform system. We can store largescale image data in streetscapes, including street information, geographic location information, etc. Also, the platform can improve the image retrieval efficiency, indexing table and index schema are designed and implemented in this paper.

3D-Grid Hierarchical Data Model
In summary, a hierarchical data model named 3D-Grid is proposed in this paper, which is showed in Figure 1. The 3D-Grid model is composed of three axes: X, Y and Z. The X axis represents the time of data acquisition; the Y is the camera ID; the Z axis is the grid ID. In the image capturing of MMS, the locations of trajectory, which is matching to street view images, were decided by GNSS (Global Navigation Satellite System). For the time consistency, GNSS ephemeris time is chosen as the X-value. For the street view, multiangle point views (with multiple cameras in MMS platform for the different point view image) bring the difficulty in street view storage and retrieve. Considering the multi-cameras of MMS platform, Y dimension indicates the arrangement of cameras with some order. For the grid, the means of grid may be different for many applications. For example, urban administration unit grid is used to adopt as Z value in urban infrastructure management. Instead, it may be replaced by street coding number in road infrastructure management. To sum up, the grid ID represents the space division and index.

Street View Image Indexing Structure
Inspiring from the MongoDB geospatial indexing structure -Geohash (MongoDB Inc., 2008), the new indexing algorithm of the street view image is proposed in this paper. Geospatial indexing algorithm was time-consuming and high computation complexity. In the novel indexing algorithm, higher efficiency is achieved through the converting from the geospatial coordinate into time indexing. Since every trajectory point match to only location at one time, the image of street view is the only corresponding. Therefore, the unique corresponding relationship is stored in indexing file. The image indexing has good performance because the two-dimension space indexing reduced to time indexing.

The Design of the 3D-Grid Model
The Mobile Mapping System can acquire image information every 8 meters or every half second, and then it will produce multiple images of different angles. In this way, even if there is only one road, the amount of data of the image reaches GB level. In particular, due to direct sunlight, weather reasons, or data update, it is sometimes necessary to repeat the collection. Thus, for a cityscale data collection, the amount of data collection is quite large, and an effective management mechanism must be established. At present, the images are generally stored in a folder mode. The core of this storage method is how to divide different folders, so that the number of images in each folder is moderate and logical, and it does not affect image retrieval. But these methods have drawbacks. It is impossible to directly determine the specific location relationship, and the space retrieval is very slow. Moreover, due to the non-repeatable time, if multiple data were collected in the same place, It must reconstruct multiple duplicate storage structures. Based on the multi-directionality and multi-period of real-life image acquisition and combined with the requirements of urban management, we created a 3D-Grid model based on urban unit grid, acquisition time and image orientation. Based on this storage structure, a real-time image retrieval mechanism and positioning algorithm based on 3D-Grid model is established to realize fast retrieval and localization of images. The specific construction contents include: (1) Develop grid division and qualification conditions for 3D-Grid storage The image data is divided into three dimensions of spatial sequence, time series and CCD camera sequence, and a three-dimensional grid is established, namely 3D-Grid. The X axis represents the time of data acquisition; the Y is the camera ID; the Z axis is the grid ID.
(2) 3D-Grid storage There are multiple data files in a custom format and each has a unique ID. The purpose of this is to avoid an unrestricted increase of a single data file, and to control its size within a certain range, which not only facilitates data maintenance, but also facilitates the transfer of data. In the real-image database, the real-image data is stored as a file, and the supporting data, such as real-image coordinate data, image index data, image description data, etc., are stored by a combination of a SQL Server database engine and a SDE spatial data engine. The whole can be quickly retrieved by the coordinate space-time conversion algorithm. The image library supports data storage as shown in the figure below. (3) 3D-Grid retrieval According to the real-image storage design, the 3D-Grid storage method is finally divided into 6 levels, from top to bottom: Grid unit, Year, Month, Day, Hour, Camera number, Only measurable multi-angle images are actually stored in the lowest camera number folder. In order to realize the rapid retrieval of massive real-time images, we establish a conversion model from position index to time index, which improves the system retrieval efficiency. For measurable images, we generate a slice data file and an index file in multiple custom formats, which can reduce the space occupied by the file on the disk. Since the original image is stored in multiple folders and occupies a considerable amount of disk space, the occupied space is larger than the actual size of the data. Custom format data files can improve the way of indexing and is convenient for fast retrieval.

Functional Requirement
The system function mainly reflects the current situation of urban operation management component data in the whole district, by collecting real-life image data covering the whole district, and producing related data such as plaques and road traffic facilities. And for large-scale data storage and retrieval, the system uses 3D-Grid model to achieve the purpose of indexing. The requirement mainly includes the following two points: (1) Panoramic image browsing Panoramic image browsing collects continuous panoramic images by using the car panoramic camera, and acquires the coordinate information and the panoramic image synchronously, and finally stores the coordinate information and the panoramic image name in a database. In the system, with the car in the forward direction, the panoramic image also changes, which forming a continuous panorama and enriching the system's browsing affect.
(2) Storage and indexing of massive images Realizing rapid storage and retrieval of geographic location data includes: Attribute query, Spatial query and Location, Attribute query uses point selection query, Spatial query is divided into polygon query and general query (displaying attributes), Location is divided into coordinate positioning, plaque positioning, road name positioning, street light positioning and road positioning.

User Requirement
For the massive image, data model for storage and indexing is an important research issue. Considering multiple cameras and GPS device in the image capturing platform, a hierarchical data model named 3D-Grid is proposed. Using the new method, we can quickly retrieve and locate images. The real-life image management platform provides users with the demand for street view imagery. By generating a panoramic image for the key roads, users can browse and query panoramas when it is necessary. At the same time, the new method can realize the storage and retrieval of large-scale streetscape images, and users can quickly locate them according to streets, courtyards, and points of interest.

EXPERIMENTS AND RESULT
The platform function implementation considers operational efficiency, especially for massive images and multi-concurrency operating efficiencies. At the same time, the platform considers the stability of the system and can easily connect other existing systems and newly developed systems. It can be well integrated with these systems to achieve data sharing, and can adapt to the needs of development and meet different needs.

Image Display
The Feature of browsing captures panoramic images with a car panoramic camera, and the coordinate information is acquired with the panoramic image, and finally the coordinate information is stored in the database in association with the panoramic image name. In the system, as the car moves forward, the panoramic image also changes, forming a continuous panoramic browsing purpose.
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W13, 2019 ISPRS Geospatial Week 2019, 10-14 June 2019, Enschede, The Netherlands Figure 5. Image browsing and playback The platform also includes additional real-life images: plaque orthophotos and columnar panoramas. The plaque orthophoto is a picture taken from the plaques on both sides of the road. Its main function is to let the managers grasp the status quo and changes of the plaque. The columnar panorama is a real-life image taken for a major event venue, and is designed to assist managers in developing a timely response in the event of an emergency. When the image panel is displayed as a live image, three images are displayed in the panel, which are three images captured by the left, middle, and right CCD cameras for browsing. The panoramic image is associated with the city component: when the user clicks on the city component, the closest panoramic image of the component is searched and the panoramic image is called out.

CONCLUSION
The validity and feasibility of the proposed hierarchical data model are validated through conducting experiments in the urban street view of Xicheng district, Beijing city. The database includes about 800GB, more than 400,000 images. The validation was conducted from two points: one is the time-consuming compare between geospatial indexing and time indexing; another is the efficiency validation with the increasing of street view data size. First of all, the time indexing is two or three times faster than geospatial indexing. Secondly, the time-consuming in geospatial indexing is increasing with the index grade when the data size increased. However, for the sake of the unique corresponding relation between image and GNSS time, the performance of time indexing is very well with the increasing of street view data size.