CREATION OF HIGH DEFINITION MAP FOR AUTONOMOUS DRIVING

High Definition Map (HD Map) is one of the essential components for autonomous driving systems. The HD Map helps localization, detection, prediction and planning of autonomous driving cars in combination with GNSS and IMU sensors. A lot of map providers and major car manufactures have been trying the development of efficient ways to create and update HD Maps with centimeter-level precision. However, there are no open standard ways well-established to produce HD Map at the present time. Since 2013, we have been being engaged in the research projects and actual production projects on the creation of HD Maps. This paper introduces practical knowledge on five key steps of our large-scale production lines for HD Map creation, which are 1) Mobile Mapping System (MMS) survey, 2) 3D plotting, 3) Feature attributes editing and road network topology building, 4) Format conversion and 5) Quality assurance.


INTRODUCTION
Autonomous driving system has been attracting a lot of attention from around the world as one of the fields of strategies of economic growth these days. In Japan, "Public-Private ITS Initiative/Roadmaps" were developed in June 2014. The main objective of the Initiative/Roadmaps is to build and maintain the world's best Intelligent Transport Systems (ITS) and thereby contribute to its people and the world. The Initiative/Roadmaps shows concrete actions to deal with issues with respect to this field by the unified public and private sector initiatives. Japan has developed and revised Public-Private ITS Initiative/Roadmaps five times and will continue to pursue the objectives. The autonomous driving system realizes autonomous mobile in combination with recognition, judgement and operation. The recognition is implemented by sensors equipped on the autonomous driving car, which are Light Detection and Ranging (LiDAR), radar, camera and GNSS/ IMU. Those sensors measure the inter-vehicular distance, and collect the situation of pedestrians and traffic information in the surrounding areas. The judgement including detection, prediction and planning needs the self-localization based on the sensors embedded on the car and HD Map, which is one of the key information to decide the next behaviour of the autonomous driving car. Then the operation will be executed based on the recognition and the judgement. The HD Map helps localization, detection, prediction and planning of autonomous driving cars in combination with GNSS and IMU sensors. A lot of map providers and major car manufactures have been trying the development of efficient ways to create and update HD Maps with centimeter-level precision. However, there are no open standard ways well-established to produce HD Map at the present time.
In this paper, we will introduce our practical knowledge on five key steps of our large-scale production lines for HD Map creation based on our experience. As Table 1 shows, the five key steps are 1) Mobile Mapping System (MMS) survey, 2) 3D plotting, 3) Feature attributes editing and road network topology building, 4) Format conversion and 5) Quality assurance.

THE OUTLINE OF HD MAP
The information required for "recognition" of an automated driving system consists of four geographical information layers called as Dynamic Map (Table 2): dynamic information, semidynamic information, semi-static information, and static information. HD MAP is corresponding to the static information part. Dynamic information, semi-dynamic information, and semi-static information are information that changes depending on the surrounding traffic conditions, weather conditions, construction schedule information, etc., but road surface information, lane information, road structures with 3D coordinates are the map that summarizes information that does not change from time to time.
Specifically, the extracted features include line objects such as road edges, lane markings, and stop lines, and point objects such as road signs and traffic lights, and area objects such as intersections and pedestrian crossings. In addition to acquiring the lane center line that represents the route that the car runs on with virtual lines, not only the three-dimensional coordinates but also the association between attributes and objects should be maintained. Figure 1 shows 3D view of HD Map.

THE HD MAP CREATION WORKFLOW
The workflow of HDMAP production starts from the Mobile Mapping System (MMS) survey, and performs 3D plotting based on the 3D point cloud data acquired by the MMS. Then, the future attributes are added, the road topology is constructed, and the road topology is converted into the format required for the automatic driving system. Finally, quality assurance is performed to reduce the errors to zero. The standard procedure for creating an HDMAP is described in the next chapter.

MMS survey
First, 3D point cloud data is captured by MMS equipped with sensors such as LiDAR, all-round camera, stereo camera, and IMU. The sensor specifications are shown in Table 3, 4 and 5.   The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B4-2020, 2020 XXIV ISPRS Congress (2020 edition)

Figure 3. Data captured by MMS
At the data acquisition stage, the measurement speed and the shooting speed changes depending on the general road and the highway. In the case of general roads, the vehicle runs at a maximum speed of 50 km at a legal speed of 60 km and takes 360-degree images in an interval of every 2 m. On expressways, the vehicle travels at a legal speed of 80 km, since the measurement speed is high, so the 360-degre images are taken in an interval of every 5 m. For the case of LiDAR survey, the ground control points for the 3D point cloud data are located at 2 or 4 points within 1 km along the road, and the accuracy of the ground control points can be set to less than 10 cm, 20cm or 30cm in both horizontal position and elevation, all those specification are on demands of clients.
According to the criteria of the manual of public survey in Japan, the minimum point density is defined as "when using threedimensional point cloud data for large-scale map (scale 1:500), 400 points / m2 or more are required" (Ministry of Land, Infrastructure, Transport and Tourism, 2016). On the other hand, we define our criteria for HD Map creation at 1,000 points / m 2 or more in order to obtain more accurate data in the process of 3D plotting. When the point density is high, the visibility is improved, and it becomes easier to extract the geometry shapes such as the edges at the phase of plotting, and the position accuracy is also high. Therefore, the point density is preferably 1,000 points / m 2 or more. Figure 4 shows the difference in point density for case of 1 m 2 . It shows the concrete and the white line of the road typically.
Comparing the appearance of the white line at a density of 1,000 points/ m 2 and 400 / m 2 , the white line can be recognized at 1,000 / m 2 , but it is difficult to recognize the white lines at a density of roughly at 400poitns / m 2 . The low point density and coarseness are consider as the fact that the laser irradiation was insufficient due to the distance from the MMS at the time of measurement, or it was shielded by people or cars. Therefore, it is very important to make a well-planned survey to measure in a time period with less people or a better route with less noise, or in the case of a highway motorway, make a round trip measurement of the up lane and the down lane so that the laser reaches the edge of the road. In this way, the point density can be assured to meet the needs of clients. 3.2.1 3D plotting method: Three-dimensional plotting is executed in the flowing steps: interpretation, plotting, and structuring. Firstly, geographic features on the road are identified using the reflection intensity and height of the 3D point cloud data acquired by MMS. If the reflection intensity is low and the feature cannot be extracted, or if the shape is similar and the type cannot be determined only by the point cloud, we can use a 360degree camera image taken by an all-round camera as a supplementary judgement material for the operators. Next, all the road features are captured as lines, polygons, and points. The height must be acquired not only if the horizontal position is correct, but also the vertical position is accurate when viewed from the top direction. Finally, the data structuring will be carried out based on the specifications and the actual road structure. Skill level is required because it is difficult if operators involved do not fully understand the specifications. In the case of large-scale production, all the plotting works must be shared and cooperated by all the CAD operators that are classified in different groups depending on their 3D extraction skills and experiences. The brief 3D plotting workflow is shown in Figure 5.  (Figure 6). At this time, a curve or the lines may be obscured at the edge of the road, and thus can be obtained as an offset from the outside line of the roadway. Next, the lane center line is acquired with an offset from the acquired outside road line and the acquired road center line. The lane center line is not a real feature, but those virtual lines are necessary for maintaining the position where the vehicle will run along with. Finally, the features such as stop lines, intersections and pedestrian crossings are extracted accordingly. We usually use aerial photos captured by UAV as supplementary material to grasp the whole target area (Figure 7). The example of feature extraction is shown in Figure 8. The lines to be virtually maintained are not limited to the center lines of the lane, and the special code numbers are set and maintained for the cases of road edges if there are parts of them outside the roads or the road edges do not end at the boundary of the maintenance range of roads. White lines that are outside roads are needed in order to determine the traveling position of the car, but there are many exceptions for the cases when the white lines disappear on the actual roads due to aging etc., white lines are not extracted for those cases. For some cases, although white lines do not exist on the road, if the white lines are specially required by the clients, we set up the special code numbers for those kinds of white lines for easy management (Figure 9). Since there are many types of line objects, when a line is drawn, it is automatically drawn by using a tool that gives the line itself an attribute simultaneously, so that the miss-assignment for attributes are reduced. Also, in the case of large-scale production, depending on the skill levels of the operators, the plotting works are shared between the operators in charge of inputting the line objects and the operators in charge of structuring the input feature to ensure the data quality. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B4-2020, 2020 XXIV ISPRS Congress (2020 edition)

Acquisition of point features by AI extraction:
Pole objects such as road signs and traffic lights are captured as point objects. Attributes such as the contents of road signs and the colours of traffic lights are added to the objects extracted. The types and positions of the road signs are extracted by AI. All the road signs in Japan are broadly divided into four categories: guiding signs, warning signs, regulatory signs, and instruction signs, and totally, there are more than 100 types of signs. Previously, in this sign acquisition process, the all-round camera images were all visually checked to acquire the sign positions and the type attributes. Since the manual acquisition efficiency was not good, in order to improve work efficiency, we switch the manual extraction of road signs to AI method to automate the feature extraction process. Firstly, in order to identify the road sign types, we apply AI tools to automate the recognition from the all-round camera image. In addition to the actual image, the training data is constructed using a pseudo-composite image that are based on a sign template, the constructed data for the road signs to be trained are without biases. The recognition rate for road signs with high frequency exceeds 90% (Lin et al., 2018). In an acquisition framework, when a road sign appears on the all-round camera image, firstly, the approximate position of the road sign is acquired. Then, the AI tool recognizes the road signs, presents the highly probable road sign candidates in a ranking list, and selects the corresponding type from the ranked candidates list ( Figure 10). The approximate position of the road signs acquired from the all-round camera image cannot meet the required specifications of the clients, so we created a workflow of correcting approximate position to a more accurate position using a 3D point cloud as the automatic sign extraction method ( Figure 11). In the conventional method of visually checking all all-round camera images, we can only check about 1,800 images per day, but it is possible to check about 3,500 images per day by applying the ranking list method recognized and recommended by AI. In the case of the road signs identified on a road section of about 44 km in length with 87,350 all-round camera images, it took 49 days by the previous method, but it is possible to acquire the road signs in 25 days using ranking list method, it is possible to shorten the work period in half (Table 6 and Figure 12).  There is a tendency for road signs installed by roads, and there are many types of instruction signs and guiding signs such as "stop", "pause", and "no parking" on general roads with high frequency. It is a problem that there is a difference in the quantity of training data between signs with high appearance frequency and signs with low appearance frequency. On the other hand, there are signs for restrictions such as "maximum speed" and "height restriction" on highways, and the number of road signs installed tends to be fewer than that on general roads. This is because the road structure is different between a general road and a highway. Since there are many intersections on general roads, there are many road signs. On the other hand, the expressway is used only by vehicles, and there are many lanes and there are also road sections where the slopes are steep, so there are many speed limit signs and regulatory signs, but the number of signs is fewer than general roads. Due to the difference in the appearance of the road signs and the tendency of the installation situation, it is considered that there are fewer numbers of sign types on highways, and the probability of erroneous recognition is low, so AI method is very effective for the case of expressway and highway. Also, on general roads, there is a problem that it is difficult to construct training data because there are many types of signs, we are working on a solution that creates a pseudo-sign image with a higher recognition rate using an illustration template and combine them together as the training data. We have been applying this method for the HD Map creation in Taiwan.

Feature attributes editing and road network topology building
Since the definition of feature attributes differs depending on the requirements and some special needs in each country, we have developed a rule-based framework for HD MAP creation that can be easily customized. In addition, for large-scale production at the national level, it is necessary to consider the size of the dataset in each road section in order to create a seamless road network using the topology.
Although the error checking is conducted after the plotting process, the error checking is also executed in the steps of the data structuring process. Final checking of connection between objects, miss-assignment of attributes and topology structure must be executed. All those checks require the total understanding of the road structure and specifications, so checks in each step have a great impact on the total quality assurance.
The topology creation is the essential part for building the seamless road network in the HD Map creation. We add different rules for different layers depending on the spatial relationships that are required and defined by the clients for monitoring the road layers that participate in the topology. All those rules are realised by the automatic Python tools created based on ArcGIS soft package from ESRI (Environment System Research Institute, CA, USA). The topology rule can be very specific, for example, the traffic lights and stop lines must share the same ID of traffic lights. It is impossible to assure the quality of HD Map without those kinds of topology rules built on the fly.

Format conversion
We designed and developed our own intermediate exchange format and conversion tools for creating various HD Maps defined by different clients. The formats are designed as ASCII based text, since they are easy to read, write, check and exchange with other formats of HD Maps defined by different clients and organizations worldwide. All the layers are finally exported from ArcGIS shape files to csv files defined by clients for the case of Autoware (Nagoya University, 2016) clients. The reason why csv files are used is that csv files are easy for understanding and checking. Since OpenDrive's data format (VIRES Simulationstechnologie GmbH, 2019) is xml, we create a special tool to convert csv files from shape files to xml. We can also provide other formats defined by clients for some special purposes and cases.

Quality assurance
Quality evaluation is carried out in accordance with ISO 19157 (Geographic information -Data quality). This international standard stipulates aspects of the quality of geographic data and fundamental framework of data quality evaluation processes. The acceptance quality limit (AQL) for the absolute positional accuracy is 25 centimetres of root mean squared error (RMSE). The AQL for the relative positional accuracy is 15 centimetres of RMSE.
The following are the summary of quality control and management in each data processing steps. In the MMS survey, in order to obtain more accurate data, we conduct the round-trip surveys on the target road and create a quality control table to manage the point cloud data. If the accuracy is not sufficient, we will re-do the survey. In 3D plotting, human errors are reduced by automating line attribute assignment and errors checking process by providing the customized ArcGIS tools. Quality is maintained by leaving a review log of error checking results and a revision history so that we can reduce and minimize the reworks.
Here is the brief summary for quality management and assurance in each steps in the HD Map creation. We provide the detailed checking steps and tools needed to check the data in each process, since the human errors are not inevitable in the steps of digital plotting and attribute assignment. All the checking tools must be applied to each step to assure the quality required by the clients, for example, the profile slope, lateral slope much be in a proper range defined by the road construction laws, the colours, and types of traffic lights must be the types allowed by the traffic regulation rules and laws, etc. The tools should be provided in an easy-to-use way in order to indicate where and what the errors are, for example, we provide the tools to check if the lines of lane centrelines are in the right direction and perfectly connected without any dangles and mismatch in the x, y, z coordinates.

CONCLUSION
Our HD Maps are created by the system that starts from MMS survey, acquisition of road features, topology creation and data format conversion. We have applied our production lines to large-scale production on HD Maps in Japan and Taiwan. Our large-scale production lines and systems have been highly evaluated by our clients in terms of efficiency, productivity, and quality.