Generation of effective orthophotos for road surfaces using MMS

In recent years, high precision and high resolution road surface orthophotos have been generated using video cameras mounted on surveying vehicles. However, there is a serious issue in generating an orthophoto from this image. The shadows of the surrounding structures and vehicles on the road surface cause a lack of information and decrease in visibility. Therefore, the shadows should be removed from the images for exact road management. On the other hand, the Mobile Mapping System with a laser scanner mounted on vehicles has been receiving more attention because the laser scanner intensity is almost unaffected by shadows. This paper presents shadow extraction and shadow correction for generating road surface orthophotos using the laser scanner intensity.


INTRODUCTION
The proper maintenance of social infrastructure, such as roads, is a public project indispensable to safety and security.Therefore, the importance of the current state and maintenance of roads has been recognized.High precision and high resolution road surface orthophotos have been generated using video cameras mounted on surveying vehicles.However, the shadows of roadside trees and buildings exist on the road surfaces in urban areas.These shadows cause a lack of information and decrease in visibility.Therefore, they must be removed from the images for exact road management.By the way, the Mobile Mapping System that uses a laser scanner mounted on vehicles has been receiving more attention because the laser scanner intensity is almost unaffected by shadows.This paper confirms the characteristics of the laser scanner intensity, shadow extraction, and shadow removal for generating road surface orthophotos using the laser scanner intensity.

CURRENT ISSUES AND LASER INTENSITY
The current road surface orthophotos have decreased visibility from shadows on the road surface.Figure 1 shows a road surface orthophoto with a 2.5 cm ground sampling distance.The shadow of the telephone pole (sample A) and the shadow of a building (sample B) can be confirmed on the road surface.Furthermore, a manhole cover exists at the position indicated by the squares in sample B, but its visibility is reduced by the shadow.Thus, a current problem in road surface orthophotos is the shadows generated on the road surface affecting its quality.On the other hand, the reflection intensity of laser data (i.e., reflection intensity value), the influence of sunlight causing the shadow is very small.Furthermore, the reflection intensity value tends to exhibit a reflection characteristic of the feature further.Table 1 shows the features of a typical road surface and materials.The reflection intensity value is changed by the material, For example, the reflection intensity value of road markings with a high reflectance compared to the road surface is strong, whereas that of the manhole cover is weak.This implies that the difference in material is enhanced by the laser data reflection intensity image (i.e., reflection intensity image) represented by the grey scale image of the reflection intensity value.If this characteristic is used, manhole covers (Figure 1, sample B) can be clearly deciphered (Figure 2).Incidentally, Figure 2 is an 8-bit image of reflection intensity values as described in Section 4.1.4.

Laser scanner
The instruments used in this study are shown in Table 2.The sensor mounted on survey vehicle B shows a higher performance than that on survey vehicle A, and the point density in the traveling direction of the laser scanner is the same ground sampling distance (2.5 cm) as the road surface orthophoto.Figure 3 shows the point density of the data acquired by the laser scanners of the survey vehicles, and it is understood that the survey vehicle B acquired the high density point data.The difference in point density is inferred to affect shadow correction in the road surface orthophoto.
In this study, in addition to the effectiveness of the reflection intensity value for shadow correction, the effect of the difference between the point densities on results is also considered.Sample A is obtained by survey vehicle A and sample B is obtained by survey vehicle B. .

Video camera
In order to obtain a high resolution image of the road surface, a video camera was set up at an angle horizontal to the direction of the road.Figure 4(a) shows the original image recorded by a video camera in the survey vehicle.Figure 4(b) shows the range of resolution that satisfies 2.5 cm to generate orthophotos from the original image.Effects from other vehicles in the surveying area are possible to avoid by the captured image of the adjacent lane or by ensuring appropriate inter-vehicle distance.However, as shown in Figure 5, avoiding the influence of parked vehicles is difficult, and the processing for parked vehicle was not assumed in this study.This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XL-5-621-2014

THE PROPOSED APPROACH
The proposed approach of shadow removal for the road surface orthophotos in this study consisted of generating an reflection intensity image with a 2.5 cm ground sampling distance corresponding to road surface orthophotos from laser data and extracting the shadows using the property differences between the reflection intensity image and road surface orthophoto.A correction of the shadows was processed after the extraction of shadows.The proposed approach is shown in Figure 6.

Generating an reflection intensity image
The reflection intensity image is generated by a road surface orthophoto with the shadow part and laser data from the random point group.Details of the process are given in the following section.

Extraction of the road surface area
The extraction of the road surface area is a process of excluding the laser data for the out of road area to speed up the process in this study.Because the scanning range of the laser scanner is wider than the imaging range of the video camera, the laser data acquired is present in a wide range of the road surface orthophoto.Therefore, the XY coordinates calculated from the pixel coordinates of the four corners of the road surface orthophoto are used to extract the laser data for the road surface.

Grid of reflection intensity value
In order to compare the discrete laser data with the road surface orthophoto, the laser data is interpolated to the same ground sampling distance as the road surface orthophoto in a grid of the reflection intensity value.As shown in Figure 7, the laser data around the target pixel is sequentially searched until the number of search points set in advance is reached, and the reflection intensity value of the target pixel is calculated by equation ( 1) with consideration for the distance ratio.Figure 8 shows the laser data before gridding and the reflection intensity image created by gridding.
Because the reflection intensity value to be interpolated will change on the basis of the search range and count of the search points, it is necessary to consider an appropriate number for the search count during gridding.Meanwhile, as shown in Table 2, the laser data average points per pixel of sample A was 0.34 points and 6.25 points for sample B. That is, if the count of search points is set to six points, sample A requires 18 pixels on average to reach the count of search points.However, Sample B will reach the count within the target pixels.Therefore, the reflection intensity image of Sample B is expected to be close to the visibility of a road surface orthophoto.Therefore, the gridding of the reflection intensity value is processed by six points, the count of search points, in this study; and it is also confirmed the effect on the results of the point density differences of sample A • B. The reflection intensity value of each grid (IR) is calculated by equation (2).
(2) dis i ：Distance to laser data around target pixel p i ：Distance weighting factor of laser data Ir i :Intensity value of laser data Therefore, it was determined that most of the non-shadow part of the road surface is included in the 2-3 class, and the average brightness of the 2-3 classes is considered the average brightness of the road surface orthophoto.As described in the next section, the shadow part of the road surface orthophoto is extracted by comparing the average brightness of the road surface orthophoto and the reflection intensity value after conversion to 8-bit.

Conversion to an 8-bit reflection intensity value
The conversion to an 8-bit reflection intensity value is the concentration change process governing the range of 0-255 (8-bit) reflection intensity value.This process is performed to allow a comparison of the brightness value because the range of reflection intensity values is different in survey vehicle A and B, as shown in Table 2. Figure 10 shows the distribution of the gridded reflection intensity value in Section 4.1.2.The horizontal axis is the reflection intensity value, and the vertical axis represents the number of pixels.
The reflection intensity value outlying significantly from the peak of distribution is estimated to be noise.Thus, the range in which the number of pixels is less than the size of the road feature (about 60 pixels: small manhole cover) is excluded.The lower, upper, and average values of the intensity are calculated in this manner.Thereafter, the reflection intensity value is converted to 8-bit as the average brightness is the average value of the road surface orthophoto, the upper limit is 255 and the lower limit is 1.In addition, the reflection intensity value is consolidated with a lower limit of 0 and upper limit value of 255.As shown in figure 11

Comparison and correction processing
The comparison and correction process compares the characteristics of the shadows of the road surface orthophoto and the reflection intensity image after extracting the shadows.The specific process is described in the next section.

Extraction of shadows
The extraction of shadows is a process of comparing each pixel of 8-bit reflection intensity value and the brightness value of the road surface orthophoto, and the shadow pixel of the road surface orthophoto is extracted when the brightness value is lower than the reflection intensity value.Figure 12 is an image colored in red for the extracted shadows.This image is represented by subtracting the magnitude of the difference between the reflection intensity value and the brightness value.The figure shows that the difference value is a larger dark red.The negative difference value is a correction for the brightness of shadows (i.e., brightness correction value).However, if the same brightness correction value is added to the pixel value of the RGB color space (i.e., RGB value), the bluish pixels are seen in the shadow part as shown in Figure 13.Therefore, it is necessary to consider the color tone in the correction of shadows.

Calculation of color correction coefficient
For correction of the shadows in consideration of the color tone, identifying trends in each of the RGB values for the shadow and non-shadow part was attempted.Figure 14 shows a distribution of the RGB value for shadow and nonshadow parts.The horizontal axis represents the RGB value, and the vertical axis represents the number of pixels.Table 3 shows a comparison of the average RGB value for shadow and non-shadow parts (i.e., average RGB value).It can be read from Table 3 that the pixel value R is strong than B in non-shadow areas, and the pixel value B is equal or strong than R in the shadow area.Therefore, the strength of the RGB value is determined to be different in the shadow and non-shadow parts.Therefore, to correct the RGB value for shadows in the road surface orthophoto, a color correction coefficient must be used.After calculating the difference between the average RGB values for the shadow and non-shadow part using equation (3), the color correction coefficient is calculated for each RGB value with equation (4).

Calculation of brightness correction value
The brightness correction value of shadows using a color correction coefficient (i.e., RGB correction value) is calculated by equation ( 5) per RGB value for each pixel extracted as the shadow part.

Replacement of brightness
The replacement of brightness is a process of replacing the reflection intensity value for the brightness value using equation ( 6) for the pixels of the non-shadow part.The image of combined RGB values for the road surface orthophoto and reflection intensity value of the reflection intensity image (i.e., the correction result image) is generated by this process after the correction process by the RGB correction value for the pixels in shadows.

RESULT AND CONSIDERATION
Figure 15 shows the results of correcting the road surface orthophoto of Figure 1 using the proposed approach.The shadows of the buildings and telegraph poles presented in Figure 1 are corrected in Figure 15.In addition, the manhole cover in a box is clearly interpreted.Further, it is understood that the difference between the color tones of the shadow and non-shadow parts, which was confirmed in Figure 13, is correct in Figure 15.Therefore, it is confirmed that the shadow correction of the proposed approach is effective for sample A and B irrespective of any laser point density difference.
Furthermore, Figure 16 shows the visibility of the road surface orthophoto and image correction result.Sample A confirmed a decrease in the readability of the feature shape from the roughness of the point density, but sample B did not have a decrease in readability.Furthermore, Figure 17 shows a comparison visibility of the image edge.Because the shape of feature is reproduced faithfully in the result image, it was confirmed that the shape distortion generated in the image edge part of the road surface orthophoto is also corrected by the reflection intensity image.
This study assumes that the average brightness of the road surface orthophoto and reflection intensity values are matched when generating a reflection intensity image.However, considering the features of different materials on the road surface, the effects of differences in material on the reflection intensity images were again confirmed.Figure 18 shows the distribution of the road shoulder and road surface.The vertical axis is the number of pixels, and the horizontal axis is the brightness value and reflection intensity value.For both values, the road shoulder is greater than the road surface.Furthermore, Table 4 shows the results of the brightness and reflection intensity value calculated from the ratio of the average value of the shoulder relative to the This contribution has been peer-reviewed.doi:10.5194/isprsarchives-XL-5-621-2014average value of the road surface.The brightness value of non-shadow part is 142%, the brightness value of shadow part is 137%, and the reflection intensity value is 135%, it is confirmed as the ratio is close to both.Furthermore, it was interpreted that the brightness value and reflection intensity value shown have the same tendency for the manhole cover and road surface markings (Figure 19).From these results, the applicability of the reflection intensity value in the shadow correction of the road surface orthophoto was confirmed based on the comparison of materials.

CONCLUSION
In this paper, the shadow correction of road surface orthophotos using the characteristics of laser data intensity was considered.As a result, the proposed approach made it possible to extract and correct the shadows on the road surface of the orthophoto using the laser data reflection intensity image.As shown in sample B, if the point density and ground sampling distance of the road surface orthophoto are almost equal, the visibility of features in the shadows are improved by the proposed method.Furthermore, it was confirmed that improvement in the image quality at the image edge part can also be obtained as a by-product.On the other hand, even if there is a big difference in the point density of the laser data and ground sampling distance of the road surface orthophoto, shadow correction is still effective, but there was some effect on the interpretation.This problem is expected to be solved by improving the point density of laser scanners, but the decline in interpretation is still an issue in the future.
The proposed approach has little influence on measurement conditions such as survey time and weather, and it can retain color information from the road surface orthophoto.Therefore, this method would be effective for creating high quality road surface orthophotos including important road marking color information.

FigureFigure 5 .
Figure 3. Point density of laser data

Figure
Figure 15.Correction result image Figure 19.Comparison of intensity by material

Table 1 .
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XL-5, 2014 ISPRS Technical Commission V Symposium, 23 -25 June 2014, Riva del Garda, Italy Features of road surface

Table 4 .
Average Ratio