RAY-BASED DETECTION OF OPENINGS IN URBAN AREAS USING MOBILE LIDAR DATA

The detection of openings like windows or doors is of great interest in the field of urban modeling. Mobile LIDAR data provides valuable 3D information for that purpose. There are generally two main problems: The estimation of wall’s surface, and the distinction between opening areas and occluded areas. Indeed, openings may be visible or occluded with regard to the sensor. The method presented in this paper focuses on the detection of visible openings using intersections between laser rays and walls. In particular, it shows that detection of visible openings can be reduced to a single distance threshold once the surface of the wall is computed. Thus all the complexity is actually in the estimation of the wall’s surface. The opening contours are then obtained by clustering the visible opening points and fitting them with rectangles. The main advantage of ray-based detection is its robustness to occlusions. This method requires the LIDAR sensor positions and angles for every laser point. Results are evaluated quantitatively on two datasets with ground truth. Qualitative results on larger datasets are also given. The results show good precision. The recall (or completeness) depends on the number of occluded openings.


INTRODUCTION
One activity of Siradel is to produce 3D city models either for 3D simulation of physical phenomenon (e.g.radio propagation for network optimization) or 3D visualization (e.g.urban planning).M ore than 400 urban units have been produced worldwide (more than 100 in France).M odels can be produced with different level of details (LOD) defined in the cityGM L model.LOD1: The buildings are modeled as flat-roofed blocks.LOD2: The roof structures are modeled.LOD3: The building's superstructures are modeled: balconies, chimneys, openings.
The word "opening" in this paper corresponds to a perforation in the wall.It includes windows, doors, loggias, shop windows, and also windows with closed shutters or grates.The detection of openings is useful for many applications: refinement of indoor or outdoor radio propagation estimations; improvement of the realism for 3D visualization; interactions with the buildings for video games or intervention simulation (entering into a building, breaking a window …).
Several data can be used for automatic detection of openings such as images or LIDAR data, with either terrestrial or aerial sensors.Both images and LIDAR are actually complementary.LIDAR data enables detecting transparent openings and intrusions in the walls.On the other hand, images enable detecting openings by their appearance, including also closed shutters that are not detected using LIDAR data.In this paper, only LIDAR data is used.It is acquired with mobile LIDAR sensors mounted on a vehicle.
Two main issues are encountered when detecting openings with LIDAR data: The estimation of the wall's surface, and the distinction between opening areas and occluded areas.Indeed, when a building is scanned, laser points that reach a wall give 3D information about its surface.However, some laser points don't reach the wall either because they have been occluded by an object or because they passed through an opening.Transparent or empty openings like windows and loggias don't diffract laser rays.In addition, some openings like doors are not transparent but they are often intruded in the wall, e.g.25cm behind, and are therefore not in the wall's plane.On the opposite, occluding objects diffract laser rays before they reach the wall resulting in holes.Therefore, additional analysis of the data is needed to separate opening holes and occlusion holes.
Clutters and occlusions occur frequently in natural environments.Openings may be visible or occluded with regard to the sensor.Some methods enable estimating occluded openings using statistics and architectural constraints.On the opposite, the method presented in this paper focuses on the detection of visible openings.In particular, it shows that the detection of visible openings can be reduced to a single distance threshold once the surface of the wall is computed.

RELATED WORK
The domain of urban modelling is widely studied.The survey (M usialski et al., 2013) covers this domain and includes the subject of facade parsing and openings detection.This section focuses on openings detection methods using LIDAR data.Two types of methods are presented: methods that study the shape of holes in the walls (shape-based methods) and methods that study the intersections of laser rays with the wall (ray -based methods).

S hape-based methods
M any existing methods for detecting openings with LIDAR data consist in finding holes in the wall and deciding whether they are due to an opening or an occlusion by studying their shapes and repetitions.(Becker and Haala, 2007) sample the facade plane into a 2D binary image with empty or non-empty pixels depending on the presence of a laser point.The empty pixels are used to detect openings.The problem of false opening detection due to occlusion holes is avoided by using multiple view-points of the same wall.Additional constraints are added to obtain a wellstructured result.First, the façade is decomposed into 2D cells using opening borders.To do so, the opening borders are detected with laser points that have no neighbour only on oneside (half-disc method).Then horizontal and vertical lines are drawn resulting into 2D cell decomposition.Cells are classified as wall or opening according to the ratio of empty VS nonempty pixels.Uncertain cells are further processed using constraints about neighbourhood relationships.(Pu and Vosselman, 2007) triangulate wall segments into a TIN mesh model.Then, hole boundaries are detected with the long edges of the mesh and boundary points are clustered together to obtain a closed contour per opening.Doors and small extrusions can be detected because they form plane segments near the main wall plane, thus they are separated from windows.Finally, in order to filter wrong windows detected in holes formed by occluding objects, windows with irregular shapes are removed.(Wang et al., 2011) approximate the major plane in facades with RANSAC algorithm.This algorithm is run with points selected according to their normal directions.Next, potential opening border points are detected with half-disk method, and crossbars are removed if no neighbours are found on both sides of a point (left and right or up and down).Then, potential points are accumulated into vertical and horizontal histograms, and local peaks in the histograms give the openings localizations.Thus occlusion holes are filtered with the assumptions of vertical and horizontal alignments of multiple openings.In the last step, architectural rules like window size and spacing are computed and form a constraint to help identify the windows boundaries more robustly.The authors p recise that 10% to 30% of the bottom of the facade is ignored because the first floor is often different than the next floors.
(M esolongitis and Stamos, 2012) estimate the major plane of the facade with a plane fitting method.This major plane is used to classify all points as behind the plane, on the plane, or in front of the plane.Then a point occupancy binary map of the wall is created.Empty areas in the plane give an initial estimate of windows.They are filtered by width and area in order to remove small noise or occlusion holes.Lastly, the 2D windows centres are iteratively refined using a local lattice fitting and lattice voting scheme.This last step is based on the assumption that the windows are arranged in multiple unknown periodic structures.
(Truong-Hong and Laefer, 2014) use vertical plane fitting to approximate the facade.The points on the boundaries of a hole are used to detect openings.Then, the shapes of the openings are studied using vertical/horizontal histograms of boundary points.False openings due to occlusion holes are filtered depending on their shapes (height, width and height/width).(Aijazi et al., 2014) project laser points onto the facade plane and sample the plane into a 2D image.Then openings are detected using empty pixels in the image.They filter small and large openings to remove noise and occlusion holes.In order to reduce occlusions and find more openings, multiple passages are performed and temporal correspondences are exploited as well as symmetrical correspondences.
Shape-based methods require dedicated filtering to remove occlusion holes.Tuning the filter parameters is a complex task.Occlusions and openings may have the same shape, involving false positives.They also may overlap such that only one hole with irregular shape appears, involving false negative.It is particularly true on the ground floor where many objects might lie between the wall and the facade.It might also be the case in the upper floors where trees, balconies or others may create occlusions.M ethods that assume repetitions of openings are mostly restricted to high buildings with sufficient repetitions, and cannot handle the ground floor which is usually different than the others.
Shape-based methods focus on the modeling of opening structures using empirical parameters or statistics.They may be adapted to the estimation of occluded openings.In the following section, ray-based methods are presented as an easy technique for the detection of visible openings.

Ray-based methods
Ray-based methods study the intersections of laser rays with walls to detect openings.(Tuttas and Stilla, 2011) use aerial LIDAR sensor mounted on a helicopter.They select indoor points that lie behind the facade's wall.These points are projected into the façade plane regarding the incidence angle of the laser.This method has been developed for aerial laser scanning with low and variable point density, therefore a test of repetition of the openings is added to detect more openings.(Xiong et al., 2013) detect openings in highly cluttered indoor environments using static LIDAR sensor.The method mixes shape-based and ray-based opening detection.First, opening hypotheses are created from depth edges extracted from a range image of the wall.The range image is created by projecting points within a given distance to the surface.In addition, an occupancy image of the wall is created with ray -tracing where each pixel is labelled 'full', 'empty' or 'occluded'.This occupancy map is then used to validate or not the opening hypotheses.Other features are also used like size or aspect ratio (14 features in total) and a machine learning method is used to validate opening hypotheses.
The advantage of ray-based methods is that a simple distance test is sufficient to detect occlusions.The only required parameter is the minimum distance to the surface that determines if a laser point occludes the wall or not.On the contrary to shape-based methods, no shape filtering is needed.This enables detecting openings with very different shapes and size like wide shop windows.This is a great property since such variable shape openings might occur a lot in very dense urban areas.The drawback is that it requires storing the position of the LIDAR sensor and the angles for every laser point.This information is usually not stored in the widely used LAS 1.2 format.However, when using mobile mapping systems with LIDAR sensors, this information is usually stored in other formats.

OUR APPROACH
Previous section highlighted the advantages of using a raybased method.To our knowledge, such method has not been used yet to detect visible openings using mobile LIDAR data.
First, we estimate the wall plane using a 3D polyhedral model as input and RANSAC estimation.Then, we select all laser rays that cross the wall plane and pass through it.The intersections of these rays with the plane are called "opening points".Final opening models are obtained by clustering the opening points and fitting them with rectangles.
The detection is reliable as soon as the wall plane is accurately estimated, which shows that all the complexity is actually in the estimation of the wall's surface.Except for evaluation of the results, all computations are in 3D.

DATA DES CRIPTION
The LIDAR data is acquired with our mobile mapping system (Figure 1).Two sensors SICK LM S 221 scan the scene vertically on both sides of the vehicle.This sensor has a frequency of 13000 pts/sec, 180° angular aperture, and 0.5° angular step.The system is georeferenced using GPS, IM U and odometer.A 360° panoramic camera ladybug 5 captures images of the scene.
Figure 1: Siradel mobile mapping system In addition to these acquired data, georeferenced 3D building models LOD2 produced by Siradel are used to find laser points belonging to building walls.Our approach is run on two datasets: Nimes and M ulhouse.The results are evaluated on two test datasets: "rue de la république" in Nimes, and "rue des boulangers" in M ulhouse.
The positioning accuracy of the system greatly depends on the quality of the GPS reception.It varies with the weather conditions and the configuration of the buildings.Indeed high buildings in narrow streets generally lead to limited GPS reception and thus higher errors.To improve accuracy, the point cloud is registered on the 3D models using correspondences selected manually.A 3D interface enables the user to visualize both 3D models and laser points, and to select pairs of corresponding points.Two points per street are selected.The user must find roof corners where a laser point is available.Roof corners are preferred to bottom corners because they are visible in aerial or satellite ortho-images and thus they are more accurate.Selected points are then used as control points.They provide the local translation needed to register the point cloud on the 3D model.Then the registration is computed as follows.
For a given laser point, the previous and next control points (in GPS time) are used to interpolate the translation  needed at the current point: The use of control points and interpolation result in a non-rigid registration that adapts to the variations of the GPS errors.Therefore it is more flexible than a global ICP registration.
Figure 2 shows laser points registered on the 3D model.The gutter position is aligned with the wall because the 3D model is reconstructed with aerial or satellite images where gutters hide the wall planes.In the following, we assume a maximum registration distance between laser points and planes   = 2m (because of gutters and roof eaves) and a maximum registration angle   = 15°.The surface normals are used to find points on the main plane and separate them from other parts like windows recess, balconies, cornices, eaves.For each laser point, the local surface is selected using neighboring points in a search radius and the normal is computed using the eigen decomposition method implemented in Point Cloud Library PCL (Rusu and Cousins, 2011).The radius must be small enough to obtain good details about the wall's local geometry, but it can't be lower than the point sampling.Since the point sampling is not constant, the radius is computed for each point, similarly to (Frueh et al., 2005).Vertical and horizontal samplings are computed (Figure 3  The minimum search radius is set to   = 20cm.Note that this adaptive radius computation may not be necessary with LIDAR sensors with higher frequency (millions of points per second), as soon as the point sampling is good enough.
Wall's plane and contour.For each plane  in the 3D model, the corresponding plane  ̂ is estimated using the laser points.First, a subset of laser points is selected by checking that the distance to  is smaller than   and the normal deviates less than   from the plane's normal.The plane  ̂ is then computed using the RANSAC algorithm with vertical constraint implemented in PCL (Rusu and Cousins, 2011).The RANSAC maximum distance threshold is set to   = 15cm.The contour of the wall is obtained with the bounding box of all RANSAC inliers, i.e. points with distance to  ̂ smaller than   (Figure 4).Openings detection.In order to detect visible openings, we want to find all laser rays that intersect the plane  ̂.The GPS timestamp of every point is useful to reduce the search complexity.Indeed, the time range of the inlier points obtained previously is used to select all laser points scanned within this range +/-1 second.Then, all intersections of the laser rays that intersect the plane and lie in the wall's contour are computed.
Only laser rays whose end point lies behind the plane are selected.Whether a point is declared to be lying behind the plane or not is dependent on the wall roughness (flat surface or many extrusions/intrusions).Because of this, statistics about the distances to the plane are computed in order to find a suitable threshold.Only points within the distance 30cm to the plane are used.The mean distance and standard deviation are computed and the threshold is set to: Therefore, the minimum distance threshold  ℎ falls within 15cm to 30cm.There is no need for a maximum distance threshold since all laser rays that cross the plane are used even if they reach an object 10m behind the plane.Figure 5 shows the selected laser rays.Finally, the intersections of the selected laser rays with  ̂ give the opening points as shown in Figure 6.
Openings contours.In order to model detected openings, the next step is to find opening contours and to fit them with 3D polygons.First, opening points are clustered using region growing with the same adaptive search radius as for the computation of the normals.Each cluster is fitted with a rectangle as shown in Figure 6.M oreover, a minimum size is set to 50cm, and smaller openings are removed.There is no maximum size for openings.This enables to detect a large variety of openings, even very wide ones like shop windows.
Occlusions.There is no special filtering step for handling occlusions.Figure 7 shows the detection in the presence of occlusions.A tree and a balcony partially occlude the wall.
Since the laser rays corresponding to these occluding objects don't cross the plane, there is no false detection of openings.
The detected opening under the balcony is a door intruded in the main wall.The openings behind the balcony are only partially detected because they are partially occluded by the balcony.No-measurement laser rays.In our data most laser rays that pass through a window glass are refracted (refracting index close to 1, i.e. no deviation) and then diffracted in many directions when they reach an object, including the direction of the LIDAR sensor.Therefore a laser point is measured.On the other hand laser rays that don't reach any objects are not returned to the sensor (e.g.sky).In addition, laser rays may also be reflected in only one direction when they reach a specular surface (e.g. a mirror) and then no laser point is measured.M any objects in the scene may reflect laser rays including windows.Indeed, we observed in our data that windows may reflect some laser rays.It seems to be windows treated with heat-reflecting coating and with a certain viewpoint from the LIDAR sensor.Such no-measurement laser rays can still be analysed with our ray/plane intersection approach because they have a start point and a direction.However they are rather sparse and noisy, it is not straightforward to decide whether it is noise (road sign, car, mirror, any reflecting material, sky …) or opening.Therefore, we decided not to use these laser rays for our data, although they may be useful in some situation.

RES ULTS
Test datasets.Two datasets acquired in the city centres of Nimes and M ulhouse in France are evaluated (Figure 8).The "Nimes" dataset contains 20 walls and 163 openings from the street "rue de la république" with average height 9m (between 2 and 5 floors).The "M ulhouse" dataset contains 19 walls and 155 openings from the street "rue des boulangers" with average height 10.5m (3 and 4 floors).Each wall corresponds to a plane in the input 3D polyhedral model.These walls respect the hypothesis that it can be approximated by a single plane.The distance between the walls and the sensors is between 3m to 6m.Various types of openings are represented: windows; shop windows; doors; windows with closed shutters; windows with grates.Also, various objects or extrusions are present: balconies; shop advertising and awnings; gutters.Lastly, several objects occlude the walls like trees, pedestrians, cars, and plants.
Figure 8: Datasets used for evaluating the method.Colorized point clouds and 3D polyhedral models are displayed.Top: M ulhouse "rue des boulangers".Bottom: Nimes "rue de la république".Object-based evaluation.The results are first evaluated in terms of object detection, i.e. we check visually if an opening is detected or not and if detection is true or false.Table 1 shows the confusion matrix and precision/recall of detected openings.
There is no false detection and therefore 100% precision.This is because occlusions in front of the plane don't cause any false detection thanks to the ray/plane intersection check.M oreover, walls respect the planarity assumption thus all laser rays that cross the plane and reach a surface behind the plane correspond to opening points.Then, the recall is around 80%.The missing 20% mainly correspond to closed shutters or occluded openings.Pixel-based evaluation.The results are now evaluated pixel-by -pixel using ground truth on the wall textures.Indeed, sampling the wall into a 2D image is convenient to evaluate the precision of the opening boundaries.In addition, since the ground truth is created manually, openings are more easily identified and labelled by a user in the images rather than in the 3D point cloud.Lastly, converting the results in the image domain enables to compare LIDAR-based methods with image-based methods and also to fuse these methods for better results.The texture of a wall is automatically computed using the panoramic images projected on the estimated plane P'.The resolution is 2cm. Figure 9 shows some images of the computed textures with detected openings superimposed in green.Table 3 shows the confusion matrix and precision/recall results.The precision is about 90% although there is no false positive as explained in the object-based section.It shows that some detected openings are not perfectly aligned with the ground truth in the textures.This misalignment mainly comes from small errors in the plane approximation (position and orientation).It induces errors in the textures projected on the wall and in the openings positions computed with ray/plane intersection.For example, we observed alignment errors toward the top of high walls because the plane approximation is less accurate in this area due to lower point density.In addition, misalignment may also come from small calibration errors between the camera and the LIDAR sensors.The recall is 63% for Nimes and 72% for M ulhouse which is smaller than the recall observed with the object-based evaluation (80%).In addition to the alignment error explained above, detected openings may be smaller than ground truth due to partial occlusions like guardrails or poor viewpoint at the top of the walls.Figure 10 shows a building from Nimes where false openings are detected because the assumption of a single plane is not respected.Indeed, the ground floor of this building is not in the same plane as the main wall, therefore almost the whole floor is detected as opening.The small walls in front of the loggias are also in the recess of the main wall and this is why they are detected as openings.

DIS CUS S IONS
We have seen that the assumption of a single plane per wall is not always respected.Although most building walls lie in a single plane, some may have complex structures and some may just simply not be planar.A possible improvement would be to search for multiple planes with a region growing method or reconstruct a 3D mesh that would better model the surface of the wall.
The results show good precision.However, many openings are still not detected because of occlusions.A first solution is to increase point density and the number of viewpoints.In addition, images can be used to detect openings.As mentioned in the introduction section, images and LIDAR are complementary.Indeed, panoramic images may offer a wider visibility on the walls while enabling detection of openings based on their appearances (including closed shutters).Good results of facade parsing in images are reported in (Kozinski et al., 2015;M artinović et al., 2012).The use of aerial data is also very interesting because it is less occluded than terrestrial data while capturing much more walls (i.e.not only in the main streets).Finally, shape-based methods and architectural rules could be exploited to estimate remaining occluded openings and to improve the detection of partially occluded openings.
Point clouds which stem from dense image matching may also be used for openings detection with the proposed method.However, the 3D reconstruction of windows from images is based on appearance and therefore may give different results than laser-based reconstruction, mainly because of reflections.For example, non-extruded windows with reflections are likely to be reconstructed as a smooth surface aligned with the wall's plane because of the regularisation property of image-based methods whereas laser rays are more likely to pass through the glass.
3D visualisation is usually the first application mentioned.
Figure 11 shows textured buildings where openings have been intruded by 25cm.The openings clearly improve the realism of the scene.In Siradel, the detection of openings is also used for simulation of physical phenomenon like radio propagation.Indeed radio waves are less attenuated if a wall contains openings.For example, the quality of indoor reception can be more accurately estimated using the surface of openings in the wall.

CONCLUS ION
We have presented a method for detecting visible openings using mobile LIDAR data.It is based on the intersection between laser rays and wall planes.The main advantage is its robustness to occlusions.We show that detection can be reduced to a single distance threshold once the surface of the wall is computed.Thus all the complexity is actually in the estimation of the wall's surface.This method requires LIDAR sensor positions and angles for every laser point.The results on two test datasets show good precision.Recall (or completeness) depends on the number of occluded openings.Therefore, the next logical step is to focus on the detection of occluded openings.

Figure 2 :
Figure 2: Laser points registered on the 3D model 5. METHOD DETAILS S urface normals.The surface normals are used to find points on the main plane and separate them from other parts like windows recess, balconies, cornices, eaves.For each laser point, the local surface is selected using neighboring points in a search radius and the normal is computed using the eigen decomposition method implemented in Point Cloud Library PCL(Rusu and Cousins, 2011).The radius must be small enough to obtain good details about the wall's local geometry, but it can't be lower than the point sampling.Since the point sampling is not constant, the radius is computed for each point, similarly to(Frueh et al., 2005).Vertical and horizontal samplings are computed (Figure3left).The vertical sampling between points increases toward the top of the wall.This distance ∆ is computed using the sensor distance  and angle step ∆∝ (Figure 3 right): Figure 3: Left: Horizontal and vertical sampling.Right: The vertical sampling ∆ depends on the sensor distance  and angle step ∆∝.

Figure 4 :
Figure 4: Plane and contour (red) approximated with the laser points

Figure 5 :
Figure 5: Laser rays that intersect plane and whose laser points lie behind the plane.Blue points show the sensor positions.

Figure 7 :
Figure 7: Tree and balcony partially occlude the wall but don't cause false detections.

Figure 9 :
Figure 9: Examples of wall with automatic textures and detected openings in green.Top: M ulhouse.Bottom: Nimes.

Figure 10 :
Figure 10: Failure example.Top: detected openings overlaid on the texture image.Bottom: detected opening points (green) and openings contours (red).The ground floor and the walls of the loggias are not in the same plane as the main wall.

Table 1 :
Object-based evaluation with confusion matrixThis is interesting to study more precisely which openings have been detected or not.Table2shows the detected openings classified into 5 classes: windows, shop windows, doors (intrusions), closed shutters (no intrusions), grates.Intrusions are non-transparent surfaces intruded in the wall like doors.Grates are security grids in front of windows or shop windows.They are differentiated from other openings because fewer laser rays pass through the wall plane since some of them are stopped by the grate.The results in the table show that most windows are detected (90% for Nimes and 96% for M ulhouse).The windows that are not detected are occluded by other objects like trees or balconies.Also some windows on top of buildings are not detected due to low visibility with regard to the sensor's view point.Then, only 2 shop windows are not detected because of curtains on the glass.8 doors are not detected.This is because the depth of intrusion in the wall is smaller than the threshold  ℎ computed whith statistics over the wall points.No shutters are detected (0/19 for Nimes, 0/25 for M ulhouse), which is normal since they are not transparent and lie in the same plane as the wall.Lastly all openings with grates are detected although they are partially occluded.

Table 2 :
Object-based detection results for 5 classes of openings

Table 3 :
Pixel-based evaluation with confusion matrix

Table 4 :
Nimes and M ulhouse datasets and results