SHADOW DETECTION FROM VERY HIGH RESOLUTON SATELLITE IMAGE USING GRABCUT SEGMENTATION AND RATIO-BAND ALGORITHMS

Very-High-Resolution (VHR) satellite imagery is a powerful source of data for detecting and extracting information about urban constructions. Shadow in the VHR satellite imageries provides vital information on urban construction forms, illumination direction, and the spatial distribution of the objects that can help to further understanding of the built environment. However, to extract shadows, the automated detection of shadows from images must be accurate. This paper reviews current automatic approaches that have been used for shadow detection from VHR satellite images and comprises two main parts. In the first part, shadow concepts are presented in terms of shadow appearance in the VHR satellite imageries, current shadow detection methods, and the usefulness of shadow detection in urban environments. In the second part, we adopted two approaches which are considered current state-of-theart shadow detection, and segmentation algorithms using WorldView-3 and Quickbird images. In the first approach, the ratios between the NIR and visible bands were computed on a pixel-by-pixel basis, which allows for disambiguation between shadows and dark objects. To obtain an accurate shadow candidate map, we further refine the shadow map after applying the ratio algorithm on the Quickbird image. The second selected approach is the GrabCut segmentation approach for examining its performance in detecting the shadow regions of urban objects using the true colour image from WorldView-3. Further refinement was applied to attain a segmented shadow map. Although the detection of shadow regions is a very difficult task when they are derived from a VHR satellite image that comprises a visible spectrum range (RGB true colour), the results demonstrate that the detection of shadow regions in the WorldView-3 image is a reasonable separation from other objects by applying the GrabCut algorithm. In addition, the derived shadow map from the Quickbird image indicates significant performance of the ratio algorithm. The differences in the characteristics of the two satellite imageries in terms of spatial and spectral resolution can play an important role in the estimation and detection of the shadow of urban objects. * Tel.: +447427618125. E-mail addresses: MohammedSalihNM@cardiff.ac.uk, nada.flower@gmail.com


INTRODUCTION 1.1 Shadows in the VHR Satellite Imageries
Very High Resolution (VHR) satellite imagery is considered one of the highest quality currently available from remote sensing satellites because of its ability to offer sub-meter resolution.That means the VHR satellite imageries are capable of providing a high level of detail, which make them a reliable and highly vital source of information.Therefore, the VHR satellite imageries support a range of services, especially in urban areas, for city planning and monitoring, urban change detection, estimation of human activities/population, and urban object/feature detection.However, due to urban constructions which are built above ground, such as buildings and bridges, inter alia, shadows are the most common component accompaniments for these constructions that can be seen in VHR images.In the fact that the incident light rays, typically sunlight, when intercepted by an off-terrain object, shadows will be generated and cast on other urban surfaces/objects at the moment of image capture by optical satellites.Figure 1 presents three conditions (a light source, an object to cast the shadow line, and a surface to receive the shadow line and shadow) that must be met to produce a shadow.In this context, a shadow indicates the shape of the object casting it, and in many ways it can indicate the texture of the surface receiving the shadow (Yee, 2013).In contrast, shade can be defined as the side of an object which is opposite the direction of illumination, which has less colour tone of the full blackness (the value intensity of darkness) compared to the objects' shadows that have very low values of brightness in VHR images, or mostly have zero value (pure black), Figure 1.The fact is that the line that locates and separates the light and shade areas on the object determines the shadow line on a receiving surface.The shadow line, in turn, determines the dark area cast onto the surface on which the object rests and which receives the cast shadow.Therefore, the shadow line and the dark area (shadow regions) in the optical satellite imageries have become one of the crucial topics that has attracted researchers' attention because the dark pixels that belong to the shadow regions in image have received two main aspects of analysis.These two aspects depend on how the shadow regions in VHR images have been treated by researchers: as a curse must be addressed or as a blessing must be exploited.Basically, shadow concept can, therefore, be understood through grasp shadow progressions which start from determining points to lines (being composed of points) to planes (being composed of lines) and lastly to solids (being composed of planes), as shown in Figure 2. In fact, the shadow of a line, a plane, or a solid is most efficiently determined by locating the shadows of the critical points of the line, plane or solid (Yee, 2013).Moreover, if we draw parallel lines (paralines) in the virtual world extended from an object to its cast shadow on a mere and flat surface with sufficient distance between them, the identical size, shape and orientation of the object will be obtained.The basis of this fact is that the greater the distance, the greater the shadow will display.However, the situation is completely different with objects' shadows within urban areas in VHR satellite imageries in terms of urban fabric which often exhibit compact areas of buildings as one factor.This factor can affect the actual size and shape of urban objects that are presented by their shadows.The reason behind this is the shadow line of the object will be shorter in the case of a rise in a horizontal receiving surface and longer where there is a drop in this surface for the other objects within urban areas.The second factor is the existence of trees and their shadows or even shadows from non-built-up areas, such as vehicles, which can distort the real boundaries of shadows (the geometric form) and give an arbitrary shape to the built-up objects, when they combine with the objects' shadows.Another factor is the shadow regions of objects in images will be larger in the summer season than their counterpart in the winter due to the sun's position in the sky vault and its angle and altitude.Spectrally, non-shadow regions, for instance, water, can present the same pixel intensity values or darkness with shadow regions of the objects in images that can cause an error in detecting shadows.Therefore, shadow detection and its extraction from VHR satellite images are not easy tasks and all image processing techniques for detecting shadows still depend on the estimation of shadow areas but using various developed methods.

Previous Work of Shadow Detection Approaches
Remote sensing with very high spatial resolution, such as Quickbird and WorldView-3, among others, can provide clearly detailed features of cities, for instance, roads, buildings, parks and trees.However, the appearance of shadow increases with the spatial resolution and whether the purpose of determining shadow regions for removing these regions or exploiting their information, the main and first process in the shadow analysis studies is to detect the shadows.Therefore, this paper reviews the current shadow detection approaches applied on VHR satellite imageries and examines the performance of the two state-of-the-art shadow detection approaches.Many pioneering studies for the automated detection of shadows from VHR satellite images were devoted to improve the quality of image after detecting and handling shadows or to utilise shadows as evidence to detect man-made structures within the urban environment.Although, the automated detection of shadow regions from VHR satellite imageries is a challenging task, automatic object detection and extraction is a very active scope of research.However, we confine and summarise the literature review for only the present studies that aimed to detect and extract the shadows of buildings from the VHR optical images automatically.A review paper on shadow analysis was produced by Dare (2005).The paper presents the methods of detection, removing the problems of shadowing in high resolution satellite imagery.Four different techniques for separating shadow pixels from non-shadow pixels were described in more detail in the article which are: thresholding, classification, region growing segmentation and three dimensional modelling.The segmentation approach was proposed to detect shadow regions with four step processing included on density slicing, thresholding, region encoding and region filtering which applied on IKONOS and Quickbird panchromatic images.Thereafter, radiometric enhancement technique was chosen between two other presented techniques It is concluded that although, the quality of image can be improved through the techniques of detection and removal of shadow areas, results can be image-dependent.In one of the latest studies, Wakchaure and Raut (2014) reviewed a range of techniques for shadow detection and de-shadowing from VHR images.The study included the proposal of a processing chain based on binary classification.Canny edge detection algorithm was used to detect shadows and differentiate their areas from non-shadow areas in the image with clear boundaries.Thereafter, the calculation of mean and standard deviation values for both pixels in the shadow and non-shadow areas and the normalisation process were suggested as potential improvements for the final results of removing shadow pixels and appearing non-shadow pixels by increasing their brightness and shadow intensity values.However, the algorithm has not been applied on any VHR image in order to evaluate its performance and its suitability in detecting shadows.The segmentation approach is one of many different approaches that are used to detect shadow regions from single VHR multispectral images.Ma et al. (2008) presented an approach in shadow segmentation and compensation that was implemented on an IKONOS image through a normalising Saturation-Value difference index (NSVDI).The approach was based on analysis Hue-Saturation-Value (HSV) colour space to detect shadow regions of buildings and a histogram matching technique was exploited to retrieve the information under shadow.The results of segmentation illustrate that the shadow areas were effectively extracted from the IKONOS image.Nevertheless, the approach was not able to distinguish between dark objects, such as water and shadow regions due to the similarity of spectral values of pixels and the lack of the validation of final results.It was concluded that multiple images can be better in use for compensating the information under the shadow regions because it was found that the retrieval information from a single image is not sufficient to this end.In the same context, segmentation algorithm based on contour model and isolation was proposed by Elbakary and Iftekharuddin (2014) to detect man-made structures by their shadows.The approach was applied on a panchromatic image, which was created from a VHR multispectral image with spatial resolution of 11 metre for this application.Thereafter, post-processing based on optimal thresholding and geometric filter were implemented to determine the real shadows from other objects.Although, the algorithm was used to detect and segment the shadow regions using the gray-level satellite image without depending on the colour information and to handle the difficulty of shadow detection in satellite images, the problem of separating water bodies, such as lake, from shadow regions is not solved.This is because they exhibit similar intensity feature characteristics.In other recent studies, a classification-based approach has been used to detect shadows regions.Wen and Yamazaki (2012) proposed a method in shadow detection using an object-based classification that employs brightness values and their relationship with the neighbouring area, which has been applied on Quickbird and WorldView-2 (PAN and multispectral images).A method for the correction of these detected shadow areas has also been proposed using a linear function to remove shadows from images.The idea is to classify images into four classes (vegetation, road, roof and shadow) and modify shadow pixels through three levels of darkness (dark, medium and light), which is derived from the twenty-six radiance measurements in order to demonstrate the nature variability of the radiance ratio.A linear-correlation function and NDVI were used to attain a smoothly restored image.The overall classification accuracy for sunlit areas was 74.94%, while for the corrected shadows areas it was 71.87%, and the comparison between these results illustrates that the proposed shadow correction method is effective in correcting the radiance in shadow areas.Nevertheless, the detection step was completely based on the results of the supervised classifier, which was the main reason for the moderate shadow regions.Moreover, a bright roof in light-shadow, small shadows cast by trees, and material with high reflectivity in shadow areas were not extracted and observed properly by the object-based classification method.Later, the morphological filtering approach for shadow detection with a shadow reconstruction approach was suggested by Song et al. (2014) to improve the classification performance applied on Quickbird and WorldView-2.Shadow detection comprises three steps: thresholding, morphological filtering and edge compensation for the derived shadow mask as a first stage in this approach.In the second stage, a Markov random field (MRF) was used to sample shadow and non-shadow pixels manually to produce a library for both types of pixels as a training stage.The final stage was to compensate and reconstruct the underlying land cover pixels using the Bayesian belief propagation algorithm to address the MRF.The supervised ISODATA classification method was adopted and applied before and after shadow construction to verify the performance of the proposed shadow removal approach.The results indicate that the constructed images can be classified more accurately than images before shadow reconstruction.However, the chain of image processing for shadow detection might affect the quality of the actual geometric shape and edge of shadows areas.Because of the morphological filtering in the shadow detection stage, some small shadow areas were not removed and were remained in the classification map after shadow reconstruction.Another different recent work, a new index for detecting shadows regions, was introduced by Teke et al. (2011).In this approach the use of near-infrared information in combination with visible information (especially, green and red bands) of VHR satellite images is utilised to generate a false colour image.The processes of normalisation and transformation were implemented on this false colour image to compute the Hue-Saturation-Intensity (HIS) space and a ratio map.In their approach, Otsu's method was applied to the histogram of the ratio map in order to detect the shadow regions because the used thresholding scheme is able to detect both shadow and vegetation regions at the same time.The final step in the approach is to derive a binary shadow mask through subtracting the regions of vegetation cover in an image.Although, the results exhibited success in detecting shadow regions, postprocessing is required to improve the shape and boundaries of the derived shadow mask from VHR satellite image.This is so especially in cases where the shadow regions are used as evidence of the structure and location of the elevated objects within the urban area, such as buildings.The same approach has been used by Ok et al. (2013) to detect shadow regions for identifying the regions recognised as buildings and roads, implemented on a set of Quickbird and Geoeye-1 VHR images.A post-processing shadow mask, which included a constrained region growing process on detected shadow regions and probabilistic landscape approach (Ok, 2013a), was achieved using three different thresholds: intensity, ratio and height in order to obtain a regular shape identical to reality and eliminate the unwanted shadow areas.Overall, image thresholding techniques are essential in the processes of object detection.This is in spite of the fact that thresholding techniques are a simple pattern of partitioning an image into isolated features, they are yet effective.Nonetheless, the thresholding techniques which applied on VHR satellite images (multispectral or PAN image) for shadow detection are required to be more effective, taking into account the different characteristics of the used images.In addition, the fully automated algorithms of shadow detection often require further constraining assumptions about extracting shadow regions from VHR satellite images in order to run them appropriately and obtain reliable precision of the shadow masks.

The Importance of Shadow Detection in Urban Environments
All derived information and details from VHR satellite images can offer vital source data for understanding the complexity of urban landscapes or for addressing the ambiguity or problems in VHR satellite images within urban areas using the development of computer vision and image processing techniques.The derived information from shadows can play an important role to this end.Many studies have considered shadows in VHR satellite images as unwanted features and drawbacks that must be removed.In fact, the studies of shadow detection and analysis have incited the development of new methods in improving image quality and in enhancing the algorithms of image segmentation and partitioning as well as object extraction.Although, the appearance of shadows can affect the performance accuracy of applications, such as image classification and registration, shadow provides information about the characteristics of surfaces and light source, shape and location of objects.Furthermore, because shadows cover a considerable portion of an image, they play a supporting role on automated analysis.In this context, the shadow presence in single VHR multispectral images has been exploited as strong evidence of the existence of a different building structure, such as buildings detection (Ok, 2013a), arbitrarily shaped buildings in complex environments (Ok et al., 2013), the extraction above ground circular structures (Ok, 2014), and automated extraction of buildings and roads (Ok, 2013b).Shadows are an important cue for information not only about man-made structures but also about supporting urban sustainability.The assessment of solar photovoltaic (PV) energy potential as a form of renewable energy within urban environments is an important application to secure the needs of energy for cities in the future.In this regard, shadows information within urban environments is a key element required for the analysis of solar radiation across daylight.By using the different datasets, Redweik et al. (2013) developed a shadow algorithm to calculate shadow maps for solar potential assessment of all surfaces (roofs and facades) of buildings within an urban landscape.Bergamasco and Asinari (2011) proposed a method to compute the available roof surface for integrated solar (PV) installations through the analysis of shadow zones, space, brightness and suitability for PV installation.Jo and Otanicar (2011) analysed the patterns of shadow cast and their effects on the rooftop in order to estimate the available rooftop area for PV installation.A number of researchers have also published works and approaches that exploit the derived information from shadow analysis.Izadi and Saeedi (2012) introduced a method for automatic detection and height estimation of buildings with polygonal shape roofs from single satellite images based on shadows and acquisition geometry.In this study, a fuzzy logicbased approach was used to estimate building heights according to the strength of shadows and the superposition between the identified shadows in the image and expected true shadows of the buildings in order to generate a 3D polygonal building model.Raju et al. (2014) proposed an approach to extract the height of the building using shadow, which includes two stages: rooftop and shadow extraction and height estimation, which were achieved by example-and rule-based approaches on the basis of the derived information and relationship between rooftop, shadow and sun-satellite geometry.Based on a volumetric shadow analysis (VSA) method, Lee and Kim (2013) presented a scheme to extract building height automatically.This scheme was implemented through examining the location change of projected shadow lines with respect to the actual shadow regions with increasing building heights progressively.The approach has been applied on IKONOS, KOMPSAT-2, Quickbird and WorldView-1 images.It seems likely that the derived information from shadow regions using VHR satellite images are not limited for achieving one purpose or specific application within urban environments using advanced techniques in image processing and computer vision, if appropriately exploited.Employing shadow information in different applications does not need to build a distinct model for inferring an object's structure because shadows can support the automated processes for deducing the structure of man-made objects.However, in order to make this information highly useful and effective in every application, shadow regions should be precisely detected and extracted from images.

Shadow Detection Using Near-Infrared Information
The main, second part of this paper is to test the performance of two current state-of-the-art shadow detection algorithms in terms of their ability to detect shadows from VHR satellite images.The two selected approaches were implemented remarkably well and their performance has been validated as well as their results verified.However, these approaches and their advanced algorithms were applied on a wide variety of kinds of images excepting VHR optical satellite images.Therefore, this article aims to examine these algorithms on the optical satellite images in order to apply them afterwards on different types of VHR satellite images for applications that might exploit and benefit from the derived information of shadow detection.We have selected a simple condition for detecting shadows in this experiment, which is a cast shadow of a detached building within an urban area.The first approach was accomplished by Rufenacht et al. (2014).The approach is to compute ratios between the nearinfrared (NIR) and visible bands on a pixel-by-pixel basis, which allows disambiguation between shadows and dark objects.The principle ideas behind this ratio algorithm are (1) objects that are dark in the visible spectrum often have a much higher reflectance in the NIR band, and (2) most of the considered illuminants in the shadow formation process have a very distinct spectra in the NIR band.Because the captured images that have been used to detect shadows were entered separately into algorithm, the algorithm has been modified to fit with VHR satellite images.Additionally, in order to extract the accurate shadow regions after applying the multi-image approach (the ratio algorithm) on our own images, we developed and refined the algorithm for obtaining accurate outputs.The shadow detection algorithm framework is presented in Figure 3.
The whole approach and all equations are already welldescribed in Rufenacht et al. (2014).The multi-image approach was applied on the Quickbird satellite imagery (visible and INR bands) with spatial resolution 60 cm, the study area being in Ankara, Turkey.The image includes some isolated buildings, plants, roads, and bare soil.Using the implementing ratio algorithm on the Quickbird image, the obtained outcomes are provided in Figure 4.

Shadow Detection Using Kernel Graph Cut
The second approach that was tested to detect shadow regions from the optical satellite image is the multiregion image segmentation based on parametric kernel graph cuts by Salah et al. (2011).The approach has been implemented with different types of data, such as the Berkeley database, medical images, and Radar satellite image (SAR), colour image (but not RGB satellite image) and motion maps.The effectiveness of the multiregion image segmentation approach has also been verified by a quantitative and comparative performance evaluation over numerous experiments on aforementioned data.Accordingly, the performance of the algorithm was considerably well in regard to segmentation accuracy and flexibility.Therefore, this algorithm was selected in this paper to examine its performance with another dataset, such as optical satellite image with very advanced sensor and very high spatial resolution, and on the contrary, to test whether shadow regions can be segmented and extracted from other urban objects without using INR information utilising only visible bands (RGB colour satellite image).The basic idea of the multiregion image segmentation approach is to increase the dimension of the feature space image datamaking it higher -via a kernel function in order to accomplish a better separability, ideally linear.Briefly, the main concept of this approach is illustrated in Figure 5.The explanation of the whole method and its equation can be found in the article presented by Salah et al. (2011).The algorithm was implemented with VHR satellite image © WorldView-3 Image Copyright 2014, DigitalGlobe.In this paper, WorldView-3 image was used to test the kernel graph cut segmentation algorithm in terms of detection shadow regions that are cast by above ground objects within an urban area, such as buildings.WorldView-3 satellite is one of the most advanced satellites that was successfully launched on August 13, 2014 by DigitalGlobe, with very high spatial resolution of 30 cm and twenty-seven super-spectral bands.The used image is a three band pan-sharpened image (natural colour includes R, G and B bands).The image has been downloaded from free satellite imagery samples, which are provided on the DigitalGlobe website.The used WorldView-3 image was collected of Madrid, Spain, and this true colour image includes bar soil, a few plants and other urban features, such as a detached building, roads, cars, and sports courts, as shown in Figure 1.The shadow detection result was provided in Figure 6.Further refinement was applied to attain a segmented shadow map provided in Figure 6c.

DISCUSSION
The detection of shadows by testing and implementing two different approaches with different VHR satellite image characteristics within an urban area was feasible and outcomes were considerable.In the Quickbird image case, the results illustrate that the ratio algorithm has the ability to detect shadow regions of isolated buildings completely.However, the algorithm has been improved and developed to fit with VHR satellite image and to attain an accurate shadow mask.In the WorldView-3 image, the kernel graph cut algorithm presents an adequate performance in detecting and separating the shadow of the building.Nevertheless, in Figure 6c, the result indicates that the segmentation was not only for the shadow region but also for the dark objects that have the same intensity values as the shadow region, such as trees and the cast shadows of cars.This problem can be solved through the exploitation of the geometric pattern of shadow, which has a distinct regular shape between the dark urban objects.To this end, further refinements are required to make the kernel graph cut algorithm capable of extracting the shadow mask without similar pixels of segmentation that belong to other objects.Additionally, the same parts of segmentation that are similar to the segmented shadow regions can increase the complexity of this problem within dense urban areas.Nonetheless, we believe that significant shadow regions have been detected via the use of both algorithms and we also believe the ratio and kernel graph cut algorithms can be achieved with a better implementation using VHR satellite images.Accordingly, the results from implementing the two current state-of-the-art shadow detection and segmentation algorithms on the different VHR images illustrate the accurate performance and efficient computation in terms of the automatic application within an urban area that includes solely the isolated buildings.

CONCLUSIONS
This paper reviews a number of current pioneering studies in the scope of shadow detection techniques from VHR satellite images in the first part of the paper and the second part comprises the test for the two advanced algorithms performance.The use of the VHR image from advanced satellites, such as WorldView-3, illustrates strong support for the process of object detection and extraction within urban areas.Shadow detection from VHR satellite images an urban feature and provides vital information that can assist in further understanding of the built environment.The cast shadow provides strong evidence of the elevated objects above the ground line which confirms the existence of a building structure.Therefore, the automated detection of shadows from VHR satellite images is an important process in applications for building detection, illumination direction analysis from the sun, and the spatial distribution of the object casting the shadow.
Although the detection of shadow regions is a very difficult task when they are derived from a VHR satellite image that comprises a visible spectrum range (RGB true colour), the results demonstrate that the detection of shadow regions in the WorldView-3 image is a reasonable separation from other objects by applying the GrabCut algorithm.In addition, the derived shadow map from the Quickbird image indicates significant performance of the ratio algorithm.The differences in the characteristics of the two satellite imageries in terms of spatial and spectral resolution can play an important role in the estimation and detection of the shadow of urban objects.
In the future, we plan to carry out tests to determine whether these used algorithms are able to detect shadow regions in dense urban areas.In this regard, the different scenarios of the cast shadows of buildings will be taken into account to examine the computational efficiency and limitations of the used algorithms.More importantly, our future work will concentrate efforts towards improving shadow detection results with the existence of the water bodies within urban areas through employing the same used algorithms.

Figure 1 .
Figure 1.Shadow conditions using real satellite image.

Figure 2 .
Figure 2. Shadow progressions, (a) Shadow of a point, (b) Shadow of a line, (c) Shadow of plane, (d) Shadow of a solid.
areas from IKONOS and Quickbird images.
(a) The Quickbird image, (b) Visible shadow candidate map, (c) NIR shadow candidate map, (d) Shadow map, (e) Shadow mask, (f) the final result of the shadow mask after enhancing the current algorithm
Figure 6.(a) The WorldView-3 image, (b) The result of segmentation obtained by the kernel graph cut algorithm, (c) The detection of shadow region of the building by the graph cut segmentation.