IMPROVEMENT AND EXTENSION OF SHAPE EVALUATION CRITERIA IN MULTI-SCALE IMAGE SEGMENTATION

From the last decade, the multi-scale image segmentation is getting a particular interest and practically being used for object-based image analysis. In this study, we have addressed the issues on multi-scale image segmentation, especially, in improving the performances for validity of merging and variety of derived region’s shape. Firstly, we have introduced constraints on the application of spectral criterion which could suppress excessive merging between dissimilar regions. Secondly, we have extended the evaluation for smoothness criterion by modifying the definition on the extent of the object, which was brought for controlling the shape’s diversity. Thirdly, we have developed new shape criterion called aspect ratio. This criterion helps to improve the reproducibility on the shape of object to be matched to the actual objectives of interest. This criterion provides constraint on the aspect ratio in the bounding box of object by keeping properties controlled with conventional shape criteria. These improvements and extensions lead to more accurate, flexible, and diverse segmentation results according to the shape characteristics of the target of interest. Furthermore, we also investigated a technique for quantitative and automatic parameterization in multi-scale image segmentation. This approach is achieved by comparing segmentation result with training area specified in advance by considering the maximization of the average area in derived objects or satisfying the evaluation index called F-measure. Thus, it has been possible to automate the parameterization that suited the objectives especially in the view point of shape’s reproducibility.


INTRODUCTION
The image segmentation technique is regarded as a very important and efficient elemental technology which is utilized for detection, classification or recognition of objectives included in images.Since the last decade, multi-scale image segmentation has attracted attention to realize object-based image analysis (Blaschke, 2010) and has been easily and widely used because the processing tool is implemented as commercial products such as eCognition supplied by Trimble Navigation Ltd. (Belgiu and Drăguţ, 2014;Laliberte and Rango, 2009;Kavzoglu and Yildiz, 2014).
In the multi-scale image segmentation, the process is performed by considering not only the similarity of the spectral information but also estimating the derived shape's stability of divided region (Baatz and Schäpe, 2000;Blaschke and Hay, 2001;Li et al., 2009).Each region obtained by this process is called as an object which is formed as an aggregation of spatially adjacent pixels.By adjusting the index of shape's property, objects with shapes of various characteristic can be derived.Therefore, the suitable selection of the shape criteria is needed to obtain the objects fit to the target of interest and the improvement of the accuracy in succeeding object-based analysis.
In this study, the key points to be discussed are summarized as follows.First of all, improvements and extensions on the conventional approach were addressed in the application of both spectral and shape evaluation.The constraint for spectral evaluation was introduced so as not to make the improper merging of adjacent regions.This operation brought apparent improvements in excessive merging between dissimilar regions.Secondly, we demonstrated that significantly different shape properties of object could be derived by modifying definition on shape's extent in the evaluation of shape criteria called smoothness.
Thirdly, we have introduced new shape criterion called aspect ratio.This criterion regulates the aspect ratio for objects by keeping the property of conventional criteria such as compactness and smoothness.Thus we could realize to obtain more accurate, flexible and diverse image segmentation results which reflected the shape characteristics of the objectives.We demonstrated that these improvements and extensions contributed to improve the performance of the reproducibility on the shape of objectives in both of the qualitative and quantitative evaluations.
Finally, we have addressed the issue of selection of the optimum parameterization.The proposed approach is achieved by comparing training area, which are marked by evaluator in advance, with the derived objects which include training area.This can realize the calculation of an optimum parameter in the state of maximizing the resulting average area size of objects and/or F-measure over a predefined threshold.Thus, it has been possible to automate the parameterization that suited the objectives especially in the view point of shape's reproducibility, which was often relied on the experience or intuition of the evaluators conventionally.
We performed a qualitative and quantitative evaluation for the proposed approaches.The results showed the effectiveness and advantages to the conventional approaches.

Principle of multi-scale image segmentation
Here the summary of multi-scale image segmentation method is described and the improvements and extensions are introduced in the next chapter.
The multi-scale image segmentation is a kind of region merge approach.Each region (object) starts from one pixel at the initial stage.Whether or not to merge the adjacent regions are achieved by evaluating the changes of heterogeneity including both spectral component and shape component in objects as defined in Eq. ( 1).
where F color = heterogeneity in spectral component F shape = heterogeneity in shape component w color = weight for F color w shape = weight for F shape F color is defined by using standard deviation derived from each band of pixels within objects as follows.
where p, q = suffix which means two objects before merge r = suffix which means object after merging p and q n = number of pixels within object σ i = standard deviation of i-th band in object F shape is defined by using two criteria called compactness and smoothness which regulate the shape properties of objects.
where f cmpct = heterogeneity in compactness f smth = heterogeneity in smoothness w cmpct = weight for f cmpct w smth = weight for f smth The compactness is defined by perimeter and number of pixels in objects and the smoothness is defined by perimeter and shorter side length of boundary box in objects respectively.A degree of merging process is regulated by scale parameter (SP), which also affects the area size of the derived regions indirectly.As a concrete procedure, merging of neighbouring two regions is executed when the evaluation value by Eq. ( 1) does not exceed the square value of SP.The SP started from small value (by ordinary, value of 1) and then all the possible merging of the objects is performed.The same processes are applied repeatedly until SP reached the predefined value.The weights for each parameter are allowed to be changed in arbitrary stage of SP if necessary.

Policies of our unique implementation
We have implemented our unique executable module on multiscale image segmentation in order to conduct several improvements and extensions of the algorithm.In our approach, the following policies have been applied.

Definition on perimeter of objects:
The perimeter of objects has been defined as distance of four per one pixel, which means four-neighbour distance in image coordinate.This makes it possible to count up the exact perimeter easily for any objects which has no matter how complex shapes.

Search direction in merge process:
Fourneighbourhood search has been applied during merge inspection between any two neighbouring objects.This handling is for satisfying the consistency of definition on perimeter of objects.

Selection of merging regions:
In the merging process, there are typically four types of options for merging two neighbouring regions.These options are classified as (1) Fitting, (2) Best fitting, (3) Local mutual best fitting, and (4) Global mutual best fitting (Baatz, 2000).When an arbitrary object A is merged with object B from its neighbouring objects B i , the first method is selecting B immediately which satisfies homogeneity criterion at the first.The second method is selecting B having best evaluation of satisfying condition (1) from B i .The third is selecting A and B which satisfies condition (2) mutually.The last is selecting A and B from all possible combinations in evaluating SP, but this is not a selectable option from the view point of practical use or processing time.
Fig. 1 shows the comparison of results for above options (1) to (3).We can confirm that more latter cases, the more distinct improvements on the region separation were realized (depicted by circles).Therefore we adopted the option of (3) for the selection of merging regions in this study.

Introduction of constraint on spectral evaluation
When the conventional evaluation was applied in the spectral heterogeneity as in Eq. ( 2), inadequate merging of regions occurred in some cases.Fig. 2 shows an example of such a case.In this illustration, (a) indicates the merged two regions of p and q, and (b) and (c) denote the histograms of those spectral distribution.Similarly, (d) shows the histograms after merging of p and q.
In this case, when two regions were merged, spectral distribution i.e. standard deviation was apparently improved despite the merging of completely dissimilar regions.The fact implies that simple evaluation using standard deviation is insufficient on spectral evaluation.To solve such situation, we have introduced the constraints for the operation on spectral evaluation expressed as follows.
where p→q = evaluative operation from object p to q i μ = mean of i-th band within object This operation is attempted when the number of pixels within each evaluated region exceeds predefined value.Because, in the case of small object size, merge process would not proceed at all.By introducing this, it is possible to suppress the occurrence of improper region merging.

Extension for smoothness criterion
As shown in Eq. ( 5), the smoothness criterion can be regarded as an index which means simplicity or complexity of region's outline on extent of object.Therefore, the behaviour in merge process would be changed by modifying the definition on the region's extent.In order to extend the diversity for the shape of derived objects, we considered introducing derivative indexes such as length of longer side or a diagonal in bounding box of object instead of shorter side.
If we consider the behaviours theoretically, in the case of selecting shorter side of boundary box, the most ideal shape of object by the evaluation formula is to become slimmed and diagonally distributed to the boundary box.Similarly, if we choose longer side, objects which are elongated along the longer side of direction in the boundary box are most likely generated.On the other hand, if we adopt the diagonal length, it is likely close to rhomboid shape of objects.Because the perimeter of the rhombus corresponds to the twice of the diagonal length, and is the shortest length of ideal object derived.
Incidentally, if the perimeter of the boundary box is applied as the index, this is equal to the twice of the sum of the shorter and longer side, and shows a state of compromise to their properties.

Introduction of aspect ratio
In this study, we have proposed a newly introduced shape criterion called aspect ratio (f aspct ) which is defined as follows.
where a = user-defined aspect ratio (normally a = 1) dx = length of x-direction in bounding box of object dy = length of y-direction in bounding box of object By introducing this index and its weight w aspct , Eq. ( 3) is extended as follows.
This criterion can provide a new property, which constrain the aspect ratio against the bounding box of the objects so as not to be ranged in a particular direction by keeping properties controlled with conventional shape criteria.Thus we can obtain more flexible and diverse segmentation results according to the shape characteristics of the target of interest.

Experimental results and discussion
Fig. 3 shows the results for comparing the effectiveness of introducing the constraint on spectral evaluation described in section 3.1.In order to facilitate the comparison, the process was performed with higher weights setting of shape criterion, that is w shape = 0.7 in Eq. ( 1) and w cmpct = 0.5 in Eq. ( 2).We can observe in the conventional processing result of (a), several p q regions which have obviously different colour were merged into same objects (indicated by yellow circles).On the other hand by applying the proposed method, as demonstrated in (b), such excessive merged objects could be suppressed properly.
Figure 3. Effectiveness of constraint on spectral evaluation Fig. 4 indicates the comparisons on extension for smoothness criterion by modifying the definition used for extent of objects discussed in section 3.2.For the purpose of comparing the differences clearly, an image shown in Fig. 4(a) with relatively uniform texture was processed in weights setting of w shape = 0.8 in Eq. ( 1) and w smth = 0.8 in Eq. ( 2).We can confirm that these results are consistent with the presumed behaviours on the ideal shape of objects which are argued in section 3.2.It is possible to obtain more desirable segmentation results for different types of objectives by changing these definitions.In such situations, the kind of the to be employed comes to be one of the annoying issues.We also discussed about automatic parameterization to cope with such a subject in the later chapter.Fig. 5 shows the segmentation results generated by different criteria including aspect ratio which is newly proposed in our research.In the processing, we set the weight of w shape = 0.8 in Eq. ( 1), the diagonal length as the definition for extent of object, and the aspect ratio a = 1 in Eq. (4).
Figure 5. Segmentation results by the different weights for three shape criteria Fig. 5(a) is the result when no shape criterion was considered which shows significantly complex shapes of objects.In the case when only smoothness criterion was applied the result shows a strong tendency to generate extremely long distribution in a particular direction (Fig. 5(b)).In the case of using only compactness criterion, as shown in Fig. 5(c), rather compact objects were generated.However, the reproducibility of the Fig. 5(d) indicates a result which was obtained by using only aspect ratio criterion proposed in this study.In the following figures, yellow frame indicates object those with the aspect ratio value near 1, purple frame indicates object those with the aspect ratio value far from 1.In this case, we could observe numerous objects with aspect ratio close to 1 having complex outlines of objects.Fig. 5(e) is a result of case by conventional approach, and the outlines of objects having both shorter perimeter and compactness.
Fig. 5(f) and Fig. 5(g) are results by the combinatorial use of aspect ratio and conventional shape criteria.These results showed that derived shape of objects reflected the property due to aspect ratio while keeping existing property observed in Fig. 5(b) and Fig. 5(c) respectively.In Fig. 5(h), we can find the result which has hybrid characteristics of all shape criteria.

Basic concept for automatic parameterization
Automatic estimation of the optimal parameters is important in developing an objective method which does not rely on the experience or intuition of the evaluators.Furthermore, by establishing a technique, it is also possible to utilize for the quantitative evaluation of the newly introduced improvements.
Challenges on this issue for multi-scale image segmentation has been addressed in several researches (Drăguţ et al., 2014;Fourie et al., 2012;Johnson et al., 2015;Lübker and Schaab, 2009;Zhang et al., 2010).Some of these studies focused on optimizing parameters for their own tasks by batch processing with commercially available software, therefore controllable procedures were limited.Also some of conventional approaches which determine the optimum parameters based on SP values have a potential problem in applying in general purposes.
Because SP can only control the size of derived object indirectly and its parameter setting is sensitively affected by the changes in weight balance of parameters and especially the image contrast.
When performing the object-based image analysis, the size of derived objects is considered as one of the most important factor.That is, if the sizes of the objects are smaller, they become close to a conventional pixel-based analysis and cannot exert the advantage of object-based approach.Therefore, we proposed a method for generating optimum parameters which maximize the size of derived objects without missing the nature on target of interest as much as possible.

Automatic parameterization scheme
In this study, in order to realize the optimum selection of processing parameters quantitatively, two-step process is applied.The first procedure is specifying training area from the target image, in advance, whose area matches to a partial region of target of interest.Then in the second step, parameter estimation which gives the best fitting to the training area is performed while considering the reproducibility of shapes of the derived objects.
To evaluate the degree of coincidence between the shape of derived object and training area, we adopted the indicators defined as follows.The indicators are frequently used in the area of pattern recognition (see also where tp = number of pixels in true positive region fp = number of pixels in false positive region fn = number of pixels in false negative region Pr = precision, which indicates detection performance Rc = recall, which indicates completeness F = F-measure, which is calculated as harmonic mean value between precision and recall In this study, when the ratio of the number of pixels which corresponded to the training area within a derived object exceeded the specified value (e.g.50%), then that object was regarded as having positive prediction.The relationship between prediction for derived object and training area is classified as shown in Table 1.Since precision and recall are in a conflicting relationship, F-measure which is a harmonic mean value of these indicators is used as actual indicator in accuracy evaluation.Table 1.Relationship between prediction and training area Fig. 6 shows the flowchart of automatic parameterization in our proposed approach.In this flow, parameters to be adjusted consist of weights for balancing spectral component and shape component, and weights for balancing three criteria on shape, that is, compactness, smoothness and aspect ratio.Because the combination of these parameters becomes too huge, we built a flow for calculating the quasi-optimal solution.
Figure 6.Process flow for automatic parameterization User-specified parameter required for the process is either of the following i.e. the one is the lower limit of the average area size (Sc) in all objects after segmentation, and the other is target value of F-measure.Even if either parameter is specified, global parameter search is executed as the first common process.Another parameter to be specified is step size of weight parameters, which regulates fineness of the parameterization and might be set as hard-coded (e.g.value of 0.05).
In the parameter search, image segmentation process is executed for all combination of parameters set by the coarse step size (e.g.value of 0.2).In each combination of parameters, the process is continued until the average area size of objects exceeds Sc, and when the condition is satisfied, F-measure and SP value are recorded.After all combinations are checked, the combination of parameters having best F-measure is selected and the process shifts to fine-tuning of parameters.
In the fine-tuning step, searching of optimum parameterization is executed around the already estimated parameters while gradually narrowing the step size of parameters.In this step, same scheme of global parameter search is repeatedly applied.When the step value reached to a predefined threshold, the process is terminated.
On the other hand, when the target value of F-measure is specified instead of Sc, searching of optimum Sc which satisfies minimum targeted F-measure is also added to the process.This procedure is started by assuming an appropriate initial value of Sc, then the refinement of Sc is carried out by using bisection method while applying global parameter search.This procedure is considerably time-consuming task but enables to realize higher versatile approach.
As described above, the proposed method does not depend on the settings of SP, and this technique is possible to extend to the process of multiple levels which means using different userspecified parameters in each stage by continuing the process from the optimized front level.

Experimental results and discussion
In order to estimate the optimum parameters which was suitable for reconstitution of the shape of the building region, several experiments were conducted in different targeted F-measures from 0.8 to 0.95.
In Fig. 7, (a) and (b) show the processing results by different Fmeasures of 0.9 and 0.95 respectively.Fig. 7(c) illustrates the training area used for search of optimum parameterization, and Fig. 7(d) indicates the estimated building area by optimum parameters when target F-measure was specified as 0.9.We can ascertain that setting of too large F-value (=0.95) lead to excessive segmentation, therefore F value is necessary to set a reasonable value (e.g.value of 0.9).We can also confirm that reconstituted shapes of buildings shows giving acceptable processing results.
In Table 2, estimated optimum parameters are listed respectively.When the targeted F-measure is different, we can find quite diverse combination of parameters were selected.In smaller targeted F-measure, weight of shape component is dominative.By contrast, in higher targeted F-measure, the influence of colour component is almost dominant.We can also confirm that the size of SP does not affect directly on the average size of derived objects, and remarkable effects on introducing aspect ratio proposed in this study to reconstitute the given shape of targeted buildings.

CONCLUSIONS
In this study, we have addressed several extensions or improvements on multi-scale image segmentation especially from the view point of improving the reproducibility for derived shapes of objects.
At first, we have introduced the constraint for spectral evaluation so as not to make the improper merging of adjacent regions.The application of this constraint brought apparent improvements in excessive merging between dissimilar regions compared to the conventional approach.In second, we have extended the evaluation for smoothness criterion by modifying the definition on extent of the object.It was demonstrated that more diversity could be realized in shapes of derived objects by this extension.
In the third, we have developed shape criterion called aspect ratio as a new index for controlling shape's property.This criterion enabled to give a new property which regulates the aspect ratio on derived objects.This criterion contributes to realize more accurate, flexible and diverse image segmentation results and also showed the effectiveness in several experiments on selecting optimum parameterization.Finally, we have proposed an approach for automatic selection of optimum parameterization.This approach is based on comparing the processing results with the training area marked in advance.It can obtain an optimum parameterization in the state of satisfying either the resulting average area size of objects or F-measure over a predefined threshold.From the qualitative or quantitative evaluation, we could confirm the effectiveness and advantages of our proposed methods compared to the conventional approaches.
length of object d = shorter side length of boundary box in object

Figure 1 .
Figure 1.Application results based on different rules of merging objects

iσ
= standard deviation of i-th band within object μ Δ , σ Δ = thresholds Here thresholds are fixed or monotonically increased based on scale parameter.
fitting (d) Local mutual best fitting (b) Histogram of p (c) Histogram of q (d) Histogram of p and q (a) Target objects (p and q) Fig.4indicates the comparisons on extension for smoothness criterion by modifying the definition used for extent of objects discussed in section 3.2.For the purpose of comparing the differences clearly, an image shown in Fig.4(a) with relatively uniform texture was processed in weights setting of w shape = 0.8 in Eq. (1) and w smth = 0.8 in Eq. (2).Fig.4, (b), (c) and (d) are the results obtained by using length of shorter side, longer side and diagonal in bounding box, respectively.

Figure 7 .
Figure 7. Processing results by automatic parameterization

Table 2 .
Estimated optimum combination of parameters in different F-measures