EXTENDED MULTISCALE IMAGE SEGMENTATION FOR CASTELLATED WALL MANAGEMENT

Castellated walls are positioned as tangible cultural heritage, which require regular maintenance to preserve their original state. For the demolition and repair work of the castellated wall, it is necessary to identify the individual stones constituting the wall. However, conventional approaches using laser scanning or integrated circuits (IC) tags were very time-consuming and cumbersome. Therefore, we herein propose an efficient approach for castellated wall management based on an extended multiscale image segmentation technique. In this approach, individual stone polygons are extracted from the castellated wall image and are associated with a stone management database. First, to improve the performance of the extraction of individual stone polygons having a convex shape, we developed a new shape criterion named convex hull fitness in the image segmentation process and confirmed its effectiveness. Next, we discussed the stone management database and its beneficial utilization in the repair work of castellated walls. Subsequently, we proposed irregular-shape indexes that are helpful for evaluating the stone shape and the stability of the stone arrangement state in castellated walls. Finally, we demonstrated an application of the proposed method for a typical castellated wall in Japan. Consequently, we confirmed that the stone polygons can be extracted with an acceptable level. Further, the condition of the shapes and the layout of the stones could be visually judged with the proposed irregular-shape indexes.


INTRODUCTION
Castellated wall is a type of masonry building and is positioned as tangible cultural heritage, which require regular maintenance to preserve their originality and to conserve them for ages to come.To protect these cultural heritages from deterioration, repair works are performed when required.Moreover, in the worst case scenario of natural calamities such as severe earthquakes, the collapsed castellated walls might require complete restoration as well.For the repair or restoration of the castellated walls to its original form, the archived database of the original architectural design of the castellated wall is a prerequisite.Traditionally, two major approaches are used to create these databases.
The first one is to generate a front view of the stone walls by laser scanning them (Křemen et al., 2011;Vacca et al., 2012).With this approach, although it is possible to manage the distribution state of the stones with respect to the surface of the stone wall, the individual properties of each stone (e.g., surface texture, planer dimension, georeferenced position, etc.) cannot be properly managed.
Another approach is to attach integrated circuits (IC) tags for identifying the individual stones constituting the stone wall before dismantling the castellated walls for restoration (Ryu et al., 2014).In this second approach, each individual stone has to be photographed before demolition.With this approach, the individual properties of the stones can be managed; however, it is cumbersome to attach IC tags on each stone and to capture the images individually, and to sort them uniformly.Additionally, another shortcoming of this technique is that the IC tag attachment cannot be performed in advance in a situation where the stone walls collapse suddenly due to natural disasters.
We herein propose a novel method to solve these problems.In our proposed method, an orthographic projected elevation view of a castellated wall is used to extract the individual stone polygons.To realize this procedure automatically, an extended multiscale image segmentation technique is applied.To achieve preferable results, we propose an original shape criterion named as convex hull fitness in this process, which is newly introduced by considering the shape property of the stone.The extracted stone polygons are stored as fundamental components of the castellated wall in the stone management database and are utilized for an efficient restoration work.Moreover, we propose three kinds of indexes named as irregular shapes that evaluate the shape of the stone and the stability of the stone arrangement state in a castellated wall.Finally, we discuss and introduce an image-segmentation based stone management database and the applications for an efficient castellated wall management.We applied the proposed method to an actual castellated wall image and confirmed its practicality.

Principle of Multiscale Image Segmentation
The multiscale image segmentation technique is used for extracting individual stone polygons from a castellated wall image.Here, the summary of this method is described and the improvements and extensions are introduced.Fig. 2 illustrates the flowchart of this procedure.
The multiscale image segmentation is a kind of region-merge approach.Each region (object) starts from one pixel at the initial stage.The decision to merge the neighboring regions is based on evaluating the changes in heterogeneity including both the spectral component and shape component in objects, as defined in the following: where F color = heterogeneity in spectral component F shape = heterogeneity in shape component w color = weight for F color w shape = weight for F shape The variable F color is defined using the average and/or standard deviation calculated from each band of pixels within the objects as follows: where p, q = suffix that means two objects before merge r = suffix that means object after merging p and q n = number of pixels within an object σ i = average and/or standard deviation of the i-th band in an object wc i = weight for the i-th band N = number of bands The variable F shape is calculated using several shape criteria that regulate geometric properties for the derived shapes of objects.
where f i = heterogeneity in the i-th shape criterion ws i = weight for the i-th shape criterion K = number of shape criteria In the original study (Baatz and Schäpe, 2000) and subsequent researches (e.g., Chen et al., 2005;Esch et al., 2008;Li et al., 2009), two types of shape criteria were defined: compactness (f cmpct ) and smoothness (f smth ).
where n = number of pixels in an object l = boundary length of an object d = shorter side length of boundary box in an object Compactness is defined by the perimeter and number of pixels in objects and is used for evaluating the roundness of an object.This means that if the weight of compactness is set larger, the shape of an object approaches a circle.
Smoothness is defined by the perimeter and the shorter side length of the minimum boundary box in objects.This criterion attempts to generate an object with shorter perimeter for the same-sized region.
In our previous study, we indicated the effectiveness of using the diagonal length in the bounding box of an object as d instead of the shorter side length (Sakamoto et al., 2016).Both theoretically and practically, when the shorter side length is applied, it tends to generate slimmed and diagonally distributed shapes of objects for their boundary boxes.However, when using the diagonal length, the rhomboid shape is theoretically generated.As this study aims to extract the shape of stones in a castellated wall, we adopt the latter indicator as d.
Additionally, in our previous work, we introduced another criterion called aspect ratio (f aspct ), which is defined as follows: where a = user-defined aspect ratio dx = length of x-direction in bounding box of object dy = length of y-direction in bounding box of object This criterion can be used not for the purpose of defining the outline of the object, but to prevent its existence range from varying.Therefore, this criterion has a remarkable effect when used in combination with other shape criteria.
The degree of object merge is regulated by a parameter called the scale parameter (SP), which also affects the size of the derived objects indirectly.The merge process of two neighboring regions is performed when the following two conditions are satisfied.The first one is that the evaluation value by Eq. ( 1) does not exceed the square value of SP.The other is that the neighboring objects satisfy the condition of the local mutual best fitting.The latter condition means that two objects under evaluation give the best evaluation value to each other in all their combination of neighboring regions.
The SP starts from a small value (e.g., value of 1) and then all the possible merging of the objects is performed.If no objects can be processed, the SP is incremented.The same processes are applied repeatedly until the SP reaches the predefined value.
The weights for each parameter (w color , w shape , and ws i ) can be changed in an arbitrary stage of the SP, if necessary.

Introduction of Size Restriction on Merged Object
As the SP cannot specify the size of the generated objects directly, it can cause large objects with similar colors or textures to be generated frequently.Therefore, we have introduced constraints on the size of the generated object by considering inherent features derived from the targeted stones of a castellated wall. ( (3) This procedure is realized by checking in advance whether the candidate object generated after merging exceeds the predefined upper limit values of the dimensions of a circumscribing rectangle (width and height) and the area of the stone polygon, in which the parameters are specified based on the actual processed stones.

Introduction of New Shape Criterion
By observing the stone appearance to be processed, we found that almost all stones have a convex shape.To realizing a more desirable extraction of individual stone polygons, we have considered introducing this property as the fourth shapeevaluation criterion.As a result of examining and evaluating several combinations assumed as indexes, which characterize the properties for the convex shape, we found that the formulation shown in the following is effective.Here, we name this criterion as the convex hull fitness (f cnvx ).
where c = number of pixels in the convex hull of an object m = boundary length of the convex hull of an object This formula incorporates and expands the conventional criterion of compactness.If a reasonable interpretation is given , the roundness equals to the ideal convex hull; therefore, it also appears to be effective for evaluating the convex hull property.The related extended variable c is for minimizing the area deviated from the convex hull, and the term including variable m is introduced for the constraint on the perimeter in an object with respect to the perimeter of the convex hull.

Determination of Upper Limit of Scale Parameter
The determination of the SP effective for the extraction of stone polygons can be performed by the following approaches.One is based on the automatic method that we proposed previously (Sakamoto et al., 2016).In this approach, the optimum selection of processing parameters is realized quantitatively using the training area and user-specified evaluation parameter.The training area is obtained by cutting out part of the target image, and the user-specified parameter required for the process is either of the lower limits of the average area in all objects after the segmentation process or the target value of F-measure.By this approach, the SP is calculated simultaneously with other optimal processing parameters.
In another approach, parameters other than the SP are set in advance in processing, and the segmentation results for each SP are maintained and overlaid on the target image.Subsequently, the SP is adopted when the acceptable segmentation is obtained by visual inspection.Although it lacks the objectivity of processing, the training data does not need to be prepared in advance; therefore, it is easy to operate and is advantageous for practical uses.

Contour Tracing and Smoothing
After the image-segmentation processing is completed, contour tracing is applied for extracting the shape of each object as the polygon data.Further, the contour smoothing of polygons is executed in the next step as necessary.

Comparison of Image Segmentation Results
To evaluate the performance of the stone polygon extraction, multiscale image segmentation including the newly proposed shape criterion was applied to an actual castellated wall, as shown in Fig. 3. Here, we compared the processing results for the combinations of shape criteria excluding the aspect ratio.This is because the aspect ratio primarily affects only the existence range of the object to be generated.
Fig. 4 shows the segmentation results generated by different criteria.In the processing, except for the case of Fig. 4(a), we set the weight of w shape = 0.5 in Eq. ( 1), and the conditions other than the ratio of weights for the shape criterion were set to be the same.Fig. 4(a) is the result when no shape criterion is considered and shows that the shape of the stones is not well reflected.Similarly, in the case when only the smoothness criterion was applied, we observed that the extraction of the stone shape is slightly inferior.
As shown in Fig. 4(c) and Fig. 4(d), both compactness and convex hull fitness reproduce the stone shape properly even when used alone.When comparing these indexes, the latter tend to cause region merging comparatively.From Fig. 4(e) and Fig. 4(f), we confirmed that the effectiveness of the smoothness criterion is increased by combining with other indicators especially in the case of convex hull fitness.
Fig. 4(g) is the result of combining two indexes that show acceptable results even when used individually; however, we found that by combining these, the stone polygons can be extracted with a better aggregate state.In Fig. 4(h), which is a result of mixing three indexes equally, we confirmed that most stone polygons are extracted satisfactorily.
In the processing results, cases in which the stone polygons that should ideally be integrated remain divided into a plurality of regions exist.The primary reason is that whether the state is improved after merging only two objects is a criterion for judgment.Therefore, we considered that improvements such as introducing the state change when merging multiple neighboring objects into the evaluation is effective, and will be investigated further as the future work.

Extraction of Stone Polygons in Castellated Wall
In our proposed method, the castellated walls are initially photographed from multiple viewpoints with a digital camera on hand or mounted on a UAV.Subsequently, the aerial triangulation is executed for estimating the position and orientation of the camera by applying the structure from motion (SfM) technique.Next, the front view orthographic projection of the castellated wall is generated utilizing the multiview stereo (MVS) processing technique.Subsequently, each stone polygon from the front-view orthographic projected images is extracted by adapting the extended multiscale image segmentation method proposed in this study.Stones that were not properly processed can be manually corrected as necessary.

Stone Management Database and Its Application
From the extracted stone polygons, the stone identification (ID) number, three-dimensional contour, center of gravity, area size, planar dimension, neighboring relationship between stones, image texture, etc. can be obtained as direct information.
Further, additional information such as weight, stone quality, processing status, placement position, archival record, etc. obtained before dismantling the stone wall can also be recorded in the stone management database.
The stone management database created based on the proposed method enables the efficiency improvement in the castellated stone wall repair.For example, if the castellated walls were collapsed by an earthquake, the images of the collapsed stone wall are taken and the feature points of each stone of the collapsed wall are derived based on invariant feature extractors such as scale invariant feature transform (SIFT), speed up robust feature (SURF), and features from accelerated segment test (FAST) (El-gayar et al., 2013;Karami et al., 2017).Similar feature points could also be derived from the front-view orthographic projected image in the database.By matching these feature points, the position of the collapsed stone to be arranged by restoration work can be efficiently specified.
As the individual stones in the stone management database are managed in association with the georeferenced position, the number of stones to be checked and verified can be reduced.As a concrete means for matching the extracted feature points, affine-SIFT (ASIFT) (Morel et al., 2009) can be applied, which is tolerant to deformation by the shooting direction of the stone image.

Introduction of Irregular-shape Index
As the contour of each stone could be obtained by the proposed method, it is also possible to calculate the indexes that contribute in evaluating the stone shape or the stability of the stone arrangement state in a castellated wall.We propose three kinds of indexes named as irregular-shape indexes I k , typically defined as the following: where k = suffix that means the index type S k = area used for calculating the k-th index S p = area of the stone polygon The first index is the horizontal rectangular irregularity (HRI) calculated using the area of the horizontal circumscribed rectangle of the stone polygon (depicted as S h in Fig. 5(a)) as S k .This index is an indicator that exhibits how ideally the stones are placed horizontally with respect to a flat land.The smaller the value, the more ideally are the stones stacked horizontally.
The second index is the minimum rectangular irregularity (MRI), which is calculated when the minimum area of the rotated circumscribed rectangle to stone polygon is given (S m is used as S k in Fig. 5(b)).This index represents whether the stone itself is close to a rectangle irrespective of the stack angle of the stone.
The last index is the convex hull irregularity (CHI) that is calculated as the ratio of the non-polygon area to the convex hull of the polygon (S c is used as S k in Fig. 5(c)).This index indicates the deviation of the polygon from the convex hull and shows the similar trend as the MRI.
Figure 5. Relationship between stone polygon and circumscribed region for calculating irregular-shape indexes Fig. 6 shows the examples in which the indexes above are calculated with respect to the processing result of Fig. 4(c) and Fig. 4(d).As shown from these processing results, the visualization based on irregular-shape indexes can also be effectively used for the detection of objects with inappropriate segmentation.

Advanced Usage of Irregular-shape Index
As a further developmental usage, by using these indexes, it is possible to evaluate the local difference of the shape and the state of each stone from comparing with the overall trend in the castellated wall.This is realized by comparing these indicators for each stone with respect to the average values of the irregular-shape indexes obtained from all stone groups constituting the stone wall (shown in Fig. 10(a) and Fig. 10(b)).These indicators can also be used for the comparison or analysis with the existing survey results (shown in Fig. 10(c)).

Application Examples of Irregular-shape Index
Here, examples of the calculation result of the irregular-shape index described in section 3.3 with respect to Fig. From the visualized results of the HRI and CHI, we confirmed that these indexes also contribute to determining the polygons that need to be modified owing to the defects from the multiscale image segmentation processing.7 shows the relative frequency to the convex hull occupancy of the stone polygons for the result of Fig. 6(e) and Fig. 6(f).By comparing these graphs, we verified that the convex hull occupancy ratio by the shape criterion of convex hull fitness is increased as a whole as compared with the conventional criterion of compactness.

Example of Castellated Wall Management
Fig. 8 shows the result of applying the proposed method to a typical castellated wall in Japan.Even for a practical image, most stone polygons could be extracted satisfactorily.The managed properties include geometric information (e.g., georeferenced position, three-dimensional contour, and planar dimension), weight, stone quality, processing state, placement state, archival record, etc. Fig. 10(a) and Fig. 10(b) indicate the results of the extended usage of the irregular-shape index described in section 3.4.Further, Fig. 10(c) shows the existing survey chart, where the illustrated lines indicate the boundaries of several types of discontinuous surfaces.These boundaries depend on the original approach of loading stones and the repair work performed in the past.We found some parts where the irregular-shape indexes respond to these lines.This implies that the proposed indexes are useful for visually interpreting the situation of a masonry wall.

CONCLUSIONS
Herein, we have proposed an efficient multiscale imagesegmentation-based castellated wall management, in which stone polygons are extracted from the orthographic-projected castellated wall image and are associated with the stone management database.
First, we addressed the introduction of a new shape criterion named convex hull fitness, which aimed at realizing a more desirable extraction of individual stone polygons with convex shape.We confirmed that the combinational use of this criterion and/or conventional shape criteria brought acceptable results.
Next, we described the data acquisition means and preprocessing for the castellated wall images used for extracting the stone polygons, and the procedure of polygon extraction.In addition, we discussed the stone management database and its effective utilization in restoration work.Subsequently, we proposed three kinds of irregular-shape indexes, which are effective for grasping the stone shape and the stability of the stone arrangement state in a castellated wall.The visualization with these indexes is also useful for determining the stone polygons that need to be modified owing to defects in image segmentation processing.
Finally, we demonstrated an example of castellated wall management for a typical castellated wall in Japan.In this application, the properties of each stone polygon are managed in a database and used for periodic restoration work or other purposes in the future.Based on the proposed irregular-shape indexes, we confirmed that the individual shapes and stacking conditions of the stones can be visually grasped in contrast to the existing survey chart.

Figure 1 .
Figure 1.Example of a typical castellated wall

Figure 2 .
Figure 2. Flowchart of stone polygon extraction by extended multiscale image segmentation Figure 4. Image segmentation results by the different weights for combination of three shape criteria Fig. 6(a)  andFig.6(b)  are the calculated results of the HRI.From these figures, the distribution of the stacked stones with inclination can be well grasped.Similarly, Fig.6(c) andFig.6(d)  show the calculated results of the MRI.Many stones with rectangular shape were present in the used test image; from these figures, the stones having a non-square shape can be

Figure 6 .
Figure 6.Comparison of irregular-shape indexes for different shape criteria

Fig. 9
Fig. 9 represents a database example for the castellated wall management.Fig. 9(a) and Fig. 9(b) show the center of gravity and the given ID of each stone polygon, respectively.Each stone polygon is associated with a stone database by its ID.Fig. 9(c) is an example of the stone properties stored in the database.The managed properties include geometric information (e.g., georeferenced position, three-dimensional contour, and planar dimension), weight, stone quality, processing state, placement state, archival record, etc.

Figure 8 .
Figure 8. Extraction of individual stone polygons by the proposed method