SURFACE HANDWRITING ENHANCEMENT OF ARTIFACTS BASED ONMANIFOLD LEARNING AND MIXED PIXEL DECOMPOSITION

Written information on the surface of cultural relics can record important historical events. Due to the influence of natural and human factors, the surface of cultural relics fades and the words are difficult to identify. Take advantage of the hyperspectral data image and spectral unity and wide spectral range, a cultural relics surface handwriting enhancement method based on manifold learning and mixed pixel decomposition was proposed. First, the minimum noise fraction (MNF) transformation was carried out on the hyperspectral image, and then the top 10 bands were selected for inverse MNF transformation to reduce noise of the hyperspectral image. Then, the reconstructed image was dimensionally reduced by locally linear embedding (LLE) to obtain a gray image with the maximum amount of information. At the same time, the spectral features of the handwriting and background area in the reconstructed image were analysed. The automatic target generation process (ATGP) was adopted for endmember extraction on the reconstructed image to identify the endmember of handwriting. The abundance map of handwriting area was obtained by the fully constrained least squares (FCLS). Finally, the gray image and the abundance map of the handwriting region were weighted together to obtain the handwriting enhanced image. The true color image was synthesized from the reconstructed image, Then the true color image and the handwriting enhancement image were fused to obtain the handwritting fusion image. The hyperspectral image of a faded text in Shuozhou City, Shanxi Province, China, was used as an example for verification, and the results showed that the method can effectively enhance the text on the surface of the artifacts.


INTRODUCTION
Cultural relics are relics with historical, artistic and scientific value left by human beings in social activities. They are the precious historical and cultural heritage of mankind. Murals, calligraphy and painting, as the majority kinds of cultural relics, can record the spiritual and cultural life, local customs and important historical events of the ancients. The surface of these kinds of cultural relics is usually described and modified by words. We can find the real history and understand the unique culture of different periods through the text information on the surface of culture relics. With the passage of time, most cultural relics are faced with the dual influence of environmental and human factors, resulting in the fading of the pigment layer on the surface. Although the texts information plays an important role in identifying the age of cultural relics and analyzing the historical background, it is difficult to identify the faded words with the naked eye. Therefore, the exploration of the text information on the surface of cultural relics has become a hotspot in the protection of cultural relics.
In recent years, many scholars have carried out a series of researches on the extraction and enhancement methods of handwriting on the surface of cultural relics. Han Y (2019) proposed a combination method suitable for Tibetan character contour extraction, which improved the poor binarization effect of Tibetan historical document images. A Tournié (2019) obtained the hidden ancient Greek characters by processing the short wave infrared hyperspectral imaging data. Lv J Z (2013) acquired the text skeleton of the inscription image through image segmentation and homomorphic filtering, extracted the text outline by using a text trunk extraction method, and finally obtained the clearer text in the inscription image through image fusion. K.Durga Devi (2019) effectively distinguished the foreground features of stone carving images by using the adaptive threshold processing method based on fuzzy entropy. Wu T (2019) used sequential maximum angle convex cone (SMACC) to obtain endmembers, analyzed the changes of spectral curves of wood and ink, extracted them using density segmentation, and finally found more than 60 Chinese characters on the wood chips of unearthed tombs. Dai T (2019) analyzed the edge width and brightness distribution of the seal scanning image to established a decision tree classification model using the image edge features and structural features. They combined the boundary width features and seal category labels to remove the seal boundary. The characters were extracted from the seal image to better preserve the text area. Narang S (2019) developed a system for recognizing ancient Sanskrit, including image binarization, character segmentation and character recognition. Yang X Y (2020) extracted the faded characters on the surface of stone carvings and tombs by constructing the black handwriting enhancement index, and recognized the extracted characters by convolution neural network.
At present, few studies use spatial information and spectral information to extract handwriting at the same time. The research on text extraction mostly focuses on digital images, and the spectral features used were limited. On the other hand, although some method extracted the texts by the handwriting enhancement index, they failed to make full use of the spatial features of the image. Thus, taking advantage of the hyperspectral data image and spectral unity and wide spectral range, we proposed a handwriting enhancement method on the surface of cultural relics based on manifold learning and mixed pixel decomposition. The character features were extracted by LLE nonlinear dimensionality reduction. Then, ATGP was adopted for endmember extraction on the reconstructed image to identify the endmember of handwriting. The abundance map of handwriting area was obtained by FCLS. This paper combined the spatial and spectral characteristics of hyperspectral data, which can effectively enhance the faded text and provide a reference for the archaeological research of cultural relics.

Data Acquisition
The hyperspectral images were obtained from a burial text material in Shuozhou City, Shanxi Province, China. Because the material was placed in the burial area to ensure the land ownership of the deceased, it was called tomb contract. It originated in the Han Dynasty (B.C. 202) and has been used until the Qing Dynasties (A.D. 1912). Tomb contracts were usually made of green and square bricks, and the words on them were ink books, which belonged to the standard writing form. The contents of the words generally included the location of the characters and the time of the construction of the tomb. However, with the passage of time, the surface of tomb contract has faded, which makes it difficult to enhance and extract useful information.
The VNIR400H ground-based hyperspectral imager was used to acquire the hyperspectral image with the spectral resolution of 2.8 nm, 1040 bands, and the spectral range spanning from 400 nm to1000 nm. The true color image combined by red, green and blue band selected from the hyperspectral bands was shown in Figure 1, where most of the words are blurred and difficult to distinguish. Therefore, a handwriting enhancement method that takes into account the spatial and spectral features of hyperspectral images was proposed. It provides a scientific basis for archaeological research.

Data Preprocessing
In the process of acquiring hyperspectral images, the imager is about 1 m away from the target. At such short distance, the influence of the atmosphere on the radiance is negligible. The preprocess mainly includes reflectance correction and image denoising. The reflectance correction is shown in equation (1).
is the standard reflector data under the same environment; 槨 E is the dark current data.
The original hyperspectral data has 1040 bands. Due to the influence of the instrument itself, the band noise at both ends is large. Therefore, the first and last 50 bands are removed. The hyperspectral image with remained 940 bands was transformed by MNF. Then, the top 10 bands selected from the data after MNF were transformed by inverse MNF to produce a new hyperspectral image with the original band dimension of 940 and lower noise.

METHODOLOGY
The overall process is shown in Figure 2.

Figure 2. Overall process
The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France Firstly, the raw digital number (DN) of hyperspectral image was rectified to reflectance by the image of standard reflector and dark current data collected under the same condition. The MNF transformation was carried out on the reflectance image, and then the top 10 bands were selected for inverse MNF transformation for reconstruction to reduce noise of the hyperspectral image. Secondly, the reconstructed image was dimensionally reduced by locally linear embedding LLE to obtain ten bands, from which the band with maximum gradient was selected and stretched to obtain the normalized image. At the same time, the spectral features of the handwriting and background area in the reconstructed image were analysed. The ATGP was adopted for endmember extraction on the reconstructed image to identify the endmember of handwriting. The abundance map of handwriting area was obtained by the FCLS. Finally, the normalized image and the abundance map of the handwriting region were weighted together to obtain the handwriting enhanced image. The true color image was synthesized from the reconstructed image. Then the true color image and the handwriting enhancement image were fused to obtain the handwritting fusion image.

Nonlinear Dimensionality Reduction Based on LLE
LLE is a nonlinear dimensionality reduction algorithm based on manifold learning. Compared with traditional data dimensionality reduction methods such as PCA and LDA, LLE pays more attention to maintaining the local linear characteristics of samples during dimensionality reduction (Li et al., 2021;Wang et al., 2021). Therefore, LLE is widely used in image recognition, high-dimensional data visualization and other fields (Yao et al., 2019). The basic idea of LLE algorithm assumes that each data sample point and its nearest neighbour in the high-dimensional observation space are located in a linear local neighbourhood of the manifold, then each data point can be linearly represented by the nearest neighbour in its neighbourhood. This method minimizes the reconstruction error and obtains the reconstruction weight. Then, by keeping the same reconstruction weight, the corresponding internal low dimensional spatial coordinate representation is calculated. In this way, the high-dimensional observation space data can be projected into the data in a low-dimensional manifold space to reduce dimension. (Qiu J R, 2019).
The LLE algorithm is used for nonlinear dimensionality reduction of handwriting hyperspectral images. X = is the data set of high-dimensional observation space , which is in the low-dimensional space with inherent dimension D. Therefore, Y = is the coordinate representation on the low dimensional embedding space .
First, the neighborhood points are selected. For a given high-dimensional data set X = , = , use the Euclidean distance formula to find k adjacent points in the neighborhood of each data point , which are recorded as .
Secondly, the reconstruction weight is calculated. For each data point , LLE algorithm needs to calculate the reconstruction weight between it and the neighborhood point , which minimizes the square of the local reconstruction error of .
Where ⦻ is the weight between and . The reconstruction weight W of X is calculated through formula (3), and the reconstruction weight of each data point forms the reconstruction weight matrix.
Finally, the low dimensional embedding y is obtained. Through the obtained weight matrix W, the low-dimensional embedded coordinate y of each data point is found to minimize the square sum function of reconstruction error.

Mixed Pixel Decomposition
Each pigment has unique spectral characteristics. In ancient times, such characters were mostly written with cinnabar pigment. Because the surface was covered with soil, the observed spectrum should be the mixed spectrum formed by cinnabar, surface soil and brick below.
Therefore, ATGP was used for endmember extraction (Shang et al., 2021;Lei et al., 2019). Firstly, a feature space composed of noise in the image or other feature vectors that need to be suppressed is constructed. Then, the pixel is projected to the orthogonal space orthogonal to the characteristic subspace composed of interference vector by orthogonal projection operator. Finally, according to the theory of convex geometry, the brightest pixel in the image is calculated.
After the endmember spectrum is known, the accuracy of abundance inversion has an important impact on the unmixing results. The least squares algorithm with abundance constraints is the most widely used abundance inversion method at present. According to the degree of abundance constraints, the algorithm can be divided into four kinds: unconstrained least square (UCLS), sum-to-one constrained least square (SCLS), nonnegative constrained least square (NCLS) and FCLS (Qi et al., 2021). When the constraint on abundance is not considered, the unconstrained solution is as follows: Where Y is the image matrix and is the endmember matrix.
The constraint solution with sum to one is as follows: Where I is the identity matrix of order p. I = is a column vector with all 1.
The iterative relationship of NCLS is as follows: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B2-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France The solution satisfying both "nonnegative" and "sum to one" constraints is called fully constrained least squares solution. Just bring (6) into (7).
Because FCLS fully considers the practical significance of abundance, the solution accuracy is the best. This study applies this method to solve the abundance of handwriting area.

Handwriting Enhancement
LLE was used to reduce the dimension of the hyperspectral image after MNF forward and inverse transformation, and the images of the first 10 bands were output. Then, an image with the most abundant information was selected through the average gradient, which was the fourth band image after LLE, as shown in Figure 3. In addition, in order to better observe the effect of nonlinear dimensionality reduction, it was compared with the first band after MNF inverse transformation through the evaluation indexes of average gradient and edge intensity.
The results were shown in Table 1. It can be found that the image with the maximum amount of information after LLE was better than the first band image after MNF noise reduction, and the handwriting was enhanced.  ATGP was used for endmember extraction. After extracting the endmembers of each color region, the abundance map of handwriting endmember was obtained by FCLS (Song et al., 2021;Khajehrayeni and Ghassemian, 2021), shown in Figure 4. In order to further enhance handwriting, spatial information and spectral information were combined. Firstly, the image with the maximum amount of information after LLE was normalized. Figure 3 and Figure 4 were weighted together to obtain the handwriting enhanced image. During the band addition operation, the weight of the two is set to 0.5. Then, HSV fusion was performed between the handwriting enhanced image and the true color image obtained from the reconstructed image to produce the fusion image. The comparison of handwriting enhanced image, handwriting fusion image, and ortho-image obtained by Nikon D850 digital camera were shown in Figure  5. Compared with the true color image in Figure 1, the text in the handwriting fusion image has been significantly enhanced. The average gradient and edge intensity are used to evaluate the enhancement effect of the text. The average gradient G is used to objectively describe the clarity of the image as follows: Where m and n are the width and height of the image, respectively, is the gradient in the x direction, and is the gradient in the y direction.
Edge intensity is an objective description of image edge texture information. The larger the value of E, the better and clearer the image. On the contrary, the image is more blurred. The formula is as follows: Where and are the first-order difference of image edge pixels in x and y directions respectively.

Image
Average Gradient As can be seen from table 2, compared with the true color image, the handwriting fusion image has greatly improved in the two indicators of average gradient and edge intensity, which shows the effectiveness of the handwriting enhancement method proposed.

Method Comparison
Figure 6 compared the results produced by the PCA feature extraction algorithm and the handwriting enhancement method. The result of PCA dimensionality reduction is a gray image, which is compared with the handwriting enhancement image. Table 3 shows the calculation results of average gradient and edge intensity of handwriting enhancement images obtained by two methods.  Table 3. Comparison of image information of different methods It can be seen from table 3 that the value of handwriting enhancement image is higher than that of PCA. It shows that the method proposed in this paper can better enhance the faded text on the surface of cultural relics.

CONCLUSION
Aiming at the problem of text fading and blurring on the surface of cultural relics, a method combining spatial information and spectral information of hyperspectral image is proposed to enhance handwriting. By using the advantages of hyperspectral data image, LLE nonlinear dimensionality reduction and FCLS are combined to reveal the faded text on the surface of tomb contract. The blurred text is effectively enhanced and the readability of the text is increased. The experimental results show that the method can make the image brighter and the text lines clearer. Compared with other methods, the method has higher average gradient and edge intensity. The proposed method can be applied to the surface of cultural relics with fading pigment layer, and lay a technical foundation for cultural relics protection workers to analyze cultural relics and interpret characters. Using historical documents to assist text recognition and improve the recognition rate will be the research direction in the future.