AUTOMATIC IDENTIFICATION OF MINERALS IN IMAGES OF POLISHED SECTIONS

Automatic identification of minerals in images of polished section is highly demanded in exploratory geology as it can provide a significant reduction in time spent in the study of ores and eliminate the factor of misdiagnosis of minerals. The development of algorithms for automatic analysis of images of polished sections makes it possible to create of a universal tool for comparing ores from different deposits, which is also much in demand. The main contribution of this paper can be summed up in three parts: i) creation of LumenStone dataset (https://imaging.cs.msu.ru/en/research/geology/lumenstone) which unites high-quality geological images of different mineral associations and provides pixel-level semantic segmentation masks, ii) development of CNNbased neural network for automatic identification of minerals in images of polished sections, iii) implementation of software tool with graphical user interface that can be used by expert geologists to perform an automatic analysis of polished sections images.


INTRODUCTION
Practical mineragraphy is a time-consuming discipline that requires, on the one hand, a highly qualified specialist, and on the other, a large number of analyzes. Creation of a mechanism for automated identification and counting of minerals in samples of geological ores has two main goals: • elimination of the human factor in the analysis of deposits at the stage of prospecting and operational exploration and a significant reduction in the time spent on prospecting and exploration of deposits; • the formation of an extensive database of oremineralogical features, reduced to a single standard, which will allow statistically analyzing large volumes of geological data in order to conduct ore-formation analysis and metallogenic constructions.
The second aspect of the application of such a mechanism is the description of the structural and textural relationships between ore minerals, which play an important role in the formation of the beneficiation chain, where it is important to clearly understand how the ore and host minerals are related to each other and how they can be separated.
Attempts to create software for diagnostics of ore minerals based on the results of analysis of micrographs have been undertaken for a long time. The first pioneering works appeared in the second half of the 90s of the XX century (Marschallinger, 1997). All major existing software solutions for determining minerals from photographs of polished sections can be divided into two types: 1. Using reflection intensity together with color characteristics expressed in RGB or LAB color space (López-Benito et al., 2020); * Corresponding author 2. using statistical principles for determining minerals in a specific sample (Berrezueta et al., 2016), (Köse et al., 2012).
Although both type of methods can deal with some problems of mineral identification, each of them has a number of significant disadvantages. The method based on color and reflectivity cannot detect all minerals, since there are "overlapping" or extremely similar pairs of minerals, e.g. pyrite-marcasite, pyritearsenopyrite, covelline-chalcopyrite, etc. The methods, that are based on statistical principles, can only work within a certain sample of minerals and require a new calibration taking into account chemical impurities in minerals for each new geological object.
One of the most effective ways to overcome this shortcoming and to achieve the desired result is applying convolutional neural networks, as it is still possible to use single algorithm when working with different mineral associations.

USED DATA
In order to create and compare different algorithms for automatic mineral identification in images of polished sections we created a LumenStone dataset (https://imaging.cs.msu. ru/en/research/geology/lumenstone), which provides pixel-level segmentation masks for images containing several minerals. LumenStone dataset consists of several subsets that correspond to different mineral associations. One of the emerging problems in geological image segmentation is data imbalance. Some minerals are found in nature much less often than others. The percentage of minerals in S1 and S2 subsets of LumenStone in pixels is shown in Table 1.
It is worth noting that despite the amount of minerals analyzed by an expert geologist is much bigger, in this work we consider only 10 classes (ores) of S1 and S2 subsets of LumenStone, since collecting a large dataset of polished sections with manual annotation of all used minerals is unavailable and too resource intensive.

PROPOSED CNN
In this work we use a convolutional neural network based on a popular U-Net architecture (Ronneberger et al., 2015) with batch normalization layers (Ioffe and Szegedy, 2015) to perform semantic segmentation of polished sections images. To increase the learning speed and overcome the problem of vanishing gradients we add residual connections inside convolution blocks similar to (Zhang et al., 2018). The difference is in the order of normalization and activation layers. Figure 1 demonstrates the arhitecture of the used ConvRes block, which is the fundemantal building block for the proposed 5-layered CNN architecture ( Figure 2). In order to match the number of channel dimensions of inputs for summation layer in ConvRes block we use 1 × 1 convolution with number of filters n corresponding to the second convolution layer in the block. The encoder way of the used architecture ( Figure 2) doubles the number of channels for each subsequent ConvRes block and halves it in the decoder part of the network.
As it was already mentioned, one of the main emerging problems in geological image segmentation is data imbalance. For example, as it can be seen from Table 1, in LumenStone S1 the number of pixels, corresponding to chalcopyrite (Ccp) is 17 times smaller than number of pixels, corresponding to background (BG). Using conventional patch-based approaches while training a neural network with unbalanced data will lead to poor results.
In order do overcome this problem of data imbalance we use a modified version of the special data balancing method proposed earlier in (Kochkarev et al., 2020) while training CNN.
In particular in the step of creation the patch probability map the function of distance to the nearest class was replaced by the area of certain class in patch, so that the probability map is calculated as: where M is the semantic mask of image, H, W are the height and the width of the image, s is the size of patch, c is the target class probability map is build for.
The normalized probability map, which is used to generate patches is defined as: The step of choosing class for patch generation was also updated by using the weighted choice function of all present classes instead of choosing the minority class at each step.
Wherein the weights are calculated as the total number of pixels for each class accumulated in patch generator at the current moment. Figure 2. Architecture of the proposed neural network. ConvRes(m) stands for ConvRes block with m filters, MaxPool (2 × 2) for max pooling layer with strides (2, 2), concat stands for depth-wise concatenation layer, upsample stands for 2 times upsample of tensor using transposed convolution, Conv(1 × 1, k) for 1 × 1 convolution with k filters, where k is the number of output channels (classes).
The first modification improves the performance of calculating probability maps for each image, while the second one improves the variance of classes attending on sequentially generated patches, which leads to better training with mini-batches.
The described algorithm allows us to create patch generator, which generates patches of needed size from the train set of images with balanced class distribution on the fly while training segmentation CNN.
The unpleasant feature of this data balancing algorithm is the necessity of keeping all dataset, including images and masks, as well as calculated patch probability maps in memory, which can be rather costly in the case of large datasets. Although, this problem can be slightly eliminated using caching techniques and loading the dataset by parts, we do not use it in the current work.

IMPLEMENTATION DETAILS AND RESULTS
Applying the described above data balancing method while training the model allowed to smooth data balance across classes and increase the recognition accuracy of rarely present minerals. We also used simple data augmentation of flipping and rotation of patches while training CNN.
The used CNN model was implemented using Python 3 programming language and open-source software library for ma-  chine learning Tensorflow 2 (Abadi et al., 2016). The training of the model was performed on a personal computer with Intel Core i7-6700HQ, 16GB RAM and Nvidia GTX 960m with 2 GB of video memory.
The model was trained on 59 images of S1 subset of Lumen-Stone dataset for 26 epochs with Adam optimizer, initial learning rate value of 10 −3 and automatic 10 times decrease of learning rate on plateau. We chose the starting number of filters n = 8 for the proposed CNN model due to the limited hardware resources. The input data for each iteration of training was a batch of 12 patches of 256 × 256 resolution, generated from the training dataset using described above data balancing algorithm.  Figure 3.
After each epoch of training we also calculated the overall accuracy on the test subset of LumenStone S1, which is defined as the ratio of correctly identified pixel to the total number of pixels in test set. We achieved the level of 0.892 for overall accuracy of recognition. The plot of overall accuracy on test subset over epoch is shown in Figure 4.
The result of prediction on one of the images of polished sections from test subset of LumenStone S1 is shown in Figure.5. In error map green color corresponds to the correctly identified pixels and red color corresponds to the misidentified pixels.
As it can be seen from the plots and the final achieved values of IoU and overall accuracy, the minority classes such as galena (Gl), bornite (Brt) and tenantite-tetrahedrite group (Tnt/Ttr) are identified much worse than the other classes. This is quite expected since even though the proposed data balancing algorithm enhances the segmentation results (Kochkarev et al., 2020), yet it is just another oversampling technique that is not able to generate fundamentally new image patches.
The only exception to the described tendency is chalcopyrite mineral (Ccp), which is the most minority class in LumenStone S1 (2.2% of all pixels), but the corresponding IoU value of 0.663 is rather big. This can be explained with the fact that this mineral substantially differs from all other minerals in Lu-menStone S1 in color aspect and it is harder to confuse it with other minerals from dataset.
It also can be noticed (Figure 5), that a lot of errors in prediction happen near the border between neighboring minerals. This can be explained as color distortions at the boundaries of minerals are of a systemic nature. The top border is skewed in green, the bottom in orange. This is caused by the peculiarity of the camera installation. Due to the complex design of the microscope, it is impossible to direct the camera perfectly perpendicular to the sample surface and the camera has a tilt of 89.99 degrees. The manual annotation in these boundary areas is not stable, thus the prediction can differ from the ground-truth annotation.

DEVELOPED SOFTWARE TOOL
For the convenience of using the developed techniques by expert geologists, we implemented a software tool for testing and results' visualization of the proposed CNN-based method of mineral identification. This tool is cross-platform and has a graphical user interface and can be run on a PC or laptop. It consists of two parts: the front-end part, which is implemented with JS and Electron framework, and the back-end part, which is implemented in Python 3. While loading the front-end part the back-end server is run as a child process and the communication between front-end and back-end parts is organized using IO streams.
The back-end part is responsible for server tasks, such as loading CNN model, performing predictions, splitting images into patches and uniting them back. Another task that is also delegated to back-end part is splitting the semantic masks into instance masks. This is done for both ground-truth masks and predictions and allows to get information about each instance Figure 5. The result on one of the images from test subset of LumenStone S1. Up to down: source image, ground-truth mask, predicted mask, error visualization. of mineral on the image, which is very useful for practical tasks of geology.
The front-end part is responsible for loading and rendering the source images of polished sections, visualizing colorized human-friendly ground-truth masks, visualize the predictions of CNN model as well as visualize error maps, if the groundtruth image for the prediction exist.
Besides that the developed tool allows user to see the mineral distribution statistics both for the whole dataset and a single image as well as display statistics of a single mineral instance on a mouse hover.
The screenshot of the developed software tool is shown in Figure 6.

CONCLUSION
In this paper we for the first time present the LumenStone dataset -a dataset of minerals in images of polished section with pixel-level semantic annotation.
We also propose a CNN model for automatic identification of minerals in images of polished sections, which is trained with the help of special oversampling data balancing patch generator. The developed software tool with graphical user interface that can be used by expert geologists to perform an automatic analysis of polished sections images.
The achieved level of overall accuracy of mineral identification is 0.892 for the test subset of LumenStone S1. It should be noted that the results of mineral identification obtained in this paper are preliminary and are planned to be improved in the future.
Further improvements of current work will be focused on the improving the accuracy of recognition of minority minerals by applying class-weighted loss function as well as enlarging the dataset of polished sections of ores and increasing the number of supported minerals for automatic identification.