NON-LINEAR MULTI-FRAME IMAGE DENOISING USING WEIGHTED NUCLEAR NORM MINIMIZATION

: We address the problem of constructing single low noise image from a sequence of multiple noisy images. We use the approach based on ﬁnding and averaging similar blocks in the image and extend it to multiple images. Unlike traditional multi-frame super-resolution algorithms, the block-matching approach does not require computationally expensive motion estimation for multi-frame image denoising. In this work, we use an algorithm based on weighted nuclear minimization for image denoising. The evaluation of the algorithm shows noticeable improvement of image quality when using multiple input images instead of single one. The improvement is the most noticeable in the areas with complex non-repeated structure.


INTRODUCTION
Image denoising is one of the most important image processing problems. Many approaches are used to solve this problem including simple Gaussian filter, anisotropic filter (Perona and Malik, 1990), total variation minimization (Rudin et al., 1992), and more sophisticated algorithms based on finding and averaging similar blocks in the image: Non-local Means (Buades et al., 2005), BM3D (Dabov et al., 2007), LSSC (Mairal et al., 2009), NCSR (Dong et al., 2011). These algorithms assume that the noise is random while the details are similar so block averaging results in noise reduction. Learning-based algorithms including neural networks are also used (Tian et al., 2020), however, they require prior knowledge about the image.
Finding similar blocks using simple mean squared error between pixel intensities may be ineffective in the case of strong noise because it does not take into account image contents. In order to improve the choice of similar blocks, the blocks can be transformed into different basis, for example into frequency domain using discrete cosine transform (Candès and Recht, 2009).
In this paper, we address the problem of constructing single low noise image from a sequence of multiple noisy images. This problem occurs, for example, when taking images in low light conditions. It could be better to take several shots with short exposure and combine multiple images into a high-quality one rather than making a single shot with long exposure and dealing with motion blur.
Image denoising algorithms based on block matching can use blocks from multiple images. It results in better noise suppression. Unlike traditional multi-frame super-resolution algorithms (Farsiu et al., 2004), this approach does not require motion estimation which is computationally expensive. * Corresponding author

WEIGHTED NUCLEAR NORM MINIMIZATION
Let the input noisy image y be divided into a set of overlapping blocks yj of equal size with m pixels in each block. For any block yj we can find similar blocks using either 1 or 2 metric. In the case of strong noise, preliminary noise suppression can be used (Jain et al., 1999).
The first n blocks with the least distance to the block yj including the block itself form the matrix Y ∈ R n×m . It can be represented as a sum where the matrix X is composed of blocks from the noise-free image and N is noise.
The matrix Y can be factorized using the SVD decomposition: where U ∈ R n×n and V ∈ R m×m are real orthogonal matrices, Σ ∈ R n×m is a diagonal matrix with non-negative elements on the main diagonal named singular values: σi ≥ 0. It is always possible to choose the decomposition so that the singular values σi are in descending order. In this case, Σ is uniquely determined by Y . The number of non-zero elements σi equals to the rank of the matrix Y .
The highest singular values correspond to the main information while the smallest values correspond to noise. Since similar blocks in the image have the same base structure, the matrix composed from them should have a low rank for images without noise (Wang et al., 2012). As a result, noise reduction problem is posed as finding a low rank matrix that approximates the matrix constructed from noisy blocks: where λ > 0 is the method parameter, X * is the nuclear The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIV-2/W1-2021 4th Int. Worksh. on "Photogrammetric & computer vision techniques for video surveillance, biometrics and biomedicine", 26-28 April 2021, Moscow, Russia norm: and X F is the Frobenius norm: In (Cai et al., 2010) it is shown that the problem (1) can be reduced toX In order to improve the algorithm (Wang et al., 2012), minimization of the weighted nuclear norm is suggested in (Gu et al., 2014): where s is the standard deviation of noise, X ω, * is the weighted nuclear norm: with weights w = [w1, . . . , wn], wi ≥ 0.
Since high singular values correspond to structures in the matrix X while low values correspond to the noise component N , the authors (Gu et al., 2014) propose using weights inversely to corresponding singular values: where c ≥ 0 is some constant value, > 0 is a small value to prevent division by zero.
Similarly to (1), the weighted nuclear norm minimization problem (2) has the direct solution Singular values σi(X) that correspond to noise-free image are unknown. Using an assumption that the noise is uncorrelated in both subspaces of U and V , the singular values σi(X) can be estimated asσ The algorithm is applied for each block yj resulting in restoring blocks xj that form the denoised image. The denoising procedure can be executed several times to strengthen the denoising effect. Fig. 1 demonstrates the results for these methods.

AUTOMATIC PARAMETER CHOICE
The algorithm based on minimization of weighted nuclear norm has two parameters: c and Niter -the number of iterations. In real conditions, the reference image is not known and we are unable to optimize c and Niter by maximizing PSNR and SSIM.
There exist algorithms that can estimate the quality of image restoration without using reference image. An analysis of Local image statistics (Mittal et al., 2012), (Moorthy and Bovik, 2011) or frequency analysis (Saad et al., 2012) can be used to assess the overall image quality.
We use the algorithm described in (Mamaev et al., 2018) to find the optimal parameters for image denoising. Its idea is based on the assumption that noise and structures are uncorrelated, and that a perfect image denoising algorithm removes only noise while keeping structures intact so the difference between noisy and restored images contains only noise. Therefore, the goal is to minimize the mutual information -the correlation between the difference image and the source image in the area along edges and ridges. We have previously used this algorithm for single-frame image denoising with automatic parameter choice (Volodina et al., 2020).
Laplas operator is used to find linear structures in the noisy image: L σ (x, y) = σ 2 · I(x, y) * Gσ(x, y) Gσ(x, y) = 1 2πσ 2 exp x 2 +y 2 2σ 2 The direction of linear structures is calculated using Hessian matrix: Its eigenvector corresponding to the lower eigenvalue corresponds to the direction of a linear structure.
The linear structure mask is calculated at different scales σ. In order to find the mutual information, we construct the random value p(k, m), where K and M are values in pixels along edges and ridges p(k, m) = 1 P # (x, y) : where I d is the difference image, #{...} is the carnality, P is the normalization constant, T is linear structure detection threshold, N is the parameter of quantization, and is the scale of linear structure.
The mutual information can be used as a measurement of independence between random values in mutual distribution Higher the value µ is, the more correlated the difference image and the noisy image are. Varying the set of parameters c and Niter, we find the minimal µ which corresponds to optimal denoising parameters.

RESULTS
We have evaluated the proposed algorithm using images from TID2013 database (Ponomarenko et al., 2015) and WNNM algorithm (Gu et al., 2014).
For every reference image we generated a series of noisy images. Small affine transform was applied to every image except the first one to model the real conditions when a camera or an object are moving. Different noise levels (4, 8, 16, 32) were considered.
During the experiments, we have found that using multiple images result in improvement of both visual quality and objective metric values. The improvement is clearly visible in the areas with complex non-repeated structure. In these areas, only few similar blocks exist for each processed block in a single image, and using multiple images significantly increases the number of similar blocks.
An example of image denoising is shown in Fig. 2.

CONCLUSION
A method for multi-frame image denoising using the weighted nuclear norm minimization has been developed. The evaluation of the algorithm has shown noticeable improvement of image quality when using multiple input images instead of single one. The improvement is the most noticeable in the areas with complex non-repeated structure.
The reported study was funded by RFBR, CNPq and MOST according to the research project 19-57-80014 (BRICS2019-394).