JP2023169526A - Video denoising method and device by twin-domain process in space-time domain and frequency conversion domain - Google Patents

Abstract

To provide a video denoising device and a video denoising method that correspond to a complex movement such as a scale change and rotation of an image frame in a video image.SOLUTION: The video denoising method is for removing noise included in a video by combining a spatiotemporal process directly using pixel values of an image in a video and a process performed in a conversion domain in which the image is converted into a frequency domain. Preferably, the spatiotemporal process directly using pixels in the image is a method for performing weighted average according to similarity of the images through block matching. Furthermore, the method applies a frame cyclic filter and performs weighted addition of results of block matching between results of self-matching within current frame input images and results of the block matching between the current frame input images and previous frame output images.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video denoising method using dual-domain processing in the spatio-temporal domain and frequency transform domain, and an apparatus therefor.

Image and video denoising processing is one of the basic video processing processes, and it is both old and new. A huge amount of research has been carried out so far, and various methods are still being proposed, but processing methods can be broadly divided into "methods in the spatial domain" and "methods in the transformation domain."

A "method in the spatial domain" is a method that directly handles pixel values in the spatial domain and focuses on the temporal and spatial similarities of images. Typical methods include a bilateral filter [Non-patent Document 17] and a non-local mean filter (NLM) [Non-patent Document 3] [Non-patent Document 4].

The present inventors [Non-Patent Document 10] [Non-Patent Document 12] simultaneously corrected rolling shutter movement distortion deformation and shaking in images captured by a camera using a CMOS sensor, and estimated that in the case of a moving camera, We heuristically introduced a recursive bilateral filter, which is an extension of the bilateral filter [Non-Patent Document 17], for time-series changes in translational parameters.

The inventor called this an "enhanced bilateral filter" and also used it for image noise removal processing [Non-Patent Document 11]. Although these methods are relatively simple, their noise removal performance is high. This corresponds to predictive coding in image and video compression [Non-Patent Document 7].

On the other hand, the "method in the transform domain" performs inverse transformation after reducing the signal level in the transform domain and removing noise components [Non-Patent Document 6] [Non-Patent Document 16] [Non-Patent Document 16] Document 14] [Non-patent document 5] [Non-patent document 18]. It corresponds to transform coding in the compression of images and videos [Non-Patent Document 7], and the Fourier transform that takes trigonometric functions as a basis is probably the most familiar. As for other transformations, the Walsh-Hadamard transformation has historically been frequently used because of its simplicity as it can be calculated only from sums and differences. The Karhunen-Loewe transformation is often theoretically treated as an optimal transformation in the sense of least squares.

In recent years, cosine transformation has been adopted as the international standard for image/video compression, JPEG [Non-Patent Document 1]/MPEG [Non-Patent Document 2]. Wavelet transform has been proposed since the 1980s and has been used in various fields.

In denoising processing, various "lets" (-lets) that go beyond the framework of orthogonal transformation have been proposed and used [Non-patent Document 16], [Non-patent Document 14], [Non-patent Document 5], [ Non-Patent Document 8].

[Multi-frame expansion and residual image transformation domain level reduction]
Normally, a non-local average filter is used for one image, but it is extended to multiple frames for a moving image sequence of continuous frames. FIG. 1 shows the configuration of a multi-frame non-local average filter. FIG. 5A shows a frame acyclic configuration, in which the current frame I _n is used as a reference frame, and a search area is taken in the previous frame I _n-1 . FIG. 4B shows a frame cyclic configuration. Z ^-1 represents a one frame delay, and with the current frame I _n as a reference frame, a search area is taken in the previous frame output O _n-1 . Liu et al. [Non-patent Document 9] also extended the non-local average filter in the frame time direction. They performed video denoising processing using a frame acyclic configuration with 11 frames.

Frame cyclic filter processing has long been used as a noise reducer in video signals [Patent Document 1] [Non-patent Document 15] [Non-patent Document 6] [Non-patent Document 13] [Non-patent Document 7]. Compared to a frame acyclic configuration, the frame memory and frame delay required for processing are reduced. This is an Infinite Impulse Response (IIR) filter in the frame time direction, and Fukinuki [Patent Document 1] [Non-patent Document 7] reduces the level of inter-frame differences in frame cyclic filter processing. By performing the processing, a configuration of a first-order lag IIR filter that does not use a multiplier was shown (FIG. 2(a)). FIG. 2 illustrates a frame cyclic filter processing block diagram, and FIG. 2(a) illustrates frame cyclic and level reduction [Patent Document 1] [Non-Patent Document 7].

An attempt is made to distinguish between noise and interframe motion based on prior knowledge that the level of noise is smaller than interframe motion in interframe differences (residual images). The threshold function for level reduction processing is used to extract noise components from inter-frame differences, rather than to obtain an image with noise components included in the image removed. ] Please note that

A difficulty in dealing with video signals that are moving images is the disturbance caused by motion. The level reduction of the inter-frame difference, which is a simple configuration, does not sufficiently separate motion and noise, and when there is inter-frame motion, an afterimage due to the motion of the previous frame appears as interference depending on the threshold value. Ebihara et al. [Non-Patent Document 6] separated motion and noise by using Hadamard transform for inter-frame differences. Furthermore, Ninomiya et al. [Non-Patent Document 13] proposed a motion-corrected frame cyclic filter that estimates motion between frames by block matching (FIG. 2(b)). FIG. 2 illustrates a frame cyclic filter processing block diagram, and FIG. 2(b) illustrates motion compensation frame cyclic and level reduction [Non-Patent Document 13].

Here, filter processing using a frame cyclic configuration is unified and redefined as a problem of estimating the pixel value in the current frame from the output of the previous frame. In the case of frame rotation and level reduction, it can be written as follows (Fig. 2(a)).

Block matching for estimating the motion between the previous frame output and the current frame input in a frame cyclic configuration can be written as follows.

This is "maximum likelihood estimation" in the presence of interframe motion.
Furthermore, if we introduce a non-local average filter,

(FIG. 2(c)), which can be considered as "Bayesian estimation." FIG. 2 illustrates a frame cyclic filter processing block diagram, and FIG. 2(c) illustrates motion compensated frame cyclic non-local averaging and level reduction.
Finally, by changing the level reduction process in the motion compensation frame cyclic non-local average filter to the transform area level reduction process, the following is achieved (FIG. 2(d)). FIG. 2(d) illustrates motion compensated frame cyclic non-local averaging and transform domain level reduction.

Takahiko Fukinuki, Noise suppression device for television signals, Special Publication No. 59-17580 (filed on June 2, 1978)

ITU-T Recommendation T.81, “Information technology - Digital compression and coding of continuous-tone still images - Requirements and guidelines”, Retrieved 2009-11-07 (aka ISO/IEC 10918-1:1994). ISO/IEC 13818-2:2003, “Information technology - Generic coding of moving pictures and associated audio information - Part 2: Video”, Retrieved 2013-09-27 (aka ITU-T Recommendation H.262). A. Buades, B. Coll, and J.-M. Morel, A non-local algorithm for image denoising, IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR 2005), San Diego, CA, U.S.A., Vol. 2, pp. 60-65, June 2005. A. Buades, B. Coll, and J.-M. Morel, A review of image denoising methods, with a new one, Multiscale Modeling and Simulation, Vol. 4, No. 2, pp. 490-530, 2006. M. N. Do and M. Vetterli, The contourlet transform: an efficient directional multiresolution image representation, IEEE Transactions on Image Processing, Vol. 14, No. 12, pp. 2091-2106, December 2005. Norio Ebihara, Tadao Fujita, Kaichi Tatsuzawa, Tsutomu Takamori, Noise reducer for television signals using Hadamard transform, Journal of the Society of Television Engineers, Vol. 37, No. 12, pp. 1030-1036, December 1983. Takahiko Fukinuki, “Multidimensional signal processing of TV images”, Nikkan Kogyo Shimbun, November 1988. E. Le Pennec and S. Mallat, Sparse geometric image representations with bandelets, IEEE Transactions on Image Processing, vol.14, no.4, pp.423-438, April 2005. C. Liu and W. T. Freeman, A high-quality video denoising algorithm based on reliable motion estimation, Proceedings of the 11th European Conference on Computer Vision Conference on Computer Vision(ECCV'10):Part III, Heraklion, Crete, Greece, pp. 706-719, September 2010. Tsutomu Matsunaga, Rolling shutter distortion correction and image stabilization without using corresponding points, Proceedings of the 19th Image Sensing Symposium (SSII2013), Yokohama (Pacifico Yokohama), June 2013. Tsutomu Matsunaga, Enhanced bilateral filter using infinite impulse response system, Proceedings of the 19th Image Sensing Symposium (SSII2013), Yokohama (Pacifico Yokohama), June 2013. Tsutomu Matsunaga, Rolling shutter distortion correction and image stabilization without using corresponding points - From translation to rotation, Proceedings of the 21st Image Sensing Symposium (SSII2015), Yokohama (Pacifico Yokohama), June 2015. Yuichi Ninomiya, Yoshimichi Otsuka, Motion-compensated noise reducer, Journal of the Society of Television Engineers, Vol. 39, No. 10, pp.956-962, October 1985. J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, Image denoising using scale mixtures of Gaussians in the wavelet domain, IEEE Transactions on Image Processing, Vol. 12, No. 11, pp.1338-1351, November 2003. J. P. Rossi, Digital techniques for reducing television noise, SMPTE Journal, Vol. 87, No. 3, pp. 134-140, March 1978. J.-L. Starck, E. J. Candes, and D. L. Donoho, The curvelet transform for image denoising, IEEE Transactions on Image Processing, Vol. 11, No. 6, pp. 670-684, June 2002. C. Tomasi and R. Manduchi, Bilateral filtering for gray and color images, Proceedings of the Sixth IEEE International Conference on Computer Vision (ICCV’98), Bombay, India, pp. 839-846, January 1998. G. Yu and G. Sapiro, DCT image denoising: a simple and effective image denoising algorithm, Image Processing On Line, Vol. 1 (2011).

How to separate image components and noise components, especially in the case of video, is a problem that involves not only the movement of the camera that photographs the subject, but also the presence of movement of the object in the image frame. So far, separation has been performed based on pixel value level differences due to noise and motion, separation by estimating and correcting motion, etc., but conventional techniques cannot be said to be sufficient. To begin with, it has traditionally been difficult to accurately estimate various movements between image frames, including noise components, and processing costs and delays occur. In particular, motion estimation based on block matching involves translational motion that combines horizontal and vertical motion, and it has traditionally been difficult to handle complex motions such as zooming, scaling, and rotation of the camera. There is.

Rather than directly estimating the movement between image frames, block matching is "speculatively executed" between image frames that have undergone geometric transformations such as predetermined minute movements, scale changes, and rotations. We will respond accordingly. The movement between successive image frames is almost minute and can be sufficiently approximated by minute parameter values in each image frame geometric transformation. It would be possible and easy to increase the number of "speculative executions" in which parameter values are changed, as long as computational resources permit.

In the present invention, in the temporal frame cyclic filters [Non-Patent Document 15], [Non-Patent Document 6], [Non-Patent Document 13], and [Non-Patent Document 7] that have been used as noise reducers for video signals, "spatial domain We propose a method that combines the "method in the transformation domain" and the "method in the transform domain." The frame cyclic filter in conventional research is redefined as the problem of estimating the pixel value in the current frame from the output of the previous frame using denoising processing methods that have been proposed so far.

According to the present invention, it is possible to cope with complex movements such as scale changes and rotations of image frames in a video, and it is possible to perform highly accurate noise removal. Since the motion in the video is not directly estimated, it is possible to deal with complex motion while reducing the cost of motion estimation. The frame cyclic configuration eliminates the need for extra frame memory and eliminates frame delays. “Speculative execution” allows processing to be performed independently and in parallel as long as computational resources allow, and can be said to be the best method for denoising processing for video.

FIG. 2 is a diagram illustrating a multi-frame non-local average filter. FIG. 2 is a diagram illustrating a frame cyclic filter processing block diagram. FIG. 2 is a schematic block diagram illustrating the entire video denoising process using dual-region processing according to the present invention. FIG. 3 is a block diagram illustrating details of video denoising processing using dual region processing. FIG. 3 is a diagram illustrating a graph (T=50) of a level reduction operator S[ξ]. This is a further expanded version of the detailed block diagram of the video denoising process using dual area processing in FIG. 4 . This is a conceptual diagram illustrating the operation of non-local average filter processing, in which a 5x5 pixel size reference block is generated around the pixel of interest (A), and a search area range of 13x13 pixel size is block matched. There is. This is a diagram explaining the base image of 64 discrete cosine transforms of an 8×8 image. The further you go, the more intense the changes in the horizontal direction, and the more you go down, the more intense the changes in the vertical direction. It is a diagram illustrating an example of video denoising processing, in both the upper and lower rows, the left side is the input image containing noise (Before (σ _n =20)), and the right side is the denoising processed image (After (2D denoised/3D ₊₊ denoised)). ), the upper row is the first frame of the video sequence, and the lower row is the result for 120 frames (see FIG. 10). FIG. 4 is a diagram showing a graph of the peak SN ratio (PSNR [dB]) of a true image that does not include noise and a difference image as a result of the denoising process for a moving image sequence frame number of the video denoising process.

(overview)
In the present invention, denoising processing for removing noise included in a video is performed using bi-region processing. Bi-domain processing is a denoising process that combines two processes: one that performs spatiotemporal processing directly using pixel values in an image, and the other that performs processing in a transformation domain that transforms the image into the frequency domain. It is.

Spatiotemporal processing that directly uses pixels in an image uses block matching to perform weighted averaging based on image similarity, and then applies a frame cyclic filter to perform self-matching within the current frame input image. The results of block matching between the results of and the output image of the previous frame are weighted and added.

By performing block matching on the output image of the previous frame with the results of "speculatively executing" minute scale transformations and rotation transformations determined in advance, it is possible to perform not only simple translational movement but also image enlargement.・Supports reduction, rotation, etc.

For the residual image obtained by subtracting the spatiotemporal processing results of such an image from the current frame input image, the image is transformed into the frequency domain using discrete cosine transformation, and the noise component is removed by level reduction processing in the transformation domain. By separating the and image components, the noise removal performance is further improved.

The new characteristic processing configuration includes a non-local average filter that performs weighted averaging based on image similarity, and a differential residual image between the non-local average filter and the input noise image, which is converted to the frequency domain by discrete cosine transformation. An example of this is to introduce a frame cyclic configuration to bi-domain processing that extracts noise with higher precision using a level reduction operator in the transformation domain, and to extend it to three-dimensional processing in the time direction between the output of the previous frame.

As a new point of view, we have added a "speculative method" in which the results of predetermined geometric transformations such as scale transformation and rotation transformation are applied to the output of the previous frame in block matching to measure the similarity of images in a non-local average filter. The objective is to extract noise with high precision from various complex movements in images by "execution of the same method".

In addition, as a new and important characteristic technical idea for configuring and constructing the method and device of the present invention,
・Frame cyclic configuration multi-frame non-local average filter processing unit that performs spatiotemporal processing directly using pixel values in an image ・Weighted summation unit that takes the weighted sum of the frame cyclic configuration multi-frame non-local average filter processing results ・A first residual image generation unit that subtracts the output from the weighted summation unit of the frame cyclic configuration multi-frame non-local average filter processing result from the input noise image.A frequency domain that converts the residual image into the frequency domain by discrete cosine transformation. Conversion unit - Conversion level reduction unit that distinguishes noise components and image components from the residual image containing the noise component and image component frequency-converted by the above frequency domain conversion using the conversion domain level threshold and extracts the noise component. Inverse frequency domain transform unit that returns the output from the transform domain level reduction unit in the frequency domain to the original pixel value by inverse discrete cosine transform. Subtracts the final noise component output from the above inverse frequency domain transform unit from the input noise image. Second residual image generation unit - Frame memory unit that stores the noise-removed output image from the second residual image generation unit - Scale conversion and rotation conversion of the previous frame image stored in the frame memory Examples include a geometric transformation section that performs geometric transformations such as . Note that, as described later, the geometric transformation section may be omitted or skipped (see, for example, FIG. 4).

The methods and devices described above can be implemented by hardware devices that process baseband video signals, or by software that processes MXF (Material Exchange Format) files and a computer that executes them. It is possible to realize this using a device based on the baseband video signal, or it can be realized with any configuration using a device that converts an MXF file into a baseband video signal or inversely converts it. It is also possible to transmit compressed camera images or MXF files via IP (Internet Protocol) and process them on the cloud. Various system configurations are possible, such as decoding IP-transmitted compressed video into a baseband video signal, denoising the result, compressing it again, and delivering it as a stream.

FIG. 3 is a schematic block diagram of the entire video denoising process using bi-region processing according to the present invention. Spatio-temporal domain denoising processing is performed using the previous frame output O _n-1 obtained by rotating the input video (Input I _n ) containing noise and the video denoising processing output (Output O _n ) by frames using a frame memory (Frame Memory). Spatio-Temporal Domain Denoising). Subscripts _{n-1 and n} represent frame numbers in the video.
Further, the difference video between the spatio-temporal domain denoising processed video and the input video is subjected to frequency transform domain denoising processing (Frequency Transform Domain Denoising). Then, the result is subtracted from the input video and the result is the final video denoising processing output (Output _On ). It is preferable to apply a geometric transformation (Geometric Transform) to the previous frame output O _n-1 from the frame memory to change the scale or rotate the image.

FIG. 4 is a detailed block diagram of video denoising processing using dual region processing. Spatio-temporal domain denoising processing is performed using the previous frame output O _n-1 obtained by rotating the input video (Input I _n ) containing noise and the video denoising processing output (Output O _n ) by frames using a frame memory (Frame Memory). Spatio-Temporal Domain Denoising).

A non-local means (NLM) [Non-patent Document 3] [Non-patent Document 4] is used for the spatio-temporal domain denoising process (details will be described later). Therefore, the video denoising output (Output O _n ) is

Here, T is a threshold value for distinguishing noise components from image components. The relationship of number 7 can be expressed in a graph as shown in FIG.

FIG. 6 is a further expanded detailed block diagram of the video denoising process using the bi-region process shown in FIG. The weighted summation (Σ) output Z _n of the nonlocal mean filter (NLM), which is a spatiotemporal domain denoising process, is as follows.

Geometric transformation is a transformation using predetermined parameters, and is "speculative execution" that predicts the movement of a camera or an object in a video. For example, if the image is a video that is zoomed in while the camera is panning, it is sufficient to perform translation conversion to move the image in the horizontal direction and scale conversion to enlarge the image.

A search equivalent to translational transformation by block matching is performed in the non-local average filter itself, but such translational transformation is predicted and speculatively performed for the previous frame output O _n-1 from the frame memory. By implementing this method, it is possible to cope with camera panning and zooming that require large amounts of movement exceeding the block matching search range in the non-local average filter, and it is expected that the noise removal performance may be improved.

Even if the prediction is wrong, the matching residual in the non-local mean filter becomes significantly large, and such a search block does not contribute to the denoising process (the weighted output of such search block pixels in the non-local mean filter is approximately becomes 0).

Each geometric transformation and non-local mean filter is processed independently and in parallel. Due to “speculative execution,” the accuracy is inferior to estimating true camera movement, but it is difficult to accurately estimate camera movement from images containing noise, and The computational cost and delay required for processing are large.

The movement between successive image frames is almost minute and can be sufficiently approximated by minute parameter values in each geometric transformation. It would be possible and easy to increase the number of "speculative executions" in which parameter values are changed, as long as computational resources permit. “Speculative execution” means carrying out multiple executions or preparations for something that is not known as a “hit”, and only validating it for the one that later turns out to be a “hit” (or correct answer). It means to make use of it.

FIG. 7 is a conceptual diagram explaining the operation of non-local average filter processing, in which a reference block of 5×5 pixel size is generated around the pixel of interest (A), and the range of the search area of 13×13 pixel size is determined. Block matching. The center pixel of each block is weighted by the matching residual and then averaged. This processing is performed while moving the block pixel by pixel.

[Nonlocal average filter]
[Pixel-wise processing]

The block matching residual SSD(u,v) is as follows.

This is a block matching process for a search area of (2W+1) x (2W+1) pixel size using a reference block of (2N+1) x (2N+1) pixel size in the image itself, and weights are applied to similar neighboring block pixels. This is weighted averaging processing. FIG. 7 is a conceptual diagram showing the operation of non-local average filter processing. A 5×5 pixel block is generated around the pixel of interest (A), and block matching is performed within a 13×13 pixel search area. The center pixel of each block is weighted by the matching residual and then averaged. This processing is performed while moving the block pixel by pixel.

The larger h in Equation 11, the stronger the degree of smoothing. As C, in addition to the three dimensions of RGB or the three dimensions of luminance and color difference YUV, one dimension of Y1 + two dimensions of UV, and one dimension each of Y/U/V can be used. In the present invention, C ₁ (Y) one dimension + C ₂ C ₃ (UV) two dimensions are used for the result of orthogonal color conversion (described later) of an RGB color image. The SSD in the one-dimensional and two-dimensional cases is set to k=1 and 2, respectively, and the normalization coefficients are also 1 and 2.

[Block-wise processing]
The non-local average filtering of all pixel values in a block {B _k |k=1,2,3} in a color image is as follows.

The block B _k has a (2N+1)×(2N+1) pixel size, similar to pixel-by-pixel processing, and the weight ω( u, v). If processing is performed while moving the block pixel by pixel, the block pixels in the estimation result will be duplicated, so the final output obtained by taking the average of all the duplicate pixel values will be as follows.

Note that although it is written as "non-local," the search area in actual filter processing is a reference block centered on the pixel of interest and "local" blocks around it. Therefore, it should actually be a "local" average filter. However, in [Non-Patent Document 3] and [Non-Patent Document 4], a "non-local" area of the entire image is used as a search area, and it is not impossible to perform such processing.

In the present invention, the search area is a "local" block within a realistic range, but according to [Non-patent Document 3] and [Non-patent Document 4], the name is "Non-local means (NLM)". ) filter” is used.

[Orthogonal color conversion of RGB color images]
In the present invention, an RGB color image is orthogonally converted as follows [Non-Patent Document 18].

This transformation matrix is a three-point cosine transformation basis, and the transformed C1, C2, and C3 approximately correspond to the luminance chrominance YUV in that order. A non-local average filter and frequency domain denoising processing (described in detail later) are performed on each transformed channel, and the image is converted into an RGB color image by inverse transformation. Since this transformation matrix is an orthogonal transformation, its inverse transformation matrix is the transpose of the transformation matrix.

FIG. 8 is a diagram illustrating 64 base images of discrete cosine transformation of an 8×8 image, where the upper left when facing the page is a constant, the whiter value is larger, the value is larger, and the darker the value is, the smaller the value. The horizontal changes become more intense as you go to the right in the paper, and the vertical changes become more intense as you go down.

[Denoising processing by level reduction in frequency domain]
The basis function of Discrete Cosine Transform (DCT) is defined as follows.

When the discretized signal is _fn , the discrete cosine transform and inverse discrete cosine transform are as follows.

In the case of a two-dimensional image, the result of the one-dimensional DCT described above in the horizontal direction may then be subjected to one-dimensional DCT in the vertical direction. Therefore, in the case of a two-dimensional image, the basis function of Equation 15 is expressed as the two-dimensional basis image in FIG. 8. FIG. 8 is a base image of 64 discrete cosine transforms of an 8×8 image.

The DCT coefficient {ξ _k } usually approaches 0 quickly when k is large. Therefore, an image can be expressed well with only a small number of terms for small k in the first equation of Equation 16. Because of this property, it is also used for image compression.

By multiplying the base image by an arbitrary DCT coefficient {ξ _k } and adding them together, an arbitrary image can be expressed. That is, the inverse DCT of the second equation of equation 16 uses the DCT coefficients {ξ _k } of the first equation of equation 16 and the base image to restore the original image.

Therefore, if only the noise component can be separated from the DCT coefficients {ξ _k } of the image component including the noise component, an image from which noise has been removed can be obtained by subtracting the separated noise component from the input noise image.

Since the non-local average filter processing result, which is the denoising process in the spatiotemporal domain in the previous stage, and the residual image of the input noise image are processed, the noise component is expected to have a relatively low level compared to the image component. Therefore, it is expected that only the noise component can be extracted with higher precision by applying the level reduction operator S[ξ] of Equation 7 to the transformed image of the residual image.

The transformation is performed block by block, such as an 8x8 image, and the results of the denoising process are obtained block by block. Therefore, while shifting the processing blocks one pixel at a time, the "collective average" of all the processing blocks that have been finally subjected to the denoising process is taken. This is the same as non-local average filter processing on a block-by-block basis.

FIG. 9 is a diagram illustrating an example of video denoising processing. In both the upper and lower rows, the left side is the input image containing noise (Before (σ _n =20)), and the right side is the denoising processed image (After (2D denoised/3D ₊₊ denoised)), the upper row is the first frame of the video sequence, and the lower row is the result of 120 frames (see FIG. 10).

[Video sequence simulation]
In FIG. 9, the first frame of the upper moving image sequence is the result of denoising processing of only the current frame input without using the previous frame output (2D denoised in FIG. 10). The non-local average filter in the spatiotemporal domain of the lower 120 frames is the result of the current frame input, the previous frame output by frame rotation, and the speculative execution number 2 (scale coefficients 0.99, 0.98) by scaling the previous frame output ( 3D ₊₊ denoised in Figure 10). The ratio of weighting coefficients in speculative execution is set to 1 for the current frame input, 99 for the previous frame output by frame cycling, and 99 for scale conversion of the previous frame output by frame cycling, respectively. The parameters of the non-local average filter were a reference block size of 5×5, a search area of 15×15, and a smoothing parameter h=22. In the denoising process in the frequency transform domain, the residual image obtained by subtracting the non-local average filter processing result from the input noise image is subjected to discrete cosine transform of the 16×16 image domain, and the level reduction operator S[ ξ] was set to 55. For the input noise image containing noise, noise following a normal distribution with an average of 0 and a standard deviation of 20 was added to the true image containing no noise.

FIG. 10 is a graph of the peak SN ratio (PSNR [dB]) of the true image without noise and the difference image as a result of the denoising process for the moving image sequence frame number of the video denoising process. When the non-local average filter uses the current frame input and the previous frame output by frame rotation (3D denoised), and when the previous frame output is scaled and the number of speculative executions is 1 (3D ₊ denoised), the number of speculative executions is 2. (3D ₊₊ denoised), and only current frame input (2D denoised).

The input noise image containing noise is the one in which noise that follows a normal distribution with a mean of 0 and standard deviation of 20 is added to the true image that does not contain noise, and the PSNR value of the input noise image is 22.54 (±0.0418) [dB] on average. ] (Standard deviations are in parentheses).

Generally speaking, 3D denoised uses the current frame input and the previous frame output by frame cycling, and 3D ₊ increases the number of speculative executions by scaling the previous frame output, rather than 2D denoised where the non-local average filter only uses the current frame input. The PSNR values of denoised and 3D ₊₊ denoised are increasing.

There is a frame (near frame 70) where the PSNR value is maximum in 3D denoised in which a partial non-local average filter uses the current frame input and the previous frame output by frame cycling, but this is due to the local movement of the camera and the image frame. This is because neither the object nor the object is scale-transformed (in this case, the image frame is reduced) and can be regarded as only a translational movement. Furthermore, the increase in the PSNR value in the latter half of the frame (approximately after the 100th frame) is due to the increase in the flat area of the road surface at the bottom of the image.

(supplementary explanation)
In the above explanation, the frame cyclic configuration multi-frame non-local average filter processing unit that performs spatio-temporal processing directly using pixel values in an image, and the weighted sum of the frame cyclic configuration multi-frame non-local average filter processing results. The weighted summation section to be taken corresponds to "Spatio-Temporal Domain Denoising" shown in FIG. 3, and corresponds to "NLM" (Non-local mean filter/Non-localmeans) shown in FIG. 4, w ₁ , w ₂ and Σ. This corresponds to "NLM", w ₁ , . . . , _n and Σ shown in FIG.

In addition, the first residual image generation unit that subtracts the output from the weighted summation unit of the frame cyclic configuration multi-frame non-local average filter processing result from the input noise image is located immediately after the “Spatio-Temporal Domain Denoising” shown in Figure 3. Corresponds to the subtraction process "-" shown on the right side, corresponds to the subtraction process "-" shown immediately to the right side of Σ shown in Figure 4, and corresponds to the subtraction process "-" shown immediately to the right side of Σ shown in Figure 6. handle.

In addition, a frequency domain transform unit converts the residual image into the frequency domain by discrete cosine transform, and a transform domain level threshold is used to convert the residual image containing the noise component and image component frequency-converted by the frequency domain transform into noise by using a transform domain level threshold. a transform level reduction unit that discriminates between the image component and the image component and extracts the noise component; an inverse frequency domain transform unit that returns the output from the transform domain level reduction unit in the frequency domain to the original pixel value by inverse discrete cosine transformation; corresponds to “Frequency Transform Domain Denoising” shown in FIG. 3, corresponds to “T”, “S”, “T ⁻¹ ” shown in FIG. 4, and corresponds to “T”, “S”, “ T ⁻¹ ”.

In addition, the second residual image generation unit that subtracts the final noise component output from the inverse frequency domain transformation unit from the input noise image performs the subtraction process shown immediately to the right of “T ⁻¹ ” shown in FIG. Corresponding to "-", the subtraction process "-" shown immediately to the right of "T ^-1 " shown in FIG. 4 corresponds to the subtraction process "-" shown immediately to the right of "T ^-1 " shown in FIG. ” corresponds to

Further, the frame memory section that stores the noise-removed output image from the second residual image generation section corresponds to the "Frame Memory" shown in FIGS. 3, 4, and 6, respectively.

Furthermore, the geometric transformation section that performs geometric transformation such as scale transformation and rotation transformation on the previous frame image stored in the frame memory corresponds to "Geometric Transform" shown in FIG. 3, and "Geometric Transform" shown in FIG. 6. Translation” and “Rotation” and “Scale”. Although "Geometric Transform" is not shown in FIG. 4, it is also possible to skip the processing without using "Geometric Transform" as shown in FIG.

The disclosure described in the above-mentioned embodiments is not limited to the specific examples thereof, and can be made by appropriately applying known techniques known to those skilled in the art or well-known techniques within the scope of the technical idea of the present invention. , or/and can be arranged and used.

The present invention is also suitable for video equipment in general, and in particular for video captured by security monitoring and the like.

Claims (14)

In video denoising equipment,
a frame cyclic multi-frame non-local average filter processing unit that performs spatiotemporal processing directly using pixel values in an image;
a weighted summation unit that calculates a weighted summation of processing results by the frame cyclic configuration multi-frame non-local average filter processing unit;
a first residual image generation unit that subtracts the output from the weighted summation unit of the frame cyclic configuration multi-frame non-local average filter processing result from the input noise image;
a frequency domain transformation unit that transforms the first residual image into a frequency domain by discrete cosine transformation;
a conversion level reduction unit that distinguishes noise components and image components from a residual image containing noise components and image components frequency-converted by the frequency domain conversion unit using a conversion domain level threshold, and extracts the noise components;
an inverse frequency domain transform unit that returns the output from the transform domain level reduction unit in the frequency domain to the original pixel value by inverse discrete cosine transform;
a second residual image generation unit that subtracts the final noise component output from the inverse frequency domain transformation unit from the input noise image;
A video denoising device comprising: a frame memory unit that stores an output image from which noise has been removed from the second residual image generation unit. In the video denoising method,
a frame cyclic multi-frame non-local average filter processing stage that performs spatio-temporal processing directly using pixel values in the image;
a weighted summation step of taking a weighted summation of the frame cyclic configuration multi-frame non-local average filter processing results;
a first residual image generation step of subtracting the output from the weighted summation result of the frame cyclic configuration multi-frame non-local average filter processing results from the input noise image;
a frequency domain transformation step of transforming the first residual image into the frequency domain by discrete cosine transformation;
a transform level reduction step of discriminating noise components and image components from a residual image including noise components and image components frequency-converted by the frequency domain transform using a transform domain level threshold and extracting the noise components;
an inverse frequency domain transform step of returning the transform domain level reduction result in the frequency domain to the original pixel value by inverse discrete cosine transform;
a second residual image generation step of subtracting a final noise component obtained as a result of the inverse frequency domain transform from the input noise image;
A video denoising method, comprising: storing the output image from which noise has been removed in the second residual image generation step in a frame memory. In the video denoising method,
Noise contained in the video is processed by combining two processes: one that performs spatiotemporal processing directly using the pixel values of the image in the video, and the other that performs processing in a transformation domain where the image is converted into a frequency domain. A video denoising method characterized by performing denoising processing to remove. The video denoising method according to claim 3,
Spatio-temporal processing that directly uses pixels in an image uses block matching to perform weighted averaging based on image similarity.Furthermore, a frame cyclic filter is applied to calculate the self-image in the current frame input image. A video denoising method characterized by weighted addition of block matching results between a matching result and a previous frame output image. The video denoising method according to claim 4,
By performing block matching on the output image of the previous frame with the results of "speculatively executing" small predetermined scale transformations and rotation transformations, it is possible to perform not only simple translational movements but also image transformations. A video denoising method characterized by being compatible with enlargement/reduction, rotation, etc. The video denoising method according to claim 5,
The residual image obtained by subtracting the spatiotemporal processing result of the image from the current frame input image is transformed into the frequency domain using discrete cosine transformation, and the noise component and the image are reduced by level reduction processing in the transformation domain. A video denoising method characterized by processing that further improves noise removal performance by separating components. In video denoising equipment,
Noise contained in the video is processed by combining two processes: one that performs spatiotemporal processing directly using the pixel values of the image in the video, and the other that performs processing in a transformation domain where the image is converted into a frequency domain. A video denoising device comprising: a denoising processing unit that removes. The video denoising device according to claim 7,
The spatio-temporal processing by the denoising processing unit that directly uses pixels in the image is to perform weighted averaging according to the similarity of the images by block matching, and further apply a frame cyclic filter to calculate the current frame. A video denoising device characterized by weighted addition of a result of self-matching within an input image and a result of block matching between an output image of a previous frame. The video denoising device according to claim 8,
The denoising processing unit further performs block matching with the results of "speculatively executing" predetermined minute scale transformations and rotation transformations on the output image of the previous frame, thereby performing simple translation processing. A video denoising device that is characterized by being able to handle not only movement, but also image enlargement/reduction, rotation, etc. The video denoising device according to claim 9,
The denoising processing unit further transforms the residual image obtained by subtracting the spatiotemporal processing result of the image from the current frame input image into a frequency domain using discrete cosine transformation, and performs level reduction in the transformation domain. A video denoising device characterized by processing that further improves noise removal performance by separating noise components and image components. The video denoising device according to claim 1,
A video denoising device, further comprising: a geometric transformation unit that performs geometric transformation on a previous frame image stored in the frame memory unit. The video denoising device according to claim 11,
The video denoising device, wherein the geometrical transformation is a scale transformation or a rotational transformation. The video denoising method according to claim 2,
A video denoising method, further comprising a geometric transformation step of performing geometric transformation on the previous frame image stored in the frame memory. The video denoising method according to claim 13,
A video denoising method, wherein the geometric transformation is a scale transformation or a rotation transformation.