CN111260580A

CN111260580A - Image denoising method based on image pyramid, computer device and computer readable storage medium

Info

Publication number: CN111260580A
Application number: CN202010051403.4A
Authority: CN
Inventors: 杨帆; 彭刚
Original assignee: Allwinner Technology Co Ltd
Current assignee: Allwinner Technology Co Ltd
Priority date: 2020-01-17
Filing date: 2020-01-17
Publication date: 2020-06-09
Anticipated expiration: 2040-01-17
Also published as: CN111260580B

Abstract

The invention provides an image denoising method based on an image pyramid, a computer device and a computer readable storage medium, wherein the method comprises the steps of obtaining an initial image, calculating an output chromatic value of each pixel: acquiring a layered search area of a pixel to be denoised, performing down-sampling on the layered search area, acquiring a layer of down-sampled image after each down-sampling, and forming a first image pyramid by the multi-layer down-sampled image; each down-sampling image is up-sampled, the up-sampled image is subtracted from the next layer of image of the down-sampling image to obtain a layer of subtraction image, and the plurality of layers of subtraction images form a second image pyramid; carrying out mean value denoising on each layer of subtraction image of the second image pyramid and the highest layer down-sampling image of the first image pyramid to obtain a denoising chromatic value; an output chroma value is calculated using the initial chroma value and the denoised chroma value. The invention also provides a computer device and a computer readable storage medium for realizing the method. The invention can reduce the calculated amount of image denoising and the hardware realization difficulty, and improve the denoising quality.

Description

Image denoising method based on image pyramid, computer device and computer readable storage medium

Technical Field

The invention relates to the technical field of image processing, in particular to an image denoising method based on an image pyramid, a computer device for realizing the method and a computer readable storage medium.

Background

Many existing intelligent electronic devices have an image shooting function, for example, a smartphone, a tablet computer, a vehicle data recorder, and the like are provided with a camera device, and the camera device is usually provided with a CMOS sensor to acquire an image. Generally, an image includes a large number of pixels, and color information of each pixel may be represented by RGB values or YUV values.

For example, the CMOS image sensor commonly used at present generally adopts a BAYER arrangement format, and the color information of each pixel is usually RGB values, but the RGB values are not information of three primary colors, and the image has color distortion, so that it is necessary to perform a demosaic (demosaic) process on each pixel to obtain RGB three primary color information to restore the original color of the image.

With the increase of image resolution, the reduction of the photosensitive quantity of a single pixel and the wider application of low-illumination scenes, the image noise output by the CMOS image sensor is greatly increased. In the process of converting the RAW image into the RGB image, the demosaicing operation needs to refer to all image pixels in a certain area to obtain three primary colors of RGB of one pixel, and noise of each color component is diffused mutually in a large range, so that a large block (from several pixels to hundreds of pixels) of color spots appear in the final image, and the appearance of human eyes is seriously affected. Therefore, it is generally necessary to perform a denoising process on an image output from the CMOS image sensor.

The information in the image has high correlation, and each pixel does not exist in isolation, so that the similarity of not only brightness but also the similarity of image texture is realized. A Non-Local Means (NLM) method is a denoising method based on image Local similarity, on the basis of a neighborhood average denoising method, a similarity weight coefficient of the NLM method is determined by the similarity between a current pixel point to be denoised and an image slice taking other pixel points in a neighborhood as a center, the calculation of the weight has no substantive relation with the space positions of two pixels and is only related to the similarity of the two image slices, and therefore false information can be well prevented from being introduced. Since the noise of the image can be equivalent to additive white gaussian noise, the weighted average of the similar pixels can better remove the noise of the image. The NLM method has the characteristics of simple algorithm, superior performance, and easy improvement and expansion, and thus becomes one of the mainstream methods for denoising the current image.

The basic principle of the NLM method is as follows: taking a pixel point to be denoised as a center to obtain a neighborhood window, wherein the size of the window is NxN (N is generally 3, 5, 7, 9 and the like), taking an image area with a certain size around the pixel point as a search area, marking the image area as MxM (the whole image can be selected, but the calculation amount is overlarge, and generally M is less than 41), searching for similar image blocks in the search area, and calculating the Gaussian weighted Euclidean distance between the image blocks by using the following formula:

wherein, Ga is a Gaussian kernel with standard deviation a, u (Ni) and u (Nj) are respectively corresponding pixels in the image block of the central window and the image block of the search area. Then, the denoised pixel estimation value is calculated by using the following formula:

wherein w (i, j) ═ exp (-d (i, j)/h²) In order to be the weight coefficient,

for the normalization factor, Ω (i) represents the search area of the central pixel i, and h is a similarity weight parameter, which determines the balance after denoising the image.

Currently, the denoising method based on the non-local mean value mainly has the following three modes:

the first mode is to transform the image and denoise the image by an NLM method in a transform domain instead of a space domain, and the specific steps are as follows: transforming the initial image, such as wavelet transformation, Contourlet transformation, Laplace transformation, etc., and decomposing into a low-frequency image and a high-frequency image; then, denoising or correcting the low-frequency image and the high-frequency image by an NLM method respectively, and finally reconstructing and inversely transforming the obtained low-frequency image and the high-frequency image to obtain a denoised image.

The second mode is to self-adaptively select the size and the weight coefficient of the neighborhood block of the NLM method by utilizing image information, and the specific steps are as follows: carrying out pixel coarse classification on the noise image by using an evaluation operator; then, for each pixel in the noise image, according to the coarse classification result of the surrounding neighborhood pixels, the classification of the pixel is finely classified into low-noise high texture, medium texture, high-noise secondary texture, smooth area and the like; and finally, for each category after the fine classification, adaptively selecting a filtering parameter and a neighborhood block size, and carrying out pixel denoising by using a non-local mean denoising algorithm.

The third mode is a BM3D denoising method, which is improved for realizing real-time video denoising, and includes the following specific steps: dividing an image into a plurality of N multiplied by N pixel blocks, selecting N pixel blocks which are most matched with a reference block in a certain area of a current frame and a plurality of frames before and after the current frame with the reference block as the center to form an N multiplied by N three-dimensional pixel array, carrying out three-dimensional discrete cosine transform, hard threshold filtering and three-dimensional inverse discrete cosine transform on the three-dimensional pixel array, and carrying out reconstruction processing on an inverse transform result to obtain a denoising result.

However, the denoising method based on the non-local mean value has the following problems:

firstly, the denoising result of the algorithm is closely related to the size of a denoising local window, when the image noise is large, a large local window is needed to obtain a good denoising effect, but the algorithm complexity and the hardware realization cost are obviously increased. For scenes with low signal-to-noise ratio such as low illumination, the search area needs to be at least 33 × 33, which means that for each pixel to be denoised, more than 1088 times of similar image block matching operation and other related operations are required, and the calculation amount is very large, which is not favorable for real-time denoising and hardware implementation.

Secondly, when the image noise is large, the NLM algorithm can effectively remove the low-frequency noise, but at the same time, the high-frequency details of the image are lost, so that the image definition is reduced. This is because the searched area is larger and there are more similar image blocks, which causes some pseudo-similar pixels to participate in the denoising process, resulting in weakening of image texture.

Thirdly, none of the current NLM denoising methods has a complete technical solution for the RAW image denoising. The current shooting equipment is generally a CMOS camera, the output format is a RAW format, a Bayer color filter array is generally adopted, each pixel point only has R, G, B color information, and the number of the pixels of G is twice of that of R, B.

Disclosure of Invention

The invention mainly aims to provide an image pyramid-based image denoising method which reduces the calculation amount of intelligent electronic equipment and has better denoising effect.

Another object of the present invention is to provide a computer apparatus for implementing the image pyramid-based image denoising method.

Still another object of the present invention is to provide a computer-readable storage medium for implementing the image pyramid-based image denoising method.

In order to achieve the main purpose of the invention, the image pyramid-based image denoising method provided by the invention comprises the steps of obtaining an initial image, and calculating the output chromatic value of each pixel of the initial image; wherein calculating the output chrominance value of each pixel comprises: acquiring a layered search area of pixels to be denoised, acquiring initial chromatic values of all pixels in the layered search area, performing at least one down-sampling on the layered search area, acquiring a layer of down-sampled image after each down-sampling, and forming a first image pyramid by the multi-layer down-sampled image; each down-sampling image is up-sampled, the up-sampled image is subtracted from the next layer of image of the down-sampling image to obtain a layer of subtraction image, and the plurality of layers of subtraction images form a second image pyramid; carrying out mean value denoising on each layer of subtraction image of the second image pyramid and the highest layer down-sampling image of the first image pyramid to obtain a denoising chromatic value; an output chroma value is calculated using the initial chroma value and the denoised chroma value.

According to the scheme, the number of the pixels of each layer of image in the image pyramid is smaller than that of the initial image, so that the image pyramid is used for denoising calculation, the calculation amount of denoising calculation can be greatly reduced, and the hardware implementation cost of the intelligent electronic equipment is reduced. In addition, each layer of image of the second image pyramid is obtained by calculating two adjacent layers of images of the first image pyramid, so that the high-level image of the second image pyramid can reflect small-scale image texture information, the high-level low-level image of the second image pyramid can reflect large-scale image edge information, and the images are divided into different frequency bands in a pyramid layering mode, so that subsequent fine denoising processing is facilitated, and the denoising effect is more ideal.

Preferably, obtaining the denoised chrominance values comprises: and setting a denoising search area of each subtraction image and the highest-layer down-sampling image, calculating a preset distance value in a preset range, and calculating an average value of chromatic values of all pixels of which the preset distance values are smaller than a noise threshold value.

Therefore, the distance value of the denoising search area in the preset range can be conveniently calculated by setting the noise threshold, the chroma values of the pixels can be conveniently determined to be available, and the chroma values of the pixels can not be used, namely, the image block with the closer color is determined to be used as the reference of denoising through searching.

Further, the noise threshold is positively related to the characteristic value of the image sensor and/or the luminance value of the pixel.

Therefore, the set noise threshold can reflect the characteristics of the image sensor, and the image is subjected to targeted denoising according to the characteristics of the image sensor. Moreover, because color noise has a certain relationship with the brightness of the pixel, for example, the reason for generating the color noise may be caused by insufficient illumination, therefore, the noise threshold is related to the brightness value of the pixel, and the color reality of the denoised image can be improved

In a further scheme, the denoising search area of at least one layer of subtraction image is larger than that of the highest layer of down-sampling image.

Therefore, according to the size of each layer of image, a corresponding denoising search area is determined, for example, if the number of pixels of a certain layer of image is more, a larger search area is set, and if the number of pixels is less, a smaller search area is set, so that the denoising quality of the image is improved.

Preferably, the calculating the preset distance value within the preset range, and the calculating the average value of the chrominance values of all the pixels of which the preset distance value is smaller than the noise threshold value includes: and calculating a preset distance value between the denoising search area of each layer of subtraction image and a preset range, calculating a preset distance value between the denoising search area of the highest layer of down-sampling image and the preset range, and calculating an average value of chromatic values of all pixels of which the preset distance values are smaller than a noise threshold value.

Therefore, the calculation of the preset distance value is performed for each layer of image, and the average value of the chroma values of the pixels of each layer is calculated, so that more parameters can be used when the output chroma values of the pixels are calculated, and the accuracy of denoising calculation is improved.

Further, calculating an output chroma value using the initial chroma value and the denoised chroma value comprises: and calculating an output colorimetric value by using the initial colorimetric value, the denoised colorimetric values of the layers and a preset weighting coefficient.

Therefore, the output colorimetric values are calculated through the preset weighting coefficients, and the appropriate weight values can be determined according to the specific weights of the denoising colorimetric values of different layers, so that the accuracy of calculating the output colorimetric values is improved, the denoised colorimetric values are closer to the real colors, and the denoising quality is improved.

Further, the predetermined distance value is one of a simplified euclidean distance, a manhattan distance, a minkoch distance, a chebyshev distance, and a cosine distance.

It can be seen that simplified euclidean distance, manhattan distance, minkoch distance, chebyshev distance, and cosine distance are common distance values for image filtering calculation, and the workload of image denoising calculation can be simplified by using the distance values.

Further, when down-sampling the layered search area, performing Gaussian filter calculation; and/or performing a gaussian filter calculation when upsampling the downsampled image.

Therefore, Gaussian filtering is carried out during both up-sampling and down-sampling, the smoothness of the image in the layered search area can be improved, and the denoising calculation effect is more ideal.

In order to achieve the above another object, the present invention provides a computer apparatus including a processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the steps of the image denoising method based on the image pyramid.

To achieve the above-mentioned further object, the present invention provides a computer-readable storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the steps of the image denoising method based on image pyramid.

Drawings

FIG. 1 is a flowchart of an embodiment of an image denoising method based on an image pyramid according to the present invention.

Fig. 2 is a schematic diagram of four BAYER format arrangements of a common RAW image.

FIG. 3 is a flowchart illustrating a calculation process of an image pyramid in an embodiment of an image denoising method based on an image pyramid.

FIG. 4 is a flowchart of a non-local mean denoising calculation in the embodiment of the image denoising method based on the image pyramid.

The invention is further explained with reference to the drawings and the embodiments.

Detailed Description

The image pyramid-based image denoising method is applied to intelligent electronic equipment, preferably, the intelligent electronic equipment is provided with a camera device such as a camera and the like, the camera device is provided with an image sensor such as a CMOS and the like, the intelligent electronic equipment acquires an initial image by using the camera device, and the method is a method for processing color noise of the initial image acquired by the image sensor. Preferably, the intelligent electronic device is provided with a processor and a memory, the memory stores a computer program, and the processor implements the image pyramid-based image denoising method by executing the computer program.

The embodiment of the image denoising method based on the image pyramid comprises the following steps:

the embodiment mainly aims at an initial image acquired by an image sensor to perform a denoising method, and particularly, in the embodiment, an image area in a range of a center nxn of a pixel to be denoised is used as a processing area on an original RAW image, pixels in the area are downsampled for multiple times to form a gaussian pyramid image, a 2-M layer (M is determined according to the area size) gaussian pyramid image can be obtained according to the area size, then the gaussian pyramid image is processed to obtain 1-M-1 layer laplacian pyramid images, the center pixel is denoised by using an NLM method on each layer of laplacian pyramid image, and finally, a final denoised image is obtained by synthesis. The texture of the image is reserved by the method, the high-quality de-noising image can be obtained, the de-noising calculated amount is less, and the requirement on hardware is low.

The present embodiment will be described in detail with reference to fig. 1. First, step S1 is performed, an initial image is acquired, and the initial image is preprocessed. In the present embodiment, the initial image is an image output by a CMOS image sensor, and the color information of the initial image is RGB information. Since the present embodiment processes an image having YCbCr information, if an initial image output by the CMOS image sensor is an RGB image, the initial image needs to be preprocessed to obtain YCbCr information of each pixel. Where Y is the luminance value of the pixel, Cb is the blue chrominance value of the pixel, and Cr is the red chrominance value of the pixel. The present embodiment deals with color noise of an image based on Cb and Cr of each pixel.

In step S1, the RAW image needs to be preprocessed to convert the RAW image into approximate luminance and chrominance images. Referring to fig. 2, fig. 2 is a four BAYER format arrangement of the most common RAW image, with each pixel in the figure, using pixels within its surrounding 3 × 3 area, resulting in approximate luminance Y and chrominance Cr/Cb information.

Specifically, for the format arrangement of fig. 2(a), the following formula can be used for calculation:

Y＝(4×G11+B01+R10+R12+B21)/16

Cr＝(R10+R12)/2

cb ═ B01+ B21)/2 (formula 3)

For the format arrangement of fig. 2(b), the following formula can be used for calculation:

Y＝(4×G11+B10+R01+R21+B12)/16

Cr＝(R01+R21)/2

cb ═ B10+ B12)/2 (formula 4)

For the format arrangement of fig. 2(c), the following formula can be used for calculation:

Y＝(4×R11+B00+B02+B20+B22+2(G01+G10+G12+G21))/16

Cr＝R11

cb ═ B00+ B02+ B20+ B22)/4 (formula 5)

For the format arrangement of fig. 2(d), the following formula can be used for calculation:

Y＝(4×B11+R00+R02+R20+R22+2(G01+G10+G12+G21))/16

Cr＝(R00+R02+R20+R22)/4

cb ═ B11 (formula 6)

After the image is preprocessed, step S2 is executed to obtain a layered search area of the pixels to be denoised. In this embodiment, each pixel of the image needs to be denoised, so that one pixel in the image is obtained first, and the pixel is the current pixel to be denoised. In this embodiment, a region where M × M pixels centered on a pixel to be denoised are located is used as a hierarchical search region, and the embodiment describes that the value of M is 33.

Then, step S3 is executed to generate a gaussian image pyramid based on the pixels of the hierarchical search area, where the gaussian image pyramid is the first image pyramid in this embodiment. Specifically, image pyramid processing is performed on the color information of the pixels in the hierarchical search area, that is, the Y/Cr/Cb information of the pixels is subjected to pyramid processing. Referring to fig. 3, the following describes in detail the procedure of performing pyramid processing on luminance value information of a pixel, i.e., Y information, and the red color concentration information Cr and the blue color concentration information Cb are processed in the same manner as the luminance value information.

First, step S21 is executed to acquire chroma value information, i.e., Y information, of each pixel in the hierarchical search area, and the first-layer image of the gaussian pyramid having the entire-layer pixels of the hierarchical search area as the gaussian pyramid is executed to step S22, since the range of the hierarchical search area is 33 × 33, the range of the gaussian pyramid first-layer image is also 33 × 33.

Then, step S23 is executed to perform gaussian filtering on each pixel in the hierarchical search area, where the gaussian filtering uses a 5 × 5 gaussian kernel with Sigma ═ 1.4, and the filtering template is:

then, step S24 is executed to perform downsampling on the gaussian-filtered hierarchical search area, specifically, to perform 1/2 downsampling. Specifically, the 1/2 downsampling method is to use the pixel to be denoised as a central point, and to keep the pixels in the even rows and the even columns, that is, to remove the pixels in the odd rows and the odd columns. The downsampling is a sampling method in which the number of pixels after sampling is smaller than the number of pixels of the original image, and the 1/2 downsampling is a sampling method in which the number of pixel rows and the number of columns of the sampled image are 1/2 of the number of rows and the number of columns of the original image.

After the first time of 1/2 downsampling, the gaussian pyramid second layer image is obtained, that is, step S25 is executed, it is seen that the number of rows and the number of columns of the pixel of the gaussian pyramid second layer image are both 17, and thus the range of the gaussian pyramid second layer image is also 17 × 17.

By analogy, gaussian filtering is continuously performed on the second-layer image of the gaussian pyramid, that is, step S29 is executed, filtering is performed by using the filtering template adopted in step S23 when gaussian filtering is performed, step S32 is executed, downsampling is performed on the second-layer image of the gaussian pyramid by 1/2, and the third-layer image of the gaussian pyramid is obtained, that is, step S33 is executed, the number of rows and the number of columns of pixels of the second-layer image of the gaussian pyramid are both 9, so that the range of the third-layer image of the gaussian pyramid is 9 × 9.

Continuing to perform gaussian filtering on the third-layer image of the gaussian pyramid, that is, executing step S35, performing gaussian filtering by using the filtering template adopted in step S23, and executing step S38, performing 1/2 down-sampling on the third-layer image of the gaussian pyramid to obtain the fourth-layer image of the gaussian pyramid, that is, executing step S39, where the number of rows and the number of columns of pixels of the fourth-layer image of the gaussian pyramid are both 5, so that the range of the fourth-layer image of the gaussian pyramid is 5 × 5.

In this embodiment, each layer of the gaussian image pyramid is an image obtained by down-sampling, and thus each layer of the image of the gaussian image pyramid is a layer of down-sampled image.

After obtaining the gaussian pyramid image, step S4 is executed to obtain a laplacian image pyramid from the gaussian image pyramid. Specifically, each downsampled image of the gaussian image pyramid is upsampled, the upsampled image is subtracted from the next layer of image of the downsampled image to obtain a layer of subtraction image, and the plurality of layers of subtraction images form a second image pyramid, namely a laplacian pyramid image.

For example, step S26 is performed for the gaussian pyramid second layer image, and the gaussian pyramid second layer image is up-sampled by 2 times. The sampling mode that the number of the sampled pixels is larger than that of the original image is adopted, and the 2-up-sampling mode is a sampling mode that the number of pixel rows and the number of columns of the sampled image are 2 times of the number of the pixel rows and the number of the columns of the original image. In general, upsampling requires interpolation.

In this embodiment, when upsampling the image of the gaussian pyramid, gaussian filtering is required, for example, filtering is performed by using a 5 × 5 gaussian kernel with Sigma ═ 1.4, and the filtering template is:

after the gaussian pyramid second layer image is up-sampled by 2 times, step S27 is executed, and the image obtained by up-sampling the gaussian pyramid first layer image and the gaussian pyramid second layer image is subtracted from each other, so as to obtain a laplacian pyramid first layer image, that is, step S28 is executed. Since the range of the gaussian pyramid first layer image is 33 × 33 and the range of the gaussian pyramid second layer image after upsampling is 33 × 33, the range of the laplacian pyramid first layer image is 33 × 33.

In this way, 2 times of upsampling is performed on the third layer image of the gaussian pyramid, and a subtraction operation is performed on the second layer image of the gaussian pyramid and the upsampled image of the third layer image of the gaussian pyramid, that is, step S30 is performed, so that the second layer image of the laplacian pyramid is obtained, that is, step S31 is performed. Since the range of the second layer image of the gaussian pyramid is 17 × 17 and the range of the third layer image of the gaussian pyramid after upsampling is also 17 × 17, the range of the second layer image of the laplacian pyramid is also 17 × 17.

Finally, 2 times of upsampling is performed on the image of the fourth layer of the gaussian pyramid, and a subtraction operation is performed on the image of the third layer of the gaussian pyramid and the image obtained by upsampling the image of the fourth layer of the gaussian pyramid, that is, step S36 is performed, so that an image of the third layer of the laplacian pyramid is obtained, that is, step S37 is performed. Since the range of the first-layer image of the gaussian pyramid is 9 × 9 and the range of the fourth-layer image of the gaussian pyramid after upsampling is also 9 × 9, the range of the third-layer image of the laplacian pyramid is also 9 × 9. As can be seen, each layer of image of the laplacian image pyramid is obtained by subtracting the images of the two layers of the laplacian image pyramids through subtraction calculation, so each layer of image of the laplacian image pyramid is a layer of subtraction image, and the plurality of layers of subtraction images form the laplacian image pyramid serving as the second image pyramid. It should be noted that, in the down-sampling and the up-sampling, gaussian filtering is not necessary, i.e. gaussian filtering may not be performed, or similar unimodal functions may be used to achieve similar effects.

After the processing, a Laplacian image pyramid with a three-layer structure is obtained, a high-level image of the Laplacian image pyramid can reflect small-scale image texture information, and a low-level image can reflect large-scale image edge information, so that the image is divided into different frequency bands in a pyramid layering mode, and subsequent fine denoising processing is facilitated.

Then, step S5 is executed to perform non-local mean denoising calculation on each layer image of the laplacian image pyramid and the highest layer image (fourth layer) of the gaussian image pyramid. Specifically, referring to fig. 4, step S51 is executed first to obtain each layer of image of the laplacian image pyramid and the highest layer of image of the gaussian image pyramid, and then step S52 is executed to obtain a denoising search area corresponding to each layer of image.

Taking the luminance value Y information as an example, the denoising search regions corresponding to the first, second, and third level images of the laplacian image pyramid and the highest level image of the gaussian image pyramid are set to be 9 × 9, 5 × 5, and 5 × 5, respectively. As can be seen, the first denoising search region of the laplacian image pyramid is larger than the denoising search region of the highest-level image of the gaussian image pyramid.

Next, step S53 is executed to calculate the preset distance value in each denoising search region within the preset range. The preset range set in this embodiment is a 3 × 3 range, that is, a 3 × 3 range with a pixel to be denoised as a center is used as the preset range, and the preset distance is a simplified gaussian euclidean distance, so that the preset distance value d (i, j) is calculated by using the following formula:

wherein, y (ni) and y (nj) are corresponding pixels in a 3 × 3 area image block centered on a pixel to be denoised and a 3 × 3 area image block centered on an arbitrary pixel of a search area, respectively, and Ga is a 3 × 3 gaussian template, for example, the following templates are used:

next, step S54 is executed to determine whether the preset distance value is smaller than the preset noise threshold NP (Yi, K), i.e. the simplified gaussian euclidean distance is compared with the preset noise threshold NP (Yi, K).

In this embodiment, variables SUM _ Y, SUM _ Cr, SUM _ Cb and CNT are defined, the variables SUM _ Y, SUM _ Cr and SUM _ Cb are the accumulated values of the chrominance values of the pixels meeting the requirement, that is, the accumulated values of the luminance value Y, the red density information Cr and the blue density information Cb of the pixels meeting the requirement, CNT is the number of the pixels meeting the requirement, and the initial values of the variables SUM _ Y, SUM _ Cr, SUM _ Cb and CNT are all 0. Step S54 needs to compare the calculated distance value d (i, j) with a preset noise threshold NP (Yi, K), and determine whether the preset distance value d (i, j) is smaller than the noise threshold NP (Yi, K), if yes, indicating that the 3 × 3 range in the search area is a satisfactory range, step S55 is executed to accumulate the chroma values of each pixel in the 3 × 3 range of the denoised search area to the variables SUM _ Y, SUM _ Cr and SUM _ Cb, and the variable CNT is incremented once. After traversing all 3 × 3 ranges in the target denoising region, the values of the variables SUM _ Y, SUM _ Cr and SUM _ Cb are the SUM of the chrominance values of all pixels in the required 3 × 3 range, and the value of the variable CNT is the number of all pixels in the required 3 × 3 range.

Step S55 may be implemented using the following function:

If(d(i,j)<NP(Yi,K))

{SUM_Y+＝Y(j)；SUM_Cr+＝Cr(j)；SUM_Cb+＝Cb(j)；CNT++；}

preferably, the noise threshold NP (Yi, K) is obtained by calibration in advance according to the characteristics of the CMOS sensor, and the noise threshold NP (Yi, K) is related to the brightness value of the pixel. More preferably, the noise threshold NP (Yi, K) is positively related to the characteristic value of the image sensor and the luminance value of the pixel. The K value of the noise threshold NP (Yi, K) is 0, 1, 2, 3, which respectively represents the first, second, and third layer images of the laplacian pyramid and the highest layer image of the gaussian pyramid, so that the smoothness of each frequency band can be controlled independently.

Of course, the simplified gaussian euclidean distance is adopted in equation 7, and other distances may be used as the predetermined distance value in practical applications, such as a manhattan distance, a minkoch distance, a chebyshev distance, a cosine distance, and the like, and the same effect may be obtained.

If the determination result in step S54 is negative, it indicates that the preset distance value does not meet the set requirement, and the chrominance values of the pixel values in the range are not increased by the value variables SUM _ Y, SUM _ Cr and SUM _ Cb.

Then, step S56 is executed to determine whether each 3 × 3 area in the denoising area has been traversed completely, if not, step S58 is executed to obtain the next 3 × 3 area in the denoising search area, and step S53 is executed again to calculate the preset distance value between the new 3 × 3 area image block and the 3 × 3 area image block centered on the pixel to be denoised.

If the traversal is completed for all 3 × 3 area image blocks within the whole de-noised area, step S57 is executed to calculate the average value of the chrominance values, for example, using the following formula:

Y(i)_NR＝SUM_Y/CNT

Cr(i)_NR＝SUM_Cr/CNT

cb (i) _ NR ═ SUM _ Cb/CNT (formula 8)

Through the steps, chroma values Y (i, K) _ NR, Cr (i, K) _ NR and Cb (i, K) _ NR of the de-noised images of the first layer, the second layer and the third layer of the Laplace pyramid and the image of the highest layer of the Gaussian pyramid are obtained, wherein K is 0, 1, 2 and 3 and respectively represents the images of the first layer, the second layer and the third layer of the Laplace pyramid and the image of the highest layer of the Gaussian pyramid.

As can be seen, in this embodiment, the preset distance value between the denoising search area of each layer of the laplacian pyramid image and the preset range is calculated, the preset distance value between the denoising search area of the highest layer of the laplacian pyramid image and the preset range is calculated, and the average value of the chroma values of all pixels of which the preset distance values are smaller than the noise threshold value is calculated.

The range of 3 × 3 is used as the preset range, and in practical application, the range is not limited to the range of 3 × 3 image blocks, and the preset range may be enlarged or reduced according to practical situations.

To this end, the denoising calculation in step S5 is completed to obtain the denoising chromaticity values Y (i, K) _ NR, Cr (i, K) _ NR, and Cb (i, K) _ NR, and then step S6 is performed to perform image synthesis of the pixels. In this embodiment, the chroma value of each layer of pixels before denoising and the chroma value of each layer of pixels after denoising are used for weighted synthesis. Specifically, the following formula may be used to calculate the synthesized chroma value of each pixel:

Y(i)_final＝W0×Y(i,0)+(1-W0)×Y(i,0)_NR+

W1×Y(i,1)+(1-W1)×Y(i,1)_NR+

W2×Y(i,2)+(1-W2)×Y(i,2)_NR+

w3 XY (i,3) + (1-W3) XY (i,3) _ NR (formula 9)

Cr(i)_final＝W0×Cr(i,0)+(1-W0)×Cr(i,0)_NR+

W1×Cr(i,1)+(1-W1)×Cr(i,1)_NR+

W2×Cr(i,2)+(1-W2)×Cr(i,2)_NR+

W3 XCR (i,3) + (1-W3). times.Cr (i,3) _ NR (formula 10)

Cb(i)_final＝W0×Cb(i,0)+(1-W0)×Cb(i,0)_NR+

W1×Cb(i,1)+(1-W1)×Cb(i,1)_NR+

W2×Cb(i,2)+(1-W2)×Cb(i,2)_NR+

W3 XCb (i,3) + (1-W3). times.Cb (i,3) _ NR (formula 11)

W0, W1, W2 and W3 are preset weighting coefficients, and the denoising degree of each frequency band and the subjective feeling of the final image can be controlled independently.

Since the initial image is a RAW image, that is, an image with RGB color information, step S7 needs to be executed to perform conversion calculation of color information on the output chromaticity value, and obtain the RAW pixel information after denoising through linear transformation. Since the RAW image has a plurality of BAYER format arrangements, for example, the 4 most common BAYER format arrangements shown in fig. 2, conversion is required for different BAYER format arrangements. For example, for the cases of fig. 2(a) and 2(b), i.e., if the center pixel is G, the following formula can be used for calculation:

g _ final ═ 4 × y (i) _ final- (cr (i) _ final + cb (i) _ final)/2 (formula 12)

For the case of fig. 2(c), i.e. if the center pixel is R, the following formula can be used for calculation:

r _ final ═ cr (i) _ final (formula 13)

For the case of fig. 2(d), i.e. if the center pixel is B, the following formula can be used for calculation:

b _ final ═ cb (i) _ final (formula 14)

And then, completing denoising calculation of a pixel to be denoised to obtain an output chromatic value of the pixel to be denoised.

Finally, step S8 is executed to determine whether the denoising calculation of all pixels of the initial image is completed, if the denoising calculation is completed, the data of the image may be output, and if not, step S9 is executed to obtain the next pixel, and step S2 is returned to obtain the layered search area corresponding to the pixel, and the denoising calculation is performed on the pixel until all pixels are calculated, that is, the denoising of the entire image is completed by a sliding window method.

Therefore, in the embodiment, the denoising calculation is performed in the image pyramid mode, so that the overall calculation amount can be reduced by more than 80%, the occupied storage space of the memory is reduced, and the calculation amount of the mean value filtering is also greatly reduced. In addition, the original image is divided into different frequency bands by using the image pyramid, similar image blocks are respectively searched on each frequency band, the accuracy is greatly improved, the denoising strength and the final synthesis coefficient of each frequency band can be independently controlled, and the detail texture and the definition of the image can be effectively maintained while the noise is removed. In addition, the embodiment can be applied to denoising of the RAW image, and meets the use requirements of different sensors.

The embodiment of the computer device comprises:

the computer apparatus of this embodiment may be an intelligent electronic device, and the computer apparatus includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the steps of the image denoising method based on the image pyramid. Of course, the intelligent electronic device further includes a camera device for acquiring an initial image.

For example, a computer program may be partitioned into one or more modules that are stored in a memory and executed by a processor to implement the modules of the present invention. One or more of the modules may be a series of computer program instruction segments capable of performing certain functions, which are used to describe the execution of the computer program in the terminal device.

The Processor may be a Central Processing Unit (CPU), or may be other general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, or the like. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor being the control center of the terminal device and connecting the various parts of the entire terminal device using various interfaces and lines.

The memory may be used to store computer programs and/or modules, and the processor may implement various functions of the terminal device by running or executing the computer programs and/or modules stored in the memory and invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.

A computer-readable storage medium:

the computer program stored in the computer device may be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as a separate product. Based on such understanding, all or part of the processes in the method according to the above embodiments may be implemented by a computer program, which may be stored in a computer readable storage medium and used by a processor to implement the steps of the image denoising method based on the image pyramid.

Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying computer program code, recording medium, U.S. disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution media, and the like. It should be noted that the computer readable medium may contain other components which may be suitably increased or decreased as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, in accordance with legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunications signals.

Finally, it should be emphasized that the present invention is not limited to the above-described embodiments, such as the variation of the filtering template, or the variation of the specific algorithm for performing the mean filtering, and such variations should also be included in the protection scope of the present invention.

Claims

1. An image pyramid-based image denoising method comprises the following steps:

acquiring an initial image, and calculating an output chromatic value of each pixel of the initial image;

wherein calculating the output chrominance value for each pixel comprises:

acquiring a layered search area of pixels to be denoised, acquiring an initial chromatic value of each pixel in the layered search area, performing at least one down-sampling on the layered search area, acquiring a layer of down-sampled image after each down-sampling, and forming a first image pyramid by a plurality of layers of the down-sampled images;

performing up-sampling on each down-sampling image, subtracting the down-sampling image from the up-sampling image to obtain a layer of subtraction image, and forming a second image pyramid by a plurality of layers of subtraction images;

carrying out mean value denoising on each layer of subtraction image of the second image pyramid and the highest layer down-sampling image of the first image pyramid to obtain a denoising chromatic value;

calculating the output chroma value using the initial chroma value and the denoised chroma value.

2. The image pyramid-based image denoising method of claim 1, wherein:

obtaining a denoised chroma value comprises: and setting a denoising search area of each subtraction image and the highest-layer down-sampling image, calculating a preset distance value in a preset range, and calculating an average value of chromatic values of all pixels of which the preset distance values are smaller than a noise threshold value.

3. The image pyramid-based image denoising method of claim 2, wherein:

the noise threshold is positively correlated to the characteristic value of the image sensor and/or the luminance value of the pixel.

4. The image pyramid-based image denoising method of claim 2, wherein:

and the denoising search area of at least one layer of the subtraction image is larger than that of the highest layer of the down-sampling image.

5. The image pyramid-based image denoising method of claim 4, wherein:

calculating preset distance values within a preset range, wherein calculating the average value of the chroma values of all pixels of which the preset distance values are smaller than a noise threshold comprises:

and calculating a preset distance value between the denoising search area of each layer of the subtraction image and a preset range, calculating a preset distance value between the denoising search area of the highest layer of the down-sampling image and the preset range, and calculating an average value of chromatic values of all pixels of which the preset distance values are smaller than a noise threshold value.

6. The image pyramid-based image denoising method of any one of claims 1 to 4, wherein:

calculating the output chroma value using the initial chroma value and the denoised chroma value comprises: and calculating the output colorimetric value by using the initial colorimetric value, the denoised colorimetric values of all layers and a preset weighting coefficient.

7. The image pyramid-based image denoising method of any one of claims 1 to 4, wherein:

the predetermined distance value is one of a simplified gaussian euclidean distance, a manhattan distance, a minkoch distance, a chebyshev distance, and a cosine distance.

8. The image pyramid-based image denoising method of any one of claims 1 to 4, wherein:

executing Gaussian filtering calculation when the hierarchical search area is subjected to down-sampling; and/or

And executing Gaussian filtering calculation when the downsampled image is upsampled.

9. Computer arrangement, characterized in that it comprises a processor and a memory, said memory storing a computer program that, when executed by the processor, performs the steps of the image pyramid based image denoising method according to any one of claims 1 to 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when being executed by a processor, implements the steps of the image pyramid-based image denoising method according to any one of claims 1 to 8.