CN113935923A - Image noise reduction method, device, equipment and storage medium - Google Patents
Image noise reduction method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113935923A CN113935923A CN202111306038.8A CN202111306038A CN113935923A CN 113935923 A CN113935923 A CN 113935923A CN 202111306038 A CN202111306038 A CN 202111306038A CN 113935923 A CN113935923 A CN 113935923A
- Authority
- CN
- China
- Prior art keywords
- frame image
- current frame
- image
- motion information
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000007499 fusion processing Methods 0.000 claims abstract description 34
- 238000012545 processing Methods 0.000 claims abstract description 33
- 230000004927 fusion Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 3
- 238000006073 displacement reaction Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/207—Analysis of motion for motion estimation over a hierarchy of resolutions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the application provides an image noise reduction method, an image noise reduction device, image noise reduction equipment and a storage medium, wherein the method comprises the steps of receiving a current frame image collected by a camera; if the reference frame image is stored in the cache, acquiring the reference frame image; performing motion estimation according to the reference frame image and the current frame image, acquiring motion information of the current frame image, and storing the motion information of the current frame image and the current frame image into a cache; and acquiring the current frame image, the reference frame image and the motion information of the current frame image in a cache, and performing fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image to obtain the noise-reduced current frame image. According to the method and the device, the data processing amount in the online mode can be reduced, so that online resources occupied by 3D noise reduction are reduced, power consumption is reduced, the cost of a chip is reduced, and user experience is improved.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image denoising method, apparatus, device, and storage medium.
Background
At present, various noises are inevitably polluted in the process of collecting, transmitting and displaying videos/images, for example, an image sensor in a camera can generate noises under the influence of the working environment, the quality of a sensing component and the like when the image sensor acquires images. The noise reduction not only can make the subjective feeling of the image/video better, but also can make the image/video compressed without wasting the code rate on the coding noise. The noise reduction algorithm includes 2d noise reduction and 3d noise reduction. The 2D denoising is only a denoising process in the spatial domain. 3D noise reduction increases temporal noise reduction relative to 2D noise reduction. Compare 2D and fall the noise, the effect of 3D making an uproar is fallen is better, therefore prior art, the 3D that generally adopts falls the noise.
In the prior art, in order to reduce display delay, an image processor performs correlation processing on an image in an online (online) mode. That is, at this time, after the image sensor of the camera acquires the image, the image is not cached in the memory, but is directly transmitted to the relevant processing module for processing. Based on this, the 3D noise reduction processing of the image by the image processor is also carried out in the online mode, so that the acquisition and the related processing can be carried out at the same time, the display is carried out at the same time, and the real-time performance is higher. However, with the improvement of user requirements, the resolution of images acquired by the image sensor is higher and higher, and when 3D denoising processing is performed on images with higher resolution, more online resources are occupied, power consumption is increased, even overflow occurs, display delay is increased, and user experience is reduced.
Disclosure of Invention
In view of this, the present application provides an image denoising method, apparatus, device and storage medium, so as to solve the problems in the prior art that when performing 3D denoising on an image, more online resources are occupied and power consumption is increased.
In a first aspect, an embodiment of the present application provides an image denoising method, where the method includes: receiving a current frame image collected by a camera;
if the reference frame image is stored in the cache, acquiring the reference frame image;
performing motion estimation according to the reference frame image and the current frame image, acquiring motion information of the current frame image, and storing the motion information of the current frame image and the current frame image into a cache;
and acquiring the current frame image, the reference frame image and the motion information of the current frame image in a cache, and performing fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image to obtain the noise-reduced current frame image.
Preferably, the method further comprises the following steps:
and when the reference frame image is stored in the cache, updating the reference frame image stored in the cache according to the current frame image after noise reduction.
Preferably, the method further comprises the following steps:
and when the reference frame image is not stored in the buffer memory, storing the current frame image as the reference frame image into the buffer memory.
Preferably, the performing motion estimation according to the reference frame image and the current frame image to obtain motion information of the current frame image, and storing the motion information of the current frame image and the current frame image in a buffer includes:
dividing the current frame image into at least two target image blocks;
for each target image block of the at least two target image blocks, determining a reference image block corresponding to the target image block in the reference frame image;
acquiring motion information of a target image block according to the target image block and the reference image block, and storing the motion information of the target image block and the target image block into a cache;
the obtaining, in the cache, the motion information of the current frame image, the reference frame image, and the current frame image, and performing fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image, the reference frame image, and the current frame image to obtain the noise-reduced current frame image includes:
acquiring motion information of the target image block, the reference image block and the target image block in a cache, and performing fusion processing on the target image block and the reference image block according to the motion information of the target image block, the reference image block and the target image block to obtain a target image block subjected to noise reduction;
and obtaining the current frame image subjected to noise reduction according to the at least two target image blocks subjected to noise reduction.
Preferably, the method further comprises the following steps:
and storing the current frame image subjected to noise reduction into a cache.
In a second aspect, an embodiment of the present application provides an image noise reduction apparatus, including:
the receiving unit is used for receiving the current frame image collected by the camera;
the device comprises an acquisition unit, a storage unit and a processing unit, wherein the acquisition unit is used for acquiring a reference frame image when the reference frame image is stored in a cache;
the motion estimation unit is used for carrying out motion estimation according to the reference frame image and the current frame image, acquiring the motion information of the current frame image and storing the motion information of the current frame image and the current frame image into a cache;
and the fusion unit is used for acquiring the current frame image, the reference frame image and the motion information of the current frame image in a cache, and performing fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image to obtain the current frame image subjected to noise reduction.
Preferably, the fusion unit is further configured to update the reference frame image stored in the buffer according to the current frame image after noise reduction when the reference frame image is stored in the buffer.
Preferably, the merging unit is further configured to store the current frame image as a reference frame image into a buffer when the reference frame image is not stored in the buffer.
Preferably, the motion estimation unit is specifically configured to divide the current frame image into at least two target image blocks;
for each target image block of the at least two target image blocks, determining a reference image block corresponding to the target image block in the reference frame image;
acquiring motion information of a target image block according to the target image block and the reference image block, and storing the motion information of the target image block into a cache;
the fusion unit is specifically configured to obtain motion information of the target image block, the reference image block, and the target image block in a cache, and perform fusion processing on the target image block and the reference image block according to the motion information of the target image block, the reference image block, and the target image block to obtain a denoised target image block;
and obtaining the current frame image subjected to noise reduction according to the at least two target image blocks subjected to noise reduction.
Preferably, the fusion unit is further configured to store the noise-reduced current frame image into a buffer.
In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores a computer program, and when the computer program is executed, the electronic device is caused to execute the method of the first aspect.
In a fourth aspect, an embodiment of the present application provides a storage medium, where the computer-readable storage medium includes a stored program, where when the program runs, the apparatus where the computer-readable storage medium is located is controlled to execute the method in the first aspect.
By adopting the scheme provided by the embodiment of the application, the current frame image collected by the camera is received, and if the reference frame image is stored in the cache, the reference frame image is obtained; the method comprises the steps of carrying out motion estimation according to a reference frame image and a current frame image, obtaining motion information of the current frame image, storing the motion information of the current frame image and the current frame image into a cache, obtaining the current frame image, the motion information of the current frame image and the reference frame image from the cache, and carrying out fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image to obtain the current frame image subjected to noise reduction. Therefore, in the embodiment of the application, when the current frame image is acquired by the camera, the reference frame image is acquired, and the motion estimation is performed in real time according to the received current frame image and the reference frame image, so that the online real-time motion estimation is realized, and the motion information of the current frame image is acquired. In order to reduce the data processed online in real time, the current frame image and the motion information of the current frame image can be stored in a cache, the current frame image, the reference frame image and the motion information of the current frame image are subsequently acquired from the cache, and the current frame image and the reference frame image are fused according to the motion information of the current frame image, so that the noise reduction of the current frame image is realized, and the current frame image after the noise reduction is obtained. That is to say, when the noise of the current frame image is reduced, the motion estimation processing can be performed on line in real time, and the fusion processing can be performed in an off-line mode, so that the data processing amount in the on-line mode can be reduced, the on-line resources occupied by the 3D noise reduction are reduced, the power consumption is reduced, the cost of a chip is reduced, and the user experience is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
Fig. 1 is a schematic flowchart of an image denoising method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of another image denoising method according to an embodiment of the present application;
fig. 3 is a schematic flowchart of another image denoising method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of another image denoising method according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an image noise reduction apparatus according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For better understanding of the technical solutions of the present application, the following detailed descriptions of the embodiments of the present application are provided with reference to the accompanying drawings.
It should be understood that the embodiments described are only a few embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the examples of this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be understood that the term "and/or" as used herein is merely one type of associative relationship that describes an associated object, meaning that three types of relationships may exist, e.g., A and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
Before specifically describing the embodiments of the present application, terms applied or likely to be applied to the embodiments of the present application will be explained first.
User experience (UX): which may also be referred to as the UX feature, refers to the user's experience during the shooting process using the electronic device.
The 3DNR (Noise Reduction) monitoring camera can find out the position of a Noise point by comparing and screening the images of the front frame and the rear frame and perform gain control on the position of the Noise point, and the 3D digital Noise Reduction function can reduce the Noise interference of a weak signal image.
In the prior art, in order to reduce display delay, an image processor performs correlation processing on an image in an online (online) mode. That is, at this time, after the image sensor of the camera acquires the image, the image is not cached in the memory, but is directly transmitted to the relevant processing module for processing. Based on this, the 3D noise reduction processing of the image by the image processor is also carried out in the online mode, so that the acquisition and the related processing can be carried out at the same time, the display is carried out at the same time, and the real-time performance is higher. However, with the improvement of user requirements, the resolution of images acquired by the image sensor is higher and higher, and when 3D denoising processing is performed on images with higher resolution, more online resources are occupied, power consumption is increased, even overflow occurs, display delay is increased, and user experience is reduced.
In order to solve the above problem, an embodiment of the present application provides an image denoising method, which receives a current frame image acquired by a camera, and if a reference frame image is stored in a cache, acquires the reference frame image; the method comprises the steps of carrying out motion estimation according to a reference frame image and a current frame image, obtaining motion information of the current frame image, storing the motion information of the current frame image and the current frame image into a cache, obtaining the current frame image, the motion information of the current frame image and the reference frame image from the cache, and carrying out fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image to obtain the current frame image subjected to noise reduction. Therefore, in the embodiment of the application, when the current frame image is acquired by the camera, the reference frame image is acquired, and the motion estimation is performed in real time according to the received current frame image and the reference frame image, so that the online real-time motion estimation is realized, and the motion information of the current frame image is acquired. In order to reduce the data processed online in real time, the current frame image and the motion information of the current frame image can be stored in a cache, the current frame image, the reference frame image and the motion information of the current frame image are subsequently acquired from the cache, and the current frame image and the reference frame image are fused according to the motion information of the current frame image, so that the noise reduction of the current frame image is realized, and the current frame image after the noise reduction is obtained. That is to say, when the noise of the current frame image is reduced, the motion estimation processing can be performed on line in real time, and the fusion processing can be performed in an off-line mode, so that the data processing amount in the on-line mode can be reduced, the on-line resources occupied by the 3D noise reduction are reduced, the power consumption is reduced, the cost of a chip is reduced, and the user experience is improved. The details will be described below.
Referring to fig. 1, a schematic flow chart of an image denoising method according to an embodiment of the present application is shown. As shown in fig. 1, the image noise reduction method includes:
and S101, receiving a current frame image acquired by a camera.
In the embodiment of the application, after the camera collects the current frame image, the collected current frame image is directly transmitted to the image noise reduction device, namely the real-time collection and real-time transmission are carried out, and the current frame image is obtained in an online mode.
The current frame image can be a video frame image in a YUV space, YUV is an image format and is composed of Y, U, V parts, and Y represents brightness, namely a gray-scale value; u and V represent the chromaticity of the color, respectively, and serve to describe the color affecting the color and saturation for a given pixel.
And step S102, if the reference frame image is stored in the cache, acquiring the reference frame image.
In the embodiment of the application, due to reasons such as image acquisition environment and equipment displacement, the noise basically exists in the image acquired by the camera, and in order to display a clear image, the acquired image can be subjected to noise reduction processing. Because the video image sequence frame pre-frames have stronger time domain characteristics and the two frames of images are similar, the noise information can be found out by comparing the pixel difference between the two frames of images before and after the comparison. Therefore, a reference frame image needs to be acquired. At this time, the image noise reduction apparatus may detect whether the buffer contains a reference frame image, and if there is a reference frame image, acquire the reference frame image.
Further, in the embodiment of the present application, the reference frame image may be a noise-reduced image of a previous frame of the current frame image.
It should be noted that, when the current frame image is the first frame image acquired, the reference frame image is not stored in the buffer, and at this time, the image denoising device may consider that the first frame image is an image without noise, and store the first frame image directly as a denoised image into the buffer for display. And, the first frame image is stored in the buffer memory as the reference frame image of the next frame image.
That is, when the reference frame image is not stored in the buffer, the current frame image is stored in the buffer as the reference frame image.
Step S103, performing motion estimation according to the reference frame image and the current frame image, acquiring motion information of the current frame image, and storing the motion information of the current frame image and the current frame image into a cache.
In the embodiment of the application, the image denoising device acquires the reference frame image from the cache, and after receiving the current frame image acquired by the camera, the image denoising device can perform motion estimation on the reference frame image and the current frame image to obtain a motion estimation result, namely the relative displacement between the current frame image and the reference frame image, so that the motion information of the current frame image can be acquired. And storing the acquired motion information of the current frame image and the current frame image into a cache so as to facilitate the subsequent process of noise reduction processing to be carried out in an off-line mode.
Further, in order to perform motion estimation quickly and accurately, the current frame image may be divided into a plurality of image blocks, motion estimation is performed for each image block, at this time, motion estimation is performed according to the reference frame image and the current frame image, motion information of the current frame image is obtained, and storing the motion information of the current frame image and the current frame image in a cache includes:
dividing a current frame image into at least two target image blocks; for each target image block of at least two target image blocks, determining a reference image block corresponding to the target image block in a reference frame image; and acquiring motion information of the target image block according to the target image block and the reference image block, and storing the motion information of the target image block and the target image block into a cache.
That is, the image noise reduction apparatus may divide the current frame image into at least two target image blocks, for example, into three target image blocks. The following is performed for each of the at least two target image blocks. And in the reference frame image, finding out a reference image block matched with the current target image block. And performing motion estimation according to the target image block and the reference image block, namely calculating the relative displacement between the target image block and the reference image block to obtain the motion information of the current target image block, and storing the motion information of the current target image block and the current target image block into a cache.
Further, the basic idea of motion estimation is: dividing each frame image in the image sequence into a plurality of complementary overlapped target image blocks, considering that the displacement of all pixels in the target image blocks are the same, then finding out the block which is the most similar to the image block of the current frame image in a given search range from each target image block to the reference frame image according to a matching criterion, namely the reference image block, wherein the relative displacement between the reference image block and the target image block is a motion vector, and the motion information of the target image block. When the video is compressed, the current image block can be completely restored only by storing the motion vector and the residual data.
And step S104, acquiring the current frame image, the reference frame image and the motion information of the current frame image in the cache, and fusing the current frame image and the reference frame image according to the motion information of the current frame image to obtain the current frame image subjected to noise reduction.
In the embodiment of the present application, the image denoising device obtains the current frame image, the reference frame image, and the motion information of the current frame image in the buffer, and according to the motion information of the current frame image, the current frame image and the reference frame image may be subjected to corresponding fusion processing, for example, the pixel value of the current frame image is adjusted according to the reference frame image, so as to achieve the purpose of denoising.
In a possible implementation manner, the image denoising device may determine the pixel weight of the current frame image and the pixel weight of the reference frame image according to the motion information of the current frame image, and further calculate the pixel weighting evaluation values of the pixel of the current frame image and the pixel of the reference frame image according to the pixel weight of the current frame image and the pixel weight of the reference frame image, so as to obtain the updated pixel value of the current frame image.
In the embodiment of the present application, the pixel weight of the current frame image is determined according to the motion information of the current frame image, and weight values corresponding to different motion information may be preset. For example, if the motion vector in the motion information of the current frame image is smaller than the first threshold, the pixel weight of the current frame image may be set to the weight a, and if the motion vector in the motion information of the current frame image is not smaller than the first threshold, the pixel weight of the current frame image may be set to the weight b. Wherein a and b are both numbers greater than 0 and less than 1.
It should be noted that, fusion processing may also be performed on the current frame image and the reference frame image according to the motion information of the current frame image in other manners, which is not limited in this application.
Further, when the current frame image is divided into at least two target image blocks in step 103, step 104 obtains the current frame image, the reference frame image and the motion information of the current frame image in the cache, and performs fusion processing on the current frame image and the reference frame image according to the current frame image, the reference frame image and the motion information of the current frame image, so as to obtain the noise-reduced current frame image, including:
acquiring motion information of a target image block, a reference image block and a target image block in a cache, and performing fusion processing on the target image block and the reference image block according to the motion information of the target image block, the reference image block and the target image block to obtain a target image block subjected to noise reduction; and obtaining the current frame image subjected to noise reduction according to the at least two target image blocks subjected to noise reduction.
That is, when the current frame image is divided into at least two target image blocks, the following operation may be performed for each target image block. The image noise reduction device obtains a stored target image block, motion information of the target image block and a reference image block of the target image block in a cache. And performing fusion processing on the target image block and the reference image block according to the motion information of the target image block to obtain the target image block subjected to noise reduction. And forming the current frame image subjected to noise reduction by using at least two target image blocks subjected to noise reduction.
Further, for convenience of acquisition, in the embodiment of the present application, the motion information of the current frame image may be stored in a registry file (REG). Of course, the motion information of the current frame image may also be stored in other locations, which is not limited in this application.
Thus, in the embodiment of the present application, the image noise reduction apparatus performs the steps 101-103 in real time, that is, the steps 101-103 are performed in an online mode. The image noise reduction device carries out motion estimation on the current frame image and the reference frame image in an online mode, stores the motion information of the current frame image and the current frame image into a cache, the image noise reduction device reads the current frame image, the reference frame image and the motion information of the current frame image in the buffer memory, and carries out fusion processing on the current frame image and the reference frame image, that is, the image noise reduction apparatus performs step S104 in an off-line mode, that is, after motion estimation, stores the result of motion estimation in a buffer, the current frame image and the reference frame image are not immediately subjected to the fusion process according to the result of the motion estimation, the result of motion estimation, namely the motion information of the current frame image, is firstly stored in a cache, when fusion processing is needed, the image noise reduction device reads the motion information of the current frame image in the cache, and performing fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image. The fusion processing of the current frame image and the reference frame image is set to be under-line mode processing, so that the occupation of on-line resources can be reduced, and the power consumption is reduced.
A ZSL (zero shot) shooting scene will be described as an example. Assume that a 24M single rear camera is employed in the electronic device. The ZSL shooting scene is divided into a preview stage and a capture stage. The current frame image is an Nth frame image, 24M RAW data is output by a 24M single rear camera, FHD (Full High Definition) YUV420 image data is output by a preview path of the electronic equipment, and 24MYUV420 image data is output by a capture path at the same time. In a preview stage, when the 3DNR function is started, an image noise reduction device in the electronic equipment starts to work, a 24M single rear camera transmits acquired FHD YUV420 image data to the image noise reduction device, the image noise reduction device acquires an N-1 th frame reference image from a cache, motion estimation is carried out on the N-1 th frame image and the N-1 th frame reference image, and a motion vector value is calculated, namely motion information of the N-th frame image is acquired. The image denoising apparatus stores the motion information of the nth frame image, the preview image data corresponding to the nth frame image, and the capture image data into a buffer memory, as shown in fig. 2. In the preview stage, the image denoising device only needs to process the FHD YUV420 image output by the preview path. The image noise reduction device obtains the stored N frame preview image from the buffer memory, namely the motion information of the N frame FHD YUV420 image, the N-1 frame reference image and the N frame image. And according to the motion information of the N frame image, fusing the N frame preview image and the reference image of the N-1 frame to obtain the N frame preview image after noise reduction, and simultaneously storing the N frame preview image after noise reduction into a cache to be used as the reference frame of the N +1 frame preview image.
In the capture stage, the capture path stops outputting 24M YUV420 image data, and at this time, the image denoising device of the electronic device may perform denoising and merging processing on the image in the capture image buffer, taking merging of 5 frames of images as an example for description. The image noise reduction device needs to fuse the N-4 th, N-3 th, N-2 th, N-1 th and N-frame images. When the image noise reduction device carries out fusion processing on the N-4 th frame image, if the N-4 th frame image is the first frame image, the fusion with other frame images is not needed. At this time, the image noise reduction device directly uses the YUV420 image data of the N-4 th frame as a reference frame image corresponding to the image of the N-3 rd frame. When the image noise reduction device performs fusion processing on the N-3 frame image, the image noise reduction device acquires the motion information of the N-3 frame image from the cache, and calls a function provided by the algo group to convert the motion information into the motion information corresponding to 24M. Wherein, the motion information of the N-3 frame image can be calculated in the preview phase. And the image denoising device performs fusion processing on the N-3 frame image and the N-4 frame image according to the motion information of the N-3 frame image to obtain a denoised N-3 frame image. And taking the N-3 frame image after noise reduction as a reference frame image of the N-2 frame image, and storing the reference frame image into a cache. By the mode, the corresponding fusion processing is carried out on the N-2 frame image, the N-1 frame image and the N frame image. Storing the N frame image after noise reduction into a buffer memory as a reference frame image of a subsequent N +1 frame image, and performing subsequent processing on the N frame image, as shown in fig. 3.
Fig. 4 is a schematic flowchart illustrating another image denoising method according to an embodiment of the present application. As shown in fig. 4, the method includes:
and S401, receiving the current frame image collected by the camera.
Specifically, refer to step S101, which is not described herein again.
It should be noted that the current frame image received by the image noise reduction apparatus and collected by the camera may be the first frame image collected by the camera, or may not be the first frame image. If the current frame image is the first frame image, the image denoising device does not receive other images at this time, and no reference frame image is stored in the buffer, and at this time, the current frame image is considered to be an image without noise, the current frame image is directly subjected to subsequent display processing, the current frame image is stored in the buffer as a reference frame image of the next frame image, and the current frame image is stored in the buffer as a denoised current frame image for subsequent display processing, at this time, the following step S402a is executed, and step S406 is executed. If the current frame image is not the first frame image, the following steps S402b, steps S403-S406 may be performed.
In step S402a, when the reference frame image is not stored in the buffer, the current frame image is stored in the buffer as the reference frame image.
In the embodiment of the present application, if no reference frame image is stored in the cache, it indicates that the currently acquired current frame image is the first frame image, and at this time, the reference frame image serving as the next frame image may be directly stored in the cache.
Step S402b, if the reference frame image is stored in the buffer, acquiring the reference frame image.
Specifically, refer to step S102, which is not described herein again.
Step S403, performing motion estimation according to the reference frame image and the current frame image, acquiring motion information of the current frame image, and storing the motion information of the current frame image and the current frame image in a cache.
Specifically, refer to step S103, which is not described herein again.
Step S404, obtaining the current frame image, the reference frame image and the motion information of the current frame image in the cache, and performing fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image to obtain the current frame image after noise reduction.
Specifically, refer to step S104, which is not described herein again.
And step S405, when the reference frame image is stored in the cache, updating the reference frame image stored in the cache according to the current frame image after noise reduction.
In this embodiment of the application, the image denoising device may use the current frame image after denoising as a reference frame image of the next frame image, at this time, the reference frame image stored in the cache needs to be updated, and at this time, the image denoising device may update the reference frame image stored in the cache to the current frame image after denoising so as to be used as the reference frame image of the next frame image.
And step S406, storing the current frame image subjected to noise reduction into a cache.
In this embodiment, the image denoising device may store the current frame image after denoising in a buffer memory, so as to perform corresponding display processing on the current frame image subsequently, and display the current frame image.
Therefore, in the embodiment of the application, when the current frame image is acquired by the camera, the reference frame image is acquired, and the motion estimation is performed in real time according to the received current frame image and the reference frame image, so that the online real-time motion estimation is realized, and the motion information of the current frame image is acquired. In order to reduce the data processed online in real time, the current frame image and the motion information of the current frame image can be stored in a cache, the current frame image, the reference frame image and the motion information of the current frame image are subsequently acquired from the cache, and the current frame image and the reference frame image are fused according to the motion information of the current frame image, so that the noise reduction of the current frame image is realized, and the current frame image after the noise reduction is obtained. That is to say, when the noise of the current frame image is reduced, the motion estimation processing can be performed on line in real time, and the fusion processing can be performed in an off-line mode, so that the data processing amount in the on-line mode can be reduced, the on-line resources occupied by the 3D noise reduction are reduced, the power consumption is reduced, the cost of a chip is reduced, and the user experience is improved.
Fig. 5 is a schematic structural diagram of an image noise reduction apparatus according to an embodiment of the present disclosure. As shown in fig. 5, the image noise reduction apparatus includes:
the receiving unit 501 is configured to receive a current frame image acquired by a camera.
An obtaining unit 502, configured to obtain the reference frame image when the reference frame image is stored in the buffer.
The motion estimation unit 503 is configured to perform motion estimation according to the reference frame image and the current frame image, acquire motion information of the current frame image, and store the motion information of the current frame image and the current frame image in a buffer.
Specifically, the motion estimation unit 503 is specifically configured to divide the current frame image into at least two target image blocks; for each target image block of at least two target image blocks, determining a reference image block corresponding to the target image block in a reference frame image; and acquiring the motion information of the target image block according to the target image block and the reference image block, and storing the motion information of the target image block into a cache.
The fusion unit 504 is configured to obtain the current frame image, the reference frame image, and the motion information of the current frame image in the buffer, and perform fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image to obtain the current frame image after noise reduction.
Specifically, the fusion unit 504 is specifically configured to obtain motion information of the target image block, the reference image block, and the target image block in the cache, and perform fusion processing on the target image block and the reference image block according to the motion information of the target image block, the reference image block, and the target image block to obtain a target image block subjected to noise reduction; and obtaining the current frame image subjected to noise reduction according to the at least two target image blocks subjected to noise reduction.
Further, the merging unit 504 is further configured to, when the reference frame image is stored in the buffer, update the reference frame image stored in the buffer according to the current frame image after noise reduction.
Further, the merging unit 504 is further configured to store the current frame image as the reference frame image into the buffer when the reference frame image is not stored in the buffer.
Further, the merging unit 504 is further configured to store the noise-reduced current frame image in a buffer.
Therefore, in the embodiment of the application, when the current frame image is acquired by the camera, the reference frame image is acquired, and the motion estimation is performed in real time according to the received current frame image and the reference frame image, so that the online real-time motion estimation is realized, and the motion information of the current frame image is acquired. In order to reduce the data processed online in real time, the current frame image and the motion information of the current frame image can be stored in a cache, the current frame image, the reference frame image and the motion information of the current frame image are subsequently acquired from the cache, and the current frame image and the reference frame image are fused according to the motion information of the current frame image, so that the noise reduction of the current frame image is realized, and the current frame image after the noise reduction is obtained. That is to say, when the noise of the current frame image is reduced, the motion estimation processing can be performed on line in real time, and the fusion processing can be performed in an off-line mode, so that the data processing amount in the on-line mode can be reduced, the on-line resources occupied by the 3D noise reduction are reduced, the power consumption is reduced, the cost of a chip is reduced, and the user experience is improved.
Corresponding to the embodiment, the application further provides the electronic equipment. Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, where the electronic device 600 may include: a processor 601, a memory 602, and a communication unit 603. The components communicate over one or more buses, and those skilled in the art will appreciate that the configuration of the servers shown in the figures are not meant to limit embodiments of the present invention, and may be in the form of buses, stars, more or fewer components than those shown, some components in combination, or a different arrangement of components.
The communication unit 603 is configured to establish a communication channel, so that the storage device can communicate with other devices. Receiving the user data sent by other devices or sending the user data to other devices.
The processor 601, which is a control center of the storage device, connects various parts of the entire electronic device using various interfaces and lines, and executes various functions of the electronic device and/or processes data by running or executing software programs and/or modules stored in the memory 602 and calling data stored in the memory. The processor may be composed of Integrated Circuits (ICs), for example, a single packaged IC, or a plurality of packaged ICs connected to the same or different functions. For example, the processor 601 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.
The memory 602 is used for storing instructions executed by the processor 601, and the memory 602 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
The execution of the instructions in the memory 602, when executed by the processor 601, enables the electronic device 600 to perform some or all of the steps in the embodiment shown in fig. 4.
In a specific implementation, the present invention further provides a computer storage medium, where the computer storage medium may store a program, and the program may include some or all of the steps in each embodiment of the image denoising method provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a Random Access Memory (RAM), or the like.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
The same and similar parts in the various embodiments in this specification may be referred to each other. Especially, as for the device embodiment and the terminal embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the description in the method embodiment.
Claims (12)
1. A method for image noise reduction, the method comprising:
receiving a current frame image collected by a camera;
if the reference frame image is stored in the cache, acquiring the reference frame image;
performing motion estimation according to the reference frame image and the current frame image, acquiring motion information of the current frame image, and storing the motion information of the current frame image and the current frame image into a cache;
and acquiring the current frame image, the reference frame image and the motion information of the current frame image in a cache, and performing fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image to obtain the noise-reduced current frame image.
2. The method of claim 1, further comprising:
and when the reference frame image is stored in the cache, updating the reference frame image stored in the cache according to the current frame image after noise reduction.
3. The method of claim 1, further comprising:
and when the reference frame image is not stored in the buffer memory, storing the current frame image as the reference frame image into the buffer memory.
4. The method of claim 1, wherein the performing motion estimation according to the reference frame image and the current frame image to obtain motion information of the current frame image, and storing the motion information of the current frame image and the current frame image in a buffer comprises:
dividing the current frame image into at least two target image blocks;
for each target image block of the at least two target image blocks, determining a reference image block corresponding to the target image block in the reference frame image;
acquiring motion information of a target image block according to the target image block and the reference image block, and storing the motion information of the target image block and the target image block into a cache;
the obtaining, in the cache, the motion information of the current frame image, the reference frame image, and the current frame image, and performing fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image, the reference frame image, and the current frame image to obtain the noise-reduced current frame image includes:
acquiring motion information of the target image block, the reference image block and the target image block in a cache, and performing fusion processing on the target image block and the reference image block according to the motion information of the target image block, the reference image block and the target image block to obtain a target image block subjected to noise reduction;
and obtaining the current frame image subjected to noise reduction according to the at least two target image blocks subjected to noise reduction.
5. The method according to any one of claims 1-4, further comprising:
and storing the current frame image subjected to noise reduction into a cache.
6. An image noise reduction apparatus, comprising:
the receiving unit is used for receiving the current frame image collected by the camera;
the device comprises an acquisition unit, a storage unit and a processing unit, wherein the acquisition unit is used for acquiring a reference frame image when the reference frame image is stored in a cache;
the motion estimation unit is used for carrying out motion estimation according to the reference frame image and the current frame image, acquiring the motion information of the current frame image and storing the motion information of the current frame image and the current frame image into a cache;
and the fusion unit is used for acquiring the current frame image, the reference frame image and the motion information of the current frame image in a cache, and performing fusion processing on the current frame image and the reference frame image according to the motion information of the current frame image to obtain the current frame image subjected to noise reduction.
7. The apparatus of claim 6,
and the fusion unit is also used for updating the reference frame image stored in the cache according to the current frame image after noise reduction when the reference frame image is stored in the cache.
8. The apparatus of claim 6,
and the fusion unit is also used for storing the current frame image as a reference frame image into the cache when the reference frame image is not stored in the cache.
9. The apparatus of claim 6,
the motion estimation unit is specifically configured to divide the current frame image into at least two target image blocks;
for each target image block of the at least two target image blocks, determining a reference image block corresponding to the target image block in the reference frame image;
acquiring motion information of a target image block according to the target image block and the reference image block, and storing the motion information of the target image block into a cache;
the fusion unit is specifically configured to obtain motion information of the target image block, the reference image block, and the target image block in a cache, and perform fusion processing on the target image block and the reference image block according to the motion information of the target image block, the reference image block, and the target image block to obtain a denoised target image block;
and obtaining the current frame image subjected to noise reduction according to the at least two target image blocks subjected to noise reduction.
10. The apparatus according to any one of claims 6 to 9,
and the fusion unit is also used for storing the current frame image subjected to noise reduction into a cache.
11. An electronic device, comprising a processor and a memory, the memory storing a computer program that, when executed, causes the electronic device to perform the method of any of claims 1-5.
12. A storage medium, comprising a stored program, wherein the program, when executed, controls an apparatus in which the computer-readable storage medium resides to perform the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111306038.8A CN113935923A (en) | 2021-11-05 | 2021-11-05 | Image noise reduction method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111306038.8A CN113935923A (en) | 2021-11-05 | 2021-11-05 | Image noise reduction method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113935923A true CN113935923A (en) | 2022-01-14 |
Family
ID=79285832
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111306038.8A Pending CN113935923A (en) | 2021-11-05 | 2021-11-05 | Image noise reduction method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113935923A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114782278A (en) * | 2022-04-29 | 2022-07-22 | 深圳市道通智能航空技术股份有限公司 | Image denoising method, device and system and electronic equipment |
| CN115841425A (en) * | 2022-07-21 | 2023-03-24 | 爱芯元智半导体(上海)有限公司 | Video noise reduction method and device, electronic equipment and computer readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103905730A (en) * | 2014-03-24 | 2014-07-02 | 深圳市中兴移动通信有限公司 | Shooting method of mobile terminal and mobile terminal |
| CN104809710A (en) * | 2015-05-14 | 2015-07-29 | 上海兆芯集成电路有限公司 | Image denoising method and device using image denoising method |
| CN105338221A (en) * | 2014-08-15 | 2016-02-17 | 联想(北京)有限公司 | Image processing method and electronic equipment |
| CN111556227A (en) * | 2020-05-19 | 2020-08-18 | 广州市百果园信息技术有限公司 | Video denoising method and device, mobile terminal and storage medium |
| CN113132637A (en) * | 2021-04-19 | 2021-07-16 | Oppo广东移动通信有限公司 | Image processing method, image processing chip, application processing chip and electronic equipment |
| CN113538265A (en) * | 2021-07-06 | 2021-10-22 | Oppo广东移动通信有限公司 | Image denoising method and device, computer readable medium and electronic device |
-
2021
- 2021-11-05 CN CN202111306038.8A patent/CN113935923A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103905730A (en) * | 2014-03-24 | 2014-07-02 | 深圳市中兴移动通信有限公司 | Shooting method of mobile terminal and mobile terminal |
| CN105338221A (en) * | 2014-08-15 | 2016-02-17 | 联想(北京)有限公司 | Image processing method and electronic equipment |
| CN104809710A (en) * | 2015-05-14 | 2015-07-29 | 上海兆芯集成电路有限公司 | Image denoising method and device using image denoising method |
| CN111556227A (en) * | 2020-05-19 | 2020-08-18 | 广州市百果园信息技术有限公司 | Video denoising method and device, mobile terminal and storage medium |
| CN113132637A (en) * | 2021-04-19 | 2021-07-16 | Oppo广东移动通信有限公司 | Image processing method, image processing chip, application processing chip and electronic equipment |
| CN113538265A (en) * | 2021-07-06 | 2021-10-22 | Oppo广东移动通信有限公司 | Image denoising method and device, computer readable medium and electronic device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114782278A (en) * | 2022-04-29 | 2022-07-22 | 深圳市道通智能航空技术股份有限公司 | Image denoising method, device and system and electronic equipment |
| CN115841425A (en) * | 2022-07-21 | 2023-03-24 | 爱芯元智半导体(上海)有限公司 | Video noise reduction method and device, electronic equipment and computer readable storage medium |
| CN115841425B (en) * | 2022-07-21 | 2023-11-17 | 爱芯元智半导体(宁波)有限公司 | Video noise reduction method and device, electronic equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113992861B (en) | Image processing method and image processing device | |
| CN108541374B (en) | Image fusion method and device and terminal equipment | |
| JP4653235B2 (en) | Composition of panoramic images using frame selection | |
| CN113935923A (en) | Image noise reduction method, device, equipment and storage medium | |
| JP7256902B2 (en) | Video noise removal method, apparatus and computer readable storage medium | |
| CN111127347A (en) | Noise reduction method, terminal and storage medium | |
| JP2012516637A5 (en) | ||
| CN110263699B (en) | Video image processing method, device, equipment and storage medium | |
| CN110445951B (en) | Video filtering method and device, storage medium and electronic device | |
| CN106412441B (en) | A kind of video stabilization control method and terminal | |
| CN113315884A (en) | Real-time video noise reduction method and device, terminal and storage medium | |
| US20150358547A1 (en) | Image processing apparatus | |
| US20040061795A1 (en) | Image processing apparatus and method, and image pickup apparatus | |
| CN113259594A (en) | Image processing method and device, computer readable storage medium and terminal | |
| CN106713762B (en) | Image processing method and device | |
| CN114339306B (en) | Live video image processing method and device and server | |
| CN108810509A (en) | A kind of image color correction method and device | |
| JP2021136461A (en) | Imaging device, control method, program, and storage medium | |
| CN111462021A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN114449130B (en) | Multi-camera video fusion method and system | |
| CN113572983B (en) | Cloud video processing method and system | |
| CN116668843A (en) | Shooting state switching method and device, electronic equipment and storage medium | |
| CN112581001B (en) | Evaluation method and device of equipment, electronic equipment and readable storage medium | |
| Moumene et al. | Generalized exposure fusion weights estimation | |
| CN114049288A (en) | Image generation method and device, electronic equipment and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |