CN110602467B

CN110602467B - Image noise reduction method and device, storage medium and electronic equipment

Info

Publication number: CN110602467B
Application number: CN201910848771.9A
Authority: CN
Inventors: 邵安宝
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2019-09-09
Filing date: 2019-09-09
Publication date: 2021-10-08
Anticipated expiration: 2039-09-09
Also published as: CN110602467A; WO2021047345A1

Abstract

The embodiment of the application discloses an image noise reduction method, an image noise reduction device, a storage medium and electronic equipment, wherein multiple frames of images to be processed of the same shooting scene are obtained; determining a first image serving as a reference image from a plurality of frames of images to be processed, and taking other images except the first image as second images; dividing the first image and the second image into a plurality of regions, respectively; calculating a first homography matrix of the second image relative to the first image on each area respectively; and on each region, based on the corresponding first homography matrix, performing noise reduction and fusion processing on the second image and the first image to obtain a target image.

Description

Image noise reduction method and device, storage medium and electronic equipment

Technical Field

The present application relates to the field of image denoising technology, and in particular, to an image denoising method, apparatus, storage medium, and electronic device.

Background

With the continuous development of intelligent terminal technology, the use of electronic devices (such as smart phones, tablet computers, and the like) is becoming more and more popular. Most of electronic devices are built-in with cameras, and with the enhancement of processing capability of mobile terminals and the development of camera technologies, users have higher and higher requirements for the quality of shot images.

In the related technology, in order to eliminate noise, multi-frame images are generally taken, the number and the positions of the noise of the multi-frame images are calculated and screened, the positions of the noise are replaced by image frames without the noise, and an image with the noise is obtained through repeated weighting and replacement.

Disclosure of Invention

The embodiment of the application provides an image noise reduction method and device, a storage medium and electronic equipment, which can eliminate a ghost image phenomenon in a synthesized image while realizing multi-frame synthesis noise reduction.

In a first aspect, an embodiment of the present application provides an image denoising method, including:

acquiring multiple frames of images to be processed of the same shooting scene;

determining a first image serving as a reference image from the plurality of frames of images to be processed, and taking other images except the first image as second images;

dividing the first image and the second image into a plurality of regions, respectively;

calculating a first homography matrix of the second image relative to the first image respectively on each region;

and on each region, based on the corresponding first homography matrix, performing noise reduction and fusion processing on the second image and the first image to obtain a target image.

In a second aspect, an embodiment of the present application provides an image noise reduction apparatus, including:

the image acquisition module is used for acquiring multiple frames of images to be processed of the same shooting scene;

the image selection module is used for determining a first image serving as a reference image from the plurality of frames of images to be processed and taking other images except the first image as second images;

an image segmentation module for segmenting the first image and the second image into a plurality of regions, respectively;

the matrix calculation module is used for calculating a first homography matrix of the second image on each area relative to the first image;

and the noise reduction fusion module is used for performing noise reduction fusion processing on the second image and the first image on each region based on the corresponding first homography matrix to obtain a target image.

In a third aspect, embodiments of the present application provide a storage medium having a computer program stored thereon, which, when run on a computer, causes the computer to execute an image denoising method as provided in any of the embodiments of the present application.

In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory has a computer program, and the processor is configured to execute the image denoising method according to any embodiment of the present application by calling the computer program.

According to the scheme provided by the embodiment of the application, multiple frames of images to be processed of the same shooting scene are obtained, a first image serving as a reference image is determined from the multiple frames of images to be processed, other images except the first image are taken as second images, the first image and the second images are respectively divided into multiple regions, first homography matrixes of the second images relative to the first images on each region are calculated, noise reduction and fusion processing is carried out on the second images and the first images on each region based on the corresponding first homography matrixes, clear images with noise points eliminated are obtained and serve as target images, and therefore, before noise reduction and fusion, homography matrixes among the images are calculated for different regions respectively in a mode of dividing the images into regions so as to prevent ghost images from being generated during noise reduction and fusion of the images.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic flowchart of a first image denoising method according to an embodiment of the present application.

Fig. 2 is a schematic image partition diagram of an image denoising method according to an embodiment of the present application.

Fig. 3 is a schematic diagram of a pixel unit in an image denoising method according to an embodiment of the present application.

Fig. 4 is a schematic diagram illustrating down-sampling of an image in an image denoising method according to an embodiment of the present application

Fig. 5 is a schematic diagram of region correction in the image denoising method according to the embodiment of the present application.

Fig. 6 is a schematic diagram of channel fusion in the image denoising method according to the embodiment of the present application.

Fig. 7 is a schematic diagram of a second flow of an image denoising method according to an embodiment of the present invention.

Fig. 8 is a schematic structural diagram of an image noise reduction device according to an embodiment of the present application.

Fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Fig. 10 is a schematic structural diagram of an image noise reduction circuit of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without inventive step, are within the scope of the present application.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

An execution main body of the image noise reduction method may be the image noise reduction device provided in the embodiment of the present application, or an electronic device integrated with the image noise reduction device, where the image noise reduction device may be implemented in a hardware or software manner. The electronic device may be a smart phone, a tablet computer, a palm computer, a notebook computer, or a desktop computer.

Referring to fig. 1, fig. 1 is a first flowchart illustrating an image denoising method according to an embodiment of the present disclosure. The specific flow of the image denoising method provided by the embodiment of the application can be as follows:

101. and acquiring multiple frames of images to be processed of the same shooting scene.

The scheme of the embodiment of the application can be applied to shooting night scenes or dark light environments. For example, the electronic device may receive a plurality of frames of images, which are sent by other terminals and obtained by shooting the same shooting scene, as the plurality of frames of images to be processed. Or, the electronic device starts the camera to shoot the shooting scene in the shooting mode to obtain the multi-frame image. For example, the electronic device photographs a scene to obtain multiple frames of images to be processed.

In some embodiments, the electronic device may continuously expose a shooting scene, acquire more images than the number of frames of images required for noise reduction synthesis, select several frames of images with the best definition from the images as images to be processed, and then perform noise reduction fusion processing using the images to be processed.

Wherein, in some embodiments, the plurality of frames of images to be processed may have different exposure parameters. In some embodiments, multiple frames of images to be processed may also have the same exposure parameters. For example, when acquiring an image to be processed, the electronic device determines an Exposure parameter of a normal Exposure according to an automatic photometry system of the camera, then adjusts the Exposure parameter based on the Exposure parameter of the normal Exposure to increase the Exposure degree, and then performs shooting, for example, increasing an Exposure amount of 1EV (Exposure value, which is an amount reflecting how much Exposure is), for example, increasing the Exposure amount by extending the Exposure time period. The specific number of the multiple frames of images to be processed can be set according to actual needs, and the application is not limited to this.

102. A first image serving as a reference image is determined from a plurality of frames of images to be processed, and other images except the first image are used as second images.

After obtaining multiple frames of images to be processed of the same shooting scene, determining one frame of image as a noise reduction fused reference image, marking the reference image as a first image, and marking other images except the reference image as second images.

In some embodiments, sharpness detection is performed on multiple frames of images to be processed, for example, sharpness of the images is detected through edge information, gradient information, and the like, and one frame of image with the highest sharpness is used as a reference image.

103. The first image and the second image are divided into a plurality of regions, respectively.

104. A first homography matrix is calculated for the second image relative to the first image over each region, respectively.

When shooting a night scene or a dark light environment, the multiple frames of images to be processed are shot and synthesized to reduce noise. However, multiple frames of images are easy to generate ghost images in the synthesis process, and in order to avoid the ghost images, the first image and the second image for noise reduction synthesis are subjected to the regional processing in the embodiment of the application. For each region, an independent first homography matrix is calculated.

For example, the electronic device acquires 5 frames of images to be processed, respectively labeled A, B, C, D, E, F. The image a is preferably a reference image, and the remaining image B, C, D, E, F is fused to the image a. Next, taking an example of synthesizing the image B and the image a, please refer to fig. 2, and fig. 2 is a schematic diagram of image partition of the image denoising method according to the embodiment of the present application. The image a and the image B are divided into M × N regions, wherein the size of M and N can be set according to the resolution of the image to be processed, the image synthesis efficiency, and the requirement for de-ghosting accuracy.

After the images a and B are partitioned, as shown in fig. 2, the images a and B are mapped in a one-to-one correspondence according to position, a first region of a first row in the image a corresponds to a first region of a first row in the image B, a second region of the first row in the image a corresponds to a second region of the first row in the image B, the first region of the second row in the image a corresponds to the first region of the second row in the image B, and so on, a last region of a last row in the image a corresponds to a last region of the last row in the image B.

It is to be understood that the first image and the second image have the same resolution, and when the partition processing is performed, the partition processing needs to be performed in the same manner, and the number of the areas is also equal.

Wherein, Homography (Homography) transformation is used to describe the position mapping relation of the object between the world coordinate system and the pixel coordinate system, and the corresponding transformation matrix is called Homography matrix. In the embodiment of the present application, the first homography matrix is used to represent a position mapping relationship of an object between a first image and a second image, feature point pairs matched with the two images are obtained by image registration of the second image and the first image (the feature points at corresponding positions on the two images form one feature point pair), and then a homography matrix is obtained by calculation based on the positions of the feature point pairs on the images, wherein at least four feature point pairs are obtained for calculating the homography matrix when image registration is performed.

In the embodiment of the present application, for each region, the first homography matrix of the second image with respect to the first image is calculated using the first image as a reference image, in such a manner that for image a, there is one first homography matrix corresponding to the region on each region. For example, a first homography matrix corresponding to the first region of the first row in the image a is calculated by using the feature point pairs matched with the first region of the first row in the image a and the first region of the first row in the image B.

105. And on each region, performing noise reduction and fusion processing on the second image and the first image based on the corresponding first homography matrix to obtain a target image.

And after the first homography matrix corresponding to each region is obtained, performing regional fusion on the second image and the first image based on the homography matrices.

And accumulating and synthesizing a plurality of frames of second images with the first image one by one according to the calculated first homography matrix, namely synthesizing the image B with the image A, synthesizing the image C with the synthesized image, and so on until all the second images are fused to the first image.

For example, for the first region in the first row in image a, the pixel points in the first region in the first row in image B are mapped to the region according to the first homography corresponding to the region, and the noise reduction fusion process is performed.

In the above manner, for each divided region in image a, based on the first homography matrix (calculated from image B and image a) corresponding to the region, the corresponding region in image B and the region are subjected to noise reduction fusion processing to obtain image a 1; then, for each region in the image a1, based on the first homography matrix (calculated from the image C and the image a) corresponding to the region, performing noise reduction and fusion processing on the region corresponding to the image C and the region to obtain an image a 2; then, for each region in the image a2, based on the first homography matrix (calculated from the image D and the image a) corresponding to the region, performing noise reduction and fusion processing using the corresponding region in the image D and the region to obtain an image A3; in this way, all the second images and the first image are subjected to noise reduction and fusion processing to obtain a final target image.

The specific noise reduction algorithm is not limited in the embodiment of the present application. For example, in some embodiments, the mean of pixel values at the same location in image A and image B are calculated for noise reduction fusion. Or, calculating the median of the pixel values at the same position in the image A and the image B to perform noise reduction fusion.

In some embodiments, further luminance boost processing may be performed on the target image to obtain a shooting output, for example, the preset convolutional neural network model is used to perform luminance boost processing on the target image to obtain the shooting output in a night view mode.

In particular implementation, the present application is not limited by the execution sequence of the described steps, and some steps may be performed in other sequences or simultaneously without conflict.

As can be seen from the above, the image noise reduction method provided in the embodiment of the present application obtains multiple frames of to-be-processed images of the same shooting scene, determines a first image as a reference image from the multiple frames of to-be-processed images, divides the first image and the second image into multiple regions by using images other than the first image as second images, calculates a first homography matrix of each region of the second image relative to the first image, on the basis of each region, the method carries out noise reduction and fusion processing on the second image and the first image based on the corresponding first homography matrix to obtain a clear image with noise points eliminated as a target image, and based on the method, the homography matrix among the images is respectively calculated aiming at different regions by adopting a mode of dividing the images into regions before noise reduction and fusion, the method can prevent ghost image phenomenon generated during image noise reduction fusion while realizing multi-frame noise reduction.

In some embodiments, the original format of the image to be processed is RAW format; calculating a first homography matrix of the second image relative to the first image respectively on each area, comprising: converting the first image and the second image from a RAW format to a gray format; a first homography matrix is calculated for each region of the second image in grayscale format relative to the first image in grayscale format.

On each region, based on the corresponding first homography matrix, the method for carrying out noise reduction and fusion processing on the second image and the first image to obtain a target image comprises the following steps: and on each region, performing noise reduction and fusion processing on the second image in the RAW format and the first image in the RAW format based on the corresponding first homography matrix to obtain a target image.

In this embodiment, the acquired image to be processed is an image in RAW format in bayer domain, and in order to reduce the amount of calculation, both the first image and the second image are converted into gray format in RAW format before the first homography matrix is calculated.

The RAW format is RAW data obtained by converting a captured light source signal into a digital signal by an image sensor, and is an unprocessed or uncompressed format, and an image in the RAW format can be understood as "RAW image encoded data" or visually referred to as "digital negative film".

There are various embodiments for converting the image from the original RAW format to the grayscale format. In some embodiments, converting the first image and the second image from RAW format to grayscale format comprises: extracting a pixel value of a green channel on each pixel unit of the first image in the RAW format to obtain a first image in the gray scale format; and extracting the pixel value of the green channel on each pixel unit of the second image in the RAW format to obtain the second image in the gray scale format.

In this embodiment, in order to increase the speed of image processing, the value of the G (Green) channel may be directly used as the pixel of the current point, so as to obtain the image in the grayscale format. Since the pixels in the RAW format image are arranged in the bayer color filter array, for example, when the pixels are output in lines, the pixels are output in the order grgrgr./BGBGBG. Referring to fig. 3, fig. 3 is a schematic diagram of a pixel unit in the image denoising method according to the embodiment of the present application, wherein four pixels of RGGB may form one pixel unit. In other embodiments, the pixel values of the R (Red) channel or the B (Blue) channel may be directly extracted to obtain a single-channel grayscale image. Or, in some embodiments, weights may be set for the three RGB pixels, and a weighting algorithm is adopted to combine the three RGB pixel values corresponding to one pixel point into one pixel value according to the weights corresponding to the three RGB pixel values, so as to obtain an image in a gray format, so as to further improve the signal-to-noise ratio of the image.

It should be noted that, converting the image from the RAW format to the grayscale format may be performed after acquiring the to-be-processed image in the multi-frame RAW format and before calculating the first homography matrix. For example, the reference image may be determined before or after the reference image is determined.

In some embodiments, calculating a first homography matrix for the second image in grayscale format over each region respectively with respect to the first image in grayscale format comprises: according to a preset sampling rate, carrying out continuous multiple downsampling processing on the first image in the gray scale format to obtain a third image, and carrying out continuous multiple downsampling processing on the second image in the gray scale format to obtain a fourth image; and calculating a second homography matrix of the fourth image relative to the third image on each area respectively, wherein the second homography matrix is used as a first homography matrix of the second image in a gray scale format relative to the first image in the gray scale format on each area respectively.

In this embodiment, in order to improve the calculation efficiency of the homography matrix, down-sampling processing is performed on each of the first image and the second image, and the size of the image is reduced. Taking the image a as an example, the image a in the gray format is subjected to continuous downsampling processing for multiple times according to a preset sampling rate. For example, when the sampling rates in the longitudinal direction and the width direction are both 1/2, the resolution of the image a obtained by downsampling is 1/4 of the image a, and downsampling is continued on the image at the same resolution to obtain an image of 1/16 size of the image a, and so on, and downsampling processing can be continuously performed 3 to 9 times. Referring to fig. 4, fig. 4 is a schematic diagram of down-sampling an image in the image noise reduction method according to the embodiment of the present application, taking the first image as an example, and performing down-sampling on the first image for multiple times continuously to obtain a third image, where multiple frame images obtained by down-sampling for multiple times continuously form a first image pyramid. The second image is down-sampled at the same sampling rate and in the same manner as the first image.

And after the first image is subjected to down-sampling processing for multiple times, the obtained images form an image pyramid according to the sizes from large to small, wherein the image with the minimum size of the bottom layer is marked as a third image. And recording the image of the bottom layer of the image pyramid corresponding to the second image as a fourth image.

It can be understood that after the downsampling process is performed on the first image or the second image with the divided regions, the first image or the second image still has the same region division as the original image, but the size of the region is reduced along with the downsampling.

When the first homography matrix is calculated, the fourth image and the third image may be subjected to image registration processing, and a second homography matrix of the fourth image relative to the third image on each region is calculated. In some embodiments, the second homography matrix of the fourth image relative to the third image on each region may be directly used as the first homography matrix of the second image relative to the first image on each region, respectively. Or, in other embodiments, the second homography matrix of the bottom layer may be utilized to perform layer-by-layer recursion to the first layer based on the image pyramid, so as to obtain the first homography matrix of the second image with the original size on each region relative to the first image. Since the homography matrix reflects the offset direction and the offset distance of the object, after the downsampling processing, the offset direction is not changed, only the offset distance is influenced, and the degree of influence of the offset distance is related to the magnitude of the downsampling sampling rate. Therefore, when the image is recursive from the lower layer to the upper layer, the first homography matrix of the second image in the gray format relative to the first image in the gray format in each region can be calculated according to the preset sampling rate and the second homography matrix of the fourth image relative to the third image in each region.

In some embodiments, calculating a second homography matrix for the fourth image relative to the third image over each region, respectively, comprises: carrying out image registration processing on the fourth image and the third image to obtain a characteristic point pair; and calculating a second homography matrix of the fourth image relative to the third image on each area respectively based on the characteristic point pairs. It is to be understood that, in some embodiments, the second image may be directly registered with the first image to obtain the feature point pairs, the first image and the second image are subjected to downsampling processing to obtain the third image and the fourth image, a part of the feature point pairs still remain in the third image and the fourth image, and the second homography matrix may be calculated based on these remaining feature point pairs. Alternatively, in other embodiments, after downsampling, image registration may be performed, for example, the third image and the fourth image are subjected to image registration processing, and the second homography matrix is calculated according to the obtained feature point pairs.

In some embodiments, after calculating the second homography matrix of the fourth image with respect to the third image respectively on each region based on the feature point pairs, the method further includes: calculating a global homography matrix of the fourth image relative to the third image; and correcting the second homography matrix based on the global homography matrix.

In this embodiment, the first homography matrix for each region is modified based on the global homography matrix for the entire image. Although in the scheme of the embodiment of the application, to avoid ghosting of an object in an image, the homography matrix is calculated by regions for the image, meanwhile, to ensure the accuracy of image fusion, the difference between the second homography matrix calculated by each region and the global homography matrix cannot be too large. And when detecting that the difference value between the second homography matrix and the global homography matrix is larger than a preset threshold value, taking the area with the difference value larger than the preset threshold value as the area to be corrected.

For example, modifying the second homography row matrix based on the global homography matrix includes: calculating a difference between the second homography matrix and the global homography matrix; determining the area with the difference value larger than a preset threshold value as an area to be corrected; and expanding the area of the region to be corrected, re-determining the characteristic point pairs from the expanded region to be corrected, and re-calculating a second homography matrix of the fourth image on the region to be corrected relative to the third image on the basis of the determined characteristic point pairs until the difference between the second homography matrix of the region to be corrected and the global homography matrix is not greater than a preset threshold value.

Referring to fig. 5, fig. 5 is a schematic diagram illustrating a region correction in an image denoising method according to an embodiment of the present disclosure. As shown in fig. 5, the area to be corrected is a rectangle of 100 × 100 (pixel points), wherein there are four feature points, the area to be corrected is expanded, 10 pixel points are expanded along each of four sides each time, three pixel points are obtained from the expanded area, the three pixel points are combined with one pixel point in the original area, a second homography matrix of the expanded area to be corrected is recalculated, then a difference between the second homography matrix and the global homography matrix is calculated, if the difference is smaller than a preset threshold, the expansion of the area is terminated, the current 120 × 120 rectangle is used as a new area, if the difference is still larger than the preset threshold or the number of feature points in the expanded area does not meet the requirement (e.g., smaller than 3), the area to be corrected is continuously expanded, and the second homography matrix is recalculated until the difference between the second homography matrix of the corrected area and the global homography matrix is not larger than the preset threshold, or until the area of the expanded region to be corrected reaches a preset threshold (for example, the area of the expanded region to be corrected reaches twice the original area), as shown in fig. 5, where the rectangular region indicated by the dotted line is the expanded region to be corrected. It will be appreciated that if a region on the first image is enlarged, the corresponding region on the second image will need to be enlarged in the same manner.

In some embodiments, the denoising and fusing the second image and the first image based on the corresponding first homography matrix comprises: determining a preset number of pixel points at the edge of the region as the edge region of the region; performing channel-division fusion processing on the edge regions connected between the adjacent regions; and for each region subjected to the channel-division fusion processing, performing noise reduction fusion processing on the second image and the first image based on the corresponding first homography matrix to obtain a target image.

In the embodiment of the application, the first homography matrix is calculated by dividing the image into regions, so that in the process of synthesis, in order to avoid the phenomena of color cast, overlapping or separation between adjacent regions, the channel-dividing fusion processing is performed on the adjacent regions of each frame of image in the first image and the second image. The specific width of the edge region may be set as required, for example, the width of 3-10 pixels at the edge of the region is determined as the edge region of the region.

Referring to fig. 6, fig. 6 is a schematic diagram of sub-channel fusion in the image denoising method according to the embodiment of the present application. Taking two adjacent regions a1 and a region a2 in the image a as an example, the widths of 3 pixels are respectively determined on the adjacent boundaries of the region a1 and the region a2 to be edge regions, and an edge region a12 of the region a1 and an edge region a21 of the region a2 are obtained. And performing channel-division transverse fusion on the A12 and the A21, taking the pixel point X as an example, calculating the distances between other five pixel points on the A12 and the A21, which are in the same row with the pixel point X, and the pixel point X, determining the weights of other pixel points in the same row with the pixel point X according to the distances, wherein the sum of the weights of the other five pixel points is 1, and the closer to the pixel point X, the greater the weight is. And calculating a weighted average value of other five pixel points according to the pixel values and the weights, calculating an average value of the obtained weighted average value and the pixel value of the pixel point X, and taking the calculated average value as the pixel value after the pixel point X is fused. And calculating the fused pixel value of each pixel point in A12 and A21 in the same way. It should be noted that, when calculating the pixel value after the fusion of the pixel point X, it is necessary to perform channel-by-channel calculation, that is, calculate the pixel value of each channel on R, G, B three channels, so as to solve the color shift problem caused by channel mixing.

The edge regions connected between adjacent regions in the image a, the image B, the image C, the image D, and the image E are respectively subjected to channel-division fusion processing according to the above manner, and then the image B, the image C, the image D, and the image E are subjected to noise reduction fusion to the image a in sequence.

The method according to the preceding embodiment is illustrated in further detail below by way of example.

Referring to fig. 7, fig. 7 is a second flowchart of an image denoising method according to an embodiment of the present invention. The method comprises the following steps:

111. acquiring multiple frames of images to be processed of the same shooting scene, wherein the original format of the images to be processed is an RAW format.

The electronic equipment starts a camera in a shooting mode, determines exposure parameters of normal exposure according to an automatic photometry system, then adjusts the exposure parameters on the basis of the exposure parameters of the normal exposure, for example, after the exposure of 1EV is increased, exposes a shooting scene to obtain multiple frames of images with the same exposure parameters, and selects multiple frames of images with the best definition as images to be processed.

112. A first image serving as a reference image is determined from a plurality of frames of images to be processed, and other images except the first image are used as second images.

And determining a frame of image from the multiple frames of images to be processed as a reference image for noise reduction and fusion, marking the frame of image as a first image, and marking other images except the reference image as second images.

113. And converting the first image and the second image from a RAW format to a gray scale format.

The acquired image to be processed is an image in a RAW format of a Bayer domain, and in order to reduce the calculation amount, the first image and the second image are converted into a gray scale format by the RAW format before the first homography matrix is calculated. For example, in some embodiments, in order to increase the speed of image processing, the value of the G (Green) channel may be directly used as the pixel of the current point, resulting in a gray format image.

114. The first image and the second image are divided into a plurality of regions, respectively.

115. And according to a preset sampling rate, carrying out continuous multi-time downsampling processing on the first image in the gray scale format to obtain a first image pyramid, and carrying out continuous multi-time downsampling processing on the second image in the gray scale format to obtain a second image pyramid.

The first image and the second image are respectively divided into a plurality of areas, and the areas in the image A and the image B correspond to each other one by one according to positions. Then, in order to improve the calculation efficiency of the homography matrix, down-sampling processing is performed on each of the first image and the second image, and the size of the image is reduced. Referring to fig. 4, taking the first image as an example, the first image is downsampled continuously for multiple times to obtain a third image, where multiple frame images obtained by downsampling continuously for multiple times form a first image pyramid. The second image is down-sampled at the same sampling rate and in the same manner as the first image. It can be understood that after the downsampling process is performed on the first image or the second image with the divided regions, the first image or the second image still has the same region division as the original image, but the size of the region is reduced along with the downsampling.

116. And calculating the image with the minimum size in the second image pyramid, and respectively calculating a second homography matrix on each area relative to the image with the minimum size in the first image pyramid.

And marking the image with the minimum size in the first image pyramid as a third image, marking the image with the minimum size in the second image pyramid as a fourth image, and calculating a second homography matrix of the fourth image on each area relative to the third image. And carrying out image registration on the fourth image and the third image to obtain corresponding characteristic point pairs, and then calculating a second homography matrix according to the characteristic point pairs.

117. And calculating a global homography matrix of the second image relative to the first image, and correcting the second homography matrix on each area according to the global homography matrix.

In order to ensure the accuracy of image fusion, the difference between the second homography matrix calculated by each region and the global homography matrix cannot be too large. When it is detected that the difference between the second homography matrix and the global homography matrix is greater than the preset threshold, the area with the difference greater than the preset threshold is taken as the area to be corrected, and the second homography matrix of the area to be corrected is recalculated.

118. And calculating a first homography matrix of the second image on each area relative to the first image according to the preset sampling rate and the second homography matrix on each area.

And based on the image pyramid, carrying out layer-by-layer recursion to the first layer by utilizing the second homography matrix of the bottom layer image according to a preset sampling rate to obtain first homography matrixes of the second image with the original size on each area relative to the first image. Compared with the method for directly calculating the homography matrix of the image with the original size, the method has the advantages that the calculation amount is reduced, and the image processing efficiency is improved.

119. And performing channel-division fusion processing on the edge areas of adjacent areas in the first image and the second image in the RAW format.

And during synthesis, in order to avoid the phenomena of color cast, overlapping or separation between adjacent regions, performing channel-division fusion processing on the adjacent regions of each frame of image in the first image and the second image.

110. And for each region subjected to the channel-division fusion processing, performing noise reduction fusion processing on the second image and the first image based on the corresponding first homography matrix to obtain a target image.

And on each region, based on the corresponding first homography matrix, performing noise reduction and fusion processing on the second image and the first image to obtain a clear image with noise points eliminated as a target image.

Therefore, the image denoising method provided by the embodiment of the invention adopts a mode of dividing the image into regions before denoising and fusing, and calculates the homography matrix between the images aiming at different regions respectively, so that the multi-frame denoising is realized, and the ghost phenomenon can be prevented from being generated during the image denoising and fusing.

An image noise reduction apparatus is also provided in an embodiment. Referring to fig. 8, fig. 8 is a schematic structural diagram of an image denoising device 400 according to an embodiment of the present application. The image denoising apparatus 200 is applied to an electronic device, and the image denoising apparatus 200 includes an image obtaining module 201, an image selecting module 202, an image segmenting module 203, a matrix calculating module 204, and a denoising and fusing module 205, as follows:

the image acquisition module 201 is configured to acquire multiple frames of images to be processed of the same shooting scene;

an image selection module 202, configured to determine a first image serving as a reference image from the plurality of frames of images to be processed, and take other images except the first image as second images;

an image segmentation module 203 for segmenting the first image and the second image into a plurality of regions, respectively;

a matrix calculation module 204, configured to calculate a first homography matrix of the second image on each region respectively relative to the first image;

and a denoising and fusing module 205, configured to perform denoising and fusing processing on the second image and the first image based on the corresponding first homography matrix in each region to obtain a target image.

In some embodiments, the original format of the image to be processed is RAW format; the matrix calculation module 204 is further configured to: converting the first image and the second image from a RAW format to a grayscale format; calculating a first homography matrix of the second image in the gray format relative to the first image in the gray format on each area respectively; the noise reduction fusion module 205 is further configured to: and on each region, performing noise reduction and fusion processing on the second image in the RAW format and the first image in the RAW format based on the corresponding first homography matrix to obtain a target image.

In some embodiments, the matrix calculation module 204 is further configured to:

extracting a pixel value of a green channel on each pixel unit of the first image in the RAW format to obtain a first image in a gray scale format;

and extracting the pixel value of a green channel on each pixel unit of the second image in the RAW format to obtain the second image in the gray scale format.

according to a preset sampling rate, carrying out continuous multiple downsampling processing on the first image in the gray scale format to obtain a third image, and carrying out continuous multiple downsampling processing on the second image in the gray scale format to obtain a fourth image;

and calculating a second homography matrix of the fourth image relative to the third image on each area respectively, wherein the second homography matrix is used as a first homography matrix of the second image in a gray scale format relative to the first image in the gray scale format on each area respectively.

calculating a second homography matrix of the fourth image relative to the third image respectively on each region;

and calculating to obtain a first homography matrix of the second image in the gray format on each area relative to the first image in the gray format according to the preset sampling rate and the second homography matrix of the fourth image on each area relative to the third image.

carrying out image registration processing on the fourth image and the third image to obtain a characteristic point pair;

and calculating a second homography matrix of the fourth image relative to the third image on each area respectively based on the characteristic point pairs.

calculating a global homography matrix for the fourth image relative to the third image;

and correcting the second homography matrix based on the global homography matrix.

calculating a difference between the second homography matrix and the global homography matrix;

determining the area with the difference value larger than a preset threshold value as an area to be corrected;

and expanding the area of the region to be corrected, re-determining a characteristic point pair from the expanded region to be corrected, and re-calculating a second homography matrix of the fourth image on the region to be corrected relative to the third image on the basis of the determined characteristic point pair until the difference between the second homography matrix of the region to be corrected and the global homography matrix is not greater than the preset threshold.

In some embodiments, the noise reduction fusion module 205 is further configured to:

determining a preset number of pixel points at the edge of a region as the edge region of the region;

performing channel-division fusion processing on the edge regions connected between the adjacent regions;

and for each region subjected to the sub-channel fusion processing, performing noise reduction fusion processing on the second image and the first image based on the corresponding first homography matrix to obtain a target image.

In some embodiments, the image denoising apparatus 200 further comprises a brightness adjusting module for:

and carrying out brightness improvement processing on the target image according to a preset convolutional neural network model.

In specific implementation, the above modules may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities, and specific implementation of the above modules may refer to the foregoing method embodiments, which are not described herein again.

It should be noted that the image noise reduction device provided in the embodiment of the present application and the image noise reduction method in the foregoing embodiment belong to the same concept, and any method provided in the embodiment of the image noise reduction method may be run on the image noise reduction device, and a specific implementation process thereof is described in detail in the embodiment of the image noise reduction method, and is not described herein again.

As can be seen from the above, in the image noise reduction apparatus provided in this embodiment of the present application, the image obtaining module 201 obtains multiple frames of to-be-processed images of the same shooting scene, the image selecting module 202 determines a first image as a reference image from the multiple frames of to-be-processed images, uses other images except the first image as second images, the image dividing module 203 divides the first image and the second image into multiple regions, the matrix calculating module 204 calculates a first homography matrix of the second image on each region with respect to the first image, the noise reduction fusion module 205 performs noise reduction fusion processing on each region with respect to the second image and the first image based on the corresponding first homography matrix to obtain a clear image with noise removed as a target image, and based on this scheme, before noise reduction fusion, calculates homography matrices between the images for different regions by dividing the images into regions, so as to prevent the ghost phenomenon generated during the image denoising and fusion.

The embodiment of the application further provides an electronic device, and the electronic device can be a mobile terminal such as a tablet computer or a smart phone. Referring to fig. 9, fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 800 may include a camera module 801, a memory 802, a processor 803, a touch display 804, a speaker 805, a microphone 806, and the like.

The camera module 801 may include Image noise reduction circuitry, which may be implemented using hardware and/or software components, and may include various Processing units that define an Image Signal Processing (Image Signal Processing) pipeline. The image noise reduction circuit may include at least: a camera, an Image Signal Processor (ISP Processor), control logic, an Image memory, and a display. Wherein the camera may comprise at least one or more lenses and an image sensor. The image sensor may include an array of color filters (e.g., Bayer filters). The image sensor may acquire light intensity and wavelength information captured with each imaging pixel of the image sensor and provide a set of raw image data that may be processed by an image signal processor.

The image signal processor may process the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the image signal processor may perform one or more image noise reduction operations on the raw image data, gathering statistical information about the image data. Wherein the image denoising operation may be performed with the same or different bit depth precision. The raw image data can be stored in an image memory after being processed by an image signal processor. The image signal processor may also receive image data from an image memory.

The image Memory may be part of a Memory device, a storage device, or a separate dedicated Memory within the electronic device, and may include a DMA (Direct Memory Access) feature.

When image data is received from the image memory, the image signal processor may perform one or more image noise reduction operations, such as temporal filtering. The processed image data may be sent to an image memory for additional processing before being displayed. The image signal processor may also receive processed data from the image memory and perform image data processing on the processed data in the raw domain and in the RGB and YCbCr color spaces. The processed image data may be output to a display for viewing by a user and/or further processed by a Graphics Processing Unit (GPU). Further, the output of the image signal processor may also be sent to an image memory, and the display may read image data from the image memory. In one embodiment, the image memory may be configured to implement one or more frame buffers.

The statistical data determined by the image signal processor may be sent to the control logic. For example, the statistical data may include statistical information of the image sensor such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, lens shading correction, and the like.

The control logic may include a processor and/or microcontroller that executes one or more routines (e.g., firmware). One or more routines may determine camera control parameters and ISP control parameters based on the received statistics. For example, the control parameters of the camera may include camera flash control parameters, control parameters of the lens (e.g., focal length for focusing or zooming), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (e.g., during RGB processing), etc.

Referring to fig. 10, fig. 10 is a schematic structural diagram of an image noise reduction circuit of an electronic device according to an embodiment of the present disclosure. For ease of illustration, only aspects of image noise reduction techniques related to embodiments of the present invention are shown.

For example, the image noise reduction circuit may include: camera, image signal processor, control logic ware, image memory, display. The camera may include one or more lenses and an image sensor, among others. In some embodiments, the camera may be either a tele camera or a wide camera.

And the image collected by the camera is transmitted to an image signal processor for processing. After the image signal processor processes the image, statistical data of the image (such as brightness of the image, contrast value of the image, color of the image, etc.) may be sent to the control logic. The control logic device can determine the control parameters of the camera according to the statistical data, so that the camera can carry out operations such as automatic focusing and automatic exposure according to the control parameters. The image can be stored in the image memory after being processed by the image signal processor. The image signal processor may also read the image stored in the image memory for processing. In addition, the image can be directly sent to a display for displaying after being processed by the image signal processor. The display may also read the image in the image memory for display.

In addition, not shown in the figure, the electronic device may further include a CPU and a power supply module. The CPU is connected with the logic controller, the image signal processor, the image memory and the display, and is used for realizing global control. The power supply module is used for supplying power to each module.

The memory 802 stores applications containing executable code. The application programs may constitute various functional modules. The processor 803 executes various functional applications and data processing by running the application programs stored in the memory 802.

The processor 803 is a control center of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing an application program stored in the memory 802 and calling data stored in the memory 802, thereby integrally monitoring the electronic device.

The touch display screen 804 may be used to receive user touch control operations for the electronic device. Speaker 805 may play sound signals. The microphone 806 may be used to pick up sound signals.

In this embodiment, the processor 803 in the electronic device loads the executable code corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 803 runs the application programs stored in the memory 802, so as to execute:

acquiring multiple frames of images to be processed of the same shooting scene;

In some embodiments, the original format of the image to be processed is RAW format; in calculating a first homography matrix for the second image, respectively on each region with respect to the first image, the processor 803 performs:

converting the first image and the second image from a RAW format to a grayscale format;

calculating a first homography matrix of the second image in the gray format relative to the first image in the gray format on each area respectively;

when the second image and the first image are subjected to noise reduction and fusion processing based on the corresponding first homography matrix in each region to obtain a target image, the processor 803 executes:

and on each region, performing noise reduction and fusion processing on the second image in the RAW format and the first image in the RAW format based on the corresponding first homography matrix to obtain a target image.

In some embodiments, in converting the first image and the second image from RAW format to grayscale format, the processor 803 performs:

In some embodiments, in calculating a first homography matrix for the second image in grayscale format over each region respectively with respect to the first image in grayscale format, the processor 803 performs:

In some embodiments, in calculating a second homography matrix for the fourth image relative to the third image, respectively, on each region, the processor 803 performs:

In some embodiments, after calculating the second homography matrix of the fourth image with respect to the third image respectively on each region based on the feature point pairs, the processor 803 performs:

In some embodiments, when modifying the second homography row matrix based on the global homography matrix, the processor 803 performs:

In some embodiments, when the second image and the first image are subjected to a noise reduction fusion process based on the corresponding first homography matrix, the processor 803 performs:

In some embodiments, after obtaining the target image, the processor 803 performs:

As can be seen from the above, an embodiment of the present application provides an electronic device that acquires multiple frames of images to be processed of the same shooting scene, determines a first image as a reference image from the multiple frames of images to be processed, uses an image other than the first image as a second image, divides the first image and the second image into multiple regions, respectively, calculates a first homography matrix for the second image with respect to the first image on each region, on the basis of the method, the homography matrixes among the images are calculated aiming at different areas respectively in a mode of dividing the images into areas before noise reduction and fusion so as to prevent a ghost phenomenon from being generated during the noise reduction and fusion of the images.

An embodiment of the present application further provides a storage medium, where a computer program is stored in the storage medium, and when the computer program runs on a computer, the computer executes the image denoising method according to any one of the above embodiments.

It should be noted that, all or part of the steps in the methods of the above embodiments may be implemented by hardware related to instructions of a computer program, which may be stored in a computer-readable storage medium, which may include, but is not limited to: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.

Furthermore, the terms "first", "second", and "third", etc. in this application are used to distinguish different objects, and are not used to describe a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or modules is not limited to only those steps or modules listed, but rather, some embodiments may include other steps or modules not listed or inherent to such process, method, article, or apparatus.

The image denoising method, the image denoising device, the storage medium and the electronic device provided by the embodiments of the present application are described in detail above. The principle and the implementation of the present application are explained herein by applying specific examples, and the above description of the embodiments is only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. An image noise reduction method, comprising:

acquiring multiple frames of images to be processed of the same shooting scene;

calculating a first homography matrix of the second image with respect to the first image respectively on each region, comprising: according to a preset sampling rate, carrying out continuous multiple downsampling processing on the first image to obtain a third image, and carrying out continuous multiple downsampling processing on the second image to obtain a fourth image; calculating a second homography matrix of the fourth image relative to the third image respectively on each region; calculating a global homography matrix for the fourth image relative to the third image; correcting the second homography matrix based on the global homography matrix; obtaining a first homography matrix of each area of the second image relative to the first image according to the corrected second homography matrix;

2. The image noise reduction method according to claim 1, wherein the original format of the image to be processed is RAW format; according to a preset sampling rate, continuous multiple downsampling processing is carried out on the first image to obtain a third image, continuous multiple downsampling processing is carried out on the second image to obtain a fourth image, and the method comprises the following steps:

the performing, on each region, noise reduction and fusion processing on the second image and the first image based on the corresponding first homography matrix to obtain a target image includes:

3. The method of image noise reduction according to claim 2, wherein the converting the first image and the second image from a RAW format to a grayscale format comprises:

4. The method for reducing image noise according to claim 2, wherein the obtaining a first homography matrix of each region of the second image relative to the first image according to the modified second homography matrix comprises:

and taking the corrected second homography matrix as a first homography matrix of the second image in the gray format relative to the first image in the gray format on each area.

5. The method for reducing image noise according to claim 2, wherein the obtaining a first homography matrix of each region of the second image relative to the first image according to the modified second homography matrix comprises:

and calculating to obtain a first homography matrix of the second image in the gray format on each area relative to the first image in the gray format according to the preset sampling rate and the corrected second homography matrix.

6. The method of image noise reduction according to claim 4 or 5, wherein the calculating a second homography matrix for each region of the fourth image relative to the third image comprises:

7. The method of image noise reduction according to claim 6, wherein the modifying the second homography matrix based on the global homography matrix comprises:

8. The image denoising method of claim 1, wherein the denoising and merging the second image with the first image based on the corresponding first homography matrix comprises:

9. The image noise reduction method according to claim 1, further comprising, after obtaining the target image:

10. An image noise reduction apparatus, comprising:

a matrix calculation module, configured to calculate a first homography matrix of the second image with respect to the first image on each region, respectively, including: according to a preset sampling rate, carrying out continuous multiple downsampling processing on the first image to obtain a third image, and carrying out continuous multiple downsampling processing on the second image to obtain a fourth image; calculating a second homography matrix of the fourth image relative to the third image respectively on each region; calculating a global homography matrix for the fourth image relative to the third image; correcting the second homography matrix based on the global homography matrix; obtaining a first homography matrix of each area of the second image relative to the first image according to the corrected second homography matrix;

11. A storage medium having stored thereon a computer program, characterized in that, when the computer program is run on a computer, it causes the computer to execute the image noise reduction method according to any one of claims 1 to 9.

12. An electronic device comprising a processor and a memory, the memory storing a computer program, wherein the processor is configured to execute the image denoising method according to any one of claims 1 to 9 by calling the computer program.