CN115942125A

CN115942125A - Multi-exposure image processing method and device and noise reduction circuit

Info

Publication number: CN115942125A
Application number: CN202111129165.5A
Authority: CN
Inventors: 李彦良
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2021-09-26
Filing date: 2021-09-26
Publication date: 2023-04-07
Also published as: WO2023045907A1

Abstract

The embodiment of the application discloses a multi-exposure image processing method, a multi-exposure image processing device and a noise reduction circuit, when the method is used for processing multi-exposure images photographed by HDR, the sizes of pixel windows used for performing noise reduction processing on at least two exposure images are respectively determined according to exposure time, and the noise reduction processing is respectively performed on the at least two exposure images according to the determined sizes of the pixel windows, so that the noise reduction processing requirements of different exposure images are met, the image noise is effectively inhibited, on the basis of ensuring the noise reduction effect, the processing resources are saved, the noise reduction cost is reduced, and the noise reduction scheme of HDR is optimized.

Description

Multi-exposure image processing method and device and noise reduction circuit

Technical Field

The present disclosure relates to image processing technologies, and in particular, to a multi-exposure image processing method and apparatus, and a noise reduction circuit.

Background

There are various methods for realizing High Dynamic Range (HDR) photography, and the currently mainstream method is Digital Overlay (DOL), which specifically includes performing long exposure, medium exposure, and short exposure on the same scene respectively, acquiring three images, fusing the three images, and finally obtaining an image with a High Dynamic Range. When three images are fused, the influence of noise in the three images on the fusion result needs to be considered, and the images generally need to be subjected to noise reduction processing. When the noise reduction is specifically implemented, the noise reduction can be selected before the fusion, or the noise reduction can be selected after the fusion. Therefore, the problem of noise reduction of multi-exposure images is a problem that needs to be studied and solved for a long time.

Disclosure of Invention

In order to solve the foregoing technical problems, embodiments of the present application desirably provide a multi-exposure image processing method and apparatus, and a noise reduction circuit.

The technical scheme of the application is realized as follows:

in a first aspect, a multi-exposure image processing method is provided, including:

acquiring at least two exposure images of a target scene shot at different exposure times;

respectively determining the sizes of pixel windows for performing noise reduction processing on the at least two exposure images based on the exposure time of the at least two exposure images;

performing noise reduction processing on the at least two exposure images respectively based on the determined size of the pixel window;

and performing fusion processing on the at least two exposure images subjected to the noise reduction processing to obtain a high dynamic range image of the target scene.

In a second aspect, there is provided a multi-exposure image processing apparatus comprising:

the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring at least two exposure images of a target scene shot at different exposure times;

the noise reduction module is used for respectively determining the sizes of pixel windows for performing noise reduction processing on the at least two exposure images based on the exposure time of the at least two exposure images; performing noise reduction processing on the at least two exposure images respectively based on the determined size of the pixel window;

and the fusion module is used for performing fusion processing on the at least two exposure images subjected to the noise reduction processing to obtain a high dynamic range image of the target scene.

In a third aspect, a noise reduction circuit is provided, the noise reduction circuit comprising:

the pixel window construction circuit is used for loading the pixel window of the exposure image according to the size of the pixel window for executing noise reduction processing on the exposure image;

the odd-row extraction circuit is used for extracting odd-row pixels from the pixel window loaded by the window construction circuit;

the even row extraction circuit is used for extracting even row pixels from the pixel window loaded by the window construction circuit;

the pixel matrix constructing circuit is used for constructing a pixel matrix according to the odd-row pixels or the even-row pixels;

and the noise reduction operation circuit is used for performing noise reduction operation on the pixel matrix and outputting noise-reduced pixel points.

In a fourth aspect, a multi-exposure image processing chip is provided, the chip comprising the noise reduction circuit of the third aspect.

In a fifth aspect, an electronic device is provided, which includes: image acquisition device and aforementioned many exposure image processing chip.

In a sixth aspect, a computer-readable storage medium is provided, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the aforementioned method.

Therefore, when the multi-exposure image shot by the HDR is processed, the sizes of the pixel windows used for executing the noise reduction processing on the at least two exposure images are respectively determined according to the exposure time, and the noise reduction processing is respectively executed on the at least two exposure images according to the determined sizes of the pixel windows, so that the noise reduction processing requirements of different exposure images are met, the image noise is effectively inhibited, the processing resources are saved, the noise reduction cost is reduced, and the noise reduction scheme of the HDR is optimized on the basis of ensuring the noise reduction effect.

Drawings

FIG. 1 is a schematic flowchart of a multi-exposure image processing method according to an embodiment of the present application;

FIG. 2 is a schematic flowchart of a denoising processing method in an embodiment of the present application;

FIG. 3 is a diagram illustrating a first dimension of a pixel window according to an embodiment of the present disclosure;

FIG. 4 is a first schematic diagram of a Gr dot pixel matrix in the embodiment of the present application;

FIG. 5 is a first schematic diagram of a Gb dot pixel matrix in an embodiment of the present application;

FIG. 6 is a first schematic diagram of an R dot pixel matrix in an embodiment of the present application;

FIG. 7 is a first schematic diagram of a B-dot pixel matrix in an embodiment of the present application;

FIG. 8 is a diagram illustrating a second dimension of a pixel window in an embodiment of the present application;

FIG. 9 is a second schematic diagram of a Gr dot pixel matrix in the embodiment of the present application;

FIG. 10 is a second schematic diagram of a Gb dot pixel matrix in an embodiment of the present application;

FIG. 11 is a second schematic diagram of an R dot pixel matrix in the embodiment of the present application;

FIG. 12 is a second schematic diagram of a B-dot pixel matrix in an embodiment of the present application;

FIG. 13 is a schematic diagram illustrating an exemplary configuration of a multi-exposure image processing apparatus according to an embodiment of the present disclosure;

FIG. 14 is a schematic diagram of the noise reduction circuit in the embodiment of the present application;

FIG. 15 is a schematic diagram of a first component structure of a noise reduction circuit according to an embodiment of the present application;

FIG. 16 is a diagram illustrating a second component structure of a noise reduction circuit according to an embodiment of the present application;

FIG. 17 is a schematic diagram of a multi-exposure image processing chip according to an embodiment of the present application;

fig. 18 is a schematic structural diagram of the electronic device in the embodiment of the present application.

Detailed Description

So that the manner in which the features and elements of the present embodiments can be understood in detail, a more particular description of the embodiments, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings.

The following provides a detailed example of a multi-exposure image processing method in an embodiment of the present application, and fig. 1 is a schematic flow chart of the multi-exposure image processing method in the embodiment of the present application, and as shown in fig. 1, the method may specifically include:

step 101: acquiring at least two exposure images of a target scene shot at different exposure times;

step 102: respectively determining the size of a pixel window for performing noise reduction processing on the at least two exposure images based on the exposure time of the at least two exposure images;

illustratively, the size of the pixel window includes a width and a height, and the determining the size of the pixel window for performing the noise reduction processing on the at least two exposure images based on the exposure time of the at least two exposure images respectively includes: at least one of a width and a height of the pixel window is made proportional to the exposure time.

That is, the longer the exposure time, the larger the size of the pixel window. Illustratively, the pixel window size for the first exposed image is configured to be a first width x a first height. The pixel window size for the second exposed image is configured to be a second width by a second height. When the first exposure time is less than the second exposure time, the first width is less than the second width, and/or the first height is less than the second height.

Step 103: performing noise reduction processing on the at least two exposure images respectively based on the determined sizes of the pixel windows;

in some embodiments, performing noise reduction processing on the at least two exposure images based on the determined size of the pixel window, respectively, comprises: and carrying out noise reduction processing on the pixel point of the central window in the pixel window based on the determined size of the pixel window.

Here, the size of the central window is 2 (width) x2 (height), and the central window includes 4 pixels rgrbgb, which are the points of the current pixel window that need to perform noise reduction operation. Further, the method further comprises: noise reduction of four Bayer-format pixel points in the central window is completed, and the pixel window is moved by two pixels along the horizontal direction to obtain a new pixel window; and carrying out noise reduction processing on the new pixel window until noise reduction of all pixel points of the exposure image is completed.

In the HDR shooting, the noise difference of the shot exposure images at different exposure time and the contribution of different exposure images to the HDR image are considered, the embodiment of the application configures pixel windows with different sizes for different exposure images, performs noise reduction operation on the image with larger noise interference by adopting the pixel window with the larger size, performs noise reduction operation on the image with smaller noise interference by adopting the pixel window with the smaller size, allocates different processing resources for different exposure images, and realizes reasonable allocation of the processing resources on the basis of ensuring the noise reduction effect, thereby improving the utilization rate of the processing resources and reducing the noise reduction cost.

Step 104: and performing fusion processing on the at least two exposure images subjected to the noise reduction processing to obtain a high dynamic range image of the target scene.

In some embodiments, the at least two exposure images include a first exposure image captured at a first exposure time, a second exposure image captured at a second exposure time, the method further comprising: acquiring a third exposure image of the target scene, wherein the third exposure image is shot in a third exposure time, and the third exposure time is longer than the first exposure time and longer than the second exposure time; and fusing the first exposure image subjected to noise reduction, the second exposure image subjected to noise reduction and the third exposure image which is not subjected to noise reduction to obtain a high dynamic range image of the target scene.

That is, for the multiple exposure images in the HDR photography, noise reduction operation may be performed on each exposure image, and then the exposure images after the noise reduction operation are subjected to fusion processing; or carrying out noise reduction operation on one part of the exposure images in the multi-exposure images, not carrying out noise reduction operation on the other part of the exposure images, and then carrying out fusion processing on the exposure images subjected to noise reduction processing and the exposure images not subjected to noise reduction processing.

For example, three exposure times are commonly used in HDR photography to obtain three exposure images, a short exposure image, a medium exposure image, and a long exposure image, which correspond to a first exposure time, a second exposure time, and a third exposure time, respectively, where the first exposure time is shorter than the second exposure time, and the second exposure time is shorter than the third exposure time. The noise reduction method provided by the embodiment of the application can be used for carrying out noise reduction operation on the short-exposure image and the middle-exposure image, and the long-exposure image is not subjected to noise reduction operation and is directly fused with the noise-reduced images of the other two images to obtain the HDR image.

By adopting the technical scheme, when the multi-exposure image of the HDR photography is processed, the sizes of the pixel windows used for executing the noise reduction processing on the at least two exposure images are respectively determined according to the exposure time, and the noise reduction processing is respectively executed on the at least two exposure images according to the determined sizes of the pixel windows, so that the noise reduction processing requirements of different exposure images are met, the image noise is effectively inhibited, on the basis of ensuring the noise reduction effect, the processing resources are saved, the noise reduction cost is reduced, and the noise reduction scheme of the HDR is optimized.

Based on the foregoing embodiment, taking the exposure image as a Bayer pattern image as an example, further illustrating a noise reduction operation, for example, as shown in fig. 2, the performing noise reduction processing on the at least two exposure images based on the pixel window respectively includes:

step 201: for a first pixel point in four Bayer format pixel points of a central window of the pixel window, constructing a first pixel matrix by using all the first pixel points in the pixel window;

here, the exposure image is a Bayer pattern image. The Bayer pattern is a pixel arrangement method named by the name of Bayer, i.e., an image is first divided into a plurality of 2 × 2 blocks, where each 2 × 2 block includes a Blue block (Blue, abbreviated as B), a Red block (Red, abbreviated as R) and two Green blocks (Green, abbreviated as G), and the two Green blocks can only be located at diagonal positions. In general, to distinguish two green blocks, one green block in the same row as a blue block is called Gb, one green block in the same row as a red block is called Gr, and Gb and Gr can be considered as different colors. All possible Bayer formats in a 2x2 block are the four modes rgrbb, bgbgbgbggrr, grRBGb, and GbBRGr. The center window is a 2x2 block at the center of the pixel window, four Bayer format pixel points are exemplified by RGrGbB, and other three formats are also applicable.

The first pixel point is a part of pixel points in the central window. Illustratively, the first pixel point is a green dot Gr or a green dot Gb among the four Bayer pattern pixel points.

For example, as shown in fig. 3, in the noise reduction operation of the intermediate exposure image in the HDR image capturing, a pixel window is a 14 (width) x10 (height) Bayer-format window, four Bayer-format pixel points rgrbgb of a center window are points of the current pixel window that need noise reduction, and other non-center points may be used as noise reduction reference points to participate in the noise reduction operation.

When performing the noise reduction operation on the green dot Gr in the central window, the noise reduction operation is performed on the first pixel matrix composed of 7 (width) × 5 (height) Gr dots with Gr as the center, as shown in fig. 4.

When performing the noise reduction operation on the green dots Gb of the central window, it is performed with the first pixel matrix composed of 7 (width) × 5 (height) Gb dots centered on Gb, as shown in fig. 5.

Step 202: performing the noise reduction operation on the first pixel matrix to obtain a noise-reduced first pixel point of the central window;

for example, the noise reduction operation may include averaging the points of the pixel matrix as the first pixel points after noise reduction, or averaging the points of the pixel matrix after filtering as the first pixel points after noise reduction in order to ensure the noise reduction effect.

In some embodiments, the noise reduction operation comprises: taking an absolute value of each pixel point in the pixel matrix, and then taking a square opening to obtain a square opening result sqrt _ abs _ cen of the central point of the pixel matrix and a square opening result sqrt _ abs _ ref of a non-central point of the pixel matrix; subtracting the central point sqrt _ abs _ cen from each non-central point sqrt _ abs _ ref, and taking an absolute value to obtain a difference value ABSDIFF; determining target non-center points and the number of the target non-center points, wherein the ABSDIFF is smaller than a first threshold; and accumulating all the target non-central points and dividing the accumulated target non-central points by the number of the target non-central points to obtain a noise reduction operation result.

For example, the first threshold may be an open-square result of the central point, sqrt _ abs _ cen, or a threshold obtained by calibrating sqrt _ abs _ cen according to a configuration parameter.

In some embodiments, the method further comprises: acquiring configured weight and offset; multiplying the square-opening result sqrt _ abs _ cen of the central point by the weight to obtain a product; and adding the product and the offset to obtain the first threshold.

Illustratively, the steps of performing the noise reduction operation on the Gr point in fig. 4 are as follows:

step 1: taking an absolute value of each Gr point in the 7x5 pixel matrix, and then taking the square of the Gr point, wherein for convenience of presentation, the square of the Gr point in the mark center is sqrt _ abs _ cenGr, and the square of the other Gr points is sqrt _ abs _ Gr;

step 2: calculating a threshold thresh _ cenGr, which is used for subsequent calculations, based on the sqrt _ abs _ cenGr value of the center Gr point, which is equal to sqrt _ abs _ cenGr _ reg _ weight + reg _ offset, where reg _ weight and reg _ offset are the parameters for the register configuration;

and step 3: subtracting the sqrt _ abs _ cenGr of the central Gr point from the 34 sqrt _ abs _ Gr values to obtain 34 difference values, then taking absolute values of the 34 difference values, naming the calculation result by ABSDIFF, and obtaining 34 ABSDIFF values in total;

and 4, step 4: and (3) comparing the 34 ABSDIFF values with a threshold value thresh _ cenGr respectively, picking out points of which the ABSDIFF values are smaller than the threshold value thresh _ cenGr, and accumulating the points to obtain an accumulated sum.

And 5: and (4) accumulating the sum and dividing the accumulated sum by the number of the pixel points participating in accumulation to obtain the average value of the pixel points participating in accumulation, and replacing the central Gr point with the average value to finish noise reduction.

The same as the Gr point noise reduction operation process, the step of performing noise reduction operation on the Gb point in fig. 5 is as follows:

step 1: taking an absolute value of each Gb point in a 7x5 pixel matrix, and then taking a square of the Gb point, wherein for convenience of presentation, the square of the Gb point at the center of the mark is sqrt _ abs _ cenGb, and the square of the other Gb points is sqrt _ abs _ Gb;

and 2, step: calculating a threshold thresh _ cenGb based on the value of sqrt _ abs _ cenGb at the central Gb point, which will be involved in subsequent calculations, the value of thresh _ cenGb being equal to sqrt _ abs _ cenGb reg _ weight + reg _ offset, where reg _ weight and reg _ offset are the parameters into which the registers are configured;

and step 3: subtracting the sqrt _ abs _ cenGb of the central Gb point from 34 sqrt _ abs _ Gb values respectively to obtain 34 difference values, then taking an absolute value of the 34 difference values, naming a calculation result by ABSDIFF, and obtaining 34 ABSDIFF values in total;

and 4, step 4: and (3) comparing the 34 ABSDIFF values with a threshold value thresh _ cenGb respectively, picking out points of which the ABSDIFF values are smaller than the threshold value thresh _ cenGb, and accumulating the points to obtain an accumulated sum.

And 5: and (4) accumulating the sum and dividing the accumulated sum by the number of the pixel points participating in accumulation to obtain an average value of the pixel points participating in accumulation, and replacing the central Gb point with the average value to finish noise reduction.

Step 203: for a second pixel point in the four Bayer format pixel points of the central window of the pixel window, forming a pixel pair by using all the second pixel points and the adjacent pixel points in the line in the pixel window;

the second pixel point is the other part of pixel points in the central window. Illustratively, the second pixel point is a red point R or a blue point B among the four Bayer pattern pixel points.

The noise reduction of the red dots in the four central points of the pixel window is performed based on all the pixel points in the row where the red dots are located, and there are 14 (width) x5 (height) points, as shown in fig. 6, R and Gr adjacent to the row can be regarded as a pair, and there are 7x5 such "pixel pairs" in total.

The noise reduction of the blue dot in the four central points of the pixel window is performed based on all the pixel points in the row where the blue dot is located, for a total of 14 (width) x5 (height) points, and as shown in fig. 7, the adjacent B and Gb in the row can be regarded as a pair, and there are 7x5 such "pixel pairs".

Step 204: summing each pixel pair, and constructing a second pixel matrix by using the sum value; calculating the difference of each pixel pair, and constructing a third pixel matrix by using the difference;

step 205: performing the noise reduction operation on the second pixel matrix and the third pixel matrix to obtain a sum value and a difference value after noise reduction;

here, the noise reduction processing is performed on the second pixel matrix and the third pixel matrix, respectively. In some embodiments, the noise reduction operation comprises: taking an absolute value of each pixel point in the pixel matrix, and then taking a square-open to obtain a square-open result sqrt _ abs _ cen of the central point of the pixel matrix and a square-open result sqrt _ abs _ ref of a non-central point of the pixel matrix; subtracting the central point sqrt _ abs _ cen from each non-central point sqrt _ abs _ ref, and taking an absolute value to obtain a difference value ABSDIFF; determining target non-center points of which the ABSDIFF is smaller than a first threshold value and the number of the target non-center points; and accumulating all the target non-central points and dividing the accumulated target non-central points by the number of the target non-central points to obtain a noise reduction operation result.

Illustratively, the noise reduction procedure for the R point in fig. 6 is as follows:

step 1: since R and Gr always appear in pairs, 35 pairs of R and Gr are summed respectively to obtain 35 (R + Gr) values, and at this time, (R + Gr) is regarded as one pixel, and there are 35 such pixels in total, which is exactly the same as the pixel matrix for Gr and Gb noise reduction, and the noise-reduced (R + Gr) can be obtained by using the same noise reduction operation;

step 2: because R and Gr always appear in pairs, respectively subtracting 35 pairs of R and Gr to obtain 35 (R-Gr) values, and considering (R-Gr) as a pixel, the total number of the (R-Gr) values is 35, which is completely the same as a pixel matrix of Gr and Gb noise reduction, and the (R-Gr) after noise reduction can be obtained by adopting the same noise reduction operation;

and step 3: averaging the two values of the denoised (R + Gr) obtained in the step 1 and the denoised (R-Gr) obtained in the step 2 again, and replacing the central R point with the calculation result at the moment to finish the denoising.

The noise reduction procedure for point B in fig. 7 is as follows:

step 1: because B and Gb always appear in pairs, 35 pairs of B and Gb are respectively summed to obtain 35 (B + Gb) values, at this time, (B + Gb) is regarded as one pixel, and then one pixel has 35 such pixels, which is completely the same as the color matrix for reducing noise of Gr and Gb, and the same algorithm is adopted, so that (B + Gb) after noise reduction can be obtained;

step 2: because B and Gb always appear in pairs, respectively subtracting 35 pairs of B and Gb to obtain 35 (B-Gb) values, and considering the (B-Gb) as a pixel, the total number of the (B-Gb) values is 35, which is completely the same as the color matrix for reducing noise of Gr and Gb, and the (B-Gb) value after noise reduction can be obtained by adopting the same algorithm;

and 3, step 3: averaging the denoised (B + Gb) obtained in the step 1 and the denoised (B-Gb) obtained in the step 2 again, and replacing the central point B with the calculation result at that time to finish the denoising.

It should be noted that the noise reduction processing at four points may be performed simultaneously, or may be performed sequentially according to a certain order.

Step 206: and averaging the sum and the difference after noise reduction to obtain a noise-reduced second pixel point of the central window.

In some embodiments, the method further comprises: noise reduction of four Bayer format pixel points in the central window is completed, and the pixel window is moved by two pixels along the horizontal direction to obtain a new pixel window; and carrying out noise reduction processing on the new pixel window until noise reduction of all pixel points of the exposure image is completed. Starting from the leftmost side of the exposed image, the above operations are repeated until the noise reduction of four points on the rightmost side in the transverse direction is completed, and the 14x10 pixel window is shifted downwards by two pixels and returns to the leftmost side of the image, and the noise reduction of a new line is started. The noise reduction operation is repeatedly carried out, and the noise reduction of all pixel points of the whole frame image can be completed.

Considering that the short exposure image contributes less to the HDR image than to the medium exposure image, a simpler noise reduction processing strategy can be adopted for the short exposure image, that is, by reducing the size of the pixel window, the calculation amount is reduced, and the noise reduction circuit is simplified, so that the noise reduction cost is reduced. For example, when the first exposure image is a short exposure image, the pixel window size may be 6 (width) x10 (height). When the second exposure image is a medium exposure image, the pixel window size may be 14 (width) x10 (height).

For example, as shown in fig. 8, in the noise reduction operation of the short-exposure image captured by the HDR, a pixel window is a Bayer-format window with a width of 6 (width) x10 (height), four Bayer-format pixel points rgrbgb of a center window are points of the current pixel window that need noise reduction, and other non-center points may be used as noise reduction reference points to participate in the noise reduction operation.

The noise reduction of the green dot Gr among the four dots in the center of the short-exposure image pixel window is performed with a pixel matrix composed of 3 (width) x5 (height) Gr dots centered on Gr, as shown in fig. 9. The step of performing the noise reduction operation on the Gr point 3 × 5 pixel matrix may refer to the noise reduction operation on the Gr point in the mid-exposure image.

The green dots Gb are de-noised in a pixel matrix of 3 (width) x5 (height) Gr dots centered at Gb as shown in fig. 10. The step of performing noise reduction operation on the Gb dot 3 × 5 pixel matrix may refer to noise reduction operation on the Gb dot in the mid-exposure image.

The noise reduction of the red dots in the four points in the center of the short-exposure image pixel window is performed based on all the pixel points in the row where the red dots are located, and there are 6 (width) x5 (height) points in total, as shown in fig. 11, R and Gr adjacent to the row can be regarded as a pair, and there are 3x5 such "pixel pairs" in total. The step of the noise reduction operation of the 3 × 5 pixel matrix may refer to the noise reduction operation of the R point in the middle exposure image.

The noise reduction of the blue dots in the four dots in the center of the short-exposure image pixel window is performed based on all the pixel dots in the row where the blue dots are located, and there are 6 (width) x5 (height) dots, and as shown in fig. 12, the adjacent B and Gb in the row can be considered as a pair, and there are 3x5 such "pixel pairs". The step of the noise reduction operation of the 3 × 5 pixel matrix may refer to the noise reduction operation of the B point in the middle exposure image.

Also, noise reduction of four points in the center of the short-exposure pixel window is completed. Next, the 6x10 pixel window is horizontally translated by two pixels to the right in the short-exposure image, a new 6x10 pixel window is constructed, and the noise reduction method is adopted to start to perform noise reduction on four points in the center of the new 6x10 pixel window. The above operations are repeated until the noise reduction of the four points on the far right side is completed, the 6 × 10 pixel window is shifted downwards by two pixels and returns to the far left side of the image, and the noise reduction of a new line is started. The noise reduction operation is repeatedly carried out, and the noise reduction of all pixel points of the whole frame image can be completed.

It should be noted that, if the long exposure image needs to be subjected to noise reduction processing, the same noise reduction operation method as that for the short exposure image and the medium exposure image may be adopted, and only the corresponding pixel window size needs to be configured according to the noise reduction requirement of the long exposure image.

In order to implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application further provides a multi-exposure image processing apparatus, as shown in fig. 13, including:

an obtaining module 131, configured to obtain at least two exposure images of a target scene captured at different exposure times;

a denoising module 132, configured to determine, based on the exposure time of the at least two exposure images, the size of a pixel window for performing denoising processing on the at least two exposure images, respectively; performing noise reduction processing on the at least two exposure images respectively based on the determined sizes of the pixel windows;

the fusion module 133 is configured to perform fusion processing on the at least two exposure images after the noise reduction processing to obtain a high dynamic range image of the target scene.

Illustratively, in some embodiments, the noise reduction module 132 includes:

a pixel window construction circuit 141 for loading a pixel window of an exposure image according to the size of the pixel window;

an odd row extracting circuit 142, configured to extract odd row pixels from the pixel window loaded by the window building circuit;

an even row extraction circuit 143 for extracting even row pixels from the window of pixels loaded by the window construction circuit;

a pixel matrix construction circuit 144, configured to construct a pixel matrix according to the odd-row pixels or the even-row pixels;

and the noise reduction operation circuit 145 is used for performing noise reduction operation on the pixel matrix and outputting noise-reduced pixel points.

The pixel matrix construction submodule is used for constructing a first pixel matrix for a first pixel point in four Bayer format pixel points of a central window of the pixel window by using all the first pixel points in the pixel window;

the noise reduction operation submodule is used for carrying out the noise reduction operation on the first pixel matrix to obtain a noise-reduced first pixel point of the central window;

the pixel matrix construction submodule is also used for forming a pixel pair by utilizing all second pixel points and the adjacent pixel points in the row in the pixel window for the second pixel point in the four Bayer-format pixel points of the central window of the pixel window; summing each pixel pair, and constructing a second pixel matrix by using the sum value; calculating the difference of each pixel pair, and constructing a third pixel matrix by using the difference;

the noise reduction operation sub-module is used for carrying out the noise reduction operation on the second pixel matrix and the third pixel matrix to obtain a sum value and a difference value after noise reduction; and averaging the sum and the difference after noise reduction to obtain a noise-reduced second pixel point of the central window.

In some embodiments, the first pixel point is a green dot Gr or a green dot Gb among the four Bayer pattern pixel points, and the second pixel point is a red dot R or a blue dot B among the four Bayer pattern pixel points.

In some embodiments, the denoising operation sub-module is specifically configured to first take an absolute value of each pixel point in the pixel matrix and then take a square-off value to obtain a square-off result sqrt _ abs _ cen of a central point of the pixel matrix and a square-off result sqrt _ abs _ ref of a non-central point of the pixel matrix; subtracting the central point sqrt _ abs _ cen from each non-central point sqrt _ abs _ ref, and taking an absolute value to obtain a difference value ABSDIFF; determining target non-center points of which the ABSDIFF is smaller than a first threshold value and the number of the target non-center points; and accumulating all the target non-central points and dividing the accumulated target non-central points by the number of the target non-central points to obtain a noise reduction operation result.

Illustratively, in some embodiments, the noise reduction operation submodule is further configured to obtain configured weights and offsets; multiplying the square-opening result sqrt _ abs _ cen of the central point by the weight to obtain a product; and adding the product and the offset to obtain the first threshold.

For example, in some embodiments, the noise reduction module 132 is further configured to complete noise reduction for four Bayer-format pixel points in the central window, and move the pixel window by two pixels along the horizontal direction to obtain a new pixel window; and carrying out noise reduction processing on the new pixel window until noise reduction of all pixel points of the exposure image is completed.

Illustratively, in some embodiments, the pixel window has dimensions including a width and a height, and the noise reduction module 132 is further configured to make at least one of the width and the height of the pixel window proportional to the exposure time.

Illustratively, in some embodiments, the at least two exposure images include a first exposure image captured at a first exposure time, a second exposure image captured at a second exposure time, and the obtaining module 131 is configured to obtain a third exposure image captured at a third exposure time for the target scene, where the third exposure time is greater than the first exposure time and greater than the second exposure time;

the fusion module 133 is configured to perform fusion processing on the first exposure image subjected to noise reduction processing, the second exposure image subjected to noise reduction processing, and the third exposure image not subjected to noise reduction processing, so as to obtain a high dynamic range image of the target scene.

Based on the method for reducing noise of an exposure image in the multi-exposure image processing method, an embodiment of the present application further provides a noise reduction circuit, which includes, for example, as shown in fig. 14:

a pixel window construction circuit 141 for loading a pixel window of an exposure image according to the size of the pixel window for performing noise reduction processing on the exposure image;

In some embodiments, the noise reduction circuitry further comprises a line buffer 146 for buffering pixels of a pixel window of row H-1, where H is the height of the pixel window, and the pixel window construction circuitry 141 loads the complete pixel window when the H-th row of pixels is fetched.

In some embodiments, the pixel matrix building circuit 144 includes: the device comprises a Gr point pixel matrix construction circuit, an R point pixel matrix construction circuit, a B point pixel matrix construction circuit and a Gb point pixel matrix construction circuit.

As shown in fig. 3, the R dots and the Gr dots are located in even rows, the Gb dots and the B dots are located in odd rows, and the Gr dot pixel matrix building circuit and the R dot pixel matrix building circuit build a Gr dot pixel matrix (i.e., a first pixel matrix) and an R dot pixel matrix according to the even-row pixels output by the even-row extracting circuit 143.

The B-point pixel matrix constructing circuit and the Gb-point pixel matrix constructing circuit construct a Gb-point pixel matrix (i.e., a first pixel matrix) and a B-point pixel matrix according to the odd-row pixels output by the odd-row extracting circuit 142.

Correspondingly, the noise reduction operation circuit 145, the Gr point noise reduction operation circuit, the R point noise reduction operation circuit, the B point noise reduction operation circuit and the Gb point noise reduction operation circuit respectively perform noise reduction operation on four points.

In some embodiments, the noise reduction circuit further includes an output circuit 147 for sorting and outputting the output order of the four points after noise reduction.

For a pixel window size of 14 (width) x10 (height), the noise reduction circuit comprises: a 9-line buffer for buffering the first 9 lines of pixels and a 14x10 pixel window construction circuit, the pixel window construction circuit constructing a 14x10 pixel window upon receiving the 10 th line of pixels. Extracting a 14x5 pixel matrix by using an odd-numbered row extraction circuit, extracting a 7x5 pixel matrix by using a Gr-point pixel matrix construction circuit, and directly acquiring the 14x5 pixel matrix by using an R-point pixel matrix construction circuit; a14 x5 pixel matrix is extracted by using an even row extraction circuit, a 14x5 pixel matrix is extracted by using a B point pixel matrix construction circuit, and a 3x5 pixel matrix is directly obtained by using a Gb point pixel matrix construction circuit.

For a pixel window size of 6 (width) x10 (height), the noise reduction circuit comprises: the 9 line buffer and 6x10 pixel window build circuit. Extracting a 6x5 pixel matrix by using an odd-row extraction circuit, extracting a 3x5 pixel matrix by using a Gr-point pixel matrix construction circuit, and directly obtaining the 6x5 pixel matrix by using an R-point pixel matrix construction circuit; a6 x5 pixel matrix is extracted by using an even row extraction circuit, a 6x5 pixel matrix is extracted by using a B point pixel matrix construction circuit, and a 3x5 pixel matrix is directly obtained by using a Gb point pixel matrix construction circuit.

Illustratively, as shown in fig. 15, the noise reduction operation circuit 145 includes:

an absolute value sub-circuit 151 for taking an absolute value for each point in the pixel matrix;

a square-open sub-circuit 152, configured to square the absolute value of each point in the pixel matrix to obtain a square-open result sqrt _ abs _ cen of the center point of the pixel matrix and a square-open result sqrt _ abs _ ref of the non-center point of the pixel matrix;

a difference and absolute value obtaining sub-circuit 153, configured to subtract the central point sqrt _ abs _ cen from each non-central point sqrt _ abs _ ref and obtain an absolute value, so as to obtain a difference value abdiff;

a comparison sub-circuit 154, configured to compare the absiff with a preset threshold, and determine a target non-center point and the number of target non-center points, where the absiff is smaller than the first threshold;

and the first averaging sub-circuit 155 is configured to accumulate all target non-center points and divide the accumulated target non-center points by the number of the target non-center points to obtain a noise-reduced pixel point.

In some embodiments, the noise reduction operation circuit 145 further includes: a threshold configuration sub-circuit 156 configured to configure the first threshold according to the square-on-square result sqrt _ abs _ cen of the center point. Illustratively, the first threshold is equal to sqrt _ abs _ cenGr _ reg _ weight + reg _ offset, where reg _ weight and reg _ offset are the weights and offsets into which the registers are configured.

For example, as shown in fig. 16, when performing a noise reduction operation on the second pixel, the noise reduction operation circuit 145 further includes: pixel pair summing sub-circuit 157 and pixel pair differencing sub-circuit 158,

the pixel pair summation sub-circuit 157 is configured to sum up pixel pairs adjacent to each other in rows in the pixel matrix output by the pixel matrix construction circuit to obtain a second pixel matrix, and input the second pixel matrix to the absolute value taking sub-circuit;

the pixel pair difference calculating sub-circuit 158 is configured to calculate a difference between adjacent pixel pairs in a row of the pixel matrix output by the pixel matrix constructing circuit to obtain a third pixel matrix, and input the third pixel matrix to the absolute value calculating sub-circuit.

Since R and Gr always appear in pairs, 35 pairs of R and Gr are summed to obtain 35 (R + Gr) values, and when (R + Gr) is regarded as one pixel, there are 35 such pixels in total, which is exactly the same as the pixel matrix for Gr and Gb noise reduction, and the noise-reduced (R + Gr) can be obtained by the same noise reduction operation.

Since R and Gr always appear in pairs, 35 pairs of R and Gr are respectively subjected to difference to obtain 35 (R-Gr) values, and at the moment, (R-Gr) is regarded as one pixel, so that the total number of the pixels is 35, and the (R-Gr) after noise reduction can be obtained by adopting the same noise reduction operation as the pixel matrix for reducing the noise of Gr and Gb.

The middle is the same as fig. 15, except that a second averaging sub-circuit 159 is added at the end for averaging the sum and difference values output by the first averaging sub-circuit to obtain the noise-reduced second pixel point of the center window.

Illustratively, the Gr noise reduction operation circuit and the Gb noise reduction operation circuit may be noise reduction operation circuits shown in fig. 15, and the R noise reduction operation circuit and the B noise reduction operation circuit may be noise reduction operation circuits shown in fig. 16.

In practical applications, the noise reduction module in the multi-exposure image processing apparatus can be implemented by the noise reduction circuit.

The present application also provides a multi-exposure image processing chip, as shown in fig. 17, where the chip 170 includes the noise reduction circuit 1701 as in any of the above embodiments.

Optionally, the multi-exposure image processing chip 170 may further include an input interface 1702. Through the input interface 1702, to communicate with other devices or chips, in particular, information or data transmitted by other devices or chips may be obtained.

Optionally, the multi-exposure image processing chip 170 may further include an output interface 1703. The output interface 1703 is used to communicate with other devices or chips, and in particular, to output information or data to the other devices or chips.

Alternatively, the multi-exposure image processing chip 170 may be applied to an electronic apparatus in the embodiment of the present application.

It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as a system-on-chip, a system-on-chip or a system-on-chip, etc.

In practical applications, the processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, and a microprocessor. It is understood that the electronic devices for implementing the above processor functions may be other devices, and the embodiments of the present application are not limited in particular.

The Memory may be a volatile Memory (volatile Memory), such as a Random-Access Memory (RAM); or a non-volatile Memory (non-volatile Memory), such as a Read-Only Memory (ROM), a flash Memory (flash Memory), a Hard Disk (HDD), or a Solid-State Drive (SSD); or a combination of the above types of memories and provides instructions and data to the processor.

The embodiment of the present application further provides an electronic device, where the electronic device described in the present application has a shooting function, and can shoot exposure images with different exposure times, and the electronic device may include a mobile phone, a tablet computer, a notebook computer, a palm computer, a Personal Digital Assistant (PDA), a Portable Media Player (PMP), a wearable device, a camera, a smart car, and the like.

As shown in fig. 18, the electronic device 180 includes: an image acquisition device 1801 and the aforementioned multi-exposure image processing chip 1802.

Exemplarily, the image capturing device 1801 is configured to capture at least two exposure images of a target scene captured in at least two exposure times;

the multi-exposure image processing chip 1802 of any one of the embodiments described above is configured to perform noise reduction processing on different exposure images by using different noise reduction processing strategies.

Optionally, the multi-exposure image processing chip 1803 is further configured to fuse the noise-reduced exposure images.

Of course, in practice, the various components of the electronic device 180 are coupled together by a bus system 1803, as shown at 18. It is understood that the bus system 1803 is used to enable communications for connections between these components. The bus system 1803 includes a power bus, a control bus, and a status signal bus in addition to a data bus. For clarity of illustration, however, the various buses are identified in fig. 18 as the bus system 1803.

In an exemplary embodiment, the present application further provides a computer readable storage medium, such as a memory including a computer program, which is executable by a processor to perform the steps of the foregoing method.

Embodiments of the present application also provide a computer program product comprising computer program instructions.

Optionally, the computer program product may be applied to a processor in the embodiment of the present application, and the computer program instructions enable a computer to execute corresponding processes implemented by the processor in the methods in the embodiment of the present application, which are not described herein again for brevity.

The embodiment of the application also provides a computer program.

Optionally, the computer program may be applied to the processor in the embodiment of the present application, and when the computer program runs on the computer, the computer is enabled to execute the corresponding process implemented by the processor in each method in the embodiment of the present application, and for brevity, details are not described here again.

It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The expressions "having", "may have", "include" and "contain", or "may include" and "may contain" in this application may be used to indicate the presence of corresponding features (e.g. elements such as values, functions, operations or components) but does not exclude the presence of additional features.

It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another, and are not necessarily used to describe a particular order or sequence. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention.

The technical solutions described in the embodiments of the present application can be arbitrarily combined without conflict.

In the several embodiments provided in the present application, it should be understood that the disclosed method, apparatus, and device may be implemented in other ways. The above-described embodiments are merely illustrative, and for example, the division of a unit is only one logical function division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit may be implemented in the form of hardware, or in the form of hardware plus a software functional unit.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application.

Claims

1. A multi-exposure image processing method, characterized in that the method comprises:

2. The multi-exposure image processing method according to claim 1, wherein the at least two exposure images are Bayer pattern images, and the performing the noise reduction processing on the at least two exposure images based on the pixel window, respectively, includes:

for a first pixel point in four Bayer format pixel points of a central window of the pixel window, constructing a first pixel matrix by using all the first pixel points in the pixel window;

performing noise reduction operation on the first pixel matrix to obtain a noise-reduced first pixel point of the central window;

for a second pixel point in the four Bayer format pixel points of the central window of the pixel window, forming a pixel pair by using all the second pixel points and the adjacent pixel points in the line in the pixel window;

summing each pixel pair, and constructing a second pixel matrix by using the sum value;

calculating the difference of each pixel pair, and constructing a third pixel matrix by using the difference;

performing the noise reduction operation on the second pixel matrix and the third pixel matrix to obtain a sum value and a difference value after noise reduction;

and averaging the sum and the difference after noise reduction to obtain a noise-reduced second pixel point of the central window.

3. The multi-exposure image processing method according to claim 2,

the first pixel point is a green dot Gr point or a green dot Gb point in the four Bayer format pixel points,

and the second pixel point is a red point R point or a blue point B point in the four Bayer format pixel points.

4. The multi-exposure image processing method according to claim 2 or 3, wherein the noise reduction operation includes:

taking an absolute value of each pixel point in the pixel matrix, and then taking a square opening to obtain a square opening result sqrt _ abs _ cen of the central point of the pixel matrix and a square opening result sqrt _ abs _ ref of a non-central point of the pixel matrix;

subtracting the central point sqrt _ abs _ cen from each non-central point sqrt _ abs _ ref, and taking an absolute value to obtain a difference value ABSDIFF;

determining target non-center points of which the ABSDIFF is smaller than a first threshold value and the number of the target non-center points;

and accumulating all the target non-central points and dividing the accumulated target non-central points by the number of the target non-central points to obtain a noise reduction operation result.

5. The multi-exposure image processing method according to claim 4, further comprising:

acquiring configured weight and offset;

multiplying the square-opening result sqrt _ abs _ cen of the central point by the weight to obtain a product;

and adding the product and the offset to obtain the first threshold.

6. The multi-exposure image processing method according to claim 2 or 3, further comprising:

noise reduction of four Bayer-format pixel points in the central window is completed, and the pixel window is moved by two pixels along the horizontal direction to obtain a new pixel window;

and carrying out noise reduction processing on the new pixel window until noise reduction of all pixel points of the exposure image is completed.

7. The multi-exposure image processing method according to any one of claims 1 to 3, wherein the size of the pixel window includes a width and a height, and the determining the size of the pixel window for performing the noise reduction processing on the at least two exposure images based on the exposure times of the at least two exposure images, respectively, includes:

at least one of a width and a height of the pixel window is made proportional to the exposure time.

8. The multi-exposure image processing method according to any one of claims 1 to 3, wherein the at least two exposure images include a first exposure image captured at a first exposure time, a second exposure image captured at a second exposure time, the method further comprising:

acquiring a third exposure image of the target scene, wherein the third exposure image is shot in a third exposure time, and the third exposure time is longer than the first exposure time and longer than the second exposure time;

and fusing the first exposure image subjected to noise reduction, the second exposure image subjected to noise reduction and the third exposure image which is not subjected to noise reduction to obtain a high dynamic range image of the target scene.

9. A multi-exposure image processing apparatus characterized by comprising:

10. A noise reduction circuit, comprising:

11. The noise reduction circuit according to claim 10, wherein the noise reduction operation circuit includes:

an absolute value sub-circuit for taking an absolute value for each point in the pixel matrix;

the square-on sub-circuit is used for square-on of the absolute value of each point in the pixel matrix to obtain a square-on result sqrt _ abs _ cen of the central point of the pixel matrix and a square-on result sqrt _ abs _ ref of the non-central point of the pixel matrix;

a difference absolute value obtaining sub-circuit, which is used for subtracting the central point sqrt _ abs _ cen from each non-central point sqrt _ abs _ ref and obtaining an absolute value to obtain a difference value ABSDIFF;

the comparison sub-circuit is used for comparing the ABSDIFF with a preset threshold value and determining the number of target non-central points and the number of target non-central points, wherein the ABSDIFF is smaller than a first threshold value;

and the first averaging sub-circuit is used for accumulating all target non-central points and then dividing the accumulated target non-central points by the number of the target non-central points to obtain the noise-reduced pixel points.

12. The noise reduction circuit according to claim 11, wherein the noise reduction operation circuit further comprises:

a threshold configuration sub-circuit configured to configure the first threshold according to a result sqrt _ abs _ cen of the square-off of the center point.

13. The noise reduction circuit according to claim 11 or 12, wherein when performing the noise reduction operation on the second pixel, the noise reduction operation circuit further comprises: a pixel pair summing sub-circuit and a pixel pair differencing sub-circuit,

the pixel pair summation sub-circuit is used for summing pixel pairs adjacent to rows in the pixel matrix output by the pixel matrix construction circuit to obtain a second pixel matrix, and inputting the second pixel matrix to the absolute value taking sub-circuit;

and the pixel pair difference calculating sub-circuit is used for calculating the difference of the adjacent pixel pairs in the row of the pixel matrix output by the pixel matrix constructing circuit to obtain a third pixel matrix, and inputting the third pixel matrix to the absolute value calculating sub-circuit.

14. The noise reduction circuit according to claim 13, wherein the noise reduction operation circuit further comprises: and the second averaging sub-circuit is used for averaging the sum and the difference output by the first averaging sub-circuit to obtain a second pixel point of the central window after noise reduction.

15. A multi-exposure image processing chip, characterized in that the chip comprises a noise reduction circuit according to any one of claims 10 to 14.

16. An electronic device, characterized in that the electronic device comprises: an image capturing device and a multi-exposure image processing chip as claimed in claim 15.

17. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.