WO2024094222A1

WO2024094222A1 - Multi-exposure image fusion method and system based on image region validity

Info

Publication number: WO2024094222A1
Application number: PCT/CN2023/137047
Authority: WO
Inventors: 林景洲; 郭奇锋; 张齐宁
Original assignee: 深圳深知未来智能有限公司
Priority date: 2022-11-02
Filing date: 2023-12-07
Publication date: 2024-05-10
Also published as: CN115409754B; CN115409754A

Abstract

The present invention relates to the technical field of image processing. Disclosed are a multi-exposure image fusion method and system based on image region validity. The method comprises: first acquiring a plurality of raw images of different exposures, and normalizing same; calculating a current noise level on the basis of the plurality of normalized raw images of different exposures; on the basis of the current noise level, acquiring validity graphs corresponding to the raw images; calculating corresponding weight graphs on the basis of the validity graphs corresponding to the raw images; and fusing the raw images according to the corresponding weight graphs to obtain a fused image. According to the present invention, during multi-exposure image fusion, the amount of calculation is small, the real-time performance is good, the fusion effect is better, and the influence during image signal processing can be reduced.

Description

A multi-exposure image fusion method and system based on image region validity

Technical Field

The present invention relates to the field of image processing technology, and more particularly to a multi-exposure image fusion method and system based on image region validity.

Background technique

With the development of computer and multimedia technology, various multimedia applications have put forward extensive demands for high-quality images. High-quality images can provide rich information and real visual experience. However, in the process of image acquisition, affected by factors such as image acquisition equipment, acquisition environment, and noise, the images presented on the display terminal are often low-quality images. Therefore, how to reconstruct high-quality images from low-quality images has always been a challenge in the field of image processing.

Multi-exposure image fusion is a technology that fuses multiple images with different exposures into one image. Multi-exposure image fusion technology can improve the imaging quality of images and avoid the loss of details in bright or dark areas caused by a single exposure. Existing multi-exposure image fusion methods mainly include Exposure Fusion (Mertens TMO) and image fusion based on guided filtering. However, the existing fusion methods still have the following defects:

Exposure Fusion—Mertens TMO: It is necessary to use the Laplace operator to calculate the three channels of the image, which consumes a lot of computing power and is prone to color cast.

Image fusion based on guided filtering: Multiple filtering operations are performed on color images. The process is complex, consumes a lot of computing power, and is prone to halo phenomenon.

Therefore, how to overcome the defects of the above-mentioned prior art and solve technical problems such as large amount of calculation and poor effect in the multi-exposure image fusion process is an issue that technical personnel in this field need to solve urgently.

Summary of the invention

In view of this, the present invention provides a multi-exposure image fusion method and system based on image region validity, which has simple method, strong scene adaptability and high speed.

In order to achieve the above object, the present invention provides the following technical solutions:

A multi-exposure image fusion method based on image region validity comprises the following steps:

Step 1: Obtain several raw images with different exposures and normalize them. Subsequent image processing is based on the normalized raw images.

Step 2: Calculate the current noise level based on several raw images with different exposures;

Step 3: Based on the current noise level, obtain the validity map corresponding to each raw image;

Step 4: Based on the validity graph corresponding to each raw image, calculate the corresponding weight graph;

Step 5: Fuse each raw image according to the corresponding weight map to obtain a fused image.

Optionally, in step 1, the plurality of raw images with different exposures are sorted in descending order according to exposure time to form a set A = {raw0, raw1, ..., rawi, ..., raw(n-1)}, and the exposure time ratio is ratio, where n represents the number of raw images in set A, and n≥2.

Optionally, in step 2, the method for calculating the current noise level is:

The calculation is performed according to the principle of scene consistency, the noise mean noise_u = mean(rawj-ratio*raw(j+1)), and the noise variance noise_std = std(rawj-ratio*raw(j+1)), where rawj and raw(j+1) are any two adjacent raw images in set A, 0≤j≤n-2.

The present invention utilizes the multi-frame exposure characteristics to evaluate the noise level, which is simpler than other noise evaluation methods.

Optionally, in step 3, the method for obtaining the validity map corresponding to each raw image is:

For the i-th raw image rawi, calculate the validity of each point except the shortest frame according to the exposure time ratio:

e_q = (ratio-1)*noise_u;

d_q = sqrt(ratio**2+1)*noise_std;

Si = rawi-ratio*raw(i+1);

Pi = (Si - (e_q - d_q)) / max (delta, 2*d_q), delta = 1e ^-6 ;

Img_valid_i＝(1-Pi);

The shortest frame validity is obtained by subtracting the validity of the second shortest frame from 1.

The image exposure effectiveness generated by the present invention is more accurate through the noise level characteristics, more suitable for the fusion of nighttime high-noise scenes, and has strong scene adaptability.

In addition, the present invention performs validity evaluation in the raw domain to obtain a validity graph, because raw data is closer to linearity and can more accurately reflect the progressive process of image brightness, thereby reducing the influence of gamma on brightness values during image signal processing.

Optionally, in step 4, for the validity graph Img_valid_i corresponding to the i-th raw image, the method for calculating the corresponding weight graph is:

Step 4.1, use the Laplace pyramid method to perform Gaussian blur on the validity map Img_valid_i;

Step 4.2, reduce the size of the validity image Img_valid_i after Gaussian blur to 1/2 of the original size;

Step 4.3: Perform several scale transformations according to step 4.2 to form a sequence of images from large to small. According to the principle that the upper image is larger than the lower image, form a Gaussian pyramid Img_valid_i_pyramid. Then, starting from the image at the bottom of the Gaussian pyramid, perform the following operations to the second-to-last top layer:

A. Use bilinear interpolation to enlarge the image to be consistent with the previous layer;

B. Calculate and synthesize a new image with the image of the previous layer;

C. Replace the original upper layer image with the new image;

The obtained top layer image becomes the weight map w_i, which can enhance the smoothness of the fused image.

The above method judges the weight by effectiveness, which is more accurate and requires less calculation than simply judging the weight by the value of the brightness map.

Optionally, in step 5, each raw image is fused according to the corresponding weight map to obtain a fused image, and the fusion formula is:

Raw_hdr＝raw0*w_0+raw1*w_1*ratio+.....+rawi*w_i*ratio ⁱ +...+raw(n-1)*w_(n-1)*ratio ^n-1 , where w_i represents the weight map corresponding to the i-th raw image.

The present invention performs image fusion in the raw domain, and compared with jpg images, the retention ratio of image information can be increased.

A multi-exposure image fusion system based on image region validity, comprising:

The exposure image acquisition module is used to obtain several raw images with different exposures;

The noise level calculation module is used to calculate the current noise level based on a number of normalized raw images with different exposures;

The validity map acquisition module is used to obtain the validity map corresponding to each raw image based on the current noise level;

The weight map acquisition module is used to calculate the corresponding weight map based on the validity map corresponding to each raw image;

The fusion module is used to fuse each raw image according to the corresponding weight map.

It can be seen from the above technical solutions that the present invention discloses a multi-exposure image fusion method and system based on image region validity, which has the following beneficial effects compared with the prior art:

Speed: Since the amount of calculation is greatly reduced, the present invention can be quickly deployed on terminal devices and can achieve good real-time performance.

Effect: The present invention fuses the unprocessed original image data in the raw domain, which can better cooperate with the later image signal processing debugging and ensure the image effect of the final output.

Applicability: The present invention is suitable for most sensors, has strong portability, can be expanded to other task areas, and can be widely used in photo-taking and video-related systems and terminal devices.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

FIG1 is a schematic flow chart of the method of the present invention;

FIG. 2 is a schematic diagram of a system module of the present invention.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The embodiment of the present invention discloses a multi-exposure image fusion method based on image region validity, referring to FIG1 , comprising the following steps:

Step 1: obtain several raw images with different exposures, normalize them, and sort them in descending order according to exposure time to form a set A = {raw0, raw1, ..., rawi, ..., raw(n-1)}, where the exposure time ratio is ratio, where n represents the number of raw images in set A, and n≥2.

Step 2: Calculate the current noise level based on the normalized raw images with different exposures:

Step 3: Based on the current noise level, obtain the validity map corresponding to each raw image.

Specifically, for the i-th raw image rawi, the validity of each point except the shortest frame (raw image with the shortest exposure time) is calculated according to the exposure time ratio:

e_q = (ratio-1)*noise_u;

d_q = sqrt(ratio**2+1)*noise_std;

Si = rawi-ratio*raw(i+1);

Pi = (Si - (e_q - d_q)) / max (delta, 2*d_q), delta = 1e ^-6 ;

Img_valid_i＝(1-Pi);

In other embodiments, the validity may also be calculated in the following manner:

Si = rawi-ratio*raw(i+1);

e_q = (ratio-1)*noise_u;

Img_valid_i = (Si - e_q) * ratio;

Step 4: Based on the validity graph corresponding to each raw image, calculate the corresponding weight graph.

Specifically, for the validity graph Img_valid_i corresponding to the i-th raw image, the method for calculating the corresponding weight graph is:

Step 4.3: Perform several scale transformations according to step 4.2 to form an image sequence from large to small. The top layer is the original size, and it is reduced to the lower layer at a fixed ratio. For example, the first layer has a width and height of 20x10, and the next layer is 10x5. According to this rule, the Gaussian pyramid Img_valid_i_pyramid is formed. Then, starting from the image at the bottom of the Gaussian pyramid, the following operations are performed to the second-to-last top layer:

B. Calculate and synthesize a new image with the image of the previous layer;

C. Replace the original upper layer image with the new image;

The resulting top layer image becomes the weight map w_i.

In other embodiments, the Laplacian pyramid may be replaced by other methods to perform smoothing operations.

Step 5: Fuse each raw image according to the corresponding weight map to obtain a fused image. The fusion formula is:

The embodiment of the present invention further discloses a multi-exposure image fusion system based on image region validity, corresponding to the above method embodiment, referring to FIG2, comprising:

The following is a specific example in which ratio=8 and n=3 to illustrate the solution of the present invention.

1. Get multi-exposure images

Set the exposure time between long and short exposure to a fixed 8 times, i.e. ratio = 8. Obtain 3 images with different exposures, record the long exposure as raw0, the medium exposure as raw1, and the short exposure as raw2. The raw images are normalized to the interval [0,1]. Subsequent steps are all image processing of the normalized raw images.

2. Noise Level Calculation

Based on the multiple exposure images obtained and the assumption of scene consistency, the estimated value of the current noise level is statistically calculated:

Noise mean: noise_u = mean(raw0-8*raw1);

Noise variance: noise_std=std(raw0-8*raw1).

3. Effectiveness Calculation

According to the noise level, from two frames of images with different exposures, the effectiveness of each point except the shortest frame is calculated according to the exposure ratio relationship:

e_q = (ratio-1)*noise_u;

d_q = sqrt(ratio**2+1)*noise_std;

S0 = raw0-8*raw1;

P0＝(S0-(e_q-d_q))/max(delta,2*d_q),delta＝1e ^-6 ;

Img_valid_0 = (1-P0);

4. Weight calculation

Apply the Laplacian pyramid method, first do Gaussian blur on the validity map Img_valid_i, and then reduce the image size to 1/2 of the original. Perform several scale transformations in the above way to form an image sequence from large to small. According to the principle that the upper layer image is larger than the lower layer image, form the Gaussian pyramid Img_valid_i_pyramid, and then start from the bottom layer image of the Gaussian pyramid to the second to last top layer to perform the following operations:

B. Calculate and synthesize a new image with the image of the previous layer;

C. Replace the original upper layer image with the new image;

The final top-level image becomes the weight map w_i.

5. Integration

The original raw image is synthesized into the final HDR raw image according to the weight:

Raw_hdr=raw0*w_0+raw1*w_1*8+raw2*w_2*64.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

A multi-exposure image fusion method based on image region validity, characterized by comprising the following steps:

Step 1, obtain several raw images with different exposures, sort them in descending order according to exposure time, form a set A = {raw0, raw1, ..., rawi, ..., raw(n-1)}, the exposure time ratio is ratio, where n represents the number of raw images in the set A, rawi represents the i-th raw image, and n≥2; and perform normalization;

Step 2: Calculate the current noise level based on several normalized raw images with different exposures. The method is:

The calculation is performed according to the principle of scene consistency. The noise mean noise_u = mean(rawj-ratio*raw(j+1)), and the noise variance noise_std = std(rawj-ratio*raw(j+1)), where rawj and raw(j+1) are any two adjacent raw images in set A, 0≤j≤n-2;

Step 3: Based on the current noise level, obtain the validity map corresponding to each raw image by:

For the i-th raw image rawi, calculate the validity of each point except the shortest frame according to the exposure time ratio:
e_q = (ratio-1)*noise_u;
d_q = sqrt(ratio*2+1)*noise_std;
Si = rawi-ratio*raw(i+1);
Pi = (Si - (e_q - d_q)) / max (delta, 2*d_q), delta = 1e -6 ;
Img_valid_i=1-Pi;

The shortest frame validity is obtained by subtracting the validity of the second shortest frame from 1;

Step 4: Based on the validity graph corresponding to each raw image, calculate the corresponding weight graph;

Step 5: Fuse each raw image according to the corresponding weight map.
The multi-exposure image fusion method based on image region validity according to claim 1 is characterized in that in step 4, for the validity map Img_valid_i corresponding to the i-th raw image, the method for calculating the corresponding weight map is:

Step 4.1, use the Laplace pyramid method to perform Gaussian blur on the validity map Img_valid_i;

Step 4.2, reduce the size of the validity image Img_valid_i after Gaussian blur to 1/2 of the original size;

Step 4.3: Perform several scale transformations according to step 4.2 to form a sequence of images from large to small. According to the principle that the upper image is larger than the lower image, form a Gaussian pyramid Img_valid_i_pyramid. Then, starting from the image at the bottom of the Gaussian pyramid, perform the following operations to the second-to-last top layer:

A. Use bilinear interpolation to enlarge the image to be consistent with the previous layer;

B. Calculate and synthesize a new image with the image of the previous layer;

C. Replace the original upper layer image with the new image;

The resulting top layer image becomes the weight map w_i.
The multi-exposure image fusion method based on image region validity according to claim 1 is characterized in that in step 5, each raw image is fused according to the corresponding weight map to obtain a fused image, and the fusion formula is:

Raw_hdr＝raw0*w_0+raw1*w_1*ratio+...+rawi*w_i*ratio i +...+raw(n-1)*w_(n-1)*ratio n-1 , where w_i represents the weight map corresponding to the i-th raw image.
A multi-exposure image fusion system based on image region validity, characterized by comprising:

An exposure image acquisition module is used to acquire a number of raw images with different exposures, sort them in descending order according to exposure time, form a set A = {raw0, raw1, ..., rawi, ..., raw(n-1)}, and the exposure time ratio is ratio, where n represents the number of raw images in the set A, n≥2; and perform normalization;

The noise level calculation module is used to calculate the current noise level based on several normalized raw images with different exposures. Flat, specific:

The calculation is performed according to the principle of scene consistency. The noise mean noise_u = mean(rawj-ratio*raw(j+1)), and the noise variance noise_std = std(rawj-ratio*raw(j+1)), where rawj and raw(j+1) are any two adjacent raw images in set A, 0≤j≤n-2;

The validity map acquisition module is used to obtain the validity map corresponding to each raw image based on the current noise level. Specifically:

For the i-th raw image rawi, calculate the validity of each point except the shortest frame according to the exposure time ratio:
e_q = (ratio-1)*noise_u;
d_q = sqrt(ratio*2+1)*noise_std;
Si = rawi-ratio*raw(i+1);
Pi = (Si - (e_q - d_q)) / max (delta, 2*d_q), delta = 1e -6 ;
Img_valid_i=1-Pi;

The shortest frame validity is obtained by subtracting the validity of the second shortest frame from 1;

The weight map acquisition module is used to calculate the corresponding weight map based on the validity map corresponding to each raw image;

The fusion module is used to fuse each raw image according to the corresponding weight map.