CN112689099B - A ghost-free high dynamic range imaging method and device for a dual-lens camera - Google Patents

Abstract

Embodiments of the present invention provide a ghost-free high dynamic range imaging method and device for a dual-lens camera, to obtain a long-exposure image and a short-exposure image collected by the dual-lens camera at the same time; The image and the short exposure image are input into a main image enhancement model for ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image: performing alignment adjustment on the exposure image to obtain an aligned image; performing exposure adjustment and noise reduction processing on the short-exposure image based on the aligned image to obtain a noise-reduced image; and fusing the short-exposure image and the noise-reduced image, The high dynamic range image is obtained. With this solution, a ghost-free high dynamic range image for the dual-lens camera can be obtained, and the imaging quality of the dual-lens camera can be improved.

Description

A ghost-free high dynamic range imaging method and device for a dual-lens camera

technical field

The invention relates to the technical field of digital image processing, in particular to a ghost-free high dynamic range imaging method and device for a dual-lens camera.

Background technique

With the development of technology, dual-lens cameras are increasingly used in mobile terminals. Since mobile terminals are usually ordinary consumer-level electronic devices, the dual-lens cameras used in mobile terminals are limited by hardware, and usually can only capture low dynamic range images with a small exposure range, far from high dynamic range images. the quality of. In this regard, in order to improve the image quality of the dual-lens camera, two low dynamic range images with different exposures captured at the same time can be used to synthesize high dynamic range images.

In the related art, two low dynamic range images can be aligned and adjusted to obtain an aligned image, and the pixel values of the aligned image and the short exposure image with shorter exposure time in the two low dynamic range images are fused into one image. image, a high dynamic range image is obtained. The alignment adjustment may include: finding a one-to-one correspondence between pixels in the two images, and according to the correspondence, using the short-exposure image to reconstruct a long-exposure image with a longer exposure time in the two low-dynamic-range images to obtain an alignment post image.

However, due to the different exposure times of the above two low dynamic range images, there are often pixels that do not correspond accurately in the aligned images, and there are incorrectly aligned areas in the aligned images, resulting in ghosting in the high dynamic range images.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present invention is to provide a ghost-free high dynamic range imaging method and device for a dual-lens camera, so as to solve the problem of ghosting in the high dynamic range image of the dual-lens camera. The specific technical solutions are as follows:

In a first aspect, an embodiment of the present invention provides a ghost-free high dynamic range imaging method for a dual-lens camera, the method comprising:

Obtain a long-exposure image and a short-exposure image captured by the dual-lens camera at the same moment;

The long-exposure image and the short-exposure image are input into a main image enhancement model for ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:

Based on the short exposure image, performing alignment adjustment on the long exposure image to obtain an alignment image;

Based on the alignment image, performing exposure adjustment and noise reduction processing on the short exposure image to obtain a noise reduction image;

The short exposure image and the noise reduction image are fused to obtain the high dynamic range image.

In a second aspect, an embodiment of the present invention provides a ghost-free high dynamic range imaging device for a dual-lens camera, the device comprising:

The input acquisition module is used to acquire a long-exposure image and a short-exposure image collected by the dual-lens camera at the same time;

An image processing module, configured to input the long-exposure image and the short-exposure image into a main image enhancement model for ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:

Based on the short exposure image, performing alignment adjustment on the long exposure image to obtain an alignment image;

Based on the alignment image, performing exposure adjustment and noise reduction processing on the short exposure image to obtain a noise reduction image;

The short exposure image and the noise reduction image are fused to obtain the high dynamic range image.

Beneficial effects of the embodiment of the present invention:

In the solution provided by the embodiment of the present invention, a long-exposure image and a short-exposure image collected by a dual-lens camera at the same time are input into the main image enhancement model of ghost-free high dynamic range imaging, and then the main image enhancement model Based on the short-exposure image, the long-exposure image can be aligned and adjusted to obtain an aligned image; based on the aligned image, the short-exposure image can be subjected to exposure adjustment and noise reduction processing to obtain a noise-reduced image; the short-exposure image and the noise-reduced image can be fused, Get high dynamic range images. Among them, exposure adjustment and noise reduction processing are performed on the short-exposure image based on the aligned image to obtain a noise reduction image, which can ensure that the aligned image contributes to exposure adjustment and noise reduction processing. Therefore, the information contained in the alignment image can be added to the denoised image through exposure adjustment and noise reduction, while ensuring that the position of the pixels in the denoised image will not be affected by the alignment image, avoiding the generation of ghosting, and further reducing short exposures. The image and the denoised image are fused, and the resulting high dynamic range image has no ghosting. It can be seen that through this solution, a ghost-free high dynamic range image for the dual-lens camera can be obtained, and the imaging quality of the dual-lens camera can be improved.

Of course, it is not necessary for any product or method of the present invention to achieve all of the advantages described above at the same time.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can also be obtained according to these drawings without creative efforts.

1 is a schematic flowchart of a ghost-free high dynamic range imaging method for a dual-lens camera according to an embodiment of the present invention;

FIG. 2 is an exemplary diagram of an acquisition flow of an aligned image in a ghost-free high dynamic range imaging method for a dual-lens camera provided by another embodiment of the present invention;

FIG. 3 is an example diagram of an acquisition process of a noise-reduced image in a ghost-free high dynamic range imaging method for a dual-lens camera provided by another embodiment of the present invention;

4 is an exemplary diagram of a ghost-free high dynamic range imaging method for a dual-lens camera provided by another embodiment of the present invention;

5 is a schematic structural diagram of a ghost-free high dynamic range imaging device for a dual-lens camera according to an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

A ghost-free high dynamic range imaging method for a dual-lens camera provided by an embodiment of the present invention can be applied to a dual-lens camera or an electronic device that uses a dual-lens to capture images, such as a dual-lens mobile terminal, a wearable device, etc. .

As shown in FIG. 1, an embodiment of the present invention provides a process of a dual-lens camera-oriented high dynamic range imaging method without ghosting, and the method may include the following steps:

S101 , acquiring a long-exposure image and a short-exposure image acquired by the dual-lens camera at the same moment.

In a specific application, the two lenses of the dual-lens camera may be lenses of the same model. Moreover, the above-mentioned one long-exposure image and one short-exposure image are images of the same subject. The exposure time of the long-exposure image is greater than the exposure time of the short-exposure image.

S102 , input the long exposure image and the short exposure image into the main image enhancement model for ghost-free high dynamic range imaging, so that the main image enhancement model performs the operations of steps S1021 to S1023 to obtain a high dynamic range image.

S1021 , based on the short-exposure image, perform alignment adjustment on the long-exposure image to obtain an aligned image.

In a specific application, based on the short exposure image, the alignment adjustment is performed on the long exposure image, and there may be various specific ways to obtain the aligned image. Exemplarily, the pixels in the short-exposure image and the long-exposure image can be aligned one-to-one, and the information of the pixels in the short-exposure image is adjusted by adjusting the information of the pixels in the long-exposure image to obtain an aligned image. Or, exemplarily, the long-exposure image may be used to adjust the exposure of the short-exposure image to obtain a first soft-exposure image; the long-exposure image and the first soft-exposure image may be input into a pre-trained image matching convolutional neural sub-network, An aligned image is obtained by adjusting the long-exposure image to an image aligned with the first soft-exposure image so that the image-matching convolutional neural sub-network. In order to facilitate understanding and reasonable layout, the second exemplary case will be specifically described in the form of optional embodiments in the following.

S1022, based on the alignment image, perform exposure adjustment and noise reduction processing on the short exposure image to obtain a noise reduction image.

When performing exposure adjustment and noise reduction processing on a short-exposure image based on the aligned image, you can specifically refer to the pixels of the aligned image to obtain information for adjusting and denoising the pixels of the short-exposure image itself, so as to perform Short exposure images undergo exposure adjustment and noise reduction. In order to facilitate understanding and reasonable layout, a specific manner of obtaining a noise-reduced image by performing exposure adjustment and noise reduction processing on a short-exposure image based on an aligned image will be described in the form of an optional embodiment.

S1023, fuse the short exposure image and the noise reduction image to obtain a high dynamic range image.

The denoised image is based on the aligned image and, therefore, contains information from the long exposure image. Based on this, the short exposure image and the noise reduction image are fused to ensure that the fused image contains the information in the short exposure and long exposure images, and is a high dynamic range image with a higher exposure range than the short exposure image and the long exposure image. . In order to facilitate understanding and reasonable layout, the process of fusing a short exposure image and a noise reduction image to obtain a high dynamic range image will be specifically described in the form of an optional embodiment in the following.

In the solution provided by the embodiment of the present invention, exposure adjustment and noise reduction processing are performed on the short-exposure image based on the aligned image to obtain a noise reduction image, which can ensure that the aligned image contributes to exposure adjustment and noise reduction processing. Therefore, the information contained in the alignment image can be added to the denoised image through exposure adjustment and noise reduction, while ensuring that the position of the pixels in the denoised image will not be affected by the alignment image, avoiding the generation of ghosting, and further reducing short exposures. The image and the denoised image are fused, and the resulting high dynamic range image has no ghosting. It can be seen that through this solution, a ghost-free high dynamic range image for the dual-lens camera can be obtained, and the imaging quality of the dual-lens camera can be improved.

In an optional implementation manner, the above-mentioned alignment adjustment is performed on the long exposure image based on the short exposure image to obtain the aligned image, which may specifically include the following steps A1 to A2:

A1, using the long-exposure image to adjust the exposure of the short-exposure image to obtain a first soft-exposure image;

A2, input the long-exposure image and the first soft-exposure image into the image-matching convolutional neural sub-network obtained by pre-training, so that the image-matching convolutional neural sub-network adjusts the long-exposure image to an image aligned with the first soft-exposure image , get the aligned image;

The image matching convolutional neural sub-network is a network trained by using multiple sample long-exposure images, multiple sample short-exposure images, and real long-exposure images corresponding to each sample short-exposure image; The real long-exposure image is an image obtained by performing physical long-exposure on the short-exposure image by the dual-lens camera.

In this optional embodiment, by using the long-exposure image to adjust the exposure of the short-exposure image, it is equivalent to simulating the physical exposure of a camera, thereby realizing soft exposure of the short-exposure image. Furthermore, by using the first soft-exposure image to obtain the alignment image, the exposure of the first soft-exposure image obtained by using the long-exposure image is more similar to that of the long-exposure image compared to directly using the short-exposure image to obtain the alignment image. Therefore, the alignment adjustment can be reduced. complexity, making alignment adjustments easier to implement.

In an optional implementation manner, the above step A1: using the long-exposure image to adjust the exposure of the short-exposure image to obtain the first soft-exposure image, may specifically include the following steps:

Obtain the histogram of the long exposure image as the target histogram;

Perform exposure adjustment on the short-exposure image, and obtain a histogram of the adjusted short-exposure image as the histogram to be adjusted;

Adjust the histogram to be adjusted so that the difference value between the histogram to be adjusted and the target histogram satisfies the preset difference condition, obtain an adjusted histogram, and use the image corresponding to the adjusted histogram as the first soft exposure image.

This optional embodiment utilizes the adjustment of the histogram, that is, the equalization of the histogram to achieve soft exposure, which can reduce the requirement for matching and alignment between the long-exposure image and the short-exposure image, and further reduce the complexity of obtaining aligned images.

In an optional implementation manner, in the above step A2: the image matching convolutional neural sub-network adjusts the long-exposure image to an image aligned with the first soft-exposure image to obtain an aligned image, which may specifically include the following steps:

extracting the depth feature of the long exposure image as the first depth feature, and extracting the depth feature of the first soft exposure image as the second depth feature;

The first depth feature and the second depth feature are mapped to a four-dimensional feature, as the first four-dimensional feature; the first four-dimensional feature is a feature used to reflect the first depth feature and the second depth feature;

Adjusting the first four-dimensional feature into a three-dimensional feature using a three-dimensional adjustment network;

For each pixel in the three-dimensional feature, weighted average is performed between the pixel and the pixel at the corresponding position in the long-exposure image to obtain an aligned image.

Illustratively, as shown in FIG. 2 . For the long-exposure image ^IL and the first soft-exposure image I ^SE' of the input image matching convolutional neural sub-network, the depth feature ^FL of the long-exposure image ^IL and the first soft-exposure image I are firstly extracted through the residual network. The depth feature F SE ^' of ^SE' . Among them, ^FL is the first depth feature, and F ^SE' is the second depth feature. Moreover, the first layer of the residual network is a 5×5 convolution with a stride of 2, followed by 8 identical residual modules, each of which consists of 2 convolution kernels with a size of 3×3 convolution layer and a connection layer, the last layer is a convolution layer with a convolution kernel size of 3 × 3. After obtaining the depth features ^FL and F ^SE' , between each pixel point (j, i) in the first soft exposure image I ^SE' and the corresponding pixel point (j, i+k) in the long exposure image ^IL During this time, a four-dimensional feature is established, and the first four ^- dimensional feature VA is obtained. The first 4D features are then fed into a 3D regularization network. The 3D regularization network sequentially includes: multiple 3×3×3 convolutional layers, 4 residual modules, a 3×3×3 transposed convolutional layer, and an activation layer with an activation function of softmax. Among them, each residual module consists of a 3×3×3 transposed convolutional layer plus a residual connection. The output of the 3D regularization network is a 3D feature W ^A . When the three-dimensional feature WA is obtained, the value of each pixel ( ^j , i) of the aligned image I ^LA can be obtained by weighted average of the corresponding candidate pixels in I ^L .

Among them, the first four-dimensional feature V ^A is shown in the following formula:

代表第一四维特征V^A的元素点(j_V,i_V,k_V)的特征值，

代表像素点(j,i)的深度特征，

代表像素点(j,i+k)的深度特征。由此，第一四维特征V^A的元素点(j_V,i_V,k_V)的特征值是根据深度特征F^L中像素点(j,i+k)的深度特征

与F^SE’像素点(j,i)的深度特征

进行张量连接得到的。由于输入图像对的像素间存在一维相对运动，对每个像素点(j,i)的对应的候选像素点的范围被定义为(j,i)到(j,i+d-1)，其中超参数d为相对最大距离，例如，可以为图像宽度的20％，也就是说k可以为d-1。Concat stands for tensor concatenation,

represents the eigenvalues of the element points ( ^j _V , i _V , _kV ) of the first four-dimensional feature VA,

represents the depth feature of the pixel point (j, i),

Represents the depth feature of the pixel point (j, i+k). Therefore, the feature value of the element point ( ^j _V , i _V , _kV ) of the first four-dimensional feature VA is the depth feature of the pixel point (j, i+k) in the depth feature ^FL

Depth feature with F ^SE' pixel point (j, i)

obtained by concatenating tensors. Since there is a one-dimensional relative motion between the pixels of the input image pair, the range of the corresponding candidate pixels for each pixel (j, i) is defined as (j, i) to (j, i+d-1), The hyperparameter d is the relative maximum distance, for example, it can be 20% of the image width, that is, k can be d-1.

Then, the first four-dimensional feature VA is input into ^a three-dimensional regularization network to estimate a three-dimensional feature ^WA of space h×w×d. When the three ^- dimensional feature WA is obtained, for each element point in the three-dimensional feature, the pixel point and the pixel point in the corresponding position in the long-exposure image are weighted and averaged to obtain an aligned image, that is, the following formula:

为像素点(j,i)在对齐图像上的像素值，

为三维特征的元素点(j_V,i_V,k_V)的特征值，

为像素点(j,i+k)在长曝光图像上的像素值。in,

is the pixel value of the pixel point (j, i) on the aligned image,

is the eigenvalue of the element point (j _V , i _V , k _V ) of the three-dimensional feature,

is the pixel value of the pixel point (j, i+k) on the long exposure image.

In a specific application, a sample long-exposure image, a sample short-exposure image, and a corresponding real long-exposure image captured at the same time can be used as a pair of sample images, and a pair of sample images can be composed of the corresponding real long-exposure images. Two cameras with different exposure times and the same model were shot at the same time. Exemplarily, the number of pairs of sample images may be 1000 pairs. And, a first loss function L ₂ can be defined by using structural similarity (Structural SIMilarity, SSIM, an index to measure the similarity of two images) as a metric function, which is used for training the image matching convolutional neural sub-network. Among them, the first loss function L ₂ is defined as the following formula (1):

L ₂ =1-SSIM(I ^LA , G ^SE );

Among them, I ^LA is the alignment image, and G ^SE is the real long-exposure image corresponding to the sample short-exposure image. Based on this, the training of the image matching convolutional neural network may specifically include: using multiple sample long-exposure images and multiple sample short-exposure images to train the pre-built image matching convolutional neural network to obtain image matching. The sample-aligned image output by the convolutional neural network; the real long-exposure image corresponding to the sample-aligned image and the sample short-exposure image used to obtain the sample-aligned image, and the first loss function L ₂ to minimize the first loss function As the goal, adjust the network parameters of the image matching convolutional neural sub-network for training, and when the goal is achieved, complete the training of the image-matching convolutional neural sub-network.

In an optional implementation manner, the above-mentioned method of performing exposure adjustment and noise reduction processing on the short-exposure image based on the aligned image to obtain a noise-reduced image may specifically include the following steps B1 to B2:

Step B1, using the alignment image to perform exposure adjustment on the short-exposure image to obtain a second soft-exposure image;

In step B2, the alignment image and the second soft exposure image are input into the pre-trained three-dimensional guided noise reduction convolutional neural sub-network, so that the three-dimensional guided noise reduction convolutional neural sub-network degrades the second soft exposure image according to the aligned image. noise to obtain a denoised image; wherein, the three-dimensional guided denoising neural sub-network is a network trained by using multiple sample alignment images, multiple sample short exposure images and multiple real long exposure images.

In a specific application, step B1 may include: acquiring a histogram of the aligned image as the second target histogram; performing exposure adjustment on the short-exposure image, and acquiring the histogram of the adjusted short-exposure image as the second to-be-adjusted histogram Fig.; Adjust the second to-be-adjusted histogram so that the difference between the second target histogram and the second target histogram satisfies the preset difference condition, obtain the adjusted histogram, and use the image corresponding to the adjusted histogram as the second soft exposure image. In this optional embodiment, the alignment image is used to obtain the second soft exposure image, which is beneficial to improve the accuracy of the soft exposure. Wherein, the preset difference condition is similar to the preset difference condition in step A1 of the present invention, and the difference is that the input images are different.

In an optional implementation manner, the above-mentioned three-dimensional guided noise reduction convolutional neural sub-network performs noise reduction on the second soft exposure image according to the aligned image to obtain a noise reduction image, which may specifically include the following steps:

The alignment image and the second soft exposure image are mapped into a four-dimensional feature as a second four-dimensional feature; the second four-dimensional feature is used to reflect the difference between each pixel in the aligned image and the corresponding pixel in the second soft exposure image feature;

Using a preset three-dimensional filter weight, weighted average processing is performed on the second four-dimensional feature to obtain a denoised image.

的每一个像素点(j,i)都是由第二软曝光图像I^SE对应的像素的邻近像素进行加权平均得到的。其中，权重矩阵W^D由一个三维U-型网络学习得到。该三维U-型网络的输入为一个由第二软曝光图像I^SE与对齐图像I^LA通过映射构建的空间为h×w×s²×m的第二四维特征V^D。其中，h为第二四维特征V^D的高，w为第二四维特征V^D的宽，s²为第二四维特征V^D的长，m为第二四维特征V^D中三维切片的数量。该三维U-型网络的前14层为相同的卷积核大小为3×3×3的卷积层，接着为4个残差模块，每个残差模块由一次3×3×3的转置卷积层加一个残差连接组成，最后为一层3×3×3转置卷积层与一个基于softmax的归一化层。The resulting denoised image

Each pixel point (j, i) of is obtained by the weighted average of the adjacent pixels of the pixel corresponding to the second soft exposure image I ^SE . Among them, the weight matrix WD is learned by a three- ^dimensional U-shaped network. The input of the three-dimensional U-shaped network is a second four-dimensional feature V ^D of space h×w×s ² ×m constructed by mapping the second soft-exposure image I ^SE and the alignment image I ^LA . Among them, h is the height of the second four-dimensional feature V ^D , w is the width of the second four-dimensional feature V ^D , s ² is the length of the second four-dimensional feature V ^D , and m is the three dimensions of the second four-dimensional feature V ^D The number of slices. The first 14 layers of the 3D U-shaped network are the same convolutional layers with the same convolution kernel size of 3×3×3, followed by 4 residual modules, each of which is composed of a 3×3×3 transformation. It consists of a convolutional layer and a residual connection, and finally a 3×3×3 transposed convolutional layer and a softmax-based normalization layer.

的每一个像素点(j,i)都是由第二软曝光图像I^SE对应的像素的邻近像素进行加权平均得到的，即如下公式(2)：Among them, the denoised image

Each pixel point (j, i) of is obtained by the weighted average of the adjacent pixels of the pixel corresponding to the second soft exposure image I ^SE , that is, the following formula (2):

针对每一个像素点(j,i)的像素层面的处理流程：以该像素点为中心的矩形区域，也就是二维临近像素区Ω(j,i)需要被重塑为一维切片；对于Ω(j,i)中的每一个像素点(j’,i’)，在第二四维特征V^D与滤波权重W^D的特征值与权重值分别为

与

每个Ω(j,i)区域的像素点(j’,i’)与中心像素点(j,i)对应的四维特征是由I^SE与I^SA中像素点(j’,i’)的像素值、I^SA的中心点(j,i)的像素值以及像素点(j,i)与像素点(j’,i’)两点的几何距离D_gm((j',i')(j,i))＝(j-j')²+(i-i')²共同决定的，即下式：Among them, Ω(j,i) is a rectangular area with length and width s, and the center of the area is the pixel point (j,i). r is the window radius. The filtering weight WD is a three- ^dimensional weight with a space of h×w×s ² , and the weight can be obtained by training a three-dimensional joint denoising convolutional neural network. Illustratively, as shown in FIG. 3 . The processing flow at the image level may include: first, using the input image: the second soft exposure image I ^SE and the alignment image I ^LA to construct a second four-dimensional feature V ^D with a space of h×w×s ² ×m; After the two- and four-dimensional features V ^D , the present invention uses a three-dimensional U-shaped network to learn the filtering weights W ^D from them. The three-dimensional U-shaped network is similar to the traditional U-shaped network, except that the present invention uses three-dimensional convolution instead of two-dimensional convolution. After obtaining the filtering weight WD, the ^denoised image can be obtained by formula (2)

The processing flow at the pixel level for each pixel point (j, i): the rectangular area centered on the pixel point, that is, the two-dimensional adjacent pixel area Ω(j, i) needs to be reshaped into a one-dimensional slice; for For each pixel (j',i') in Ω(j,i), the eigenvalue and weight value of the second four-dimensional feature V ^D and the filtering weight ^WD are respectively

and

The four-dimensional feature corresponding to the pixel point (j',i') of each Ω(j,i) area and the central pixel point (j,i) is composed of the pixel point (j',i') in I ^SE and I ^SA The pixel value, the pixel value of the center point ( ^j , i) of the ISA, and the geometric distance D _gm ((j', i')( j,i))=(j-j') ² +(i-i') ² are jointly determined, that is, the following formula:

其中，Concat代表张量连接，V_{j,i,(j'-j+r)·s+(i'-i+r)}代表第二四维特征的元素点(j_V,i_V,(j'_V-j_V+r)·s+(i'_V-i_V+r))的特征值，

代表像素点(j’,i’)在第二软曝光图像上的像素值，

代表像素点(j,i)在对齐图像上的像素值，

代表像素点(j,i)的第二软曝光图像上的像素值。其中，(j’,i’)和(j,i)均为第二软曝光图像中的像素点。D_gm((j',i')(j,i))代表像素点(j,i)与像素点(j’,i’)两点间的几何距离。

Among them, Concat represents tensor connection, V _{j,i,(j'-j+r) s+(i'-i+r)} represents the element point of the second four-dimensional feature (j _V ,i _V ,(j' The eigenvalues of _V -j _V +r) s+(i' _V -i _V +r)),

represents the pixel value of the pixel point (j', i') on the second soft exposure image,

represents the pixel value of the pixel point (j, i) on the aligned image,

Pixel value on the second soft exposure image representing pixel point (j,i). Wherein, (j', i') and (j, i) are both pixel points in the second soft exposure image. D _gm ((j',i')(j,i)) represents the geometric distance between the pixel point (j,i) and the pixel point (j',i').

In the training of the three-dimensional joint denoising convolutional neural network, the present invention uses the structural similarity SSIM as the metric function, and defines the loss function as:

和实际的真实长曝光图像G^SE之间的曝光差异最小化，从而避免这种差异影响降噪质量的评价。全局调整曲线α是由降噪图像

和实际的真实长曝光图像G^SE估算得到，包含动态范围为0-255的256个点。其中，全局调整曲线α中每个点的计算方法如下公式(3)：where L ₁ is the second loss function, G ^SE is the actual real long-exposure image of the main image, and α is the global adjustment curve. The present invention performs global adjustment to make noise-reduced images

The exposure difference between GSE and the actual real long-exposure image G ^SE is minimized, so as to avoid this difference from affecting the evaluation of noise reduction quality. The global adjustment curve α is determined by denoising the image

and the actual real long-exposure image G ^SE is estimated, including 256 points with a dynamic range of 0-255. Among them, the calculation method of each point in the global adjustment curve α is as follows:

对应于公式(3)，χ'具体取值为降噪图像中像素点(j,i)的像素值

χ具体取值为实际的真实长曝光图像G^SE中像素点的像素值。in,

Corresponding to formula (3), the specific value of χ' is the pixel value of the pixel point (j, i) in the noise reduction image

The specific value of χ is the pixel value of the pixel point in the actual real long-exposure image G ^SE .

In an optional implementation manner, the above-mentioned fusion of the short exposure image and the noise reduction image to obtain a high dynamic range image may specifically include the following steps:

The short-exposure image and the noise-reduced image are input into the pre-trained image fusion convolutional neural sub-network, so that the image fusion convolutional neural sub-network fuses the short-exposure image and the noise-reduced image to obtain a high dynamic range image;

Among them, the image fusion convolutional neural sub-network is a network trained by using multiple sample short exposure images, multiple sample noise reduction images and real high dynamic range images corresponding to each sample short exposure image; any sample short exposure image corresponds to The real high dynamic range image is the image acquired by the camera used to acquire the high dynamic range image at the acquisition moment of the short exposure image of the sample.

In an optional embodiment, the above-mentioned image fusion convolutional neural sub-network fuses the short-exposure image and the noise reduction image to obtain a high dynamic range image, which may specifically include the following steps:

The short-exposure image and the noise-reduced image are weighted by the preset harmonic weight to obtain the high dynamic range image.

该图像融合卷积神经网络直接建立了一个残差网络去学习短曝光图像I^S与降噪图像

在每个像素点的融合权重，以得到高动态范围图像I^HDR，如下式：In a specific application, the input of the above-mentioned image fusion convolutional neural network may include a short exposure image ^IS and a denoised image

The image fusion convolutional neural network directly establishes a residual network to learn the short exposure image ^IS and the denoised image

The fusion weight at each pixel point to obtain the high dynamic range image I ^HDR , as follows:

其中，W^B为融合权重。

Among them, ^WB is the fusion weight.

In the training of the image fusion convolutional neural network, the present invention uses the structural similarity SSIM as the metric function, and defines the loss function as:

L ₃ =1-SSIM(I ^HDR , G ^HDR );

Among them, L ₃ is the third loss function, and G ^HDR is the actual real high dynamic range image.

Exemplarily, the present invention can be implemented using TensorFlow, a core open source library that can be used to develop and train machine learning models. All neural networks are optimized using RMSProp (Root Mean Square prop, a method for gradient computation in deep learning) with learning rate set to 0.001. The images in the dataset used for training were randomly divided into a training set containing 700 pairs of images and a test set containing 300 pairs of images. All program codes can be run on a server with Intel I7 processor and 4 NVIDIA 1080Ti GPUs. In the training phase, the present invention uses images from the dataset with a resolution of 416x576. In the testing stage, the present invention tests three resolution levels, namely, resolution level 1: 832×1184), resolution level 2: 416×576, and resolution level 3: 192×288.

将短曝光图像I^S和降噪图像

输入图像融合卷积神经网络即对于整个成像方法的图像融合卷积神经子网络，得到高动态范围成像结果：高动态范围图像I^HDR。For ease of understanding, the above embodiments of the present invention and optional embodiments are integrated to obtain a ghost-free high dynamic range imaging method for a dual-lens camera provided by another embodiment of the present invention. The following is in the form of an exemplary description. Describe in detail. Exemplarily, as shown in FIG. 4 . Using the long-exposure image ^IL to adjust the exposure of the short-exposure image IS to achieve soft exposure, a soft-exposure result is obtained: the first soft-exposure image I ^{SE '} . The long exposure image ^IL and the first soft exposure image I ^SE' are input to the image matching convolutional neural network, that is, the image matching convolutional neural sub-network relative to the entire imaging method, to obtain an alignment result: the aligned image I ^LA . Using the alignment image I ^LA to adjust the exposure of the short-exposure image ^IS to achieve soft exposure, the first soft-exposure result is obtained: the second soft-exposure image I ^SE . In step B2, the alignment image I ^LA and the second soft exposure image I ^SE are input into the three-dimensional joint noise reduction convolutional neural network, that is, the three-dimensional guided noise reduction convolutional neural sub-network for the entire imaging method, and the joint noise reduction result is obtained: noise reduction image

The short-exposure image ^IS and the noise-reduced image

The input image fusion convolutional neural network is the image fusion convolutional neural sub-network for the entire imaging method, and a high dynamic range imaging result is obtained: the high dynamic range image I ^HDR .

Corresponding to the above method embodiments, the embodiments of the present invention further provide a ghost-free high dynamic range imaging device for a dual-lens camera.

As shown in FIG. 5 , an embodiment of the present invention provides a structure of a dual-lens camera-oriented high dynamic range imaging device without ghosting. The device may include:

An input acquisition module 501 is used to acquire a long-exposure image and a short-exposure image collected by the dual-lens camera at the same moment;

The image processing module 502 is configured to input the long-exposure image and the short-exposure image into a main image enhancement model for ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image :

Based on the short exposure image, performing alignment adjustment on the long exposure image to obtain an alignment image;

Based on the alignment image, performing exposure adjustment and noise reduction processing on the short exposure image to obtain a noise reduction image;

The short exposure image and the noise reduction image are fused to obtain the high dynamic range image.

In the solution provided by the embodiment of the present invention, exposure adjustment and noise reduction processing are performed on the short-exposure image based on the aligned image to obtain a noise reduction image, which can ensure that the aligned image contributes to exposure adjustment and noise reduction processing. Therefore, the information contained in the alignment image can be added to the denoised image through exposure adjustment and noise reduction, while ensuring that the position of the pixels in the denoised image will not be affected by the alignment image, avoiding the generation of ghosting, and further reducing short exposures. The image and the denoised image are fused, and the resulting high dynamic range image has no ghosting. It can be seen that through this solution, a ghost-free high dynamic range image for the dual-lens camera can be obtained, and the imaging quality of the dual-lens camera can be improved.

In an optional implementation manner, the above-mentioned image processing module 502 is specifically used for:

Performing exposure adjustment on the short-exposure image by using the long-exposure image to obtain a first soft-exposure image;

The long-exposure image and the first soft-exposure image are input into a pre-trained image-matching convolutional neural sub-network, so that the image-matching convolutional neural sub-network adjusts the long-exposure image to match the The first soft-exposure image is aligned with the image to obtain an aligned image;

The image matching convolutional neural sub-network is a network trained by using multiple sample long-exposure images, multiple sample short-exposure images, and real long-exposure images corresponding to each sample short-exposure image; any short-exposure image The corresponding real long-exposure image is an image obtained by performing physical long-exposure on the short-exposure image by the dual-lens camera.

An embodiment of the present invention further provides an electronic device, as shown in FIG. 6 , including a processor 601 , a communication interface 602 , a memory 603 and a communication bus 604 , wherein the processor 601 , the communication interface 602 , and the memory 603 pass through the communication bus 604 complete communication with each other,

a memory 603 for storing computer programs;

When the processor 601 is used to execute the program stored in the memory 603, the following steps are implemented:

Obtain a long-exposure image and a short-exposure image captured by the dual-lens camera at the same moment;

The long-exposure image and the short-exposure image are input into a main image enhancement model for ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:

Based on the short exposure image, performing alignment adjustment on the long exposure image to obtain an alignment image;

Based on the alignment image, performing exposure adjustment and noise reduction processing on the short exposure image to obtain a noise reduction image;

The short exposure image and the noise reduction image are fused to obtain the high dynamic range image.

In the solution provided by the embodiment of the present invention, exposure adjustment and noise reduction processing are performed on the short-exposure image based on the aligned image to obtain a noise reduction image, which can ensure that the aligned image contributes to exposure adjustment and noise reduction processing. Therefore, the information contained in the alignment image can be added to the denoised image through exposure adjustment and noise reduction, while ensuring that the position of the pixels in the denoised image will not be affected by the alignment image, avoiding the generation of ghosting, and further reducing short exposures. The image and the denoised image are fused, and the resulting high dynamic range image has no ghosting. It can be seen that through this solution, a ghost-free high dynamic range image for the dual-lens camera can be obtained, and the imaging quality of the dual-lens camera can be improved.

The communication bus mentioned in the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above electronic device and other devices.

The memory may include random access memory (Random Access Memory, RAM), or may include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In another embodiment provided by the present invention, a computer-readable storage medium is also provided, and a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any one of the above-mentioned dual-camera orientations is implemented Steps of a camera's ghost-free high dynamic range imaging method.

In yet another embodiment provided by the present invention, there is also provided a computer program product including instructions, which, when run on a computer, enables the computer to execute any of the above-mentioned embodiments of the dual-lens camera-oriented ghost-free high dynamic Range imaging method.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, Solid State Disk (SSD)), among others.

It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for related parts.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (8)

1. A double-lens camera-oriented ghost-free high dynamic range imaging method is characterized by comprising the following steps:

acquiring a long exposure image and a short exposure image which are acquired by a double-lens camera at the same time;

inputting the long exposure image and the short exposure image into a main image enhancement model of ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:

based on the short exposure image, carrying out alignment adjustment on the long exposure image to obtain an aligned image, comprising:

carrying out exposure adjustment on the short-exposure image by using the long-exposure image to obtain a first soft-exposure image;

inputting the long exposure image and the first soft exposure image into an image matching convolution neural sub-network obtained through pre-training, so that the image matching convolution neural sub-network adjusts the long exposure image into an image aligned with the first soft exposure image, and an aligned image is obtained;

the image matching convolution neural sub-network is obtained by training a plurality of sample long-exposure images, a plurality of sample short-exposure images and a real long-exposure image corresponding to each sample short-exposure image; the real long exposure image corresponding to any short exposure image is an image obtained by carrying out physical long exposure on the short exposure image by the double-lens camera;

based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image;

and fusing the short exposure image and the noise reduction image to obtain the high dynamic range image.

2. The method of claim 1, wherein performing exposure adjustment on the short-exposure image by using the long-exposure image to obtain a first soft-exposure image comprises:

acquiring a histogram of the long exposure image as a target histogram;

carrying out exposure adjustment on the short-exposure image, and acquiring a histogram of the adjusted short-exposure image as a histogram to be adjusted;

and adjusting the histogram to be adjusted until a difference value between the histogram to be adjusted and the target histogram meets a preset difference condition to obtain an adjusted histogram, and taking an image corresponding to the adjusted histogram as the first soft exposure image.

3. The method of claim 1, wherein the image matching convolutional neural subnetwork adjusts the long exposure image to an image aligned with the first soft exposure image, resulting in an aligned image, comprising:

extracting a depth feature of the long exposure image as a first depth feature, and extracting a depth feature of the first soft exposure image as a second depth feature;

mapping the first depth feature and the second depth feature into a four-dimensional feature as a first four-dimensional feature; the first four-dimensional feature is a feature reflecting the first depth feature and the second depth feature;

adjusting the first four-dimensional feature into a three-dimensional feature using a three-dimensional adjustment network;

and aiming at each pixel point in the three-dimensional characteristic, carrying out weighted average on the pixel point and the pixel point at the corresponding position in the long exposure image to obtain the alignment image.

4. The method according to claim 1, wherein performing exposure adjustment and noise reduction processing on the short-exposure image based on the alignment image to obtain a noise-reduced image comprises:

carrying out exposure adjustment on the short-exposure image by using the alignment image to obtain a second soft-exposure image;

inputting the alignment image and the second soft exposure image into a pre-trained three-dimensional guide noise reduction convolution neural sub-network, so that the three-dimensional guide noise reduction convolution neural sub-network performs noise reduction on the second soft exposure image according to the alignment image to obtain a noise reduction image;

the three-dimensional guiding noise reduction neural sub-network is a network obtained by utilizing a plurality of sample alignment images, the plurality of sample short-exposure images and the plurality of real long-exposure images.

5. The method of claim 4, wherein the three-dimensional guided denoising convolutional neural subnetwork denoises the second soft exposure image according to the alignment image to obtain a denoised image, comprising:

mapping the alignment image and the second soft exposure image into a four-dimensional feature as a second four-dimensional feature; the second four-dimensional feature is used for reflecting the features of each pixel point in the aligned image and the corresponding pixel point in the second soft exposure image;

and carrying out weighted average processing on the second four-dimensional features by utilizing a preset three-dimensional filtering weight to obtain the noise reduction image.

6. The method of claim 1, wherein the fusing the short-exposure image and the noise-reduced image to obtain the high-dynamic-range image comprises:

inputting the short-exposure image and the noise-reduced image into an image fusion convolution neural sub-network obtained by pre-training, so that the image fusion convolution neural sub-network fuses the short-exposure image and the noise-reduced image to obtain the high dynamic range image;

the image fusion convolution neural subnetwork is a network obtained by training the plurality of sample short-exposure images, the plurality of sample noise reduction images and the real high dynamic range image corresponding to each sample short-exposure image; the real high dynamic range image corresponding to any sample short exposure image is an image acquired by a camera used for acquiring the high dynamic range image at the acquisition moment of the sample short exposure image.

7. The method of claim 6, wherein the image fusion convolutional neural subnetwork fuses the short-exposure image and the noise-reduced image to obtain the high dynamic range image, comprising:

and weighting the short-exposure image and the noise reduction image by using preset harmonic weight to obtain the high dynamic range image.

8. A ghost-free high dynamic range imaging apparatus for a dual lens camera, the apparatus comprising:

the input acquisition module is used for acquiring a long exposure image and a short exposure image which are acquired by the double-lens camera at the same time;

an image processing module, configured to input the long-exposure image and the short-exposure image into a main image enhancement model of ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:

based on the short exposure image, carrying out alignment adjustment on the long exposure image to obtain an aligned image, comprising:

carrying out exposure adjustment on the short-exposure image by using the long-exposure image to obtain a first soft-exposure image;

inputting the long exposure image and the first soft exposure image into an image matching convolution neural sub-network obtained through pre-training, so that the image matching convolution neural sub-network adjusts the long exposure image into an image aligned with the first soft exposure image, and an aligned image is obtained;

the image matching convolution neural sub-network is obtained by training a plurality of sample long-exposure images, a plurality of sample short-exposure images and a real long-exposure image corresponding to each sample short-exposure image; the real long exposure image corresponding to any short exposure image is an image obtained by carrying out physical long exposure on the short exposure image by the double-lens camera;

based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image;

and fusing the short exposure image and the noise reduction image to obtain the high dynamic range image.

Images