CN112689099B - Double-image-free high-dynamic-range imaging method and device for double-lens camera - Google Patents
Double-image-free high-dynamic-range imaging method and device for double-lens camera Download PDFInfo
- Publication number
- CN112689099B CN112689099B CN202011460648.9A CN202011460648A CN112689099B CN 112689099 B CN112689099 B CN 112689099B CN 202011460648 A CN202011460648 A CN 202011460648A CN 112689099 B CN112689099 B CN 112689099B
- Authority
- CN
- China
- Prior art keywords
- image
- exposure
- short
- exposure image
- long
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 40
- 230000009467 reduction Effects 0.000 claims abstract description 63
- 238000012545 processing Methods 0.000 claims abstract description 36
- 230000001537 neural effect Effects 0.000 claims description 42
- 238000012549 training Methods 0.000 claims description 24
- 238000000034 method Methods 0.000 claims description 22
- 230000004927 fusion Effects 0.000 claims description 14
- 238000013507 mapping Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 4
- 230000009977 dual effect Effects 0.000 claims description 2
- 230000006870 function Effects 0.000 description 12
- 238000004891 communication Methods 0.000 description 9
- 238000013528 artificial neural network Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 3
- 230000009471 action Effects 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 239000004576 sand Substances 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Landscapes
- Image Processing (AREA)
Abstract
The embodiment of the invention provides a double-lens camera-oriented imaging method and device with no ghost image and high dynamic range, which are used for acquiring a long exposure image and a short exposure image which are acquired by a double-lens camera at the same time; inputting the long exposure image and the short exposure image into a main image enhancement model of ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image: based on the short exposure image, carrying out alignment adjustment on the long exposure image to obtain an alignment image; based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image; and fusing the short exposure image and the noise reduction image to obtain the high dynamic range image. By the scheme, the double-lens camera-oriented high-dynamic-range image without double images can be obtained, and the imaging quality of the double-lens camera is improved.
Description
Technical Field
The invention relates to the technical field of digital image processing, in particular to a double-lens camera-oriented imaging method and device with no ghost image and high dynamic range.
Background
As the technology is developed, a dual lens camera is increasingly applied to a mobile terminal. Since the mobile terminal is generally an electronic device of a general consumer level, the dual-lens camera used for the mobile terminal is limited by hardware, and can only capture a low dynamic range image with a small exposure range, which is far from the quality of a high dynamic range image. In contrast, in order to improve the imaging quality of the twin-lens camera, a high dynamic range image may be synthesized using two low dynamic range images with different exposures captured at the same time.
In the related art, two low dynamic range images can be aligned to obtain an aligned image, and the aligned image and the pixel values of the short exposure image with shorter exposure time in the two low dynamic range images are fused into one image to obtain a high dynamic range image. Wherein the alignment adjustment may include: and searching the one-to-one corresponding relation of the pixels in the two images, and reconstructing the long exposure image with longer exposure time in the two low dynamic range images by using the short exposure image according to the corresponding relation to obtain the aligned image.
However, because the exposure times of the two low dynamic range images are different, there are often pixels that do not correspond exactly in the aligned images, and there are regions where the aligned images are misaligned, resulting in ghosting of the high dynamic range images.
Disclosure of Invention
The embodiment of the invention aims to provide a double-lens camera-oriented imaging method and device with no ghost image and high dynamic range, which are used for solving the problem that the double image exists in the high dynamic range image of the double-lens camera. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a double-lens camera-oriented ghost-free high dynamic range imaging method, where the method includes:
acquiring a long exposure image and a short exposure image which are acquired by a double-lens camera at the same time;
inputting the long exposure image and the short exposure image into a main image enhancement model of ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:
based on the short exposure image, carrying out alignment adjustment on the long exposure image to obtain an alignment image;
based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image;
and fusing the short exposure image and the noise reduction image to obtain the high dynamic range image.
In a second aspect, an embodiment of the present invention provides a double-lens camera-oriented ghost-free high dynamic range imaging apparatus, where the apparatus includes:
the input acquisition module is used for acquiring a long exposure image and a short exposure image which are acquired by the double-lens camera at the same time;
an image processing module, configured to input the long-exposure image and the short-exposure image into a main image enhancement model of ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:
based on the short exposure image, carrying out alignment adjustment on the long exposure image to obtain an alignment image;
based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image;
and fusing the short exposure image and the noise reduction image to obtain the high dynamic range image.
The embodiment of the invention has the following beneficial effects:
in the scheme provided by the embodiment of the invention, a long exposure image and a short exposure image which are acquired by a double-lens camera at the same time are input into a main image enhancement model of ghost-free high dynamic range imaging, and the main image enhancement model can align and adjust the long exposure image based on the short exposure image to obtain an aligned image; based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image; and fusing the short exposure image and the noise reduction image to obtain a high dynamic range image. The short-exposure image is subjected to exposure adjustment and noise reduction processing based on the alignment image to obtain a noise reduction image, and the alignment image can be ensured to contribute to the exposure adjustment and the noise reduction processing. Therefore, when information contained in the alignment image is added into the noise-reduced image through exposure adjustment and noise reduction processing, the position of a pixel in the noise-reduced image is not influenced by the alignment image, the generation of a ghost image is avoided, the short-exposure image and the noise-reduced image are fused, and the obtained high-dynamic-range image does not have the ghost image. Therefore, through the scheme, the double-lens camera-oriented high-dynamic-range image without double images can be obtained, and the imaging quality of the double-lens camera is improved.
Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
Fig. 1 is a schematic flowchart of a double-lens camera-oriented ghost-free high dynamic range imaging method according to an embodiment of the present invention;
fig. 2 is a diagram illustrating an example of an acquisition flow of aligned images in a double-lens camera-oriented ghost-free high dynamic range imaging method according to another embodiment of the present invention;
fig. 3 is an exemplary diagram of a process of obtaining a noise-reduced image in a double-lens camera-oriented ghost-free high dynamic range imaging method according to another embodiment of the present invention;
fig. 4 is a diagram illustrating an exemplary ghost-free high dynamic range imaging method for a dual-lens camera according to another embodiment of the present invention;
fig. 5 is a schematic structural diagram of a double-lens camera-oriented ghost-free high dynamic range imaging apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the invention
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The double-lens camera-oriented imaging method with no ghost image and high dynamic range can be applied to double-lens cameras or electronic equipment for acquiring images by adopting double lenses, such as double-lens mobile terminals, wearable equipment and the like.
As shown in fig. 1, a flow of a ghost-free high dynamic range imaging method for a dual-lens camera according to an embodiment of the present invention may include the following steps:
s101, acquiring a long exposure image and a short exposure image which are acquired by the double-lens camera at the same time.
In a particular application, the two lenses of a dual-lens camera may be the same model lens. The one long-exposure image and the one short-exposure image are images of the same subject. Wherein the exposure time of the long exposure image is longer than the exposure time of the short exposure image.
And S102, inputting the long-exposure image and the short-exposure image into a main image enhancement model without ghost image and high dynamic range imaging, so that the main image enhancement model executes the operations of the steps S1021 to S1023 to obtain a high dynamic range image.
And S1021, carrying out alignment adjustment on the long exposure image based on the short exposure image to obtain an aligned image.
In a specific application, the alignment adjustment is performed on the long-exposure image based on the short-exposure image, and a specific manner of obtaining the aligned image may be various. Illustratively, the pixels in the short-exposure image and the pixels in the long-exposure image can be aligned one by one, and the information of the pixels in the short-exposure image is used for adjusting the information of the pixels in the long-exposure image to obtain an aligned image. Or, for example, the short exposure image may be subjected to exposure adjustment by using the long exposure image, resulting in a first soft exposure image; and inputting the long exposure image and the first soft exposure image into an image matching convolution neural sub-network obtained by pre-training, so that the image matching convolution neural sub-network adjusts the long exposure image into an image aligned with the first soft exposure image, and an aligned image is obtained. For ease of understanding and reasonable layout, the second exemplary scenario is described in detail below in the form of an alternative embodiment.
And S1022, carrying out exposure adjustment and noise reduction processing on the short-exposure image based on the alignment image to obtain a noise-reduced image.
When performing exposure adjustment and noise reduction processing on the short-exposure image based on the alignment image, information for adjusting and noise reduction of the pixels of the short-exposure image itself may be specifically acquired with reference to the pixels of the alignment image, so that exposure adjustment and noise reduction processing is performed on the short-exposure image according to the acquired information. For easy understanding and reasonable layout, a specific manner of obtaining a noise-reduced image by performing exposure adjustment and noise reduction processing on the short-exposure image based on the aligned image is described in the following with an alternative embodiment.
And S1023, fusing the short-exposure image and the noise-reduced image to obtain a high-dynamic-range image.
The noise-reduced image is obtained based on the alignment image, and therefore, contains information of the long-exposure image. Based on the above, the short-exposure image and the noise-reduced image are fused, so that the fused image can be ensured to contain information in the short-exposure image and the long-exposure image, and is a high dynamic range image with an exposure range higher than that of the short-exposure image and the long-exposure image. For easy understanding and reasonable layout, the process of fusing the short-exposure image and the noise-reduced image to obtain the high-dynamic-range image is specifically described in the following in the form of an optional embodiment.
In the scheme provided by the embodiment of the invention, exposure adjustment and noise reduction processing are carried out on the short-exposure image based on the alignment image to obtain the noise reduction image, so that the alignment image can be ensured to contribute to the exposure adjustment and the noise reduction processing. Therefore, when information contained in the alignment image is added into the noise-reduced image through exposure adjustment and noise reduction processing, the position of a pixel in the noise-reduced image is not influenced by the alignment image, the generation of a ghost image is avoided, the short-exposure image and the noise-reduced image are fused, and the obtained high-dynamic-range image does not have the ghost image. Therefore, through the scheme, the double-lens camera-oriented high-dynamic-range image without double images can be obtained, and the imaging quality of the double-lens camera is improved.
In an optional implementation manner, the performing alignment adjustment on the long-exposure image based on the short-exposure image to obtain an aligned image may specifically include the following steps a1 to a 2:
a1, carrying out exposure adjustment on the short-exposure image by using the long-exposure image to obtain a first soft-exposure image;
a2, inputting the long exposure image and the first soft exposure image into a pre-trained image matching convolution neural sub-network, so that the image matching convolution neural sub-network adjusts the long exposure image into an image aligned with the first soft exposure image, and an aligned image is obtained;
the image matching convolution neural sub-network is obtained by training a plurality of sample long-exposure images, a plurality of sample short-exposure images and a real long-exposure image corresponding to each sample short-exposure image; and the real long-exposure image corresponding to any short-exposure image is an image obtained by carrying out physical long-exposure on the short-exposure image by the double-lens camera.
The optional embodiment performs exposure adjustment on the short-exposure image by using the long-exposure image, which is equivalent to simulating physical exposure of a camera, thereby realizing soft exposure of the short-exposure image. And, the alignment image is acquired using the first soft exposure image, and the exposure of the first soft exposure image acquired using the long exposure image is more similar to that of the long exposure image than that of the alignment image acquired directly using the short exposure, so that the complexity of alignment adjustment can be reduced, and the alignment adjustment can be implemented more easily.
In an alternative embodiment, the step a 1: the method for adjusting exposure of the short exposure image by using the long exposure image to obtain the first soft exposure image specifically includes the following steps:
acquiring a histogram of the long exposure image as a target histogram;
carrying out exposure adjustment on the short-exposure image, and acquiring a histogram of the adjusted short-exposure image as a histogram to be adjusted;
and adjusting the histogram to be adjusted until the difference value between the histogram to be adjusted and the target histogram meets a preset difference condition to obtain an adjusted histogram, and taking an image corresponding to the adjusted histogram as a first soft exposure image.
The optional embodiment utilizes the adjustment of the histogram, that is, the equalization of the histogram to realize the soft exposure, so that the requirement on the matching alignment between the long-exposure image and the short-exposure image can be reduced, and the complexity of obtaining the aligned image is further reduced.
In an alternative embodiment, in the step a 2: the image matching convolutional neural subnetwork adjusts the long exposure image to an image aligned with the first soft exposure image to obtain an aligned image, and specifically may include the following steps:
extracting the depth feature of the long exposure image as a first depth feature, and extracting the depth feature of the first soft exposure image as a second depth feature;
mapping the first depth feature and the second depth feature into a four-dimensional feature as a first four-dimensional feature; the first four-dimensional feature is a feature reflecting the first depth feature and the second depth feature;
adjusting the first four-dimensional feature into a three-dimensional feature by using a three-dimensional adjusting network;
and aiming at each pixel point in the three-dimensional characteristic, carrying out weighted average on the pixel point and the pixel point at the corresponding position in the long exposure image to obtain an aligned image.
Illustratively, as shown in fig. 2. Matching long-exposure images of convolutional neural sub-networks to input images ILAnd a first soft-exposure image ISE’Firstly, respectively extracting long exposure images I through a residual error networkLDepth feature of (F)LAnd a first soft-exposure image ISE’Depth feature of (F)SE’. Wherein, FLI.e. a first depth feature, FSE’I.e. the second depth feature. And the residual network has a first level of convolution with 5 x5, step size of 2, followed by 8 identical residual blocks, each consisting of 2 convolution layers with convolution kernel size of 3 x 3One connected layer, the last layer being 1 convolutional layer with a convolutional kernel size of 3 x 3. Obtaining a depth feature FLAnd FSE’Then, in the first soft exposure image ISE’Each pixel point (j, I) of (a), and a long-exposure image ILA four-dimensional feature is established between corresponding pixel points (j, i + k) in the image to obtain a first four-dimensional feature VA. The first four-dimensional feature is then input into a three-dimensional regularization network. The three-dimensional regularization network sequentially comprises: a plurality of 3 x 3 convolutional layers, 4 residual blocks, a layer of 3 x 3 transposed convolutional layers, and an active layer with an activation function of softmax. Wherein each residual block is composed of a 3 x 3 transposed convolution layer plus a residual join. The output result of the three-dimensional regularization network is a three-dimensional feature WA. When the three-dimensional feature W is obtainedAThen, image I is alignedLAThe value of each pixel point (j, I) can be represented by ILAnd obtaining the weighted average of the corresponding candidate pixel points.
Wherein the first four-dimensional feature VAAs shown in the following formula:
concat represents the connection of the tensor,
representing a first four-dimensional feature VAElement point (j) ofV,iV,kV) Is determined by the characteristic value of (a),
representing the depth characteristics of the pixel points (j, i),
representing the depth characteristic of the pixel point (j, i + k). Thus, the first four-dimensional feature VAElement point (j) ofV,iV,kV) Is based on the depth feature FLDepth feature of middle pixel point (j, i + k)
And FSE’Depth characterization of pixels (j, i)
And carrying out tensor connection. Since there is a one-dimensional relative motion between the pixels of the input image pair, the range of the corresponding candidate pixel for each pixel (j, i) is defined as (j, i) to (j, i + d-1), where the over-parameter d is the relative maximum distance, which may be, for example, 20% of the image width, that is, k may be d-1.
Then, the first four-dimensional feature V is combinedAInputting a three-dimensional regularization network to estimate three-dimensional features W with a space of hxwxdA. When the three-dimensional feature W is obtainedAThen, for each pixel point in the three-dimensional feature, performing weighted average on the pixel point and a pixel point at a corresponding position in the long exposure image to obtain an aligned image, namely the following formula:
wherein,
for the pixel values of the pixel points (j, i) on the alignment image,
element points (j) being three-dimensional featuresV,iV,kV) Is determined by the characteristic value of (a),
is the pixel value of the pixel point (j, i + k) on the long exposure image.
In specific application, a sample long-exposure image, a sample short-exposure image and a corresponding real long-exposure image which are shot at the same time can be used as a pair of sample images, and the two sample images can be respectively corresponding to the real long-exposure images and have different exposure timeThe cameras with the same model and the same model are shot at the same time. For example, the number of pairs of sample images may be 1000 pairs. Also, the first loss function L may be defined using Structural SIMilarity (SSIM), an index for measuring the SIMilarity between two images, as a metric function2For training of image-matching convolutional neural sub-networks. Wherein the first loss function L2Defined as the following formula (1):
L2＝1-SSIM(ILA,GSE)；
wherein, ILATo align the images, GSEAnd the real long exposure image corresponding to the sample short exposure image. Based on this, the training of the image matching convolutional neural sub-network may specifically include: training a pre-constructed image matching convolution neural sub-network by utilizing a plurality of sample long exposure images and a plurality of sample short exposure images to obtain a sample alignment image output by the image matching convolution neural sub-network; using a sample alignment image and a true long exposure image corresponding to a sample short exposure image used to obtain the sample alignment image, and a first loss function L2And adjusting network parameters of the image matching convolution neural sub-network for training by taking the minimized first loss function as a target, and finishing the training of the image matching convolution neural sub-network when the target is achieved.
In an optional implementation manner, the performing exposure adjustment and noise reduction processing on the short-exposure image based on the alignment image to obtain a noise-reduced image may specifically include the following steps B1 to B2:
step B1, carrying out exposure adjustment on the short-exposure image by using the alignment image to obtain a second soft-exposure image;
step B2, inputting the aligned image and the second soft exposure image into a pre-trained three-dimensional guide noise reduction convolution neural sub-network, so that the three-dimensional guide noise reduction convolution neural sub-network performs noise reduction on the second soft exposure image according to the aligned image to obtain a noise reduction image; the three-dimensional guiding noise reduction neural subnetwork is a network obtained by training a plurality of sample alignment images, a plurality of sample short-exposure images and a plurality of real long-exposure images.
In a specific application, step B1 may include: acquiring a histogram of the aligned image as a second target histogram; carrying out exposure adjustment on the short-exposure image, and acquiring a histogram of the adjusted short-exposure image as a second histogram to be adjusted; and adjusting the second histogram to be adjusted until the difference value between the second histogram to be adjusted and the second target histogram meets the preset difference condition to obtain an adjusted histogram, and taking an image corresponding to the adjusted histogram as a second soft exposure image. In this optional embodiment, the second soft exposure image is acquired by using the alignment image, which is beneficial to improving the accuracy of soft exposure. The preset difference condition is similar to the preset difference condition in step a1 of the present invention, and the difference is that the input image is different.
In an optional implementation manner, the three-dimensional guiding denoising convolutional neural subnetwork denoises the second soft exposure image according to the alignment image to obtain a denoised image, which may specifically include the following steps:
mapping the aligned image and the second soft exposure image into a four-dimensional feature as a second four-dimensional feature; the second four-dimensional feature is used for reflecting the features of each pixel point in the aligned image and the corresponding pixel point in the second soft exposure image;
and carrying out weighted average processing on the second four-dimensional features by using the preset three-dimensional filtering weight to obtain a noise reduction image.
Resulting noise reduced image
Each pixel point (j, I) of the image is formed by the second soft exposure image ISEAnd the adjacent pixels of the corresponding pixels are obtained by weighted average. Wherein the weight matrix WDLearned from a three-dimensional U-shaped network. The input of the three-dimensional U-shaped network is a second soft exposure image ISEAnd aligning with image ILAThe space constructed by mapping is h × w × s2Second four-dimensional feature V of x mD. Wherein h is a second four-dimensional feature VDW is the second four-dimensional feature VDWidth of(s)2As a second four-dimensional feature VDM is the second four-dimensional feature VDThe number of medium three-dimensional slices. The first 14 layers of the three-dimensional U-shaped network are convolutional layers with the same convolutional kernel size of 3 x 3, then 4 residual error modules are arranged, each residual error module is formed by connecting a 3 x 3 transposed convolutional layer and a residual error once, and finally a layer of 3 x 3 transposed convolutional layer and a normalization layer based on softmax are arranged.
Wherein the image is denoised
Each pixel point (j, I) of the image is formed by the second soft exposure image ISEThe weighted average of the neighboring pixels of the corresponding pixel is obtained, which is expressed by the following formula (2):
wherein, Ω (j, i) is a rectangular region with length and width s, and the center of the region is a pixel point (j, i). r is the window radius. Filter weight WDIs a space of h x w x s2The weights can be obtained by three-dimensional combined noise reduction convolution neural network training. Illustratively, as shown in FIG. 3. The processing flow at the image level may include: first using the input image: second soft-exposure image ISEAnd aligning with image ILATo construct a space of h × w × s2Second four-dimensional feature V of x mD(ii) a Obtain a second four-dimensional feature VDThe invention then learns the filter weights W from it using a three-dimensional U-shaped networkD. A three-dimensional U-type network is similar to a conventional U-type network except that the present invention uses three-dimensional convolution instead of two-dimensional convolution. After obtaining the filtering weight WDThen, the noise-reduced image can be obtained by the formula (2)
The processing flow aiming at the pixel layer of each pixel point (j, i) is as follows: the rectangular region centered on the pixel point, i.e., the two-dimensional adjacent pixel region Ω (j, i), needs to be definedIs reshaped into a one-dimensional slice; for each pixel point (j ', i') in Ω (j, i), in the second four-dimensional feature VDAnd a filtering weight WDThe characteristic value and the weight value are respectively
And
the four-dimensional characteristic corresponding to the pixel point (j ', I') of each omega (j, I) area and the central pixel point (j, I) is represented by ISEAnd ISAPixel value, I, of the middle pixel point (j', ISAAnd the geometric distance D between the pixel point (j, i) and the two points of the pixel point (j ', i')gm((j',i')(j,i))＝(j-j')2+(i-i')2Jointly determined, i.e. of the formula:
In the training of the three-dimensional combined noise reduction convolutional neural network, the invention uses the structure similarity SSIM as a measurement function, and defines the loss function as:
wherein L is1Is a second loss function, GSEIs the actual true long exposure image of the main image and alpha is the global adjustment curve. The invention carries out global adjustment to reduce the noise of the image
And the actual real long exposure image GSEThe exposure difference between them is minimized to avoid that such difference affects the evaluation of the noise reduction quality. The global adjustment curve alpha is formed by the noise-reduced image
And the actual real long exposure image GSEThe estimate was found to contain 256 points with a dynamic range of 0-255. The calculation method of each point in the global adjustment curve α is as follows formula (3):
wherein,
corresponding to the formula (3), the specific value of χ' is the pixel value of the pixel point (j, i) in the noise-reduced image
Real long exposure image G with actual x specific valueSEThe pixel value of the middle pixel point.
In an optional implementation manner, the fusing the short-exposure image and the noise-reduced image to obtain the high-dynamic-range image specifically includes the following steps:
inputting the short-exposure image and the noise-reduced image into an image fusion convolution neural sub-network obtained by pre-training so that the image fusion convolution neural sub-network fuses the short-exposure image and the noise-reduced image to obtain a high dynamic range image;
the image fusion convolution neural subnetwork is a network obtained by training a plurality of sample short-exposure images, a plurality of sample noise reduction images and a real high dynamic range image corresponding to each sample short-exposure image; the real high dynamic range image corresponding to any sample short exposure image is an image acquired by a camera used for acquiring the high dynamic range image at the acquisition moment of the sample short exposure image.
In an optional implementation manner, the image fusion convolution neural sub-network fuses the short-exposure image and the noise-reduced image to obtain the high-dynamic-range image, which may specifically include the following steps:
and weighting the short-exposure image and the noise reduction image by using preset harmonic weight to obtain a high dynamic range image.
In a specific application, the input of the image fusion convolution neural network can comprise a short-exposure image ISAnd noise reduction image
The image fusion convolution neural network directly establishes a residual error network to learn the short-exposure image ISAnd noise reduction image
Fusing weight at each pixel point to obtain high dynamic range image IHDRThe following formula:
In the training of the image fusion convolutional neural network, the invention uses the structural similarity SSIM as a measurement function, and defines the loss function as:
L3＝1-SSIM(IHDR,GHDR)；
wherein L is3As a third loss function, GHDRIs a realistic real high dynamic range image.
Illustratively, the invention can be implemented using TensorFlow (a core open source library that can be used to develop and train machine learning models). All neural networks were optimized using RMSProp (Root Mean Square prop, a method for deep learning gradient calculation) with a learning rate set to 0.001. The images in the data set used for training were randomly divided into a training set containing 700 pairs of images and a test set containing 300 pairs of images. All program code may run on a server with an Intel I7 processor and a 4-block NVIDIA 1080Ti GPU. In the training phase, the present invention uses images from the dataset with a resolution of 416x 576. In the testing phase, the invention tests three resolution levels, namely resolution level 1: 832x1184), resolution level 2: 416x576 and resolution level 3: 192x 288.
For convenience of understanding, the above embodiments and alternative embodiments of the present invention are integrated to obtain a ghost-free high dynamic range imaging method for a dual-lens camera according to another embodiment of the present invention, which is specifically described below in an exemplary form. Illustratively, as shown in fig. 4. Using long exposure images ILFor short exposure image ISCarrying out exposure adjustment to realize soft exposure, and obtaining a soft exposure result: first soft-exposure image ISE’. Will be long exposure image ILAnd a first soft-exposure image ISE’The input image matches the convolutional neural network, i.e. the convolutional neural sub-network is matched with the image corresponding to the whole imaging method, and the alignment result is obtained: aligning images ILA. Using aligned images ILAFor short exposure image ISCarrying out exposure adjustment to realize soft exposure, and obtaining a second soft exposure result: second soft-exposure image ISE. Step B2, aligning image ILAAnd a second soft-exposure image ISEInputting the three-dimensional combined noise reduction convolution neural network, namely, for the three-dimensional guiding noise reduction convolution neural sub-network of the whole imaging method, obtaining a combined noise reduction result: noise reduced image
Short exposure image ISAnd noise-reduced image
Inputting an image fusion convolution neural network, namely, for the image fusion convolution neural sub-network of the whole imaging method, obtaining a high dynamic range imaging result: high dynamic range image IHDR。
Corresponding to the method embodiment, the embodiment of the invention also provides a double-lens camera-oriented imaging device with no ghost image and high dynamic range.
As shown in fig. 5, an embodiment of the present invention provides a structure of a double-lens camera-oriented ghost-free high dynamic range imaging apparatus, where the apparatus may include:
an input acquisition module 501, configured to acquire a long exposure image and a short exposure image acquired by a dual-lens camera at the same time;
an image processing module 502, configured to input the long-exposure image and the short-exposure image into a main image enhancement model of ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:
based on the short exposure image, carrying out alignment adjustment on the long exposure image to obtain an alignment image;
based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image;
and fusing the short exposure image and the noise reduction image to obtain the high dynamic range image.
In the scheme provided by the embodiment of the invention, exposure adjustment and noise reduction processing are carried out on the short-exposure image based on the alignment image to obtain the noise reduction image, so that the alignment image can be ensured to contribute to the exposure adjustment and the noise reduction processing. Therefore, when information contained in the alignment image is added into the noise-reduced image through exposure adjustment and noise reduction processing, the position of a pixel in the noise-reduced image is not influenced by the alignment image, the generation of a ghost image is avoided, the short-exposure image and the noise-reduced image are fused, and the obtained high-dynamic-range image does not have the ghost image. Therefore, through the scheme, the double-lens camera-oriented high-dynamic-range image without double images can be obtained, and the imaging quality of the double-lens camera is improved.
In an optional implementation manner, the image processing module 502 is specifically configured to:
carrying out exposure adjustment on the short-exposure image by using the long-exposure image to obtain a first soft-exposure image;
inputting the long exposure image and the first soft exposure image into an image matching convolution neural sub-network obtained through pre-training, so that the image matching convolution neural sub-network adjusts the long exposure image into an image aligned with the first soft exposure image, and an aligned image is obtained;
the image matching convolution neural sub-network is obtained by training a plurality of sample long-exposure images, a plurality of sample short-exposure images and a real long-exposure image corresponding to each sample short-exposure image; and the real long exposure image corresponding to any short exposure image is an image obtained by carrying out physical long exposure on the short exposure image by the double-lens camera.
An embodiment of the present invention further provides an electronic device, as shown in fig. 6, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604, where the processor 601, the communication interface 602, and the memory 603 complete mutual communication through the communication bus 604,
a memory 603 for storing a computer program;
the processor 601 is configured to implement the following steps when executing the program stored in the memory 603:
acquiring a long exposure image and a short exposure image which are acquired by a double-lens camera at the same time;
inputting the long exposure image and the short exposure image into a main image enhancement model of ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:
based on the short exposure image, carrying out alignment adjustment on the long exposure image to obtain an alignment image;
based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image;
and fusing the short exposure image and the noise reduction image to obtain the high dynamic range image.
In the scheme provided by the embodiment of the invention, exposure adjustment and noise reduction processing are carried out on the short-exposure image based on the alignment image to obtain the noise reduction image, so that the alignment image can be ensured to contribute to the exposure adjustment and the noise reduction processing. Therefore, when information contained in the alignment image is added into the noise-reduced image through exposure adjustment and noise reduction processing, the position of a pixel in the noise-reduced image is not influenced by the alignment image, the generation of a ghost image is avoided, the short-exposure image and the noise-reduced image are fused, and the obtained high-dynamic-range image does not have the ghost image. Therefore, through the scheme, the double-lens camera-oriented high-dynamic-range image without double images can be obtained, and the imaging quality of the double-lens camera is improved.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any of the above-mentioned double-lens camera-oriented ghost-free high dynamic range imaging methods.
In yet another embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the above-described embodiments of the double-lens camera-oriented ghost-free high dynamic range imaging method.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (8)
1. A double-lens camera-oriented ghost-free high dynamic range imaging method is characterized by comprising the following steps:
acquiring a long exposure image and a short exposure image which are acquired by a double-lens camera at the same time;
inputting the long exposure image and the short exposure image into a main image enhancement model of ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:
based on the short exposure image, carrying out alignment adjustment on the long exposure image to obtain an aligned image, comprising:
carrying out exposure adjustment on the short-exposure image by using the long-exposure image to obtain a first soft-exposure image;
inputting the long exposure image and the first soft exposure image into an image matching convolution neural sub-network obtained through pre-training, so that the image matching convolution neural sub-network adjusts the long exposure image into an image aligned with the first soft exposure image, and an aligned image is obtained;
the image matching convolution neural sub-network is obtained by training a plurality of sample long-exposure images, a plurality of sample short-exposure images and a real long-exposure image corresponding to each sample short-exposure image; the real long exposure image corresponding to any short exposure image is an image obtained by carrying out physical long exposure on the short exposure image by the double-lens camera;
based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image;
and fusing the short exposure image and the noise reduction image to obtain the high dynamic range image.
2. The method of claim 1, wherein performing exposure adjustment on the short-exposure image by using the long-exposure image to obtain a first soft-exposure image comprises:
acquiring a histogram of the long exposure image as a target histogram;
carrying out exposure adjustment on the short-exposure image, and acquiring a histogram of the adjusted short-exposure image as a histogram to be adjusted;
and adjusting the histogram to be adjusted until a difference value between the histogram to be adjusted and the target histogram meets a preset difference condition to obtain an adjusted histogram, and taking an image corresponding to the adjusted histogram as the first soft exposure image.
3. The method of claim 1, wherein the image matching convolutional neural subnetwork adjusts the long exposure image to an image aligned with the first soft exposure image, resulting in an aligned image, comprising:
extracting a depth feature of the long exposure image as a first depth feature, and extracting a depth feature of the first soft exposure image as a second depth feature;
mapping the first depth feature and the second depth feature into a four-dimensional feature as a first four-dimensional feature; the first four-dimensional feature is a feature reflecting the first depth feature and the second depth feature;
adjusting the first four-dimensional feature into a three-dimensional feature using a three-dimensional adjustment network;
and aiming at each pixel point in the three-dimensional characteristic, carrying out weighted average on the pixel point and the pixel point at the corresponding position in the long exposure image to obtain the alignment image.
4. The method according to claim 1, wherein performing exposure adjustment and noise reduction processing on the short-exposure image based on the alignment image to obtain a noise-reduced image comprises:
carrying out exposure adjustment on the short-exposure image by using the alignment image to obtain a second soft-exposure image;
inputting the alignment image and the second soft exposure image into a pre-trained three-dimensional guide noise reduction convolution neural sub-network, so that the three-dimensional guide noise reduction convolution neural sub-network performs noise reduction on the second soft exposure image according to the alignment image to obtain a noise reduction image;
the three-dimensional guiding noise reduction neural sub-network is a network obtained by utilizing a plurality of sample alignment images, the plurality of sample short-exposure images and the plurality of real long-exposure images.
5. The method of claim 4, wherein the three-dimensional guided denoising convolutional neural subnetwork denoises the second soft exposure image according to the alignment image to obtain a denoised image, comprising:
mapping the alignment image and the second soft exposure image into a four-dimensional feature as a second four-dimensional feature; the second four-dimensional feature is used for reflecting the features of each pixel point in the aligned image and the corresponding pixel point in the second soft exposure image;
and carrying out weighted average processing on the second four-dimensional features by utilizing a preset three-dimensional filtering weight to obtain the noise reduction image.
6. The method of claim 1, wherein the fusing the short-exposure image and the noise-reduced image to obtain the high-dynamic-range image comprises:
inputting the short-exposure image and the noise-reduced image into an image fusion convolution neural sub-network obtained by pre-training, so that the image fusion convolution neural sub-network fuses the short-exposure image and the noise-reduced image to obtain the high dynamic range image;
the image fusion convolution neural subnetwork is a network obtained by training the plurality of sample short-exposure images, the plurality of sample noise reduction images and the real high dynamic range image corresponding to each sample short-exposure image; the real high dynamic range image corresponding to any sample short exposure image is an image acquired by a camera used for acquiring the high dynamic range image at the acquisition moment of the sample short exposure image.
7. The method of claim 6, wherein the image fusion convolutional neural subnetwork fuses the short-exposure image and the noise-reduced image to obtain the high dynamic range image, comprising:
and weighting the short-exposure image and the noise reduction image by using preset harmonic weight to obtain the high dynamic range image.
8. A ghost-free high dynamic range imaging apparatus for a dual lens camera, the apparatus comprising:
the input acquisition module is used for acquiring a long exposure image and a short exposure image which are acquired by the double-lens camera at the same time;
an image processing module, configured to input the long-exposure image and the short-exposure image into a main image enhancement model of ghost-free high dynamic range imaging, so that the main image enhancement model performs the following operations to obtain a high dynamic range image:
based on the short exposure image, carrying out alignment adjustment on the long exposure image to obtain an aligned image, comprising:
carrying out exposure adjustment on the short-exposure image by using the long-exposure image to obtain a first soft-exposure image;
inputting the long exposure image and the first soft exposure image into an image matching convolution neural sub-network obtained through pre-training, so that the image matching convolution neural sub-network adjusts the long exposure image into an image aligned with the first soft exposure image, and an aligned image is obtained;
the image matching convolution neural sub-network is obtained by training a plurality of sample long-exposure images, a plurality of sample short-exposure images and a real long-exposure image corresponding to each sample short-exposure image; the real long exposure image corresponding to any short exposure image is an image obtained by carrying out physical long exposure on the short exposure image by the double-lens camera;
based on the alignment image, carrying out exposure adjustment and noise reduction processing on the short-exposure image to obtain a noise-reduced image;
and fusing the short exposure image and the noise reduction image to obtain the high dynamic range image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011460648.9A CN112689099B (en) | 2020-12-11 | 2020-12-11 | Double-image-free high-dynamic-range imaging method and device for double-lens camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011460648.9A CN112689099B (en) | 2020-12-11 | 2020-12-11 | Double-image-free high-dynamic-range imaging method and device for double-lens camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112689099A CN112689099A (en) | 2021-04-20 |
| CN112689099B true CN112689099B (en) | 2022-03-22 |
Family
ID=75449251
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011460648.9A Expired - Fee Related CN112689099B (en) | 2020-12-11 | 2020-12-11 | Double-image-free high-dynamic-range imaging method and device for double-lens camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112689099B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113835462A (en) * | 2021-09-13 | 2021-12-24 | 星宸科技股份有限公司 | Control circuit and control method of image sensor |
| CN116033274B (en) * | 2023-02-27 | 2023-06-09 | 江苏游隼微电子有限公司 | 3D-noise-reduction-compatible image width dynamic method |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020051305A1 (en) * | 2018-09-07 | 2020-03-12 | Dolby Laboratories Licensing Corporation | Auto exposure of spatially-multiplexed-exposure high-dynamic-range image sensors |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4661922B2 (en) * | 2008-09-03 | 2011-03-30 | ソニー株式会社 | Image processing apparatus, imaging apparatus, solid-state imaging device, image processing method, and program |
| JP5978949B2 (en) * | 2012-03-16 | 2016-08-24 | 富士通株式会社 | Image composition apparatus and computer program for image composition |
| JP6020199B2 (en) * | 2013-01-24 | 2016-11-02 | 株式会社ソシオネクスト | Image processing apparatus, method, program, and imaging apparatus |
| WO2019001701A1 (en) * | 2017-06-28 | 2019-01-03 | Huawei Technologies Co., Ltd. | Image processing apparatus and method |
| CN107507234B (en) * | 2017-08-29 | 2019-11-01 | 北京大学 | Cone beam computed tomography image and x-ray image method for registering |
| CN107808365A (en) * | 2017-09-30 | 2018-03-16 | 广州智慧城市发展研究院 | One kind is based on the self-compressed image denoising method of convolutional neural networks and system |
| WO2019071613A1 (en) * | 2017-10-13 | 2019-04-18 | 华为技术有限公司 | Image processing method and device |
| CN108492245B (en) * | 2018-02-06 | 2020-06-30 | 浙江大学 | Low-luminosity image pair fusion method based on wavelet decomposition and bilateral filtering |
| CN109040524B (en) * | 2018-08-16 | 2021-06-29 | Oppo广东移动通信有限公司 | Artifact eliminating method and device, storage medium and terminal |
| US11107205B2 (en) * | 2019-02-18 | 2021-08-31 | Samsung Electronics Co., Ltd. | Techniques for convolutional neural network-based multi-exposure fusion of multiple image frames and for deblurring multiple image frames |
-
2020
- 2020-12-11 CN CN202011460648.9A patent/CN112689099B/en not_active Expired - Fee Related
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020051305A1 (en) * | 2018-09-07 | 2020-03-12 | Dolby Laboratories Licensing Corporation | Auto exposure of spatially-multiplexed-exposure high-dynamic-range image sensors |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112689099A (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200387750A1 (en) | Method and apparatus for training neural network model for enhancing image detail | |
| CN112241976B (en) | Model training method and device | |
| CN112766279B (en) | Image feature extraction method based on combined attention mechanism | |
| Ruan et al. | Aifnet: All-in-focus image restoration network using a light field-based dataset | |
| CN112446380A (en) | Image processing method and device | |
| CN112689099B (en) | Double-image-free high-dynamic-range imaging method and device for double-lens camera | |
| Chen et al. | Optical aberrations correction in postprocessing using imaging simulation | |
| CN111160229B (en) | SSD network-based video target detection method and device | |
| Sil et al. | Convolutional neural networks for noise classification and denoising of images | |
| US20230215085A1 (en) | Object-centric neural decomposition for image re-rendering | |
| US20240062343A1 (en) | Image Restoration Method and Apparatus, Image Restoration Device and Storage Medium | |
| CN110032931B (en) | Method and device for generating countermeasure network training and removing reticulation and electronic equipment | |
| CN117351448A (en) | Improved polarized image road target detection method based on YOLOv8 | |
| Jiang et al. | A multi-focus image fusion method based on attention mechanism and supervised learning | |
| CN113658091A (en) | Image evaluation method, storage medium and terminal equipment | |
| CN112101456A (en) | Attention feature map acquisition method and device and target detection method and device | |
| Yang et al. | Aberration-aware depth-from-focus | |
| CN114358204A (en) | No-reference image quality evaluation method and system based on self-supervision | |
| CN116403064B (en) | Picture processing method, system, equipment and medium | |
| CN115311149A (en) | Image denoising method, model, computer-readable storage medium and terminal device | |
| CN116844008A (en) | Attention mechanism guided content perception non-reference image quality evaluation method | |
| CN116485743A (en) | No-reference image quality evaluation method, system, electronic equipment and storage medium | |
| CN115689947A (en) | Image sharpening method, system, electronic device and storage medium | |
| CN115937121A (en) | Non-reference image quality evaluation method and system based on multi-dimensional feature fusion | |
| CN116071804A (en) | Face recognition method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220322 |