CN115439384A - Ghost-free multi-exposure image fusion method and device - Google Patents
Ghost-free multi-exposure image fusion method and device Download PDFInfo
- Publication number
- CN115439384A CN115439384A CN202211079389.4A CN202211079389A CN115439384A CN 115439384 A CN115439384 A CN 115439384A CN 202211079389 A CN202211079389 A CN 202211079389A CN 115439384 A CN115439384 A CN 115439384A
- Authority
- CN
- China
- Prior art keywords
- image
- exposure
- brightness
- frequency component
- fusion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Abstract
The invention provides a ghost-free multi-exposure image fusion method, a ghost-free multi-exposure image fusion device, computer equipment and a readable storage medium, wherein alignment of a source sequence image to a reference image is realized by combining brightness mapping and consistency detection, and different exposure brightness latent image sequences with consistent moving objects are obtained; based on the weighted least square filter, the method can be combined with favorable information in different exposure image sequences to promote and inhibit the brightness of the over-dark area and the over-bright area in the image, thereby retaining the details on the brightness of different levels, finally obtaining the ghost-free image with vivid color and rich information, and effectively enhancing the visual quality and the dynamic range of the fused image. The invention can meet the requirement of multi-exposure fusion in different dynamic scenes and can effectively solve the problem of ghost image caused by the shake of a moving object or imaging equipment in the imaging scene.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a ghost-free multi-exposure image fusion method and device, computer equipment and a readable storage medium.
Background
The luminance span in natural scenes is usually large, ranging from a night star to a bright sun, with luminance covering 9 orders of magnitude. Due to the restriction of the dynamic range of imaging equipment, it is difficult to record the detail information of different brightness levels in a scene through one-time exposure imaging; the high dynamic range image obtained by the Multi-exposure image fusion (MEF) method can contain more detailed information, better matches the response of human eyes to a real scene, and has wide application prospect and important practical significance in a plurality of fields. Currently, most MEF methods assume that the source images are perfectly aligned, but inevitably there are moving objects of different laws of motion and camera shake in the actual imaged scene. Time difference exists in the image acquisition process, and if the acquired dynamic scene multi-exposure image sequence is directly fused, the problems of fuzzy, ghost or semitransparent areas and the like of a fusion result can be caused, the imaging quality is seriously influenced, and the method becomes a challenging problem in the current multi-exposure image fusion technology.
In this regard, many studies have been conducted to explore methods for removing ghosting images and improving the fusion quality of multi-exposure images from different perspectives, and generally differ in two ways: how to detect and eliminate ghosting. The current methods can be mainly divided into two major categories of moving object removal, moving object selection and registration. The method for removing the moving objects removes all the moving objects in a scene by estimating a static background, does not select a reference image, mostly static in the image scene, only a small part of the image scene comprises moving objects, performs consistency check on each pixel, models the moving objects as abnormal values, and eliminates the abnormal values to obtain a high-dynamic image without artifacts. However, this type of method has poor robustness when scene changes are frequent, and has a good effect of moving objects quickly, and for slowly moving objects, artifacts cannot be removed well. The moving object selection and registration method reserves the moving object in the output fusion result, and restores or reconstructs the pixels affected by the motion by searching the local corresponding relation of the motion area between the reference image and the sequence image, and performs space alignment on the source sequence image. However, the method depends on the registration accuracy, and has the problems of ghosting or partial area color distortion in different degrees and low efficiency.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, an apparatus, a computer device and a readable storage medium for fusing multiple exposure images without ghosting, which can meet the requirement for multiple exposure fusion in different dynamic scenes and effectively solve the problem of ghosting caused by the shake of an imaging object or an imaging device in an imaging scene.
In a first aspect, an embodiment of the present invention provides a ghost-free multi-exposure image fusion method, including:
acquiring a multi-exposure image sequence, wherein the multi-exposure image sequence is composed of images with multi-level exposure brightness;
determining a reference image from the multi-exposure image sequence, wherein the reference image is an image with a central exposure brightness;
determining a latent image for fusion based on a brightness mapping model for an overexposed image in the multi-exposure image sequence;
for underexposed images in the multi-exposure image sequence, determining the latent images for fusion based on a motion consistency principle;
filtering the latent image to obtain a low-frequency component and a high-frequency component of the latent image;
constructing a mixed weight relation corresponding to the low-frequency component and the high-frequency component and carrying out exposure appropriateness evaluation;
and carrying out weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fusion image.
As an alternative, the determining a reference image from the multi-exposure image sequence, where the reference image is an exposure brightness-centered image, includes:
arranging each image of the multi-exposure image sequence in ascending order according to the overall brightness value, and selecting a sequencing intermediate value I ref1 Is a first reference image;
counting the whole exposure level of the brightness components of all the images to obtain the exposure prior information of the images, and calculating the total number of pixels with pixel values falling in a target interval, wherein the maximum value of the total number is a second reference image I ref2 。
Comparing the first reference image I ref1 And a second reference picture I ref2 When the sorting difference is larger than 2, selecting the first reference image I ref1 For the final reference picture I ref Otherwise, selecting the second reference image I ref2 For reference picture I ref For images with lower rank value than the reference image I ref Is configured as an underexposed image, with a higher rank value than the reference image I ref Is configured as an overexposed image.
As an alternative, the determining a latent image for fusion based on a pre-trained luminance mapping model for an overexposed image in the multi-exposure image sequence includes:
and establishing a mapping of brightness relation between the images of the multi-exposure image sequence and a reference image to obtain a brightness mapping model, and mapping the brightness of the reference image into the same brightness range as the overexposed image to obtain a latent image for fusion.
As an alternative, the determining the latent image for fusion based on the principle of motion consistency for the underexposed image in the multi-exposure image sequence includes:
extracting a group of color image blocks from the same spatial position of the multi-exposure image sequence by using a moving window with fixed step length, and determining a structure vector S of the reference image ref And an image structure vector S of the sequence of multi-exposure images k Inner product of between, for the structure vector S ref And the image structure vector S k Judging the motion consistency of the inter-objects;
pre-configuration structure consistency determination threshold T ρ Detecting pixels with inconsistent motion and pixels with consistent motion between the images;
mapping the k-th exposure image to the brightness level of the reference image to obtain a latent image, and determining the corresponding latent image and the reference imageThe absolute value of the average brightness difference of the image blocks of the image at the same spatial position is preset with an average brightness difference threshold T u And judging other pixels with inconsistent motion.
Obtaining a final motion consistency detection result by utilizing multiplication processing, and reserving pixel points of the image at consistent pixel point positions; and mapping the reference image to the image pixel points in the image brightness range at the inconsistent pixel point positions.
The abnormal pixel point judgment condition is designed to restrict the latent image pixel value, and the problems of brightness saturation and color distortion of some latent image local areas are solved.
As an optional solution, the weighting processing on the latent image to obtain the low frequency component and the high frequency component of the latent image includes:
determining RGB three-color channel weighted sum of the latent images to obtain brightness image of each latent image
Estimating the latent image based on a weighted least square filter to obtain low-frequency information of different exposure latent images, and determining the low-frequency image corresponding to the low-frequency information after filtering
When a low-frequency component image of the input latent image is obtained
According to the input latent image of each channel
Obtaining high-frequency component images on red, green and blue channels
As an optional solution, the constructing a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and performing exposure moderation evaluation includes:
constructing a weight function of the low-frequency component of the image based on the local average brightness and the global average brightness to evaluate exposure appropriateness;
and performing mean filtering based on the latent image, and determining a weight function of the high-frequency components of the image to perform exposure appropriateness evaluation.
As an optional solution, the performing weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fusion image includes:
and respectively multiplying the high-frequency component and the low-frequency component of the latent image by the corresponding weights, carrying out weighted summation, and carrying out combined processing on RGB color channels during fusion to obtain a target fusion image.
In a second aspect, an embodiment of the present invention provides a ghost-free multi-exposure image fusion apparatus, including:
an acquisition component for acquiring a sequence of multi-exposure images, the sequence of multi-exposure images consisting of images of multiple levels of exposure brightness;
a determining component for determining a reference image from the sequence of multi-exposure images, the reference image being an image with a centered exposure brightness;
the determining unit is further used for determining a latent image for fusion based on a pre-trained brightness mapping model for an overexposed image in the multi-exposure image sequence;
the determining unit is further used for determining the latent image for fusion based on a motion consistency principle for the underexposed image in the multi-exposure image sequence;
the determining unit is further used for filtering the latent image to obtain a low-frequency component and a high-frequency component of the latent image;
the determining unit is further configured to construct a mixed weight relationship corresponding to the low-frequency component and the high-frequency component, and perform exposure appropriateness evaluation;
and the fusion component is used for performing weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fusion image.
In a third aspect, an embodiment of the present invention provides a computer device, including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described ghost-free multi-exposure image fusion method.
In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the above-described ghost-free multi-exposure image fusion method.
The embodiment of the invention provides a method, a device, computer equipment and a readable storage medium for fusing ghost-free multi-exposure images, which are used for realizing the alignment of a source sequence image to a reference image by combining brightness mapping and consistency detection to obtain different exposure brightness latent image sequences with consistent moving objects; based on the weighted least square filter, the method can be combined with favorable information in different exposure image sequences to promote and inhibit the brightness of the over-dark area and the over-bright area in the image, thereby retaining the details on different brightness levels, finally obtaining the ghost-free image with vivid color and rich information, and effectively enhancing the visual quality and the dynamic range of the fused image. The invention can meet the requirement of multi-exposure fusion in different dynamic scenes and can effectively solve the problem of ghost image caused by the shake of a moving object or imaging equipment in the imaging scene.
Drawings
FIG. 1 is a flowchart illustrating a method for ghost-free multi-exposure image fusion according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of latent images before and after abnormal pixel constraint by a ghost-free multi-exposure image fusion method according to an embodiment of the present invention;
FIG. 3a is a schematic diagram of a sequence of images of a dynamic scene at different exposures;
FIG. 3b is a diagram illustrating a fusion result of a ghost-free multi-exposure image fusion method according to an embodiment of the present invention;
FIG. 4 is a block diagram illustrating an embodiment of a ghost-free multi-exposure image fusion apparatus;
fig. 5 is a block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to fig. 1, an embodiment of the present invention provides a ghost-free multi-exposure image fusion method, including:
s101, acquiring a multi-exposure image sequence, wherein the multi-exposure image sequence is composed of images with multi-level exposure brightness, and inputting a multi-exposure image sequence I k (K =1, \8230;, K), K being the total number of input images, all image pixel values are normalized to [0,1;, K]Within the range.
And S102, determining a reference image from the multi-exposure image sequence, wherein the reference image is an image with central exposure brightness.
S103, determining a latent image for fusion based on a pre-trained brightness mapping model for an overexposed image in the multi-exposure image sequence.
And S104, determining the latent image for fusion based on a motion consistency principle for the underexposed image in the multi-exposure image sequence.
And S105, filtering the latent image to obtain a low-frequency component and a high-frequency component of the latent image.
And S106, constructing a mixed weight relation corresponding to the low-frequency component and the high-frequency component and carrying out exposure appropriateness evaluation.
And S107, carrying out weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fusion image.
In step S102, the determining a reference image from the multi-exposure image sequence, where the reference image is an image with a centered exposure brightness, includes:
arranging each image of the multi-exposure image sequence in ascending order according to the overall brightness value, and selecting a sequencing intermediate value I ref1 Is a first reference image;
counting the whole exposure level of the brightness components of all the images to obtain the exposure prior information of the images, and calculating the total number of pixels of which the pixel values fall in a target interval, wherein the maximum total number is a second reference image I ref2 。
Comparing the first reference image I ref1 And a second reference picture I ref2 When the sorting difference is larger than 2, selecting the first reference image I ref1 For the final reference picture I ref Otherwise, selecting the second reference image I ref2 For reference picture I ref For images with lower rank value than the reference image I ref Is configured as an underexposed image, with a higher ranking value than the reference image I ref Is configured as an overexposed image.
In step S103, determining a latent image for fusion based on a pre-trained luminance mapping model for an overexposed image in the multi-exposure image sequence includes:
and establishing a mapping of brightness relation between the images of the multi-exposure image sequence and a reference image to obtain a brightness mapping model, and mapping the brightness of the reference image into the same brightness range as the overexposed image to obtain a latent image for fusion.
In step S104, for an underexposed image in the multi-exposure image sequence, determining the latent image for fusion based on a motion consistency principle includes:
extracting a group of color image blocks from the same spatial position of the multi-exposure image sequence by using a moving window with fixed step length, and determining a structural vector S of the reference image ref And an image structure vector S of the sequence of multi-exposure images k Inner product of between, for the structure vector S ref And the image structure vector S k Judging the consistency of the motion of the objects;
pre-arrangement of structure consistency determination threshold T ρ Detecting pixels with inconsistent motion and pixels with consistent motion between the images;
mapping the k-th exposure image to the brightness level of the reference image to obtain a latent image, determining the absolute value of the average brightness difference of image blocks of the corresponding latent image and the reference image at the same spatial position, and pre-configuring an average brightness difference threshold T u And judging other pixels with inconsistent motion.
Obtaining a final motion consistency detection result by utilizing multiplication processing, and reserving pixel points of the image at consistent pixel point positions; and mapping the reference image to the image pixel points in the image brightness range at the inconsistent pixel point positions.
The abnormal pixel point judgment condition is designed to restrict the latent image pixel value, and the problems of brightness saturation and color distortion of some latent image local areas are solved.
In step S105, the filtering the latent image to obtain a low-domain component and a high-domain component of the latent image includes:
determining RGB three-color channel weighted sum of the latent images to obtain brightness image of each latent image
Estimating the latent image based on a weighted least square filter to obtain low-frequency information of different exposure latent images, and determining the low-frequency image corresponding to the low-frequency information after filtering
When a low-frequency component image of the input latent image is obtained
According to the input latent image of each channel
Obtaining high-frequency component images on red, green and blue channels
In step S106, the constructing a mixed weight relationship corresponding to the low frequency component and the high frequency component and performing exposure appropriateness evaluation includes:
constructing a weight function of the low-frequency component of the image based on the local average brightness and the global average brightness to evaluate exposure appropriateness;
and performing mean filtering based on the latent image, and determining a weight function of the high-frequency components of the image to perform exposure appropriateness evaluation.
In step S107, the performing weighted fusion processing on the low frequency component and the high frequency component to obtain a target fusion image includes:
and respectively multiplying the high-frequency component and the low-frequency component of the latent image by the corresponding weights, carrying out weighted summation, and carrying out combined processing on RGB color channels during fusion to obtain a target fusion image.
The embodiment of the invention also provides a ghost-free multi-exposure image fusion method, which combines brightness mapping and consistency detection to realize the alignment of the source sequence image to the reference image and obtain different exposure brightness latent image sequences with consistent moving objects; based on the weighted least square filter, the method can be combined with favorable information in different exposure image sequences to promote and inhibit the brightness of the over-dark area and the over-bright area in the image, thereby retaining the details on the brightness of different levels, finally obtaining the ghost-free image with vivid color and rich information, and effectively enhancing the visual quality and the dynamic range of the fused image. The invention can meet the multi-exposure fusion requirements under different dynamic scenes and can effectively solve the ghost problem caused by the shake of a moving object or imaging equipment in the imaging scene.
The embodiment of the invention also provides a ghost-free multi-exposure image fusion method, which comprises the following steps:
s201, inputting a multi-exposure source image sequence I k (K =1, \ 8230;, K), K being the total number of input images, all source image pixel values being normalized to [0, 1;, K]Within the range.
S202, the exposure quality of the reference image influences the definition, the area detection and the correction of a moving target in an imaging scene, and further influences the visual effect of the multi-exposure fusion image. In order to select a reference image with proper exposure, a strategy of combining the intermediate value judgment and the exposure information prior is adopted for judgment.
The detailed process of step S202 is as follows:
s2021, arranging the input source image sequence in an ascending order according to the brightness values, and selecting a sequencing intermediate value I ref1 For the reference image:
s2022, counting the overall exposure level of the brightness of all the input source image sequences to obtain the exposure prior information of the images, and calculating the pixel value of each image to be in [0.01,0.09 ]]The total number of pixels in the interval is the maximum value and is the reference image I ref2 。
S2023, comparison I ref1 And I ref2 If both are presentThe sorting difference is more than 2, which indicates that the whole brightness level of the image sequence is higher or lower, therefore I is selected ref1 For the final reference picture I ref Otherwise, select I ref2 For the final reference picture I ref To ensure that the reference image has a maximum number of properly exposed pixel values. And setting the source images with the ranking values lower than the reference image as underexposed images, and setting the source images with the ranking values higher than the reference image as overexposed images.
S203, establishing a brightness mapping relation between the source sequence image and the reference image for the overexposed image to obtain a latent image for fusion. The brightness mapping relation is established among the images with different exposure levels, and a brightness mapping model can be obtained:
I k ＝τ(I ref )
wherein τ (I) = g -1 (kg (I)) is the luminance mapping function, I is the image luminance; g is a camera response function; k is an exposure coefficient representing the radiation relationship between the two images. A brightness mapping relation model between the reference image and the rest overexposed images can be established by utilizing the brightness histogram of the image; according to the brightness mapping model, the brightness of the reference image can be mapped into the same brightness range as the overexposed image, thereby obtaining a latent image for final fusion.
S204, for the underexposed image with the brightness value lower than that of the reference image, the method realizes the motion consistency detection by utilizing the direction information of the image structure vector quantity; introducing a structural consistency threshold and a brightness difference threshold to maximally retain consistent pixels and correct inconsistent pixels; and adding abnormal pixel point judgment conditions to carry out constraint to solve the problem of abnormal brightness of local areas in the latent images.
The detailed process of step S204 is as follows:
s2041, motion consistency analysis and detection between images are achieved by means of direction information which is possessed by the image structure vector. Firstly, a multi-exposure image sequence I is obtained by utilizing a moving window with fixed step k Extracting a set of color image blocks { g) at the same spatial location k }＝{g k |1≤k≤K}，g k Is a vector of CWH dimensions (C is 3 channels, W and H are the width and height of the image block), inStructural vector S of calculation source image block k Calculating a structure vector S of the reference image block ref . In the motion region detection, the motion consistency detection problem between a reference image and a source sequence image is converted into a direction consistency analysis problem of a structure vector between the reference image and the source sequence image. Calculating a reference image block structure vector S ref And a source image block structure vector S k Inner product between:
in the formula u gk Is g k Average luminance of u gref Is g ref Average luminance of p k ∈[-1,1]If ρ is k The larger the value is, the S is shown ref And S k The higher the similarity. Construction of S by luminance normalization k The algorithm can be ensured to have stronger robustness to image exposure change and contrast change; due to the weak contrast and low visibility in the region of insufficient exposure, the structure vector S k Scaling to a single bit length will involve the noise structure, so an e-parameter close to 0 is introduced to ensure that the coherence ratio ρ is guaranteed regardless of the noise structure k Is close to 1.
S2042, introducing a structural consistency threshold T for detecting more motion inconsistent pixel points ρ Judging the motion consistency:
by means of binary maps
And observing the pixel points with inconsistent motion in the whole image sequence.
S2043, in order to detect and correct inconsistent pixels to the maximum extent, fusion ghost is reduced, and brightness difference judgment conditions are increased. Calculating an underexposed image I k And a reference picture I ref Mapping the k-th exposure image to the exposure level of the reference image to obtain an image with the same brightness level, calculating the absolute value of the average brightness difference between the image blocks of the image with the same brightness level and the reference image at the same spatial position, and setting an average brightness difference threshold T u And (4) judging:
in formula (II) u' gk Is the average luminance of the image blocks in the image of equal luminance level resulting from the mapping of the underexposed image to the reference image.
Step 4-4: the final motion consistency detection result is as follows:
for B k A set of pixels of =1, representing that the source sequence image and the reference image move in unison, at which time the image block at the same position in the source sequence image is retained; for B k A pixel set of =0, which represents that the source sequence image and the reference image do not move uniformly, and the non-uniform pixels need to be corrected by combining a brightness mapping function, so as to obtain a latent image Q of the reference image on a lower exposure image brightness level k 。
S2045, with reference to fig. 2, shows latent images before and after underexposed image constraint, and in some latent images, brightness saturation distortion may occur in local areas, resulting in missing or abnormal pixel colors. To this end, we add a constraint on the abnormal pixel point determination condition, if the pixel value at a certain position in the latent image is close to saturation, copy the corresponding pixel value from the source sequence image to solve the problem of pixel saturation in the latent image:
in the formula, Q k,1 、Q k,2 And Q k,3 The three color channels representing the latent image, before and after underexposed image constraint, are shown in FIG. 2. Through the above processing, a latent image without color anomaly, in which the reference image is mapped to the low-exposure-level source image, can be obtained.
S205, estimating low-frequency information of the latent images with different exposures by using a weighted least square filter, and further calculating high-frequency information of the latent images.
The detailed process of step S205 is as follows:
s2051, calculating RGB three-channel weighted sum of input latent images to obtain a brightness image of each latent image
S2052, the weighted least squares filter (WLS) is an edge-preserving filter, which can be smoothed at other places except the edge and approach the image to the maximum extent
The invention estimates low frequency information for different exposed latent images based on a WLS filter. Presenting an input image
Computing corresponding filtered low frequency component images
Can be calculated by the following formula:
in the formula (I)
Ensuring the similarity between the input image and the output image, wherein the smaller the distance between the input image and the output image is, the higher the similarity between the filtered image and the input latent image is; the second term achieves the degree of smoothing by partial derivation, where λ is a regularization factor to maintain the balance between the two sub-formulas, the larger the value of λ, the smoother the output image, w x And w y Is a smoothing factor; v k For inputting images
In logarithmic form, i.e.
The index α represents the sensitivity to the input image gradient, and e is a constant close to 0.
S2053, once the low-frequency component image of the input latent image is obtained
According to the input latent image Q on each color channel k The high-frequency component image on any channel of red, green and blue can be obtained
And S206, constructing a weight function of the low-frequency component and the high-frequency component of the image of each level based on the local average brightness and the global average brightness of the exposure latent image of each level respectively, and performing exposure appropriateness evaluation.
The detailed process of the step is as follows:
s2061, calculating the weight of the low-frequency component of the image, and constructing a weight function of the component based on the local average brightness and the global average brightness to evaluate the exposure adequacy. The weight function of the low frequency component of the kth stage input latent image is calculated as follows:
in the formula, low frequency component
Used as local brightness exposure characteristic, the global average brightness of the kth-level input image is calculated according to the total brightness of the whole image and the image size
σ L And σ G Is the gaussian standard deviation set to 0.5 and 0.2, respectively; the first item retains the local structure with good exposure, and the second item encourages the consistency of the space brightness of the whole image, and by applying the two factors, a fused image with better visual effect can be generated.
S2062, calculating the weight of the high frequency component of the image, and calculating the weight function of the high frequency component of the kth-level input latent image as follows:
in the formula (I), the compound is shown in the specification,
is a mean filtered latent image intensity map, σ H Is the gaussian standard deviation.
S207, the low-frequency component and the high-frequency component of the input latent image are multiplied by the corresponding mixing weights, and weighted summation is performed. In order to reserve more color information, the color information of the image is implicitly reserved during calculation, and RGB three color channels are jointly processed during fusion to obtain a final fused image which is free of ghost and rich in color.
Fig. 3a is a schematic diagram of an image sequence of a dynamic scene under different exposures, and fig. 3b is a fusion result diagram after the method provided by the invention is applied, so that the fused image has no ghost and has richer details after integrating the light and dark level information of the different exposure images.
The ghost-free multi-exposure image fusion method provided by the embodiment of the invention has no complex parameter setting and does not depend on the prior knowledge of a camera. Effective scene characteristics can be reserved by selecting the optimal reference image, the corresponding relation between the source sequence image and the reference image is analyzed, brightness mapping and structure consistency detection are respectively carried out on the overexposed image and the underexposed image, ghost images caused by movement region changes are effectively removed, and different aligned exposure latent image sequences are obtained. The low-frequency components of different exposure latent images are estimated by using a weighted least square filter, so that corresponding high-frequency components are obtained, and the problem of information distortion caused by inaccurate estimation of high-frequency and low-frequency components of an image is solved; and finally, carrying out weighted fusion on the high-frequency component and the low-frequency component by using different mixed weights to obtain a final fusion result. Compared with the prior art, the method can better keep the clear details of the bright area and the dark area, can reflect the integrity of different levels of information of a real scene, and effectively improves the visual quality and the dynamic range of the fused image; the invention can not only process the motion of different amplitudes of different moving objects, even the motion of large motion, but also be suitable for the slight shake of the imaging equipment, has few artifacts and has stable performance. The method can provide technical support for obtaining high-dynamic imaging of different dynamic scenes.
With reference to fig. 4, correspondingly, an embodiment of the present invention further provides a ghost-free multi-exposure image fusion apparatus, including:
an acquisition component 401 for acquiring a sequence of multi-exposure images, the sequence of multi-exposure images consisting of images of multiple levels of exposure brightness;
a determining component 402 configured to determine a reference image from the sequence of multiple-exposure images, wherein the reference image is an image with centered exposure brightness;
the determining unit 402 is further configured to determine, for an overexposed image in the sequence of multi-exposure images, a latent image for fusion based on a pre-trained luminance mapping model;
the determining unit 402 is further configured to determine the latent image for fusion based on a motion consistency principle for an underexposed image in a multi-exposure image sequence;
the determining unit 402 is further configured to perform weighting processing on the latent image to obtain a low-frequency component and a high-frequency component of the latent image;
the determining unit 402 is further configured to construct a mixed weight relationship corresponding to the low-frequency component and the high-frequency component, and perform exposure appropriateness evaluation;
and a fusion component 403, configured to perform weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fusion image.
Accordingly, the invention also provides a computer device, a readable storage medium and a computer program product according to the embodiments of the invention.
Fig. 5 is a schematic structural diagram of a computer device 12 provided in an embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention. The computer device 12 shown in FIG. 5 is only an example and should not bring any limitations to the functionality or scope of use of embodiments of the present invention.
As shown in FIG. 5, computer device 12 is in the form of a general purpose computing device. Computer device 12 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
The components of computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and/or cache memory 32. Computer device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5 and commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.
The processing unit 16 executes various functional applications and data processing, such as implementing the image fusion method provided by the embodiment of the present invention, by running a program stored in the system memory 28.
The embodiment of the invention also provides a non-transitory computer readable storage medium which stores computer instructions and stores a computer program, wherein the program is executed by a processor to perform the image fusion method provided by all the invention embodiments of the application.
Computer storage media for embodiments of the present invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
An embodiment of the present invention further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the image fusion method according to the foregoing is implemented.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A ghost-free multi-exposure image fusion method is characterized by comprising the following steps:
acquiring a multi-exposure image sequence, wherein the multi-exposure image sequence is composed of images with multi-level exposure brightness;
determining a reference image from the multi-exposure image sequence, wherein the reference image is an image with central exposure brightness;
determining a latent image for fusion based on a brightness mapping model for an overexposed image in the sequence of multi-exposure images;
for underexposed images in the multi-exposure image sequence, determining the latent images for fusion based on a motion consistency principle;
filtering the latent image to obtain a low-frequency component and a high-frequency component of the latent image;
constructing a mixed weight relation corresponding to the low-frequency component and the high-frequency component and carrying out exposure appropriateness evaluation;
and carrying out weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fusion image.
2. The image fusion method according to claim 1, wherein the determining a reference image from the multi-exposure image sequence, the reference image being an image with a centered exposure brightness, comprises:
arranging each image of the multi-exposure image sequence in ascending order according to the overall brightness value, and selecting a sequencing intermediate value I ref1 Is a first reference image;
counting the whole exposure level of the brightness components of all the images to obtain the exposure prior information of the images, and calculating the total number of pixels of which the pixel values fall in a target interval, wherein the maximum total number is a second reference image I ref2 。
Comparing the first reference image I ref1 And a second reference picture I ref2 When the sorting difference is larger than 2, selecting the first reference image I ref1 For the final reference picture I ref Otherwise, selecting the second reference image I ref2 For reference picture I ref For images with lower rank value than the reference image I ref Is configured as an underexposed image, with a higher rank value than the reference image I ref Is configured as an overexposed image.
3. The image fusion method according to claim 1 or 2, wherein the determining a latent image for fusion based on a pre-trained luminance mapping model for an overexposed image in the sequence of multi-exposure images comprises:
and establishing a mapping of brightness relation between the images of the multi-exposure image sequence and a reference image to obtain a brightness mapping model, and mapping the brightness of the reference image into the same brightness range as the overexposed image to obtain a latent image for fusion.
4. The image fusion method of claim 3, wherein the determining the latent image for fusion based on a motion consistency principle for an underexposed image in a sequence of multiple-exposure images comprises:
extracting a group of color image blocks from the same spatial position of the multi-exposure image sequence by using a moving window with fixed step length, and determining a structure vector S of the reference image ref And an image structure vector S of the sequence of multi-exposure images k Inner product of between, for the structure vector S ref And the image structure vector S k Judging the motion consistency of the inter-objects;
pre-arrangement of structure consistency determination threshold T ρ Detecting pixels with inconsistent motion and pixels with consistent motion between the images;
mapping the k-th exposure image to the brightness level of the reference image to obtain a latent image, determining the absolute value of the average brightness difference of image blocks of the corresponding latent image and the reference image at the same spatial position, and pre-configuring an average brightness difference threshold T u Judging other pixels with inconsistent motion;
obtaining a final motion consistency detection result by utilizing multiplication processing, and reserving pixel points of the image at consistent pixel point positions; mapping the reference image to image pixel points in the image brightness range at inconsistent pixel point positions;
pixel values of the latent image are constrained using abnormal pixel determination.
5. The image fusion method according to claim 4, wherein the weighting the latent image to obtain the low-frequency component and the high-frequency component of the latent image comprises:
determining RGB three-color channel weighted sum of the latent images to obtain brightness image of each latent image
Estimating the latent image based on a weighted least square filter to obtain low-frequency information of different exposure latent images, and determining the low-frequency image corresponding to the low-frequency information after filtering
When a low-frequency component image of the input latent image is obtained
According to the input latent image of each channel
Obtaining high-frequency component images on red, green and blue channels
6. The image fusion method according to claim 5, wherein the constructing a mixing weight relationship corresponding to the low-frequency component and the high-frequency component and performing exposure moderation evaluation comprises:
constructing a weight function of the low-frequency component of the image based on the local average brightness and the global average brightness to evaluate exposure appropriateness;
and performing mean filtering based on the latent image, and determining a weight function of the high-frequency component of the image to perform exposure adequacy evaluation.
7. The image fusion method according to claim 6, wherein the performing weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fusion image comprises:
and respectively multiplying the high-frequency component and the low-frequency component of the latent image by the corresponding weights, carrying out weighted summation, and carrying out combined processing on RGB color channels during fusion to obtain a target fusion image.
8. A ghost-free multi-exposure image fusion device, comprising:
an acquisition component for acquiring a sequence of multi-exposure images, the sequence of multi-exposure images consisting of images of multiple levels of exposure brightness;
a determining component for determining a reference image from the sequence of multi-exposure images, wherein the reference image is an image with a centered exposure brightness;
the determining unit is further used for determining a latent image for fusion based on a pre-trained brightness mapping model for an overexposed image in the multi-exposure image sequence;
the determining unit is further used for determining the latent image for fusion based on a motion consistency principle for the underexposed image in the multi-exposure image sequence;
the determining unit is further used for weighting the latent image to obtain a low-frequency component and a high-frequency component of the latent image;
the determining unit is further configured to construct a mixed weight relationship corresponding to the low-frequency component and the high-frequency component, and perform exposure appropriateness evaluation;
and the fusion component is used for performing weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fusion image.
9. A computer device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the ghost-free multi-exposure image fusion method of any one of claims 1-7.
10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the ghost-free multi-exposure image fusion method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211079389.4A CN115439384A (en) | 2022-09-05 | 2022-09-05 | Ghost-free multi-exposure image fusion method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211079389.4A CN115439384A (en) | 2022-09-05 | 2022-09-05 | Ghost-free multi-exposure image fusion method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115439384A true CN115439384A (en) | 2022-12-06 |
Family
ID=84246934
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211079389.4A Pending CN115439384A (en) | 2022-09-05 | 2022-09-05 | Ghost-free multi-exposure image fusion method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115439384A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116095502A (en) * | 2023-04-13 | 2023-05-09 | 芯动微电子科技(珠海)有限公司 | Method and device for fusing multiple exposure images |
-
2022
- 2022-09-05 CN CN202211079389.4A patent/CN115439384A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116095502A (en) * | 2023-04-13 | 2023-05-09 | 芯动微电子科技(珠海)有限公司 | Method and device for fusing multiple exposure images |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Berman et al. | Single image dehazing using haze-lines | |
| US10979640B2 (en) | Estimating HDR lighting conditions from a single LDR digital image | |
| US9020243B2 (en) | Image adjustment | |
| WO2021164550A1 (en) | Image classification method and apparatus | |
| JP2014520446A (en) | Automatic exposure compensation for images | |
| CN109389569B (en) | Monitoring video real-time defogging method based on improved DehazeNet | |
| Xiang et al. | Blind night-time image quality assessment: Subjective and objective approaches | |
| WO2014070273A1 (en) | Recursive conditional means image denoising | |
| CN113962859B (en) | Panorama generation method, device, equipment and medium | |
| WO2024078179A1 (en) | Lighting map noise reduction method and apparatus, and device and medium | |
| CN111784658B (en) | Quality analysis method and system for face image | |
| CN114529946A (en) | Pedestrian re-identification method, device, equipment and storage medium based on self-supervision learning | |
| WO2022206729A1 (en) | Method and apparatus for selecting cover of video, computer device, and storage medium | |
| CN115439384A (en) | Ghost-free multi-exposure image fusion method and device | |
| CN111882555A (en) | Net detection method, device, equipment and storage medium based on deep learning | |
| CN111862040A (en) | Portrait picture quality evaluation method, device, equipment and storage medium | |
| CN112651945A (en) | Multi-feature-based multi-exposure image perception quality evaluation method | |
| CN109409305A (en) | A kind of facial image clarity evaluation method and device | |
| Yan et al. | A natural-based fusion strategy for underwater image enhancement | |
| CN110197459B (en) | Image stylization generation method and device and electronic equipment | |
| CN111950457A (en) | Oil field safety production image identification method and system | |
| CN116797504A (en) | Image fusion method, electronic device and storage medium | |
| CN108416815B (en) | Method and apparatus for measuring atmospheric light value and computer readable storage medium | |
| Gao et al. | Automated parameter tuning for tone mapping using visual saliency | |
| CN112712571B (en) | Object plane mapping method, device and equipment based on video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |