CN114187213A - Image fusion method and device, equipment and storage medium thereof - Google Patents
Image fusion method and device, equipment and storage medium thereof Download PDFInfo
- Publication number
- CN114187213A CN114187213A CN202111525658.0A CN202111525658A CN114187213A CN 114187213 A CN114187213 A CN 114187213A CN 202111525658 A CN202111525658 A CN 202111525658A CN 114187213 A CN114187213 A CN 114187213A
- Authority
- CN
- China
- Prior art keywords
- image
- weight
- fusion
- sampling
- histogram
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 42
- 230000004927 fusion Effects 0.000 claims abstract description 207
- 238000004364 calculation method Methods 0.000 claims abstract description 178
- 238000005070 sampling Methods 0.000 claims abstract description 102
- 230000006870 function Effects 0.000 claims description 54
- 238000000034 method Methods 0.000 claims description 25
- 238000010606 normalization Methods 0.000 claims description 9
- 230000003247 decreasing effect Effects 0.000 claims description 5
- 230000000694 effects Effects 0.000 abstract description 7
- 230000008901 benefit Effects 0.000 description 12
- 230000007704 transition Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 11
- 238000007499 fusion processing Methods 0.000 description 10
- 230000014509 gene expression Effects 0.000 description 4
- 230000005856 abnormality Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/40—Image enhancement or restoration using histogram techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20208—High dynamic range [HDR] image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention provides an image fusion method, which comprises the following steps: providing an original image, and acquiring a histogram of the original image; extracting the features of the histogram to obtain histogram calculation parameters, and adjusting and determining a fusion weight calculation formula according to the histogram calculation parameters; acquiring brightness information in an original image, acquiring a down-sampling image according to the brightness information and the original image, and acquiring a down-sampling fusion weight according to a pixel brightness and fusion weight calculation formula of the down-sampling image; acquiring an up-sampling fusion weight according to the down-sampling fusion weight; and obtaining a fused image according to the up-sampling fusion weight and the original image. The invention combines the whole image with the local image, achieves the effects of keeping highlight and dark details of the image, reducing noise and enabling the image to transit naturally, and improves the quality of the fused image. The invention also provides a device, equipment and a storage medium for realizing the image fusion method.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to an image fusion method, an apparatus, a device, and a storage medium.
Background
The HDR image refers to a High-Dynamic Range (High-Dynamic Range) image. At present, a fixed fusion weight calculation formula is usually designed for the fusion of HDR images, and image information is substituted into the fusion weight calculation formula point by point to obtain a fusion weight through calculation. Because the same information in the image, such as the expression difference and noise influence of brightness information in different scenes, the calculated fusion weight has a certain deviation with an ideal value; the point-by-point weight has a large variation trend, and the brightness gradient in the transition region may be large, so that the transition is unnatural. In practical application, the introduction of calculation in the HDR image fusion process may cause the problems of loss of details and unnatural transition in highlight or dark places, and greatly affect the quality of images.
Chinese patent publication No. CN 111462031A discloses a method, an apparatus, a storage medium, and an electronic device for processing a multi-frame HDR image, in which at least two original images with different exposure degrees are obtained, the obtained original images are respectively preprocessed to obtain a weight fusion image, and then the weight fusion image is fused to obtain a final fusion image. And obtaining fusion weight by analyzing the detail degree of the multi-frame images, thereby fusing the multi-frame images. However, the invention does not solve the problem that the introduction of calculation in the HDR image fusion process may cause the loss of details in highlight or dark places and unnatural transition.
Therefore, it is necessary to provide an image fusion method, an apparatus, a device and a storage medium thereof to solve the above-mentioned problems in the prior art.
Disclosure of Invention
The invention aims to provide an image fusion method, an image fusion device, image fusion equipment and a storage medium, which are used for solving the problems that highlight or dark details are lost and transition is unnatural due to calculation introduced in an HDR image fusion process.
In order to achieve the above object, the image fusion method of the present invention includes the steps of:
providing an original image, and acquiring a histogram of the original image;
extracting the features of the histogram to obtain histogram calculation parameters, and adjusting and determining a fusion weight calculation formula according to the histogram calculation parameters;
acquiring brightness information in the original image, acquiring a down-sampling image according to the brightness information and the original image, and acquiring a down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula;
acquiring an up-sampling fusion weight according to the down-sampling fusion weight, so that the resolution of the image corresponding to the up-sampling fusion weight is consistent with the resolution of the original image;
and obtaining a fused image according to the up-sampling fusion weight and the original image.
The image fusion method has the beneficial effects that:
by acquiring the histogram of the original image and the histogram calculation parameters, when image fusion is carried out in different scenes, the fusion weight calculation formula is adjusted and determined through the histogram calculation parameters, and the statistical information of the histogram is fully utilized, so that the fusion weight calculation formula has higher self-adaptability, the applicability of the image fusion method is improved, the problem that fusion weight calculation is not ideal due to different relative expressions in different scenes or the same brightness value is effectively prevented, and the problems of high light overexposure, dark information loss and noise abnormality caused by the fusion weight calculation formula are improved; the integral information of the image is reflected according to the downsampling weight calculated by the downsampling image, the fused image is obtained through the upsampling fusion weight and the original image, the local and integral information of the image is obtained, the local information and the integral information of the image are combined, the effects of keeping highlight and dark details of the image, reducing noise and enabling the image to be transited naturally are achieved, and the quality of the fused image is improved; the method solves the problems that the introduction of calculation in the HDR image fusion process can cause the loss of details in highlight or dark places and unnatural transition.
Optionally, the step of obtaining the histogram of the original image includes:
in a plurality of different scenes, respectively acquiring an original image in each scene within different exposure time, and acquiring a histogram of the original image. The method has the advantages that the histograms of the original images in different exposure times under different scenes are obtained, so that the image fusion method is suitable for various different scenes, and the applicability of the image fusion method is improved.
Optionally, the histogram calculation parameters include a median, an integral, and a mean of the histogram.
Optionally, the fusion weight calculation formula includes a weight calculation curve, and the step of adjusting and determining the fusion weight calculation formula according to the histogram calculation parameter includes:
presetting a weight curve, wherein a first slope of the weight curve on a first defined interval is less than or equal to 0 and is monotonically decreased, a second slope of the weight curve on a second defined interval is less than or equal to 0 and is monotonically increased, and a right endpoint of the first defined interval is connected with a left endpoint of the second defined interval;
recording a point of the first slope changing from 0 to non-0 as a first inflection point, and recording a point of the second slope changing from non-0 to 0 as a second inflection point;
adjusting at least one of the shape of the weight curve, the first slope, the second slope, the coordinate of the first inflection point, and the coordinate of the second inflection point according to the histogram calculation parameter to obtain the weight calculation curve. The method has the advantages that at least one of the shape of the weight curve, the first slope, the second slope, the coordinate of the first inflection point and the coordinate of the second inflection point is adjusted through the histogram calculation parameters to obtain the weight calculation curve, so that the weight calculation curve is suitable for weight calculation of the original image corresponding to the histogram.
Optionally, the down-sampling fusion weight includes a first exposure image weight and a second exposure image weight, and the step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula includes:
traversing all pixel points of a down-sampling image to obtain the brightness of the pixel points, inputting the brightness of the pixel points into the weight calculation curve to obtain a first weight result, and carrying out normalization processing on the first weight result to obtain a first exposure image weight and a second exposure image weight, wherein the sum of the first exposure image weight and the second exposure image weight is 1;
the first exposure image weight corresponds to a first exposure image, the second exposure image weight corresponds to a second exposure image, and the exposure time of the first exposure image is longer than that of the second exposure image.
Optionally, the fusion weight calculation formula includes a one-dimensional weight calculation gaussian function, and the step of adjusting and determining the calculation formula of the fusion weight according to the histogram calculation parameter includes:
presetting a one-dimensional Gaussian function, and adjusting the one-dimensional Gaussian function according to the histogram calculation parameters to obtain the one-dimensional weight calculation Gaussian function. The method has the advantages that the one-dimensional weight calculation Gaussian function obtained by presetting the one-dimensional Gaussian function and adjusting the one-dimensional Gaussian function through the histogram calculation parameters can be better suitable for weight calculation of the original image corresponding to the histogram.
Optionally, the step of obtaining the downsampling fusion weight according to the pixel brightness of the downsampling image and the fusion weight calculation formula includes:
and inputting the brightness of each pixel in the downsampled image into the one-dimensional weight calculation Gaussian function to obtain a second weight result, and performing normalization processing on the second weight result to obtain the downsampled fusion weight, wherein the downsampled fusion weight comprises N weights, and N is a positive integer.
Optionally, the step of obtaining a down-sampled image from the luminance information and the original image comprises:
dividing the original image into m × n blocks, wherein m and n are positive integers, calculating the average brightness value of all pixel points in each block, and replacing the brightness of all pixel points in each block with the average brightness value corresponding to the block to obtain the down-sampling image with the resolution of m × n.
Optionally, the obtaining the upsampling fusion weight according to the downsampling fusion weight includes:
and performing up-sampling processing on the down-sampling fusion weight through a bilinear interpolation method to obtain the up-sampling fusion weight.
Optionally, the obtaining the upsampling fusion weight according to the downsampling fusion weight includes:
and performing upsampling processing on the downsampling fusion weight through a two-dimensional Gaussian function to obtain the upsampling fusion weight.
The present invention also provides an image fusion apparatus, comprising:
the image processing module is used for providing an original image, acquiring a histogram and brightness information of the original image, and acquiring a down-sampling image according to the brightness information and the original image;
the adjusting module is used for extracting the features of the histogram to obtain histogram calculation parameters, adjusting the histogram calculation parameters and determining a fusion weight calculation formula;
the weight calculation module is used for acquiring a down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula, and acquiring an up-sampling fusion weight according to the down-sampling fusion weight, so that the resolution of the image corresponding to the up-sampling fusion weight is consistent with the resolution of the original image;
and the image fusion calculation module is used for obtaining a fusion image according to the up-sampling fusion weight and the original image.
The image fusion device of the invention has the advantages that:
acquiring a histogram and brightness information of the original image through the image processing module, acquiring a down-sampling image according to the brightness information and the original image, acquiring a histogram calculation parameter through the adjusting module, adjusting and determining a fusion weight calculation formula according to the histogram calculation parameter, acquiring a down-sampling fusion weight and an up-sampling fusion weight through the weight calculation module, and acquiring a fusion image according to the up-sampling fusion weight and the original image through the image fusion calculation module; the statistical information of the histogram is fully utilized, the applicability of the fusion weight calculation formula is improved, and the application scenes of the image fusion method are enlarged; by calculating the down-sampling image, the down-sampling fusion weight and the up-sampling fusion weight, the whole image is combined with the local part, the effects of keeping highlight and dark details of the image, reducing noise and enabling the image to transit naturally are achieved, and the quality of the fusion image is improved; the method solves the problems that the introduction of calculation in the HDR image fusion process can cause the loss of details in highlight or dark places and unnatural transition.
The invention also provides a device comprising a memory, a processor and a program stored on the memory and executable on the processor, the processor implementing the image fusion method when executing the program.
The present invention also provides a storage medium having stored thereon a program which, when executed by a processor, implements the image fusion method.
The device and the storage medium of the invention have the advantages that:
the fusion weight calculation formula is adjusted and determined through the histogram, the statistical information of the histogram is fully utilized, the applicability of the fusion weight calculation formula is improved, the application scenes of the image fusion method are expanded, the down-sampling image, the down-sampling fusion weight and the up-sampling fusion weight are calculated, so that the whole image is combined with the local part, the effects of keeping highlight and dark details of the image, reducing noise and enabling the image to be transited naturally are achieved, the quality of the fusion image is improved, and the problems that the highlight or dark details are lost and the transition is unnatural due to the fact that calculation is introduced in the HDR image fusion process are solved.
Drawings
FIG. 1 is a flow chart of an image fusion method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a weighting curve according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating bilinear interpolation according to an embodiment of the present invention;
FIG. 4 is a grid diagram of bilinear interpolation according to an embodiment of the present invention;
FIG. 5 is a graph of a weight curve gradient formed by two-dimensional Gaussian functions according to an embodiment of the present invention;
fig. 6 is a block diagram of an image fusion apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below, and it is obvious that the described embodiments are a part of the embodiments of the present invention, but not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. Unless defined otherwise, technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention belongs. As used herein, the word "comprising" and similar words are intended to mean that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items.
Aiming at the problems existing in the prior art, the embodiments of the present invention provide an image fusion method, an apparatus, a device, and a storage medium thereof, so as to solve the problem that the introduction of computation in the HDR image fusion process may cause details loss and unnatural transition in highlight or dark places.
Fig. 1 is a flowchart of an image fusion method according to an embodiment of the present invention.
In some embodiments of the present invention, referring to fig. 1, the image fusion method includes the steps of:
s0: providing an original image, and acquiring a histogram of the original image;
s1: extracting the features of the histogram to obtain histogram calculation parameters, and adjusting and determining a fusion weight calculation formula according to the histogram calculation parameters;
s2: acquiring brightness information in the original image, acquiring a down-sampling image according to the brightness information and the original image, and acquiring a down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula;
s3: acquiring an up-sampling fusion weight according to the down-sampling fusion weight, so that the resolution of the image corresponding to the up-sampling fusion weight is consistent with the resolution of the original image;
s4: and obtaining a fused image according to the up-sampling fusion weight and the original image.
The image fusion method has the advantages that: by acquiring the histogram of the original image and the histogram calculation parameters, when image fusion is carried out in different scenes, the fusion weight calculation formula is adjusted and determined through the histogram calculation parameters, and the statistical information of the histogram is fully utilized, so that the fusion weight calculation formula has higher self-adaptability, the applicability of the image fusion method is improved, the problem that fusion weight calculation is not ideal due to different relative expressions in different scenes or the same brightness value is effectively prevented, and the problems of high light overexposure, dark information loss and noise abnormality caused by the fusion weight calculation formula are improved; the integral information of the image is reflected according to the downsampling weight calculated by the downsampling image, the fused image is obtained through the upsampling fusion weight and the original image, the local and integral information of the image is obtained, the local information and the integral information of the image are combined, the effects of keeping highlight and dark details of the image, reducing noise and enabling the image to be transited naturally are achieved, and the quality of the fused image is improved; the method solves the problems that the introduction of calculation in the HDR image fusion process can cause the loss of details in highlight or dark places and unnatural transition.
In some embodiments, the fused image ultimately obtained by the present invention is a high dynamic range image.
As an optional implementation manner of the present invention, in step S0, the step of acquiring the histogram of the original image includes:
in a plurality of different scenes, respectively acquiring an original image in each scene within different exposure time, and acquiring a histogram of the original image. The method has the advantages that the histograms of the original images in different exposure times under different scenes are obtained, so that the image fusion method is suitable for various different scenes, and the applicability of the image fusion method is improved.
As an optional implementation manner of the present invention, in step S1, the histogram calculation parameters include a median, an integral, and a mean of the histogram.
In some embodiments, the histogram is a grayscale histogram.
As an optional implementation manner of the present invention, in step S1, the step of adjusting and determining the fusion weight calculation formula according to the histogram calculation parameter includes:
presetting a weight curve, wherein a first slope of the weight curve on a first defined interval is less than or equal to 0 and is monotonically decreased, a second slope of the weight curve on a second defined interval is less than or equal to 0 and is monotonically increased, and a right endpoint of the first defined interval is connected with a left endpoint of the second defined interval;
recording a point of the first slope changing from 0 to non-0 as a first inflection point, and recording a point of the second slope changing from non-0 to 0 as a second inflection point;
adjusting at least one of the shape of the weight curve, the first slope, the second slope, the coordinate of the first inflection point, and the coordinate of the second inflection point according to the histogram calculation parameter to obtain the weight calculation curve. The method has the advantage that at least one of the shape of the weight curve, the first slope, the second slope, the coordinate of the first inflection point and the coordinate of the second inflection point is adjusted by the histogram calculation parameter to obtain the weight calculation curve, so that the weight calculation curve is suitable for weight calculation of the original image corresponding to the histogram.
As an optional implementation manner of the present invention, in step S2, the step of acquiring a down-sampled image according to the luminance information and the original image includes:
dividing the original image into m × n blocks, wherein m and n are positive integers, calculating the average brightness value of all pixel points in each block, and replacing the brightness of all pixel points in each block with the average brightness value corresponding to the block to obtain the down-sampling image with the resolution of m × n; and each original image is subjected to the steps to obtain a corresponding down-sampling image with the resolution of m multiplied by n.
As an alternative embodiment of the present invention, in step S3, the step of obtaining the downsampling fusion weight according to the pixel brightness of the downsampling image and the fusion weight calculation formula includes:
traversing all pixel points of a down-sampling image to obtain the brightness of the pixel points, inputting the brightness of the pixel points into the weight calculation curve to obtain a first weight result, and carrying out normalization processing on the first weight result to obtain a first exposure image weight and a second exposure image weight, wherein the sum of the first exposure image weight and the second exposure image weight is 1;
the first exposure image weight corresponds to a first exposure image, the second exposure image weight corresponds to a second exposure image, and the exposure time of the first exposure image is longer than that of the second exposure image.
Fig. 2 is a schematic diagram of a weighting curve according to an embodiment of the present invention, where the ordinate of the weighting curve in fig. 2 is weighting and the abscissa is luminance.
In some embodiments of the invention, the step of obtaining a first fused image from the downsampled image and the downsampled fusion weight comprises:
referring to fig. 2, a weight curve f (g) is preset, wherein a first slope of the weight curve f (g) in a first defined interval [0, a ] is less than or equal to 0 and monotonically decreases, and a second slope in a second defined interval [ a, b ] is less than or equal to 0 and monotonically increases;
in some embodiments, the expression of the weight curve is:
wherein G is the abscissa of the curve, specifically representing the brightness of the image, and f (G) is the ordinate of the curve.
Recording a point at which the first slope of a weight curve f (G) changes from 0 to non-0 in the first defined interval [0, a ] as a first inflection point a, recording a point at which the second slope of a weight curve f (G) changes from non-0 to 0 in the second defined interval [ a, B ] as a second inflection point B, and adjusting at least one of a shape of the weight curve, the first slope, the second slope, coordinates of the first inflection point a, and coordinates of the second inflection point B according to the histogram calculation parameter, i.e., adjusting any one or more of a shape of the weight curve, the first slope, the second slope, coordinates of the first inflection point a, and the second inflection point B by the histogram calculation parameter, to obtain the weight calculation curve w ═ f' (G);
specifically, information of an original image is converted into G channel brightness information, the image is divided into m × n blocks, the average value of the brightness of all pixel points in each block is calculated, the average value of the brightness of the pixel points in each block is used as the brightness of the block, and a down-sampling image with the resolution of m × n is obtained;
substituting the brightness of the sampling image into the weight calculation curve w ═ f' (G) to obtain a first weight result, and performing normalization processing on the first weight result to obtain the first exposure image weight w and the second exposure image weight (1-w), wherein the sum of the first exposure image weight w and the second exposure image weight (1-w) is 1;
the first exposure image weight w corresponds to a first exposure image l1, the second exposure image weight (1-w) corresponds to a second exposure image l2, and the exposure time of the first exposure image l1 is longer than that of the second exposure image l 2.
As an optional implementation manner of the present invention, in step S1, the fusion weight calculation formula includes a one-dimensional weight calculation gaussian function, and the step of adjusting and determining the calculation formula of the fusion weight according to the histogram calculation parameter includes:
presetting a one-dimensional Gaussian function, and adjusting the one-dimensional Gaussian function according to the histogram calculation parameters to obtain the one-dimensional weight calculation Gaussian function. The method has the advantages that the obtained one-dimensional weight calculation Gaussian function can be better suitable for the weight calculation of the original image corresponding to the histogram through presetting the one-dimensional Gaussian function and adjusting the one-dimensional Gaussian function through the histogram calculation parameters.
As an alternative embodiment of the present invention, in step S3, the step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula includes:
and inputting the brightness of each pixel in the downsampled image into the one-dimensional weight calculation Gaussian function to obtain a second weight result, and performing normalization processing on the second weight result to obtain the downsampled fusion weight, wherein the downsampled fusion weight comprises N weights, and N is a positive integer.
In other embodiments of the present invention, the step of obtaining a first fused image based on the downsampled image and the downsampled fusion weight comprises:
acquiring a histogram of an original image through the original image, and extracting features of the histogram to acquire histogram calculation parameters;
presetting a one-dimensional Gaussian function, wherein the formula of the one-dimensional Gaussian function is as follows:
wherein x represents the input original image information, x in the embodiment of the present invention refers to brightness, σ is a gaussian standard deviation, and μ represents an optimal pixel value;
adjusting the one-dimensional gaussian function according to the histogram calculation parameter, specifically, adjusting a gaussian standard deviation σ and an optimal pixel value μ by the histogram calculation parameter to obtain the one-dimensional weight calculation gaussian function w ═ g' (x);
inputting the brightness of each pixel in the downsampled image into the one-dimensional weight calculation gaussian function w ═ g' (x) to obtain a second weight result, and performing normalization processing on the second weight result to obtain the downsampled fusion weight, wherein the downsampled fusion weight comprises N weights, N is a positive integer, the N weights are a first weight, a second weight, an integral right weight and an integral right weight, and the weighted sum of the N weights is 1.
In some embodiments, the weighted weights of the gaussian standard deviation σ and the optimal pixel value μ are determined according to the number of points of the histogram greater than a certain value, and the values of the gaussian standard deviation σ and the optimal pixel value μ are adjusted, so that the finally obtained one-dimensional weight calculation gaussian function is more suitable for the weight calculation of the image in the current scene.
In some embodiments, the gaussian standard deviation σ is 0.5, and the optimal pixel value μ is a weighted average of the different frame images.
The invention obtains the weight calculation curve by adjusting the weight curve of the exposure image, obtains the one-dimensional weight calculation Gaussian function by adjusting the one-dimensional Gaussian function, and calculates the weight of the image by any one of the weight calculation curve and the one-dimensional weight calculation Gaussian function, thereby providing various methods and calculation ways for the calculation of the fusion weight, improving the applicability and practicability of the image fusion method, and being applicable to various scenes.
It can be said that, for the processing of the same group of original images, the same group of original images only uses one of the weight calculation curve and the one-dimensional weight calculation gaussian function to calculate the weight of the image, that is, the two methods of the weight calculation curve and the one-dimensional weight calculation gaussian function cannot be used simultaneously in the same group of original images to calculate the image weight, so as to reduce the error of weight calculation and avoid the problem of unnatural image fusion caused by using different weight calculation methods.
As an optional implementation manner of the present invention, the obtaining of the upsampling fusion weight according to the downsampling fusion weight includes:
and performing up-sampling processing on the down-sampling fusion weight through a bilinear interpolation method to obtain the up-sampling fusion weight.
In some embodiments, the upsampling the downsampling fusion weight by bilinear interpolation to obtain the upsampling fusion weight includes:
FIG. 3 is a diagram illustrating bilinear interpolation according to an embodiment of the present invention.
In some embodiments, referring to fig. 3, linear interpolation is performed in two directions by using the downsampled fusion weights of the four neighboring pixels of the pixel to be solved, and the coordinates of the four points Q11, Q12, Q21, and Q22 of the known function f are respectively: q11(x1, y1), Q12(x1, y2), Q21(x2, y1), Q22(x2, y 2);
firstly, linear interpolation is carried out in the x direction, R2 is inserted into Q12 and Q22, and R1 is inserted into Q11 and Q21 to obtain R1 and R2;
then linear interpolation is carried out in the y direction, P point is interpolated in the y direction through the R1 and the R2 calculated in the first step, P is obtained, and f (x, y) is obtained through the following calculation formula:
the coordinates (x, y) of the final interpolation point P are obtained by the above formula, and four points in each down-sampled image are combined to obtain the interpolation point of each down-sampled image.
Fig. 4 is a schematic grid diagram of a bilinear interpolation method according to an embodiment of the present invention, in which a number on a z-axis represents a weight value of a point, and numbers on an x-axis and a y-axis represent an abscissa and an ordinate of the point, respectively.
In some embodiments, the mesh diagram of bilinear interpolation as shown in FIG. 4 is obtained according to the calculation formula of f (x, y). Calculating to obtain corresponding interpolation weights of points at each coordinate in the up-sampled image of the current point, namely substituting the coordinate of the interpolation point P into a grid schematic diagram of a bilinear interpolation method shown in FIG. 4 to obtain the weight of the interpolation point P, and calculating to obtain the up-sampling weight according to the weight of the interpolation point P; normalizing the upsampling weights of the pixel points at the same position in the plurality of images to obtain the upsampling fusion weight; and obtaining a first weight gradient image according to the up-sampling fusion weight, wherein the resolution of the weight gradient image is consistent with that of the original image.
As an optional implementation manner of the present invention, the obtaining of the upsampling fusion weight according to the downsampling fusion weight includes:
and performing upsampling processing on the downsampling fusion weight through a two-dimensional Gaussian function to obtain the upsampling fusion weight.
In some embodiments, the upsampling the downsampled fusion weight by a two-dimensional gaussian function to obtain the upsampled fusion weight comprises:
presetting a two-dimensional Gaussian function, wherein the formula of the two-dimensional Gaussian function is as follows:
wherein σ0Is the gaussian standard deviation.
Fig. 5 is a curved surface diagram formed by a two-dimensional gaussian function according to an embodiment of the present invention, and the number of the Z axis in fig. 5 represents the weight value.
In some embodiments, referring to FIG. 5, the two-dimensional Gaussian function described above may be derived as (x)center,ycenter) Is used as the center of the device,
is a curved surface with a radius gradually decreasing from the center to the periphery. For each point in the down-sampled image, combining the weight value of the current down-sampled image B (X, Y) and the surrounding blocks B (X-1, Y-1), B (X, Y-1) … … B (X +1, Y +1) and the function value f (X ', Y') of the corresponding coordinates (X ', Y') of the curved surface to obtain a second weight gradually radiating from the center to the peripheryAnd the resolution of the second weight gradient graph is consistent with that of the original image.
Referring to fig. 5, for each point in the down-sampled image, substituting coordinates of left B (X, Y) of the current point of the current down-sampled image and surrounding blocks B (X-1, Y-1), B (X, Y-1) … … B (X +1, Y +1) into a surface graph formed by a two-dimensional gaussian function as shown in fig. 5, calculating corresponding interpolation weights of the current point at each coordinate in the up-sampled image, and calculating an up-sampling weight of the current point according to the interpolation weights; and normalizing the upsampling weights of the pixel points at the same position in the plurality of images to obtain the upsampling fusion weight.
In some embodiments, the step of obtaining a one-dimensional weight calculation gaussian function image according to the upsampled fusion weight and the original image in step S4 includes:
after the up-sampling fusion weight is obtained by performing up-sampling processing on the down-sampling fusion weight through the bilinear interpolation method or the two-dimensional Gaussian function, the fusion image is obtained through weighted fusion after the up-sampling fusion weight of a plurality of original images is obtained, and the specific formula is as follows:
ImgHDR＝w1×Img1+w2×Img2+…+wN×ImgN
wherein imgdhdr is a fused image, Img1 is a first original image, Img2 is a second original image, …, ImgN is an nth original image, and N is a positive integer; w1 is the upsampled fusion weight corresponding to the first original image Img1, w2 is the upsampled fusion weight corresponding to the second original image Img2, …, wN is the upsampled fusion weight corresponding to the nth original image ImN, and w1+ w2+ … + wN is 1.
The embodiment of the invention adopts a bilinear interpolation method and a two-dimensional Gaussian function to carry out up-sampling on the down-sampling fusion weight so as to obtain the up-sampling fusion weight, and provides a plurality of calculation ways for calculating the up-sampling fusion weight, thereby improving the applicability of the image fusion method, saving algorithm steps and improving the up-sampling efficiency.
It can be said that, for the processing of the original images of the same group, the original images of the same group only adopt one of the bilinear interpolation method and the two-dimensional gaussian function to calculate the upsampling fusion weight, that is, the two methods of the bilinear interpolation method and the two-dimensional gaussian function cannot be adopted simultaneously in the original images of the same group to calculate the upsampling fusion weight, so as to reduce the error of weight calculation and avoid the problem of unnatural image fusion caused by adopting different weight calculation methods.
The invention also provides an image fusion device, and fig. 6 is a structural block diagram of the image fusion device of the invention.
In some embodiments, referring to fig. 6, the image fusion apparatus includes:
the image processing module 1 is configured to provide an original image, acquire a histogram of the original image and luminance information in the original image, and acquire a down-sampling image according to the luminance information and the original image;
the adjusting module 2 is used for extracting the features of the histogram to obtain histogram calculation parameters, adjusting the histogram calculation parameters and determining a fusion weight calculation formula;
the weight calculation module 3 is configured to obtain a downsampling fusion weight according to the pixel brightness of the downsampling image and the fusion weight calculation formula, and obtain an upsampling fusion weight according to the downsampling fusion weight, so that the resolution of the image corresponding to the upsampling fusion weight is consistent with the resolution of the original image;
and the image fusion calculation module 4 is used for obtaining a fusion image according to the upsampling fusion weight and the original image.
The image fusion device of the invention has the advantages that: acquiring a histogram and brightness information of the original image through the image processing module 1, acquiring a down-sampling image according to the brightness information and the original image, acquiring a histogram calculation parameter through the adjusting module 2, adjusting and determining a fusion weight calculation formula according to the histogram calculation parameter, acquiring a down-sampling fusion weight and an up-sampling fusion weight through the weight calculation module 3, and acquiring a fusion image according to the up-sampling fusion weight and the original image through the image fusion calculation module 4; the statistical information of the histogram is fully utilized, the applicability of the fusion weight calculation formula is improved, and the application scenes of the image fusion method are enlarged; by calculating the down-sampling image, the down-sampling fusion weight and the up-sampling fusion weight, the whole image is combined with the local part, the effects of keeping highlight and dark details of the image, reducing noise and enabling the image to transit naturally are achieved, and the quality of the fusion image is improved; the method solves the problems that the introduction of calculation in the HDR image fusion process can cause the loss of details in highlight or dark places and unnatural transition.
In some embodiments, referring to fig. 6, the step of obtaining the downsampled fusion weight according to the pixel luminance of the downsampled image and the fusion weight calculation formula includes:
respectively acquiring the histograms of the original images in each scene within different exposure time in a plurality of different scenes through an image processing module 1;
presetting a weight curve through an adjusting module 2, wherein a first slope of the weight curve on a first definition interval is less than or equal to 0 and is monotonically decreased, a second slope of the weight curve on a second definition interval is less than or equal to 0 and is monotonically increased, a right end point of the first definition interval is connected with a left end point of the second definition interval, a point where the first slope is changed from 0 to non-0 is recorded as a first inflection point, a point where the second slope is changed from non-0 to 0 is recorded as a second inflection point, and at least one of the shape of the weight curve, the first slope, the second slope, the coordinate of the first inflection point and the coordinate of the second inflection point is adjusted according to the histogram calculation parameter to obtain the weight calculation curve;
dividing the original image into mxn blocks through the image processing module 1, wherein m and n are positive integers, calculating the average brightness value of all pixel points in each block, and replacing the brightness of all pixel points in each block with the average brightness value corresponding to the block to obtain the down-sampling image with the resolution of mxn;
traversing all pixel points of the down-sampling image through a weight calculation module 3 to obtain the brightness of the pixel points, inputting the brightness of the pixel points into the weight calculation curve to obtain a first weight result, and performing normalization processing on the first weight result to obtain the first exposure image weight and the second exposure image weight, wherein the sum of the first exposure image weight and the second exposure image weight is 1.
In further embodiments of the present invention, referring to fig. 6, the step of obtaining the down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula includes:
presetting a one-dimensional Gaussian function through the adjusting module 2, and adjusting the one-dimensional Gaussian function according to the histogram calculation parameters to obtain the one-dimensional weight calculation Gaussian function;
dividing the original image into m × n blocks by the image processing module 1, where m and n are positive integers, and obtaining a corresponding down-sampled image with a resolution of m × n from each original image;
the brightness of each pixel in the down-sampling image is input to the one-dimensional weight calculation gaussian function through the weight calculation module 3 to obtain a second weight result, and the second weight result is normalized to obtain a first weight, a second weight, an.
In some embodiments of the present invention, referring to fig. 6, the weight calculation module 3 performs upsampling on the downsampled fusion weight by a bilinear difference method to obtain the upsampled fusion weight.
In other embodiments of the present invention, referring to fig. 6, the weight calculation module 3 performs an upsampling process on the downsampled fusion weight through a two-dimensional gaussian function to obtain the upsampled fusion weight.
The invention also provides a device comprising a memory, a processor and a program stored on the memory and executable on the processor, the processor implementing the image fusion method when executing the program.
The present invention also provides a storage medium having stored thereon a program which, when executed by a processor, implements the image fusion method.
The advantages of the device and the storage medium of the invention are: the fusion weight calculation formula is adjusted and determined through the histogram, the statistical information of the histogram is fully utilized, the applicability of the fusion weight calculation formula is improved, the application scenes of the image fusion method are expanded, the down-sampling image, the down-sampling fusion weight and the up-sampling fusion weight are calculated, so that the whole image is combined with the local part, the effects of keeping highlight and dark details of the image, reducing noise and enabling the image to be transited naturally are achieved, the quality of the fusion image is improved, and the problems that the highlight or dark details are lost and the transition is unnatural due to the fact that calculation is introduced in the HDR image fusion process are solved.
Although the embodiments of the present invention have been described in detail hereinabove, it is apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it is to be understood that such modifications and variations are within the scope and spirit of the present invention as set forth in the following claims. Moreover, the invention as described herein is capable of other embodiments and of being practiced or of being carried out in various ways.
Claims (13)
1. An image fusion method, comprising the steps of:
providing an original image, and acquiring a histogram of the original image;
extracting the features of the histogram to obtain histogram calculation parameters, and adjusting and determining a fusion weight calculation formula according to the histogram calculation parameters;
acquiring brightness information in the original image, acquiring a down-sampling image according to the brightness information and the original image, and acquiring a down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula;
acquiring an up-sampling fusion weight according to the down-sampling fusion weight, so that the resolution of the image corresponding to the up-sampling fusion weight is consistent with the resolution of the original image;
and obtaining a fused image according to the up-sampling fusion weight and the original image.
2. The image fusion method of claim 1, wherein the step of obtaining a histogram of the original image comprises:
in a plurality of different scenes, respectively acquiring an original image in each scene within different exposure time, and acquiring a histogram of the original image.
3. The image fusion method of claim 1, wherein the histogram calculation parameters include a median, an integral, and a mean of the histogram.
4. The image fusion method of claim 1, wherein the fusion weight calculation formula comprises a weight calculation curve, and the step of adjusting and determining the fusion weight calculation formula according to the histogram calculation parameters comprises:
presetting a weight curve, wherein a first slope of the weight curve on a first defined interval is less than or equal to 0 and is monotonically decreased, a second slope of the weight curve on a second defined interval is less than or equal to 0 and is monotonically increased, and a right endpoint of the first defined interval is connected with a left endpoint of the second defined interval;
recording a point of the first slope changing from 0 to non-0 as a first inflection point, and recording a point of the second slope changing from non-0 to 0 as a second inflection point;
adjusting at least one of the shape of the weight curve, the first slope, the second slope, the coordinate of the first inflection point, and the coordinate of the second inflection point according to the histogram calculation parameter to obtain the weight calculation curve.
5. The image fusion method of claim 4, wherein the downsampling fusion weight includes a first exposure image weight and a second exposure image weight, and the step of obtaining the downsampling fusion weight according to the pixel luminance of the downsampling image and the fusion weight calculation formula comprises:
traversing all pixel points of a down-sampling image to obtain the brightness of the pixel points, inputting the brightness of the pixel points into the weight calculation curve to obtain a first weight result, and carrying out normalization processing on the first weight result to obtain a first exposure image weight and a second exposure image weight, wherein the sum of the first exposure image weight and the second exposure image weight is 1;
the first exposure image weight corresponds to a first exposure image, the second exposure image weight corresponds to a second exposure image, and the exposure time of the first exposure image is longer than that of the second exposure image.
6. The image fusion method of claim 1, wherein the fusion weight calculation formula comprises a one-dimensional weight calculation gaussian function, and the step of adjusting and determining the calculation formula of the fusion weight according to the histogram calculation parameter comprises:
presetting a one-dimensional Gaussian function, and adjusting the one-dimensional Gaussian function according to the histogram calculation parameters to obtain the one-dimensional weight calculation Gaussian function.
7. The image fusion method of claim 6, wherein the step of obtaining the downsampled fusion weight according to the pixel intensity of the downsampled image and the fusion weight calculation formula comprises:
and inputting the brightness of each pixel in the downsampled image into the one-dimensional weight calculation Gaussian function to obtain a second weight result, and performing normalization processing on the second weight result to obtain the downsampled fusion weight, wherein the downsampled fusion weight comprises N weights, and N is a positive integer.
8. The image fusion method of claim 1, wherein the step of obtaining a down-sampled image from the luminance information and the original image comprises:
dividing the original image into m × n blocks, wherein m and n are positive integers, calculating the average brightness value of all pixel points in each block, and replacing the brightness of all pixel points in each block with the average brightness value corresponding to the block to obtain the down-sampling image with the resolution of m × n.
9. The image fusion method of claim 1, wherein obtaining upsampled fusion weights based on the downsampled fusion weights comprises the steps of:
and performing up-sampling processing on the down-sampling fusion weight through a bilinear interpolation method to obtain the up-sampling fusion weight.
10. The image fusion method of claim 1, wherein obtaining upsampled fusion weights based on the downsampled fusion weights comprises the steps of:
and performing upsampling processing on the downsampling fusion weight through a two-dimensional Gaussian function to obtain the upsampling fusion weight.
11. An image fusion apparatus, comprising:
the image processing module is used for providing an original image, acquiring a histogram and brightness information of the original image, and acquiring a down-sampling image according to the brightness information and the original image;
the adjusting module is used for extracting the features of the histogram to obtain histogram calculation parameters, adjusting the histogram calculation parameters and determining a fusion weight calculation formula;
the weight calculation module is used for acquiring a down-sampling fusion weight according to the pixel brightness of the down-sampling image and the fusion weight calculation formula, and acquiring an up-sampling fusion weight according to the down-sampling fusion weight, so that the resolution of the image corresponding to the up-sampling fusion weight is consistent with the resolution of the original image;
and the image fusion calculation module is used for obtaining a fusion image according to the up-sampling fusion weight and the original image.
12. An apparatus comprising a memory, a processor and a program stored on the memory and executable on the processor, wherein the processor implements the image fusion method of any one of claims 1-10 when executing the program.
13. A storage medium on which a program is stored, the program realizing the image fusion method according to any one of claims 1 to 10 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111525658.0A CN114187213A (en) | 2021-12-14 | 2021-12-14 | Image fusion method and device, equipment and storage medium thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111525658.0A CN114187213A (en) | 2021-12-14 | 2021-12-14 | Image fusion method and device, equipment and storage medium thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114187213A true CN114187213A (en) | 2022-03-15 |
Family
ID=80543748
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111525658.0A Pending CN114187213A (en) | 2021-12-14 | 2021-12-14 | Image fusion method and device, equipment and storage medium thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114187213A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108154474A (en) * | 2017-12-22 | 2018-06-12 | 浙江大华技术股份有限公司 | A kind of super-resolution image reconstruction method, device, medium and equipment |
| CN109636767A (en) * | 2018-11-30 | 2019-04-16 | 深圳市华星光电半导体显示技术有限公司 | More exposure image fusion methods |
| US20190311223A1 (en) * | 2017-03-13 | 2019-10-10 | Beijing Sensetime Technology Development Co., Ltd. | Image processing methods and apparatus, and electronic devices |
| CN112837254A (en) * | 2021-02-25 | 2021-05-25 | 普联技术有限公司 | Image fusion method and device, terminal equipment and storage medium |
| CN113313645A (en) * | 2021-05-26 | 2021-08-27 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, terminal, and readable storage medium |
| US20210342629A1 (en) * | 2019-06-14 | 2021-11-04 | Tencent Technology (Shenzhen) Company Limited | Image processing method, apparatus, and device, and storage medium |
-
2021
- 2021-12-14 CN CN202111525658.0A patent/CN114187213A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190311223A1 (en) * | 2017-03-13 | 2019-10-10 | Beijing Sensetime Technology Development Co., Ltd. | Image processing methods and apparatus, and electronic devices |
| CN108154474A (en) * | 2017-12-22 | 2018-06-12 | 浙江大华技术股份有限公司 | A kind of super-resolution image reconstruction method, device, medium and equipment |
| CN109636767A (en) * | 2018-11-30 | 2019-04-16 | 深圳市华星光电半导体显示技术有限公司 | More exposure image fusion methods |
| US20210342629A1 (en) * | 2019-06-14 | 2021-11-04 | Tencent Technology (Shenzhen) Company Limited | Image processing method, apparatus, and device, and storage medium |
| CN112837254A (en) * | 2021-02-25 | 2021-05-25 | 普联技术有限公司 | Image fusion method and device, terminal equipment and storage medium |
| CN113313645A (en) * | 2021-05-26 | 2021-08-27 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, terminal, and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10339643B2 (en) | Algorithm and device for image processing | |
| US10580120B2 (en) | Image filtering based on image gradients | |
| US7149355B2 (en) | Image processing apparatus, image processing method, image processing program, and computer-readable record medium storing image processing program | |
| US20230080693A1 (en) | Image processing method, electronic device and readable storage medium | |
| US10410327B2 (en) | Shallow depth of field rendering | |
| CN109767413B (en) | HDR method and device for resisting motion artifacts and portable terminal | |
| KR20170031033A (en) | Methods, systems and apparatus for over-exposure correction | |
| CN111260580B (en) | Image denoising method, computer device and computer readable storage medium | |
| CN110136055B (en) | Super resolution method and device for image, storage medium and electronic device | |
| US7418130B2 (en) | Edge-sensitive denoising and color interpolation of digital images | |
| US20180047176A1 (en) | Image processing device and method, and recording medium | |
| CN113781320A (en) | Image processing method and device, terminal equipment and storage medium | |
| Park et al. | High dynamic range and super-resolution imaging from a single image | |
| US11741576B2 (en) | Image system including image signal processor and operation method of image signal processor | |
| CN113379609A (en) | Image processing method, storage medium and terminal equipment | |
| Peng et al. | Efficient image resolution enhancement using edge-directed unsharp masking sharpening for real-time ASIC applications | |
| CN112801879B (en) | Image super-resolution reconstruction method and device, electronic equipment and storage medium | |
| US8687912B2 (en) | Adaptive overshoot control for image sharpening | |
| CN115004220A (en) | Neural network for raw low-light image enhancement | |
| CN110852947B (en) | Infrared image super-resolution method based on edge sharpening | |
| Leitao et al. | Content-adaptive video up-scaling for high definition displays | |
| CN116523787A (en) | Low-illumination image enhancement method based on LAB space and multi-scale feature pyramid | |
| JP4664938B2 (en) | Image processing apparatus, image processing method, and program | |
| CN114187213A (en) | Image fusion method and device, equipment and storage medium thereof | |
| CN113781345A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |