WO2023155324A1

WO2023155324A1 - Image fusion method and apparatus, device and storage medium

Info

Publication number: WO2023155324A1
Application number: PCT/CN2022/094865
Authority: WO
Inventors: 赵尧; 于洪英; 闫奇; 顾建超
Original assignee: 烟台艾睿光电科技有限公司
Priority date: 2022-02-21
Filing date: 2022-05-25
Publication date: 2023-08-24
Also published as: CN114519808A

Abstract

Disclosed in embodiments of the present invention are an image fusion method and apparatus, an image processing device and a computer-readable storage medium. The image fusion method comprises: acquiring a visible light image and an infrared image synchronously captured for a target field of view; binarizing the infrared image to obtain a mask image, and determining a target fusion area according to the mask image; and fusing the infrared image and the visible light image on the basis of the target fusion area to obtain a fused image. By means of the determination of the target fusion area, the valid information part contained in the infrared image an be extracted, and the extracted valid information and the visible light image are fused, so that the situation that image quality is reduced because the fused image contains useless information can be effectively avoided, invalid information in the image is reduced, the amount of calculation and complexity are reduced, and the real-time performance of a system can be improved.

Description

Image fusion method, device and equipment, storage medium

technical field

The present invention relates to the technical field of image processing, in particular to an image fusion method and device, image processing equipment, and a computer-readable storage medium.

Background technique

Images are mainly divided into visible light images and infrared images. Visible light images are rich in high-frequency details, which can better reflect the overall details of the shooting scene. The background of the environment becomes blurred; while the principle of infrared imaging is mainly to display the shape and outline of the object through the thermal radiation intensity of the object, which has a good adaptability to weather and light, especially for hidden heat source targets, such as camouflaged enemies, Military targets such as weapons have good detectability, but infrared imaging has problems such as blurred image details, insufficient texture, less high-frequency information of the scene, poor contrast, and low definition.

In order to take into account the respective advantages of visible light images and infrared images, the visible light images and infrared images are fused in the application scene to obtain a comprehensive and accurate image description of the shooting scene, to achieve full use of information, and to improve system analysis and decision-making accuracy and reliability.

Image fusion methods that fuse visible light images and infrared images are mainly divided into three categories according to the complexity of information processing in the fusion process: pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion is the process of operating the pixels of an image to obtain a fused image. The advantage is that it retains more information contained in the original image. The disadvantage is that it needs to traverse, analyze and calculate the image pixel information, and the amount of data calculation and complexity is large. , The real-time performance of the system is low. Feature-level image fusion is to extract edge, shape, texture, pixel density and other feature information from the image to be fused, and then form a multi-dimensional vector space according to these extracted features, and then analyze and process the feature vector in the vector space to form The feature set of the image is then trained and the image to be fused is fused according to the training result. At present, the algorithm of artificial neural network is mostly used in feature-level image fusion, which has the advantages of fast processing speed and small amount of calculation; the disadvantage is that there is more information loss and higher requirements for the operating system. Decision-level image fusion is to first perform feature extraction, target feature recognition, and decision-making classification on the image to be fused, establish a preliminary judgment for the same target, and then fuse the decision information of visible light images and infrared images according to the fusion rules in terms of credibility. Get the result of a joint judgment. At present, decision-level fusion methods mainly include fusion algorithms based on support vector machines, neural networks, evidential reasoning, Bayesian reasoning, and fuzzy integrals, which are complex and require higher operating systems.

technical problem

In order to solve the existing technical problems, the embodiment of the present invention provides an image fusion method, device, image processing equipment, and computer-based image fusion method that can reduce invalid information in the image, reduce the amount of calculation and complexity, and improve the real-time performance of the system. Read storage media.

technical solution

In the first aspect of the embodiments of the present invention, an image fusion method is provided, which is applied to an image processing device, including:

Acquire visible light images and infrared images synchronously collected for the target field of view;

Binarizing the infrared image to obtain a mask image, and determining a target fusion area according to the mask image;

The infrared image is fused with the visible light image based on the target fusion area to obtain a fused image.

Wherein, the fusion of the infrared image based on the target fusion area and the visible light image to obtain a fusion image includes:

Channel-separating the infrared image and the visible light image respectively, and fusing the separated two brightness channel components representing image brightness according to the target fusion area to obtain a brightness channel fusion image;

The fused image of the brightness channel and the visible light image are fused to obtain a fused image.

Wherein, the binarization of the infrared image to obtain a mask image, and determining the target fusion area according to the mask image includes:

Comparing the gray value of each pixel in the infrared image with the binarization threshold, the gray value of the pixel whose gray value is less than the binarization threshold is set to the first set value, and the gray value is greater than or The grayscale value of the pixel point equal to the binarization threshold is set to a second set value to obtain a mask image;

Selecting at least a part of the pixel point distribution area of the second set value in the mask image as a target fusion area.

Wherein, before comparing the gray value of each pixel in the infrared image with the binarization threshold, it includes:

According to the gray histogram of the infrared image and the distribution characteristics of the average gradient, determine a matching binarization strategy;

The binarization threshold is determined according to the binarization strategy.

Wherein, according to the distribution characteristics of the grayscale histogram and the average gradient of the infrared image, the matching binarization strategy is determined, including:

According to the distribution characteristics of the gray histogram of the infrared image, it is judged whether the gray histogram is in a unimodal distribution;

If so, determine that the matching binarization strategy is the triangle method;

If not, according to the comparison result of the average gradient of the infrared image and the average gradient of the visible light image, it is determined that the matching binarization strategy is the Gaussian method or the Otsu method.

Determining the difference between the mode of the gray value of the infrared image and the average value of the gray value, if the difference is less than or equal to a preset value, it is determined that the gray histogram of the infrared image is in a unimodal distribution, then Determine that the matching binarization strategy is the triangle method;

At this time, the determination of the binarization threshold according to the binarization strategy is specifically:

Determining a triangle with the largest peak in the grayscale histogram as the apex;

The maximum straight-line distance is determined through the triangle, and the binarization threshold is determined according to the gray level of the histogram corresponding to the maximum straight-line distance.

Wherein, said according to the distribution characteristics of the grayscale histogram and average gradient of said infrared image, determine the binarization strategy of matching, also include:

If the difference is greater than the preset value, determine the first average gradient of the infrared image and the second average gradient of the visible light image;

If the second average gradient is greater than or equal to the first average gradient, then determine that the matching binarization strategy is the Gaussian method;

In this case, the determining the binarization threshold according to the binarization strategy specifically includes: calculating a Gaussian mean of the grayscale values of the infrared image within the target window function, and determining the binarization threshold according to the Gaussian mean.

If the second average gradient is smaller than the first average gradient, it is determined that the matching binarization strategy is the Otsu method;

Segmenting the infrared image into a foreground image and a background image;

A binarization threshold is determined according to the inter-class variance value of the foreground image and the background image.

Wherein, the channel separation of the infrared image and the visible light image is performed, and the separated brightness channel component representing the brightness of the image is fused according to the target fusion area to obtain a brightness channel fusion image, including:

The infrared image and the visible light image are respectively subjected to HSI channel separation, and the two separated I channel components are fused according to the Poisson image editing principle according to the target fusion area to obtain an I channel fusion image;

The merging of the luminance channel fused image and the visible light image to obtain a fused image includes:

Merging the I channel component of the I channel fused image with the H channel component and the S channel component separated from the visible light image to obtain a fused reference image;

Convert the fused reference image to RGB color space to obtain a fused image.

In a second aspect, an image fusion device is also provided, including:

An acquisition module, configured to acquire visible light images and infrared images synchronously collected for the target field of view;

A fusion area determination module, configured to binarize the infrared image to obtain a mask image, and determine a target fusion area according to the mask image;

A fusion module, configured to fuse the infrared image with the visible light image based on the target fusion area to obtain a fusion image.

Wherein, the fusion module is specifically used to separate the channels of the infrared image and the visible light image, and fuse the separated brightness channel component representing the brightness of the image according to the target fusion area to obtain a brightness channel fusion image; The fused image of the brightness channel and the visible light image are fused to obtain a fused image.

Wherein, the fusion area determination module is specifically used to compare the gray value of each pixel in the infrared image with the binarization threshold, and the gray value of the pixel whose gray value is smaller than the binarization threshold Set the first set value, and set the gray value of the pixel point whose gray value is greater than or equal to the binarization threshold to the second set value to obtain a mask image; select the second set value in the mask image At least a part of the fixed-value pixel point distribution area is used as the target fusion area.

Wherein, the fusion region determination module is further configured to determine a matching binarization strategy according to the gray histogram of the infrared image and the distribution characteristics of the average gradient; determine the binarization threshold according to the binarization strategy.

Wherein, the fusion area determination module is also used to judge whether the gray histogram is a unimodal distribution according to the distribution characteristics of the gray histogram of the infrared image; if so, determine that the matching binarization strategy is a triangle If not, according to the comparison result of the average gradient of the infrared image and the average gradient of the visible light image, it is determined that the matching binarization strategy is the Gaussian method or the Otsu method.

Wherein, the fusion region determination module is also used to determine the difference between the mode of the gray value of the infrared image and the average value of the gray value, if the difference is less than or equal to a preset value, determine the infrared The gray histogram of the image is a unimodal distribution; the largest peak in the gray histogram is used as the apex to determine the triangle; the maximum straight-line distance is determined by the triangle, and the two are determined according to the gray level of the histogram corresponding to the maximum straight-line distance. Value threshold.

Wherein, the fusion region determination module is further configured to determine the first average gradient of the infrared image and the second average gradient of the visible light image if the difference is greater than the preset value; if the second The average gradient is greater than or equal to the first average gradient, the Gaussian mean value of the gray value of the infrared image within the target window function is calculated, and the binarization threshold is determined according to the Gaussian mean value.

Wherein, the fusion region determination module is further configured to segment the infrared image into a foreground image and a background image if the second average gradient is smaller than the first average gradient; according to the foreground image and the background image The between-class variance value of , and determine the binarization threshold.

Wherein, the fusion module is also used to separately perform HSI channel separation on the infrared image and the visible light image, and fuse the separated two I-channel components according to the Poisson image editing principle according to the target fusion area, Obtain the I channel fused image; merge the I channel component of the I channel fused image with the H and S channel components separated from the visible light image to obtain a fused reference image; convert the fused reference image to RGB color space to obtain Blend images.

In the third aspect, there is also provided an image processing device, including a processor, a memory connected to the processor, and a computer program stored in the memory and executable by the processor, the computer program being executed by the When executed by the processor, the steps of the image fusion method described in any embodiment of the present application are realized.

In a fourth aspect, a computer-readable storage medium is provided, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the image fusion method described in any embodiment of the present application are implemented. .

Beneficial effect

In the image fusion method provided by the above embodiment, the mask image is obtained by binarizing the infrared image, the target fusion area is determined according to the mask image, and the infrared image is fused with the visible light image based on the target fusion area to obtain a fusion image In this way, through the determination of the target fusion area, the part of the effective information contained in the infrared image can be extracted, and the extracted effective information can be fused with the visible light image, which can effectively prevent the fused image from containing useless information and reduce the image quality. , reducing invalid information in the image, reducing the amount of calculation and complexity, and improving the real-time performance of the system.

In the above-mentioned embodiments, the image fusion device, image processing equipment, and computer-readable storage medium belong to the same concept as the corresponding image fusion method embodiments, and thus have the same technical effects as the corresponding image fusion method embodiments, and are not repeated here. repeat.

Description of drawings

Fig. 1 is a schematic diagram of an application scene of an image fusion method in an embodiment;

Fig. 2 is a flowchart of an image fusion method in an embodiment;

Fig. 3 is the flowchart of image fusion method in another embodiment;

Fig. 4 is the flowchart of image fusion method in another embodiment;

Fig. 5 is a schematic diagram of a grayscale histogram of an example mid-infrared image;

Fig. 6 is a schematic diagram of grayscale histogram data in an example in a unimodal distribution;

Figure 7 is a schematic diagram of the comparison of the gray histogram of the infrared image showing a unimodal distribution, using the triangle method, the Gaussian method and the Otsu method for fusion;

Figure 8 is a schematic diagram of the comparison of the gray histogram of the infrared image, which is roughly uniformly distributed, and which is fused using the triangle method, the Gaussian method, and the Otsu method;

Figure 9 is a schematic diagram of the comparison of the gray histogram of the infrared image showing a bimodal distribution, using the triangle method, the Gaussian method and the Otsu method for fusion;

FIG. 10 is a flowchart of an image fusion method in an optional specific example;

Fig. 11 is a schematic diagram of an infrared image used in the embodiment shown in Fig. 10;

Fig. 12 is a schematic diagram of a visible light image used in the embodiment shown in Fig. 10;

13 is a schematic diagram of a fusion image obtained after fusion of an infrared image and a visible light image using the image fusion method described in the present application;

FIG. 14 is a schematic diagram of a fusion image obtained by fusing an infrared image and a visible light image using a known low-rank representation principle;

FIG. 15 is a schematic diagram of a fusion image obtained by fusing an infrared image and a visible light image using a known non-subsampling shearlet transform principle;

FIG. 16 is a schematic diagram of a fusion image obtained by fusing an infrared image and a visible light image using a known non-subsampling contourlet transform principle;

Fig. 17 is a schematic diagram of a fusion image obtained by fusing an infrared image and a visible light image using a known Poisson image editing principle;

Fig. 18 is a schematic diagram of an image fusion device in an embodiment;

Fig. 19 is a schematic structural diagram of an image processing device in an embodiment.

Embodiments of the present invention

The technical solutions of the present invention will be further described in detail below in conjunction with the drawings and specific embodiments of the description.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field of the invention. The terms used herein in the description of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the protection scope of the present invention. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

In the following description, the expression "some embodiments" refers to a subset of all possible embodiments, but it should be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments , and can be combined with each other without conflict.

In the following description, the terms "first, second, and third" are only used to distinguish similar objects, and do not represent a specific ordering of objects. Understandably, "first, second, and third" are used in Where permitted, the specific order or sequence may be interchanged such that the embodiments of the application described herein can be practiced in other sequences than illustrated or described herein.

Please refer to FIG. 1, which is a schematic diagram of an optional application scenario of the image processing method provided by the embodiment of the present application, wherein the image processing device 11 includes a processor 12, a memory 13 connected to the processor 12, and a visible light shooting module 14 And infrared camera module 15. The image processing device 11 collects visible light images and infrared images synchronously and in real time through the visible light shooting module 14 and the infrared shooting module 14 and sends them to the processor 12, and the memory 13 stores the images provided by the embodiments of the present application The computer program of the fusion method, the processor 12 executes the computer program, binarizes the infrared image to obtain a mask image, determines the target fusion area through the mask image, and performs the infrared image based on the target fusion area and the visible light image. fusion to obtain a fusion image. Wherein, the image processing device 11 can be various types of intelligent terminals integrated with the visible light shooting module 14 and the infrared shooting module 15, and have storage and processing functions, such as security monitoring equipment, vehicle-mounted equipment, etc.; the image processing device 11 is also It can be a computer device connected to the visible light shooting module 14 and the infrared shooting module 15; the image processing device 11 can also be a dual-light fusion aiming device of white light and red light.

Referring to Fig. 2, the image fusion method provided by an embodiment of the present application can be applied to the image processing device shown in Fig. 1 . Wherein, the image processing method includes the following steps:

S101. Acquire a visible light image and an infrared image that are synchronously collected for a target field of view.

The visible light image and the infrared image are acquired synchronously for the target field of view, so that the visible light image and the infrared image include the imaging of objects in the same target field of view. Optionally, the image processing device includes a visible light shooting module and an infrared shooting module, and the acquiring the visible light image and the infrared image synchronously collected for the target field of view includes: the image processing device simultaneously collects the visible light image and the infrared image through the visible light shooting module and the infrared shooting module image, and send the collected visible light image and infrared image to the processor. In some other optional embodiments, the image processing device does not include an image capturing module, and the acquiring the visible light image and the infrared image synchronously collected for the target field of view includes: the image processing device acquires other images with the functions of capturing visible light images and infrared images Visible light images and infrared images synchronously collected for the target field of view sent by the smart device. Here, other smart devices may include infrared detectors, mobile terminals, and the cloud.

S103. Binarize the infrared image to obtain a mask image, and determine a target fusion area according to the mask image.

The binarization of the infrared image refers to assigning the gray value of each pixel on the infrared image to obtain a binarized image that can reflect the overall and local features of the image.

In an optional embodiment, the step S103 is to binarize the infrared image to obtain a mask image, and determine the target fusion area according to the mask image, including:

The binarization threshold can be preset, or can be calculated according to the distribution characteristics of the gray value of pixels in the infrared image. The first set value and the second set value can be respectively selected from the maximum value and the minimum value of the gray value interval, or can also be two gray values close to the maximum value and the minimum value respectively in the gray value interval. In a specific example, the first set value is 0, and the second set value is 255, so that the entire image presents a black and white image effect. The said binarization of the infrared image to obtain the mask image includes: performing binarization on the grayscale images of 256 brightness levels in the infrared image through the binarization threshold, and converting the grayscale value of each pixel in the infrared image to Compared with the binarization threshold, the gray value of pixels whose gray value is smaller than the binarization threshold is set to 0, and the gray value of pixels whose gray value is greater than the binarization threshold is set to 255, thus obtaining A binary image that reflects the overall and local features of the image, that is, the mask image. Correspondingly, determining the target fusion area according to the mask image may be based on the pixel point distribution area of the second set value in the mask image, that is, the white part determines the target fusion area, such as the white part in the mask image may be selected All as the target fusion area, or select a certain part of the white part in the mask image as the target fusion area.

S105. Fusion the infrared image with the visible light image based on the target fusion area to obtain a fusion image.

Fusing the infrared image based on the target fusion area with the visible light image to obtain a fusion image may refer to merging the infrared image and the visible light image with image parts corresponding to the target fusion area, respectively, Other parts retain the image part of the visible light image to obtain a fused image; or, extract the target fusion area of the infrared image to form an image to be fused, and fuse the image to be fused with the visible light image, etc.

In the above embodiment, the mask image is obtained by binarizing the infrared image, the target fusion area is determined according to the mask image, and the infrared image is fused with the visible light image based on the target fusion area to obtain a fusion image. In this way, through target fusion The determination of the area can extract the effective information contained in the infrared image, and fuse the extracted effective information with the visible light image, which can effectively avoid the image quality degradation caused by the useless information contained in the fused image, and reduce the invalid information in the image , reduce the amount of calculation and complexity, and can improve the real-time performance of the system.

Optionally, please refer to FIG. 3, S105, the infrared image is fused based on the target fusion area with the visible light image to obtain a fused image, including:

S1051. Channel-separate the infrared image and the visible light image respectively, and fuse the separated two brightness channel components representing image brightness according to the target fusion area to obtain a brightness channel fusion image.

For a digital image, it is a picture observed by the human eye, but from the perspective of a computer, a digital image is a bunch of points with different brightness. For example, a digital image with a size of M×N can be represented by a M ×N matrix, the values of the elements in the matrix respectively represent the brightness of the corresponding pixel at this position, and the larger the pixel value, the brighter the pixel. Usually, the grayscale image can be represented by a two-dimensional matrix, and the color image can be represented by a three-dimensional matrix (M×N×3), that is, a multi-channel image.

The hue and color of the image can be changed through the channel. For example, if only the red channel is saved, the image itself only retains red elements and information. For each single channel, it can be displayed as a pair of grayscale images (it should be noted that the grayscale image is not a black and white image), and the lightness and darkness in the grayscale image of a single channel correspond to the lightness and darkness of the color of the single channel, correspondingly representing The distribution of the color/light of the single channel on the image.

Channel-separating the infrared image and the visible light image, and merging the separated luminance channel component representing the brightness of the image according to the target fusion area to obtain the luminance channel fusion image can refer to channel separation of the infrared image and the visible light image, and the infrared image The part corresponding to the target fusion area in the luminance channel component representing the image luminance separated in , is fused with the part corresponding to the target fusion area in the luminance channel component representing image luminance separated from the visible light image, and the luminance channel fusion image is obtained . Wherein, the luminance channel components separated from the infrared image and the visible light image are fused according to the target fusion area, which can reduce the amount of calculation required for fusion, and can retain effective information in the target fusion area.

S1052. Fusion the brightness channel fusion image and the visible light image to obtain a fusion image.

Image fusion (Image Fusion) refers to the image data of the same target collected by multi-source channels through image processing technology to maximize the extraction of beneficial information in each channel, and finally synthesize high-quality images to improve image information. Improve the utilization rate of computer, improve the accuracy and reliability of computer interpretation, and improve the spatial resolution and spectral resolution of the original image, which is beneficial to monitoring. The brightness channel fusion image contains the effective information in the brightness channel component of the visible light image and the brightness channel component of the infrared image, and fuses the brightness channel fusion image and the visible light image, so that the brightness channel component in the brightness channel fusion image and other channels of the visible light image The components are combined to obtain a fused image.

In the above embodiment, the mask image is obtained by binarizing the infrared image, the target fusion area is determined according to the mask image, the channels of the infrared image and the visible light image are separated, and the separated luminance channel component representing the brightness of the image is fused according to the target The region is fused to obtain a luminance channel fusion image, and the luminance channel fusion image and the visible light image are fused to obtain a fusion image. In this way, through the determination of the target fusion area, it is possible to ensure that the part of the effective information contained in the infrared image is extracted. The luminance channel components separated from the infrared image and the visible light image are fused, and then fused with the visible light image, which can effectively prevent the fused image from containing useless information and cause image quality degradation, reduce invalid information in the image, and reduce the amount of calculation and complexity. , and can improve the real-time performance of the system; the fused image can retain the respective advantages of visible light images and infrared images at the same time, whether it is for the fused image obtained after imaging under sufficient lighting conditions or for imaging under poor lighting conditions After the fused image is obtained, the target can be better highlighted, ensuring that the target in the image is more clearly presented, and it is easier for the human eye to observe and identify.

In some embodiments, please refer to FIG. 4, the S103, binarize the infrared image to obtain a mask image, and in the step of determining the target fusion area according to the mask image, in the step of converting the infrared image Before comparing the gray value of each pixel with the binarization threshold, it includes:

S1031. Determine a matching binarization strategy according to the gray histogram of the infrared image and the distribution characteristics of the average gradient;

S1032. Determine the binarization threshold according to the binarization strategy.

Binarizing the infrared image according to the binarization threshold determined by the binarization strategy to obtain a mask image, and determining a target fusion area according to the mask image.

The principles of binarization methods adopted by different binarization strategies are different, and the applicable target images are also different. The distribution characteristics of the gray histogram and average gradient of the infrared image can be judged that the gray histogram of the infrared image is distributed with a single peak, a bimodal distribution or a roughly uniform distribution, so as to determine the matching binarization strategy. For example, the binarization strategy includes triangle method, Gaussian method, and Otsu method. If the gray histogram of the infrared image is distributed with a single peak, the triangle method is applied; if the gray histogram of the infrared image is more uniformly distributed, the Gaussian method is applied. method; if the gray histogram of the infrared image shows a bimodal distribution, the Otsu method is applied. By determining the binarization strategy applicable to the infrared image, the mask image is obtained by binarizing the infrared image through the corresponding binarization strategy, and the area of the white part in the mask image is determined as the target fusion area.

In the above-mentioned embodiment, by analyzing the gray histogram of the infrared image and the distribution characteristics of the average gradient to determine the binarization strategy adapted to it, to ensure that the target in the image can be more accurately located after binarizing the infrared image. The image area is binarized into the white part, so as to determine the target fusion area based on the mask image, and the effective information in the image can be more completely and comprehensively preserved after the brightness channel components separated from the visible light image and the infrared image are fused according to the target fusion area .

The S1031, according to the gray histogram of the infrared image and the distribution characteristics of the average gradient, determine a matching binarization strategy, including:

In the process of determining the binarization strategy suitable for the infrared image, firstly, according to the distribution characteristics of the infrared image histogram, it is judged whether the infrared image is suitable for the triangle method, if not, then the Gaussian method or Otsu method is selected according to the average gradient Law. Judging whether the binarization strategy for binarizing the current infrared image is suitable for the triangle method is to use the gray histogram data of the infrared image to judge whether it is a unimodal distribution, assuming the maximum peak of the gray histogram of the infrared image On the brightest side, use this to find the best threshold for infrared image binarization. If the gray histogram data of the infrared image is used to judge that the unimodal distribution is not satisfied, then the comparison between the average gradient of the infrared image and the average gradient of the visible light image can be used to determine whether the distribution is relatively uniform or bimodal, so as to determine the applicable than Gauss method or Otsu method.

In the above embodiment, the binarization strategy is set to include triangle method, Gaussian method and Otsu method, and the distribution characteristics of the gray histogram and average gradient of the infrared image are analyzed to determine the binarization strategy adapted to it as triangular method, Gaussian method, and Gaussian method. method or Otsu method to ensure that after binarizing the infrared image, the image area where the target in the image is located can be more accurately binarized into a white part, so that after the target fusion area is determined based on the mask image, the target fusion area can be aligned according to the target fusion area. In the luminance component fusion image obtained by fusing the luminance channel components separated from the visible light image and the infrared image, the effective information in the image can be more completely and comprehensively preserved.

In some embodiments, said S1031, according to the distribution characteristics of the grayscale histogram and the average gradient of the infrared image, determine a matching binarization strategy, including:

Determine the difference between the mode of the gray value of the infrared image and the average value of the gray value, if the difference is less than or equal to a preset value, determine that the gray histogram of the infrared image is a unimodal distribution;

According to the grayscale histogram data of the infrared image, if the grayscale values in the grayscale histogram are concentrated on a certain value, then the difference between the grayscale value mode and the grayscale average value corresponding to the infrared image is relative to The gray histogram is much smaller than other forms of infrared images. For example, the absolute value of the difference between the mode of the gray value and the average value of the gray value can be recorded as A-M (average-mode). By judging the gray Whether the difference A-M between the mode of the degree value and the gray-scale average value is less than the preset value is used to judge the degree of the gray histogram showing a unimodal distribution. If the difference A-M is less than the preset value, it means that the corresponding infrared image The gray histogram of the infrared image shows a unimodal distribution, otherwise, it means that the gray histogram corresponding to the infrared image does not meet the characteristics of a unimodal distribution. As shown in Figure 5, use the gray histogram data of the infrared image to find the optimal binarization threshold based on a pure geometric method, assuming that the largest peak in the gray histogram is on the brightest side, and find the largest straight line through the triangle Determine the gray level of the histogram corresponding to the maximum straight-line distance as the segmentation threshold.

As shown in Figure 6, the mode and average value of the gray value in the gray histogram corresponding to the infrared image are 121 and 127.734 respectively, where the mode of the gray value is the gray value with the most repetitions among the gray values , the calculation formula of the gray value AG is as follows: Formula 1:

H(i, j) represents the gray value of the pixel point whose coordinates are (i, j), M represents the maximum value of the abscissa, and N represents the maximum value of the ordinate. Assuming that the preset value is 10, if the difference A-M between the mode of the gray value and the average value of the gray value is less than 10, the infrared image will be binarized using the triangle method, with the largest peak in the gray histogram as the apex A triangle is determined, a maximum straight-line distance is determined through the triangle, and a binarization threshold is determined according to the gray level of the histogram corresponding to the maximum straight-line distance.

In the above embodiment, by calculating the difference between the mode of the gray value in the gray histogram and the average value of the gray value, based on the relative size of the difference and the preset value, it is measured whether the gray value is concentrated in a certain index, To judge whether the grayscale histogram corresponding to the infrared image has a unimodal distribution, so as to realize the fast and accurate judgment of the distribution characteristics of the grayscale histogram of the infrared image.

Please refer to Figure 7. The gray histogram of the infrared image shows a unimodal distribution. The triangle method, Gaussian method and Otsu method are used as the corresponding binarization strategies to binarize the infrared image to obtain the mask image, and the visible light image Schematic diagram of fusion comparison, in which, after using the triangle method as the binarization strategy to determine the binarization threshold, the mask image obtained by binarizing the infrared image can highlight the image target more comprehensively and completely, and finally with The loss of effective image information in the triangular method fused image obtained after visible light image fusion is the smallest.

In some embodiments, the step S1031, according to the distribution characteristics of the grayscale histogram and the average gradient of the infrared image, determining a matching binarization strategy, further includes:

If the second average gradient is greater than or equal to the first average gradient, calculate the Gaussian mean value of the gray value of the infrared image within the target window function, and determine the binarization threshold according to the Gaussian mean value.

If the difference A-M between the mode of the gray value and the average value of the gray value is greater than the preset value, it means that the gray histogram corresponding to the infrared image does not meet the characteristics of a unimodal distribution. By determining the average gradient of the infrared image and the visible light The relative size of the average gradient of the image is used to determine whether the gray histogram corresponding to the infrared image is more uniformly distributed or bimodal. For ease of distinction, the average gradient of the infrared image is referred to as the first average gradient, and the average gradient of the visible light image is referred to as the second average gradient. Wherein, the calculation formula of the average gradient of the image can be as follows formula two:

H(i, j) represents the gray value of the pixel point whose coordinates are (i, j), M represents the maximum value of the abscissa, and N represents the maximum value of the ordinate. If the second average gradient is greater than the first average gradient, it means that the gray histogram is relatively uniformly distributed, and the binarization strategy applicable to the current infrared image is the Gaussian method. The principle of determining the binarization threshold by the Gaussian method is to calculate the Gaussian mean value of the grayscale of the image within the window function, and use the Gaussian mean value as the binarization threshold value to perform binarization operation on the part of the image. The Gaussian method achieves binarization by obtaining local thresholds. By optimizing the scale of the window function, the part to be fused and the part that does not need to be fused in the corresponding window function are determined, so that the gray value in the gray histogram is not concentrated in The infrared image of a certain indicator, that is, the infrared image whose gray value is concentrated on multiple indicators, can binarize the image areas where the targets corresponding to the multiple indicators in the infrared image are respectively located into white in the mask image. Part, so that after the target fusion area is determined based on the white part of the mask image, the luminance component fusion image obtained by fusing the luminance channel components separated from the visible light image and the infrared image according to the target fusion area can retain the image more completely and comprehensively. The effective information in the fused image is strengthened and the edges are highlighted.

In the above-mentioned embodiment, the relative size of the average gradient of the visible light image and the average gradient of the infrared image is used to determine whether the gray value is distributed approximately uniformly, so as to quickly and accurately determine whether the binarization strategy of the infrared image conforms to Gaussian For the case where the average gradient of the visible light image is greater than the average gradient of the infrared image, the visible light image is clear and contains most of the effective information, so the Gaussian method is used to obtain the binarized image to ensure that the infrared image can be binarized more accurately The image area where each target in the image is located is binarized into a white part, and after the target fusion area is determined based on the mask image, the luminance component fusion image obtained by fusing the luminance channel components separated from the visible light image and the infrared image according to the target fusion area In , the effective information in the image can be preserved more completely and comprehensively.

Please refer to Figure 8. The gray histogram of the infrared image is roughly evenly distributed. The triangle method, Gaussian method, and Otsu method are used as the corresponding binarization strategies to binarize the infrared image to obtain the mask image. Schematic diagram of the comparison of image fusion, in which, the mask image obtained by binarizing the infrared image after using the Gaussian method as the binarization strategy to determine the binarization threshold, the target outline is clearer and more prominent, and finally fused with the visible light image The Gaussian method fusion image obtained after the loss of image effective information is the smallest.

In some embodiments, the determining a matching binarization strategy according to the distribution characteristics of the grayscale histogram and the average gradient of the infrared image further includes:

if the second average gradient is smaller than the first average gradient, segmenting the infrared image into a foreground image and a background image;

If the difference A-M between the mode of the gray value and the average value of the gray value is greater than the preset value, it means that the gray histogram corresponding to the infrared image does not meet the characteristics of a unimodal distribution. By determining the average gradient of the infrared image and the visible light The relative size of the average gradient of the image is used to determine whether the gray histogram corresponding to the infrared image is more evenly distributed or bimodal. In order to facilitate the distinction, the average gradient of the infrared image is called the first average gradient, and the average gradient of the visible light image is called the second average gradient. If the second average gradient is smaller than the first average gradient, it means that the gray histogram has a bimodal distribution, and the binarization strategy applicable to the current infrared image is the Otsu method. The principle of determining the binarization threshold by the Otsu method is to divide the image into two parts, the background and the target, according to the grayscale characteristics of the image. The greater the inter-class variance between the background and the target, the greater the difference between the two parts that make up the image. When part of the target is misclassified into the background or part of the background is misclassified into the target, the difference between the two parts will become smaller. Thus, The binarization threshold segmentation based on the maximum variance between classes can minimize the probability of misclassification.

In the above-mentioned embodiment, the relative size of the average gradient of the visible light image and the average gradient of the infrared image is used to judge whether the gray histogram is bimodal, so as to quickly and accurately judge whether the binarization strategy of the infrared image conforms to the Otsu method , the infrared image is divided into a foreground image and a background image by the Otsu method, wherein the foreground image contains main information, such as the energy radiated outward by other heat-generating target objects such as people, and the pixels in the area including the main information in the foreground image and the The difference between the gray values of adjacent pixels is large, so for the case where the average gradient of the infrared image is greater than the average gradient of the visible light image, it is more suitable to determine the area containing effective information according to the foreground image of the infrared image, and obtain the infrared image by the Otsu method. Binarize the image to ensure that the image area where the target in the image is located can be more accurately binarized into a white part after binarizing the infrared image. After determining the target fusion area based on the mask image, the visible light is processed according to the target fusion area. In the luminance component fusion image obtained by fusing the luminance channel components separated from the image and the infrared image, the effective information in the image can be retained more completely and comprehensively.

Please refer to Figure 9. The gray histogram of the infrared image shows a bimodal distribution. The triangle method, Gaussian method, and Otsu method are used as the corresponding binarization strategies to binarize the infrared image to obtain the mask image, and then compare it with the visible light image. Schematic diagram of the fusion comparison. After the Otsu method is used as the binarization strategy to determine the binarization threshold, the mask image obtained by binarizing the infrared image has clearer and more prominent outlines of multiple targets, and is finally compared with the visible light image. The loss of image effective information is the smallest in the fused image obtained by Otsu method after fusion.

In some embodiments, the channel separation of the infrared image and the visible light image is carried out, and the separated brightness channel component representing the brightness of the image is fused according to the target fusion area to obtain a brightness channel fusion image, including:

Convert the fused reference image to RGB color space to obtain a fused image.

Among them, HSI (Hue-Saturation-Intensity (Lightness)) refers to the color model of a digital image, the HSI color model describes the color characteristics of the image with H, S, I three parameters, H defines the frequency of the color, called hue; S Indicates the depth of color, known as saturation; I indicates intensity or brightness.

Optionally, HSI channel separation is performed on the infrared image and the visible light image respectively, and the two I-channel components separated from the target fusion regions corresponding to the infrared image and the visible light image are separated according to the Poisson image The editing principle is used for fusion to obtain the I-channel fused image, which includes: separating the visible light image from the HSI channel, and separating the H-channel component, S-channel component, and I-channel component of the visible light image; according to the gray histogram and average gradient of the infrared image distribution characteristics, determine the matching binarization strategy, binarize the infrared image according to the binarization strategy to obtain a mask image, and determine the target fusion area according to the mask image; perform HSI channel separation on the infrared image , separate the H channel component, S channel component, and I channel component of the infrared image; fuse the I channel component of the infrared image and the I channel component of the visible light image according to the target fusion area determined by the mask image according to the Poisson image editing principle , to get the I channel fusion image. Wherein, if the infrared image and the visible light image are non-HSI format images, before performing HSI channel separation on the infrared image and the visible light image, it also includes converting the infrared image and the visible light image into an infrared image and a visible light image in HSI format.

In the above-mentioned embodiment, by extracting the I channel component of the visible light image to be fused and the I channel component of the infrared image and performing fusion according to the target fusion area specified by the mask image, the obtained I channel fusion image and the visible light image are separated into H After the channel component and the S channel component are combined, they are converted to the RGB color space to obtain the fused image. In this way, by extracting the I channel component of the visible light image and the infrared image respectively, and determining the I channel component in the target fusion area for fusion, both It can ensure that the obtained fusion image can retain the effective information in the visible light image and the infrared image, retain the detailed information contained in the infrared image and the visible light image, and at the same time improve the objective evaluation index of the fusion image, reduce the amount of calculation, and improve the processing of the fusion image efficiency.

In order to have a more overall understanding of the image fusion method provided by the embodiment of the present application, please refer to FIG. 10 to FIG. 12 , and use an optional example as an example to describe the image fusion method.

S11, read the infrared image and the visible light image; as shown in Figure 11 and Figure 12, the infrared image IR_1 and the visible light image VIS_2;

Select the adaptive threshold binarization strategy for infrared image matching to binarize the infrared image; the adaptive threshold binarization strategy includes the triangle method, the Gaussian method, and the Otsu method; the methods for selecting the adaptive binarization strategy include:

S121, according to the grayscale histogram binarization distribution characteristics of the infrared image, calculate the difference A-M value between the grayscale value mode and the grayscale average value of the image grayscale value;

S122, judging whether the A-M value is greater than a preset value;

S123, if the A-M value is less than or equal to the preset value, it means that the gray histogram is in a unimodal distribution, and the infrared image is binarized using the triangle method;

S124, if the A-M value is greater than the preset value, calculate the average gradient AG1 of the infrared image and the average gradient AG2 of the visible light image;

S125, judging whether the difference between AG1 and AG2 is greater than 0;

S126, if the difference between AG1 and AG2 is less than or equal to 0, it means that the gray histogram is more evenly distributed, and the infrared image is binarized by using the Gaussian method;

S127, if the difference between AG1 and AG2 is greater than 0, it means that the gray histogram shows a bimodal distribution, and the infrared image is binarized by the Otsu method;

S13, generating a mask image according to the binarized image after binarization processing, and specifying an area to be fused according to the mask image;

S14, performing HSI channel separation on the infrared image to obtain the H channel component, the S channel component and the I channel component of the infrared image;

S15, performing HSI channel separation on the visible light image to obtain the H channel component, the S channel component and the I channel component of the visible light image;

S16, merging the I channel components of the infrared image and the visible light image according to the Poisson principle in the area to be fused according to the mask image;

S17, merging the I channel of the fused image with the H and S channels of the visible light image;

S18, converting the merged image into an RGB color space to obtain a fusion image.

The image fusion method provided by the foregoing embodiments has at least the following characteristics:

First, by selecting a matching binarization strategy for the infrared image for binarization processing, the region to be fused is determined according to the binarized image; by locking the region to be fused, the fusion calculation amount is reduced and the fusion processing efficiency is improved, and the Effective information in the image;

Second, it provides instructions on how to determine whether the gray histogram is unimodal, relatively uniform, or bimodal according to the distribution characteristics of the gray histogram and average gradient of the infrared image to select a matching binarization strategy. method, thereby ensuring that the effective information and detailed information contained in the infrared image and the visible light image can be retained after the region to be fused is determined based on the binarization result;

Third, after merging the I channel component of the infrared image and the visible light image, they are merged with the H and S channel components of the visible light image to obtain a fused image, which can effectively prevent the fused image from containing useless information and reduce the image quality and reduce the image quality. Invalid information, reducing the amount of calculation and complexity, and can improve the real-time performance of the system, as shown in Figure 13, which is the fused image obtained after using the image fusion method described in this application to fuse infrared images and visible light images, Figure 13 14 Fusion comparison effect of infrared image and visible light image fusion based on known low-rank representation principle, Figure 15 is fusion comparison of infrared image and visible light image fusion based on known non-subsampling shearlet transform principle Effect. Figure 16 shows the fusion and comparison effect of infrared images and visible light images based on the known principle of non-subsampling contourlet transformation. Fusion contrast effect for direct fusion.

The evaluation index comparison of the fused images corresponding to Figure 13 to Figure 17 is shown in Table 1 below:

Among them, IE (Information Entropy) refers to information entropy; SF (Spatial Frequency) refers to spatial frequency; RMSE (Root Mean Sqaured Error) refers to root mean square error; SSIM (Structural Similarity Index) refers to structural similarity index; TIME It refers to the fusion processing time. NSST (Non-subsampled Shearlet Transform) refers to non-subsampled shearlet transform; NSCT (Nonsubsampled contourlet transform) refers to the principle of non-subsampled contourlet transform. Combining the illustrations and Table 1, it can be seen that the fused image obtained by using the image fusion method described in this application to fuse the infrared image and the visible light image has the smallest root mean square error value, the fusion processing time is significantly reduced, and the structural similarity index It is close to 1, and the information entropy and spatial frequency still maintain a relatively large value, and the overall performance of the image performance is obviously better than the fusion image obtained by other fusion methods.

Please refer to FIG. 18 , another aspect of the present application provides an image fusion device, including: an acquisition module 131, used to acquire visible light images and infrared images synchronously collected for the target field of view; a fusion area determination module 132, used for all Binarize the infrared image to obtain a mask image, and determine the target fusion area according to the mask image; the fusion module 134 is used to fuse the infrared image based on the target fusion area and the visible light image to obtain a fusion image .

Optionally, the fusion module 134 is specifically configured to separate the channels of the infrared image and the visible light image, and fuse the separated brightness channel component representing the brightness of the image according to the target fusion area to obtain the brightness channel Fusing images: fusing the luminance channel fusion image and the visible light image to obtain a fusion image.

Optionally, the fusion region determination module 132 is specifically configured to compare the grayscale value of each pixel in the infrared image with a binarization threshold, and the grayscale value of a pixel with a grayscale value smaller than the binarization threshold is The grayscale value is set to the first set value, and the grayscale value of the pixel point whose grayscale value is greater than or equal to the binarization threshold is set to the second set value to obtain a mask image; select the mask image described in At least a part of the pixel point distribution area of the second set value is used as the target fusion area.

Optionally, the fusion region determination module 132 is further configured to determine a matching binarization strategy according to the distribution characteristics of the gray histogram and the average gradient of the infrared image; determine the binarization strategy according to the binarization strategy threshold.

Optionally, the fusion region determination module 132 is further configured to judge whether the gray histogram is a unimodal distribution according to the distribution characteristics of the gray histogram of the infrared image; if so, determine the matching binarization The strategy is the triangle method; if not, according to the comparison result of the average gradient of the infrared image and the average gradient of the visible light image, it is determined that the matching binarization strategy is the Gaussian method or the Otsu method.

Optionally, the fusion region determination module 132 is also used to determine the difference between the mode of the gray value of the infrared image and the average value of the gray value, if the difference is less than or equal to a preset value, determine The grayscale histogram of the infrared image is in a unimodal distribution; a triangle is determined with the largest peak in the grayscale histogram as the apex; the maximum straight-line distance is determined through the triangle, and the grayscale of the histogram corresponding to the maximum straight-line distance Level determines the binarization threshold.

Optionally, the fusion region determining module 132 is further configured to determine the first average gradient of the infrared image and the second average gradient of the visible light image if the difference is greater than the preset value; The second average gradient is greater than or equal to the first average gradient, the Gaussian mean value of the gray value of the infrared image in the target window function is calculated, and the binarization threshold is determined according to the Gaussian mean value.

Optionally, the fusion region determination module 132 is further configured to segment the infrared image into a foreground image and a background image if the second average gradient is smaller than the first average gradient; The inter-class variance value of the background image is used to determine the binarization threshold.

Optionally, the fusion module 134 is further configured to separately perform HSI channel separation on the infrared image and the visible light image, and edit the two separated I-channel components according to the Poisson image editing principle according to the target fusion area. Carrying out fusion to obtain an I channel fusion image; merging the I channel component of the I channel fusion image with the H and S channel components separated from the visible light image to obtain a fusion reference image; converting the fusion reference image to RGB color space to obtain a fused image.

It should be noted that: in the image fusion device provided by the above-mentioned embodiments, in the process of realizing fusion processing of visible light images and infrared images, the division of the above-mentioned program modules is used as an example for illustration. In practical applications, the above-mentioned processing can be allocated according to needs. Completed by different program modules, that is, the internal structure of the device can be divided into different program modules to complete all or part of the method steps described above. In addition, the image fusion device and the image fusion method embodiments provided in the above embodiments belong to the same idea, and the specific implementation process thereof is detailed in the method embodiments, and will not be repeated here.

Another aspect of the present application provides an image processing device. Please refer to FIG. 19 , which is a schematic diagram of an optional hardware structure of the image processing device provided by the embodiment of the present application. The image processing device includes a processor 111, and the processing The memory 112 connected to the device 111, the memory 112 is used to store various types of data to support the operation of the image processing equipment, and stores a computer program for realizing the image processing method provided by any embodiment of the present application, the computer program When executed by the processor, the steps of the image processing method provided by any embodiment of the present application can be realized, and the same technical effect can be achieved. To avoid repetition, details are not repeated here.

Optionally, the image processing device further includes an infrared shooting module and a visible light shooting module connected to the processor 111, and the infrared shooting module and the visible light shooting module are used for synchronously shooting infrared images and visible light images for the same target field of view It is sent to the processor 111 as an image to be fused.

The embodiment of the present application also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, each process of the above-mentioned image processing method embodiment is realized, and the same To avoid repetition, the technical effects will not be repeated here. Wherein, the computer-readable storage medium is, for example, a read-only memory (Read-Only Memory, ROM for short), a random access memory (Random Access Memory, RAM for short), a magnetic disk or an optical disk, and the like.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk, etc.) ) includes several instructions to make a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present invention.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above-mentioned embodiments can be completed by instructing related hardware through computer programs, and the programs can be stored in a non-volatile computer-readable In the storage medium, when the program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any reference to memory, storage, database or other media used in various embodiments of the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. The protection scope of the present invention shall be determined by the protection scope of the claims.

Claims

An image fusion method applied to an image processing device, characterized in that it comprises:

Acquire visible light images and infrared images synchronously collected for the target field of view;

Binarizing the infrared image to obtain a mask image, and determining a target fusion area according to the mask image;

The infrared image is fused with the visible light image based on the target fusion area to obtain a fused image.
The image fusion method according to claim 1, wherein the fusion of the infrared image based on the target fusion area and the visible light image to obtain a fusion image comprises:

Channel-separating the infrared image and the visible light image respectively, and fusing the separated two brightness channel components representing image brightness according to the target fusion area to obtain a brightness channel fusion image;

The fused image of the brightness channel and the visible light image are fused to obtain a fused image.
The image fusion method according to claim 1, wherein said binarizing said infrared image to obtain a mask image, and determining a target fusion area according to said mask image comprises:

Comparing the gray value of each pixel in the infrared image with the binarization threshold, the gray value of the pixel whose gray value is less than the binarization threshold is set to the first set value, and the gray value is greater than or The grayscale value of the pixel point equal to the binarization threshold is set to a second set value to obtain a mask image;

Selecting at least a part of the pixel point distribution area of the second set value in the mask image as a target fusion area.
The image fusion method according to claim 3, wherein before comparing the gray value of each pixel in the infrared image with the binarization threshold, it includes:

According to the gray histogram of the infrared image and the distribution characteristics of the average gradient, determine a matching binarization strategy;

The binarization threshold is determined according to the binarization strategy.
The image fusion method according to claim 3, wherein, determining a matching binarization strategy according to the distribution characteristics of the gray histogram and the average gradient of the infrared image includes:

According to the distribution characteristics of the gray histogram of the infrared image, it is judged whether the gray histogram is in a unimodal distribution;

If so, determine that the matching binarization strategy is the triangle method;

If not, according to the comparison result of the average gradient of the infrared image and the average gradient of the visible light image, it is determined that the matching binarization strategy is the Gaussian method or the Otsu method.
The image fusion method according to claim 4, wherein, the gray histogram according to the infrared image and the distribution characteristics of the average gradient determine the matching binarization strategy, comprising:

Determining the difference between the mode of the gray value of the infrared image and the average value of the gray value, if the difference is less than or equal to a preset value, it is determined that the gray histogram of the infrared image is in a unimodal distribution, then Determine that the matching binarization strategy is the triangle method;

At this time, the determination of the binarization threshold according to the binarization strategy is specifically:

Determining a triangle with the largest peak in the grayscale histogram as the apex;

The maximum straight-line distance is determined through the triangle, and the binarization threshold is determined according to the gray level of the histogram corresponding to the maximum straight-line distance.
The image fusion method according to claim 6, wherein, determining a matching binarization strategy according to the distribution characteristics of the gray histogram and the average gradient of the infrared image, further comprising:

If the difference is greater than the preset value, determine the first average gradient of the infrared image and the second average gradient of the visible light image;

If the second average gradient is greater than or equal to the first average gradient, then determine that the matching binarization strategy is the Gaussian method;

In this case, the determining the binarization threshold according to the binarization strategy specifically includes: calculating a Gaussian mean of the grayscale values of the infrared image within the target window function, and determining the binarization threshold according to the Gaussian mean.
The image fusion method according to claim 7, wherein, determining a matching binarization strategy according to the distribution characteristics of the gray histogram and the average gradient of the infrared image, further comprising:

If the second average gradient is smaller than the first average gradient, it is determined that the matching binarization strategy is the Otsu method;

At this time, the determination of the binarization threshold according to the binarization strategy is specifically:

Segmenting the infrared image into a foreground image and a background image;

A binarization threshold is determined according to the inter-class variance value of the foreground image and the background image.
The image fusion method according to claim 2, wherein the channel separation of the infrared image and the visible light image is carried out, and the separated brightness channel component representing the brightness of the image is fused according to the target fusion area , to obtain the luminance channel fusion image, including:

The infrared image and the visible light image are respectively subjected to HSI channel separation, and the two separated I channel components are fused according to the Poisson image editing principle according to the target fusion area to obtain an I channel fusion image;

The merging of the luminance channel fused image and the visible light image to obtain a fused image includes:

Merging the I channel component of the I channel fused image with the H channel component and the S channel component separated from the visible light image to obtain a fused reference image;

Convert the fused reference image to RGB color space to obtain a fused image.
An image fusion device, characterized in that it comprises:

Obtaining module, be used for obtaining the visible light image and the infrared image that are collected synchronously for target field of view;

A fusion area determination module, configured to binarize the infrared image to obtain a mask image, and determine a target fusion area according to the mask image;

A fusion module, configured to fuse the infrared image with the visible light image based on the target fusion area to obtain a fusion image.
An image processing device, characterized by comprising a processor, a memory connected to the processor, and a computer program stored on the memory and executable by the processor, the computer program being executed by the processor When executed, the image fusion method according to any one of claims 1 to 9 is realized.
A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the image fusion according to any one of claims 1 to 9 is realized method.