WO2024002186A1

WO2024002186A1 - Image fusion method and apparatus, and storage medium

Info

Publication number: WO2024002186A1
Application number: PCT/CN2023/103334
Authority: WO
Inventors: 李如宇; 葛成伟; 施海涛
Original assignee: 中兴通讯股份有限公司
Priority date: 2022-06-28
Filing date: 2023-06-28
Publication date: 2024-01-04
Also published as: CN117350919A

Abstract

Embodiments of the present disclosure relate to the field of image processing, and provide an image fusion method and apparatus, and a storage medium. The method comprises: obtaining a visible light image of a target object, the target object being provided with a preset marked object; obtaining an infrared image of the target object; on the basis of a preset feature extraction algorithm, determining a first position corresponding to the marked object in the infrared image; on the basis of a preset target matching algorithm, determining a second position corresponding to the marked object in the visible light image; and according to the first position and the second position and on the basis of a preset image fusion algorithm, fusing the visible light image and the infrared image to obtain a target image.

Description

Image fusion method, device and storage medium

Cross-references to related applications

This disclosure claims priority to Chinese patent application CN202210742455.5 titled "Image Fusion Method, Device and Storage Medium" submitted on June 28, 2022, the entire content of which is incorporated into this disclosure by reference.

Technical field

The present disclosure relates to the field of image processing technology, and in particular, to an image fusion method, device and storage medium.

Background technique

The visible light image captured by the 2D camera can adapt to the visual habits of the human eye, but is easily affected by occlusion, environmental brightness, etc.; the infrared image captured by the infrared camera can be imaged according to the temperature data in the environment, and will not be affected by occlusion, environment, etc. Brightness interference. Therefore, fusing visible light images with infrared images can combine the advantages of both and provide convenience for production and life. However, there are technical problems with the complicated fusion process when fusing visible light images and infrared images.

Contents of the invention

Embodiments of the present disclosure provide an image fusion method, device and storage medium.

In a first aspect, embodiments of the present disclosure provide an image fusion method, which includes: acquiring a visible light image of a target object, where the target object is set with a preset marker object; acquiring an infrared image of the target object; and determining based on a preset feature extraction algorithm. The first position corresponding to the marked object in the infrared image; based on the preset target matching algorithm, determine the second position corresponding to the marked object in the visible light image; based on the first position and the second position, based on the preset image fusion algorithm, determine the visible light The image and the infrared image are fused to obtain the target image.

In a second aspect, embodiments of the present disclosure also provide an image fusion device. The image fusion device includes a processing memory, a computer program stored in the memory and executable by the processor, and a data bus used to realize connection communication between the processor and the memory, wherein when the computer program is executed by the processor, the implementation of the present disclosure is implemented Any of the image fusion methods provided in the example.

In a third aspect, embodiments of the present disclosure also provide a storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the following: Any image fusion method provided by the embodiments of the present disclosure.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present disclosure, which are of great significance to this field. Ordinary technicians can also obtain other drawings based on these drawings without exerting creative work.

Figure 1 is a schematic flowchart of the steps of an image fusion method provided by an embodiment of the present disclosure;

Figure 2 is a schematic diagram of a scene for implementing the image fusion method provided by an embodiment of the present disclosure;

Figure 3 is a schematic flowchart of sub-steps of an image fusion method provided by an embodiment of the present disclosure;

Figure 4 is a schematic flowchart of sub-steps of an image fusion method provided by an embodiment of the present disclosure;

Figure 5 is a schematic flowchart of sub-steps of an image fusion method provided by an embodiment of the present disclosure;

Figure 6 is a schematic flowchart of sub-steps of an image fusion method provided by an embodiment of the present disclosure; and

FIG. 7 is a schematic structural block diagram of an image fusion device provided by an embodiment of the present disclosure.

Detailed ways

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this disclosure.

The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all contents and operations/steps. Nor do they have to be performed in the order described. For example, some operations/steps can also be decomposed, combined or partially merged, so the actual order of execution may change based on actual conditions.

It should be understood that the terminology used in the description of the disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise.

Visible light images captured by 2D cameras can adapt to human visual habits, but are easily affected by occlusion, environmental brightness, etc. Visible light images captured in environments with low air visibility or insufficient lighting cannot well reflect the environment. Object information; infrared graphics captured by infrared cameras can be imaged according to the temperature data in the environment and will not be interfered by occlusion and ambient brightness. However, infrared images cannot reflect the background information in the environment and do not conform to the visual habits of the human eye. Therefore, fusing visible light images with infrared images can combine the advantages of both and provide convenience for production and life. However, in some cases, pulse coupled neural networks or convolutional neural networks are usually used to fuse visible light images and infrared images. Not only is the process more complicated, but the interpretability and portability are also poor. There is an urgent need for a simpler and more capable method. Widely applicable visible light image and infrared image fusion method.

Embodiments of the present disclosure provide an image fusion method, device and storage medium. Among them, the image fusion method can be applied to mobile terminals, which can be electronic devices such as mobile phones, tablet computers, notebook computers, desktop computers, personal digital assistants, and wearable devices.

Some embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The following embodiments and features in the embodiments may be combined with each other without conflict.

Please refer to FIG. 1 , which is a schematic flow chart of an image fusion method provided by an embodiment of the present disclosure.

As shown in Figure 1, the image fusion method includes steps S101 to S105.

Step S101: Obtain a visible light image of the target object, and the target object is set with a preset marking object.

In an exemplary embodiment, a visible light image of a target object is acquired through a preset 2D camera. The target object may be a production workshop, for example. Multiple 2D cameras are set up in the production workshop to acquire top-down images of the production workshop from multiple angles. The image fusion method provided by the present disclosure generates a target image used to reflect the overall production situation of the production workshop. The image fusion method provided by this disclosure can also be applied to other scenarios, For example, the target object can also be a port, warehouse, etc., which is not limited here.

In an exemplary embodiment, the visible light image may be a photo of the target object captured by a 2D camera, or may be an image frame intercepted from a video of the target object captured by the 2D camera.

In an exemplary embodiment, the target object may be an object with a certain temperature, so that the target object can be significantly distinguished from the background area in the infrared image.

In an exemplary embodiment, the target object can be set according to the actual situation. For example, it can be a container containing hot water. Of course, it is not limited to this. For example, if the target object is a steel production workshop, it can be set in the steel production workshop. A device for placing heated steel balls, with the heated steel balls as the target object, and the target object is not limited here.

In some embodiments, the preset marking objects include multiple marking objects, and the plurality of marking objects are arranged in at least two directions, and at least two directions intersect.

Please refer to FIG. 2 , which is a schematic diagram of a scene for implementing the image fusion method provided by an embodiment of the present disclosure.

In an exemplary embodiment, as shown in Figure 2, four marking areas that intersect at the same position can be set in the target object. The intersection angle between the marking areas is 45°, that is, the marking areas are in the shape of " "meter" shape distribution. Compared with setting mark objects on the target object in the form of a checkerboard at certain distances, the method of setting mark objects provided by the embodiments of the present disclosure can be applied to target objects with a larger area, and has wider applicability, and This reduces the number of tag objects that need to be set and reduces implementation costs.

Step S102: Obtain the infrared image of the target object.

In an exemplary embodiment, the infrared image may be acquired through customer premise equipment (CPE). In an exemplary embodiment, the CPE that accesses the wireless signal or wired broadband signal provided by the operator communicates with the preset infrared camera, and the infrared image captured by the infrared camera is acquired through the CPE.

In an exemplary embodiment, the shooting angle of the infrared camera can also be adjusted through the CPE.

In an exemplary embodiment, the infrared image of the target object is acquired through a preset infrared camera. Similarly, the target object can be a production workshop. Multiple infrared cameras are set up in the production workshop. The target object can also be a port, a warehouse, etc., No limitation is made here.

In an exemplary embodiment, the infrared image may be a photo of the target object captured by an infrared camera, or may be an image frame intercepted from a video of the target object captured by the infrared camera.

In an exemplary embodiment, the order of acquiring the visible light image and the infrared image is not limited here. The visible light image may be acquired first, and then the infrared image may be acquired; the infrared image may be acquired first, and then the visible light image may be acquired; or the visible light image may be acquired first, and then the visible light image may be acquired simultaneously. Visible light images and infrared images, the order in which visible light images and infrared images are acquired is not limited here.

In some implementations, step S102 includes: acquiring an original infrared image of the target object; performing distortion correction on the original infrared image based on a preset distortion correction algorithm to obtain an infrared image.

In an exemplary embodiment, the original infrared image acquired through an infrared camera usually has relatively obvious distortion. In order to improve the quality of the target image, the original infrared image needs to be corrected for distortion.

In an exemplary embodiment, the distortion of the infrared image includes radial distortion and tangential distortion. The radial distortion is caused by the light bending more far away from the center of the lens than near the center. The radial distortion occurs along the lens. Distribution in the radial direction, the distortion at the center of the optical axis of the imager is 0. The farther away from the center of the optical axis of the imager, the more serious the radial distortion; tangential distortion is caused by the inability of the lens and the sensor plane or the image plane to be completely parallel when the camera is assembled. distortion.

Please refer to FIG. 3 , which is a schematic flowchart of sub-steps of an image fusion method provided by an embodiment of the present disclosure.

As shown in Figure 3, in some implementations, distortion correction is performed on the original infrared image based on a preset distortion correction algorithm to obtain an infrared image, including steps S1021 to S1023: Step S1021, projecting the original infrared image to a normalized normalized plane to obtain the original infrared image on the normalized plane; step S1022, based on the preset radial distortion coefficient and the preset tangential distortion coefficient, perform radial distortion correction and correction on the original infrared image on the normalized plane. Tangential distortion is corrected to obtain an infrared image on the normalized plane; step S1023, project the infrared image on the normalized plane to the pixel plane to obtain an infrared image.

In an exemplary embodiment, the radial distortion coefficient and the tangential distortion coefficient can be obtained by calibrating the infrared camera in advance, and the method of determining the radial distortion coefficient and the tangential distortion coefficient will not be described in detail here.

In an exemplary embodiment, the coordinates of the pixels in the original infrared image are x, y, and z. The original infrared image is projected onto the normalized plane to obtain the original infrared image on the normalized plane. The coordinates of the pixels in the original infrared image are x', y', where x'=x/z, y'=y/z.

In an exemplary embodiment, radial distortion correction and tangential distortion correction are performed on the original infrared image on the normalized plane according to the radial distortion coefficients k ₁ and k ₂ and the tangential distortion coefficients p ₁ and p ₂ , There are: x″＝x′·(1+k ₁ ·r ² +k ₂ ·r ⁴ )+2·p ₁ ·x′·y′+p ₂ ·(r ² +2x′ ² ); y″＝ y′·(1+k ₁ ·r ² +k ₂ ·r ⁴ )+2·p ₁ ·x′·y′+p ₂ ·(r ² +2y′ ² ), where r is on the normalized plane The polar coordinates of the pixels in the original infrared image are r ² =x′ ² +y′ ² , x”, y” are the coordinates of the pixels in the infrared image on the normalized plane.

In an exemplary embodiment, the infrared image on the normalized plane is projected onto the pixel plane to obtain the correct position of the pixel point on the infrared image, as follows: u=f _x ·x″+c _x ; v=f _y ·y″+c _y , where u and v are the coordinates of the pixels in the infrared image on the pixel screen, c _x and _cy are the projection parameters of the infrared camera. The projection parameters can be determined through the internal parameter matrix of the infrared camera and will not be described in detail here.

Step S103: Determine the first position corresponding to the marked object in the infrared image based on a preset feature extraction algorithm.

In an exemplary embodiment, since target objects with a certain temperature have higher brightness in infrared images, the first position corresponding to the marked object in the infrared image can be determined through a preset feature extraction algorithm.

In some embodiments, step S103 includes: acquiring the brightness channel image of the infrared image, determining the brightness area position in the infrared image based on the preset brightness threshold; fitting the brightness area position, and determining the corresponding position of the marked object in the infrared image. First position.

In an exemplary embodiment, the infrared image is converted into HSV space, where H, S, and V respectively represent the hue (Hue), saturation (Saturation), and brightness (Value) of the image. The H channel is generally measured in angle, and the value range is 0°-360°; the S channel value range is 0.0-1.0, and when S=0, it means only grayscale; the V channel value range is 0.0-1.0, and when it is 0.0, it means Black, 1.0 means white.

In an exemplary embodiment, after determining the values of the H channel, S channel, and V channel of the infrared image, the V channel value of the infrared image is separated to obtain a brightness channel image, so as to determine the corresponding mark object in the infrared image based on the brightness channel image. first position.

In some embodiments, determining the first position corresponding to the marked object in the infrared image based on a preset brightness threshold includes: binarizing the pixels of the brightness channel image based on the preset brightness threshold; Binarization results determine the location of the brightness area in the infrared image.

In an exemplary embodiment, a brightness threshold capable of extracting the brightness of the marked object from the brightness channel image is determined in advance based on the actual situation, and the pixels of the brightness channel image are binarized based on the preset brightness threshold. In an exemplary embodiment, if the brightness value of a pixel in the brightness channel image is greater than or equal to the brightness threshold, the gray value of the pixel is determined to be 255; if the brightness value of the pixel in the brightness channel image is less than the brightness threshold, the gray value of the pixel is determined to be 255. The gray value of the pixel is determined to be 0, and the binary image corresponding to the brightness channel image is obtained, in which the pixel with a gray value of 255 is the brightness area position.

In an exemplary embodiment, since the marked objects are arranged in at least two directions, that is, the marked objects in the same direction are approximately in a straight line, in order to improve the accuracy of determining the first position, a straight line can be performed on the brightness area position. Fitting and filtering the positions of brightness areas with poor fitting properties to prevent objects with higher temperatures in other areas of the target object from affecting the accuracy of the first position determined in step S103. The method of straight line fitting will not be described in detail here.

In an exemplary embodiment, step S103 may be implemented through OpenCV. In an exemplary embodiment, the cvCvtColor function is called to convert the infrared image into HSV space, the cvSplit function is called to separate the V channel of the infrared image, and the threshold function is called to binarize the brightness channel image according to the preset brightness threshold.

Step S104: Determine the second position corresponding to the marked object in the visible light image based on a preset target matching algorithm.

In an exemplary embodiment, since the target object is provided with a marked object, the corresponding position of the acquired visible light image also contains an image of the marked object. Based on the target matching algorithm, the second position of the marked object in the visible light image can be determined. .

Please refer to FIG. 4 , which is a schematic flowchart of sub-steps of an image fusion method provided by an embodiment of the present disclosure.

As shown in Figure 4, in some embodiments, step S104 includes step S1041-step S1043: Step S1041, based on the preset target matching algorithm, determine the targets of multiple areas in the preset marked object template image and the visible light image Matching degree; step S1042, determine the target area position in the visible light image according to the target matching degree of multiple areas; step S1043, fit the target area position, and determine the second position corresponding to the marked object in the visible light image.

In an exemplary embodiment, the target matching algorithm may be implemented based on image matching technology. exist In an exemplary embodiment, multiple visible light images of the marked object in the target object are obtained in advance as the marked object template image, and matching is performed in the visible light image of the target object according to the marked object template image. For example, use the size of the marker object template image as the figure window size, slide the marker object template image on the visible light image of the target object according to the preset figure window movement step, and determine the targets of multiple areas in the marker object template image and the visible light image. The matching degree is determined, and the area whose target matching degree is greater than the preset matching degree is determined as the target area position, and then the second position corresponding to the marked object in the visible light image is determined.

In an exemplary embodiment, the target matching degree may be determined based on the Euclidean distance between the marked object template image and the multiple areas in the visible light image: the Euclidean distance between the marked object template image and the multiple areas in the visible light image. The greater the distance, the greater the target matching degree. Of course, it is not limited to this, and the target matching degree can also be determined based on other methods, which is not limited here.

In an exemplary embodiment, the marker object template image size is smaller than the visible light image size of the target object.

In an exemplary embodiment, similar to step S103, in order to improve the accuracy of determining the second position, straight line fitting can be performed on the target area position in the visible light image, and the target area positions in the visible light image with poor fitting properties can be filtered. In order to avoid the appearance of objects similar to the marked object in other areas of the target object from affecting the accuracy of the second position determined in step S103, the straight line fitting method will not be described in detail here.

In an exemplary embodiment, step S104 may be implemented through OpenCV. In an exemplary embodiment, the matchTemplate function is called to determine the target matching degree between the marked object template image and multiple areas of the visible light image, and then the cv2.minMaxLoc function is called to determine the position of the target area with the largest target matching degree.

Step S105: According to the first position and the second position, the visible light image and the infrared image are fused based on a preset image fusion algorithm to obtain the target image.

In an exemplary embodiment, based on the pixel point coordinates of the first position and the second position, an image transformation matrix for perspective transformation of the infrared image to the same coordinates as the visible light image is determined. Of course, it is not limited to this. It may also be determined to transform the visible light image into a perspective image. The image perspective is transformed into an image transformation matrix that has the same coordinates as the infrared image, which is not limited here.

Please refer to FIG. 5 , which is a schematic flowchart of sub-steps of an image fusion method provided by an embodiment of the present disclosure.

As shown in Figure 5, in some embodiments, step S105 includes step S1051-step S1053: step S1051, determine the image transformation matrix according to the first position and the second position; step S1052, according to the image transformation matrix, combine the infrared image with The visible light image is transformed to the same coordinates; step S1053, the visible light image and the infrared image at the same coordinates are fused to obtain the target image.

In an exemplary embodiment, according to the coordinate matrix of the pixel point corresponding to the first position and the coordinate matrix of the pixel point corresponding to the second position, the optimal single mapping transformation matrix for transforming the infrared image and the visible light image to the same coordinates can be determined, that is, Determine the image transformation matrix.

In an exemplary embodiment, according to the image transformation matrix, the infrared image and the visible light image are transformed to the same coordinates, as follows: The 3×3 matrix is the image transformation matrix, [u, v, w] represents the coordinates of the pixel points in the infrared image, [X, Y, Z] represents the coordinate points of the infrared image under the coordinates of the visible light image.

In an exemplary embodiment, after the infrared image and the visible light image are transformed to the same coordinate according to the image transformation matrix, the visible light image and the infrared image at the same coordinate are superimposed on each other to obtain the target image.

In an exemplary embodiment, step S105 may be implemented through OpenCV. In an exemplary embodiment, the findHomography function is called to calculate the coordinate matrix of the pixel point corresponding to the first position and the coordinate matrix of the pixel point corresponding to the second position to obtain the image transformation matrix; warpPerspective is called to combine the infrared image and visible light according to the image transformation matrix. The image is transformed to the same coordinates; the addWeighted function is called to fuse the infrared image and the visible light image.

In some embodiments, the image fusion method further includes: acquiring at least one target image; and splicing the target images based on the overlapping area of the at least one target image based on a preset image splicing algorithm to obtain a global target image.

In an exemplary embodiment, in order to make the image fusion method applicable to a larger area of target objects, multiple 2D cameras and multiple infrared cameras can be respectively set up to obtain visible light images and infrared images, and each visible light image and infrared image can be acquired. The images are respectively fused to obtain at least one target image, and then the target images are spliced to obtain a global target image with a wider field of view.

In an exemplary embodiment, in order to facilitate stitching of target images, a 2D camera and red When using external cameras, adjust the positions of the 2D camera and the infrared camera so that the visible light images acquired by adjacent 2D cameras include overlapping areas, and the infrared images acquired by adjacent infrared cameras include overlapping areas.

Please refer to FIG. 6 , which is a schematic flowchart of sub-steps of an image fusion method provided by an embodiment of the present disclosure.

As shown in Figure 6, in some embodiments, based on the overlapping area of at least one target image, the target images are spliced based on a preset image splicing algorithm to obtain a global target image, including steps S1061-step S1063: step S1061 , the overlapping region includes a first overlapping sub-region and a second overlapping sub-region, the first splicing sequence is determined for the first overlapping sub-region, and the second splicing sequence is determined for the second overlapping sub-region; step S1062, determine the first splicing sequence according to the first overlapping sub-region. The first splicing weight of an overlapping sub-region is determined according to the second splicing sequence; the second splicing weight of the second overlapping sub-region is determined according to the first splicing weight; Step S1063: Process the first sub-overlapping region according to the first splicing weight, and process the first sub-overlapping region according to the second splicing weight. The second overlapping sub-region is processed to obtain the global target image.

In an exemplary embodiment, taking two adjacent target images on the left and right as an example, the central axis of the overlapping area can be determined, and the overlapping area can be divided into a left overlapping area and a right overlapping area according to the central axis. For the left overlapping area, The region constructs an arithmetic increasing sequence, and constructs an arithmetic decreasing sequence for the overlapping area on the right. The arithmetic increasing sequence weights the pixels in the left overlapping area, and the arithmetic decreasing sequence weights the pixels in the right overlapping area. Realize the splicing between two adjacent target images on the left and right.

In an exemplary embodiment, the splicing process between two adjacent target images is similar to the above method and will not be described again here.

The image fusion method provided by the above embodiments obtains a visible light image of a target object, which is set with a preset marked object; obtains an infrared image of the target object; and determines the corresponding third position of the marked object in the infrared image based on the preset feature extraction algorithm. First position; based on the preset target matching algorithm, determine the second position corresponding to the marked object in the visible light image; based on the first position and the second position, based on the preset image fusion algorithm, fuse the visible light image and the infrared image to obtain target image. It reduces the complexity of fusion of visible light images and infrared images, and can be applied to target objects in diverse scenes.

Please refer to FIG. 7 , which is a schematic structural block diagram of an image fusion device provided by an embodiment of the present disclosure.

As shown in Figure 7, the image fusion device 300 includes a processor 301 and a memory 302. The processor 301 and the memory 302 are connected through a bus 303, which is, for example, an I2C (Inter-integrated Circuit) bus.

In an exemplary embodiment, the processor 301 is used to provide computing and control capabilities to support the operation of the entire image fusion device. The processor 301 can be a central processing unit (Central Processing Unit, CPU). The processor 301 can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC). ), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general processor may be a microprocessor or the processor may be any conventional processor.

In an exemplary embodiment, the memory 302 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) disk, an optical disk, a USB disk, a mobile hard disk, or the like.

Those skilled in the art can understand that the structure shown in Figure 7 is only a block diagram of a partial structure related to the embodiment of the present disclosure, and does not constitute a limitation on the image fusion device to which the embodiment of the present disclosure is applied. Specifically, The image fusion device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

Wherein, the processor is used to run a computer program stored in the memory, and implement any image fusion method provided by the embodiments of the present disclosure when executing the computer program.

In one embodiment, the processor is used to run a computer program stored in the memory, and implement the following steps when executing the computer program: obtain a visible light image of a target object, where the target object is set with a preset marking object; obtain an infrared image of the target object image; based on the preset feature extraction algorithm, determine the first position corresponding to the marked object in the infrared image; based on the preset target matching algorithm, determine the second position corresponding to the marked object in the visible light image; based on the first position and the second position , Based on the preset image fusion algorithm, the visible light image and the infrared image are fused to obtain the target image.

In one embodiment, when implemented, the processor is used to: obtain an original infrared image of the target object; and perform distortion correction on the original infrared image based on a preset distortion correction algorithm to obtain an infrared image.

In one embodiment, when acquiring the infrared image of the target object, the processor is used to: project the original infrared image onto the normalized plane to obtain the original infrared image on the normalized plane; based on the preset Radial distortion coefficient and preset tangential distortion coefficient, perform radial distortion correction and tangential distortion correction on the original infrared image on the normalized plane, and obtain the infrared image on the normalized plane; The infrared image is projected onto the pixel plane to obtain the infrared image.

In one embodiment, when the processor performs distortion correction on the original infrared image based on the preset distortion correction algorithm to obtain the infrared image, it is used to: obtain the brightness channel image of the infrared image; based on the preset brightness threshold, A first position corresponding to the marked object in the infrared image is determined.

In one embodiment, when the processor implements a preset feature extraction algorithm and determines the first position corresponding to the marked object in the infrared image, it is used to: obtain the brightness channel image of the infrared image, based on the preset brightness threshold, Determine the position of the brightness area in the infrared image; fit the position of the brightness area to determine the first position corresponding to the marked object in the infrared image.

In one embodiment, the processor determines the position of the brightness area in the infrared image based on a preset brightness threshold, and is used to: binarize the pixels of the brightness channel image based on the preset brightness threshold; The binarization result of the pixel points determines the location of the brightness area in the infrared image.

In one embodiment, when the processor determines the second position of the marked object in the visible light image based on the preset target matching algorithm, the processor is configured to: determine the preset marked object based on the preset target matching algorithm. The target matching degree between the template image and multiple areas in the visible light image; determine the target area position in the visible light image based on the target matching degree in the multiple areas; fit the target area position to determine the second location corresponding to the marked object in the visible light image Location.

In one embodiment, when the processor fuses the visible light image and the infrared image according to the first position and the second position based on a preset image fusion algorithm to obtain the target image, it is used to: according to the first position and the second position, In the second position, the image transformation matrix is determined; according to the image transformation matrix, the infrared image and the visible light image are transformed to the same coordinates; the visible light image and the infrared image at the same coordinates are fused to obtain the target image.

In one embodiment, when implementing the image fusion method, the processor is also configured to: determine the image transformation matrix according to the first position and the second position; transform the infrared image and the visible light image to the same coordinates according to the image transformation matrix; The visible light image and the infrared image at the same coordinates are fused to obtain the target image.

In one embodiment, the processor implements the preset based on the overlapping area of at least one target image. The image splicing algorithm is used to splice the target image to obtain the global target image: the overlapping area includes the first overlapping sub-area and the second overlapping sub-area, the first splicing sequence is determined for the first overlapping sub-area, and the first splicing sequence is determined for the first overlapping sub-area. The second overlapping sub-region determines the second splicing sequence; the first splicing weight of the first overlapping sub-region is determined according to the first splicing sequence, and the second splicing weight of the second overlapping sub-region is determined according to the second splicing sequence; the first splicing weight is determined according to the first splicing weight. The first sub-overlapping area is processed, and the second overlapping sub-area is processed according to the second splicing weight to obtain the global target image.

In one embodiment, when implementing the image fusion method, the processor is also configured to: set a marking object based on a preset marking area in the target object, and the intersection angle between the marking areas is a preset angle.

It should be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the image fusion device described above can be referred to the corresponding process in the foregoing image fusion method embodiment, and will not be described here. Again.

Embodiments of the present disclosure also provide a storage medium for computer-readable storage. The storage medium stores one or more programs. The one or more programs can be executed by one or more processors to implement the embodiments of the present disclosure. Provides steps for any of the image fusion methods.

The storage medium may be an internal storage unit of the image fusion device in the aforementioned embodiment, such as a hard disk or memory of the image fusion device. The storage medium can also be an external storage device of the image fusion device, such as a plug-in hard disk, smart memory card (Smart Media Card, SMC), secure digital (SD) card, flash memory card (Flash) equipped on the image fusion device. Card) etc.

Those of ordinary skill in the art can understand that all or some steps, systems, and functional modules/units in the devices disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may consist of several physical components. Components execute cooperatively. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. removable, removable and non-removable media. computer Storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical disk storage, magnetic cassettes, tapes, disk storage or other magnetic storage devices, or may be used Any other medium that stores the desired information and can be accessed by a computer. Additionally, it is known to those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Embodiments of the present disclosure provide an image fusion method, device and storage medium to obtain a visible light image of a target object, where the target object is set with a preset marker object; obtain an infrared image of the target object; and determine the infrared image based on a preset feature extraction algorithm. The first position corresponding to the marked object in the image; based on the preset target matching algorithm, determine the second position corresponding to the marked object in the visible light image; based on the first position and the second position, based on the preset image fusion algorithm, determine the visible light image Fusion with infrared images to obtain the target image. The embodiments of the present disclosure reduce the complexity of the fusion of visible light images and infrared images. The present disclosure also provides a solution for splicing target images to obtain a global target image, making this method suitable for target objects with a wider area and more diverse application scenarios. ization, which improves the diversity of usage scenarios and can adapt to different types of target objects.

The above serial numbers of the embodiments of the present disclosure are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person familiar with the technical field can easily think of various equivalent modifications or modifications within the technical scope disclosed in the present disclosure. Substitutions, these modifications or substitutions should be covered by the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

An image fusion method including:

Obtaining a visible light image of a target object, where the target object is set with a preset marking object;

Obtain an infrared image of the target object;

Based on a preset feature extraction algorithm, determine the first position corresponding to the marked object in the infrared image;

Based on a preset target matching algorithm, determine the second position corresponding to the marked object in the visible light image;

According to the first position and the second position, the visible light image and the infrared image are fused based on a preset image fusion algorithm to obtain a target image.
The image fusion method according to claim 1, wherein said obtaining the infrared image of the target object includes:

Obtain the original infrared image of the target object;

Based on a preset distortion correction algorithm, distortion correction is performed on the original infrared image to obtain the infrared image.
The image fusion method according to claim 2, wherein the distortion correction is performed on the original infrared image based on a preset distortion correction algorithm to obtain the infrared image, including:

Project the original infrared image onto the normalized plane to obtain the original infrared image on the normalized plane;

Based on the preset radial distortion coefficient and the preset tangential distortion coefficient, radial distortion correction and tangential distortion correction are performed on the original infrared image on the normalized plane to obtain the infrared image on the normalized plane. image;

The infrared image on the normalized plane is projected onto the pixel plane to obtain the infrared image.
The image fusion method according to claim 1, wherein the determining the first position corresponding to the marked object in the infrared image based on a preset feature extraction algorithm includes:

Obtain the brightness channel image of the infrared image, and determine the position of the brightness area in the infrared image based on a preset brightness threshold;

The brightness area position is fitted to determine the first position corresponding to the marked object in the infrared image.
The image fusion method according to claim 4, wherein determining the brightness area position in the infrared image based on a preset brightness threshold includes:

Binarize the pixels of the brightness channel image based on a preset brightness threshold;

According to the binarization result of the pixel point, the position of the brightness area in the infrared image is determined.
The image fusion method according to claim 1, wherein the determining the second position corresponding to the marked object in the visible light image based on a preset target matching algorithm includes:

Based on a preset target matching algorithm, determine the target matching degree between the preset marked object template image and multiple areas in the visible light image;

Determine the position of the target area in the visible light image according to the target matching degree of the multiple areas;

The position of the target area is fitted to determine the second position corresponding to the marked object in the visible light image.
The image fusion method according to claim 1, wherein the visible light image and the infrared image are fused based on a preset image fusion algorithm according to the first position and the second position to obtain Target images, including:

Determine an image transformation matrix according to the first position and the second position;

According to the image transformation matrix, transform the infrared image and the visible light image to the same coordinates;

The visible light image and the infrared image at the same coordinates are fused to obtain the target image.
The image fusion method according to claim 1, wherein the image fusion method further includes:

Obtain at least one target image;

According to the overlapping area of at least one of the target images, the target images are spliced based on a preset image splicing algorithm to obtain a global target image.
The image fusion method according to claim 8, wherein the target images are spliced based on the overlapping area of at least one of the target images and based on a preset image splicing algorithm to obtain a global target image, including:

The overlapping region includes a first overlapping sub-region and a second overlapping sub-region, a first splicing sequence is determined for the first overlapping sub-region, and a second splicing sequence is determined for the second overlapping sub-region;

Determine the first splicing weight of the first overlapping sub-region according to the first splicing sequence, and determine the second splicing weight of the second overlapping sub-region according to the second splicing sequence;

The first sub-overlapping region is processed according to the first splicing weight, and the second overlapping sub-region is processed according to the second splicing weight to obtain the global target image.
The image fusion method according to any one of claims 1 to 9, wherein the preset marking objects include a plurality of marking objects, and the plurality of marking objects are arranged in at least two directions, and the at least two directions intersect.
An image fusion device, including a processor, a memory, a computer program stored on the memory and executable by the processor, and a data bus for realizing connection communication between the processor and the memory, When the computer program is executed by the processor, the steps of the image fusion method according to any one of claims 1 to 10 are implemented.
A storage medium for computer-readable storage, the storage medium stores one or more programs, the one or more programs can be executed by one or more processors to implement any of claims 1 to 10 One step of the image fusion method.