WO2022257794A1 - Method and apparatus for processing visible light image and infrared image - Google Patents

Abstract

Provided in the present invention are a method and apparatus for processing a visible light image and an infrared image. The method and apparatus are applied to a device, which is configured with a visible light camera and an infrared camera, wherein the optical axis of the visible light camera or the infrared camera is perpendicular to a front view plane of the device. The method comprises: according to the spatial relative positions of a visible light camera and an infrared camera, a conversion parameter, and the horizontal distance of a target object from the visible light camera, establishing a registration model for the coordinate position of a visible light image of the target object that is acquired by the visible light camera and the coordinate position of an infrared image of the target object that is acquired by the infrared camera; and according to the registration model, performing registration and fusion on the visible light image and the infrared image of the target object. In the present invention, by means of an established registration model, data of a visible light image and data of an infrared image in a device, which is configured with dual cameras, can be fused into one image.

Description

Method and device for processing visible light image and infrared image technical field

The present invention relates to the field of image processing, in particular to a method and device for processing visible light images and infrared images.

Background technique

At present, there are many infrared temperature measurement and face detection and recognition devices. These temperature measurement and face detection and recognition devices are fixed installation devices, and the infrared camera module is often integrated with the visible light camera module. The relative position of the two camera modules and the relative angle is fixed. For example, a dual-light (infrared and visible light or white light) module design is adopted. Because of the dual-light module design, the central optical axes of the two camera modules are fixed in parallel and do not change, and the positions of the Z-axis directions of the centers of the two camera modules are the same. , the vertical (Y-axis direction) height is the same and fixed, the horizontal (X-axis direction) relative position is fixed and the distance is very small, or the vertical (Y-axis direction) height relative position deviation of the centers of the two camera modules is fixed, and the horizontal (X-axis direction) The positions are the same and fixed. These fixed infrared temperature measurement and face detection and recognition devices have the following three characteristics: the central optical axes of the two camera modules are parallel, the zero coordinate positions of the two camera modules on the Z axis are the same, and the zero coordinate positions on the Y axis are the same or on the X axis. The coordinates of the zero point are the same.

These characteristics provide favorable conditions for the data fusion of two different camera images, but in product design and various practical solutions, the two modules of the infrared camera and the visible light camera are not designed at the same position (X, Y, Z three-axis zero point The coordinates are not the same). For example, when applied to wearable application scenarios, it is often necessary to adjust the angle of the visible light camera or the infrared camera when people of different heights wear it and apply in different scenes, and the two camera modules of the infrared camera and the visible light camera are often not binocular modules. In order to achieve this, the central optical axes of the two camera modules are not parallel (there is an included angle), and the zero point coordinates of the X, Y, and Z axes of the spatial positions of the other two camera modules are also different. There are obvious differences between the above and the three characteristics of these fixed infrared temperature measurement and face detection and recognition devices. These differences bring great challenges to the data fusion of two different camera images of wearable devices.

Although the existing equipment is equipped with an infrared camera and a visible light camera, it has not realized the registration and fusion of dual images, and can only perform data analysis and processing on one camera picture, and the equipment cannot automatically combine the data of the visible light picture with the data of the infrared picture. The data are fused into one image.

Contents of the invention

The present invention provides a method and device for processing visible light images and infrared images to at least solve the problem in the related art that equipment equipped with dual cameras cannot automatically fuse data of visible light images and infrared images.

One aspect of the present invention provides a method for processing visible light images and infrared images, which is applied to a device equipped with a visible light camera and an infrared camera, wherein the optical axis of the visible light camera or the optical axis of the infrared camera and the device The front view plane is vertical, including the following steps:

According to the spatial relative position of the visible light camera and the infrared camera, conversion parameters and the horizontal distance between the target object and the visible light camera, establish the visible light image of the target object collected by the visible light camera and the infrared image of the target object collected by the infrared camera. The registration model of the coordinate position of the image;

The visible light image and the infrared image of the target object are registered and fused according to the registration model.

In another aspect of the present invention, it is provided that the processing of visible light images and infrared images is located on a device equipped with a visible light camera and an infrared camera, wherein the optical axis of the visible light camera or the optical axis of the infrared camera is in line with the front view of the device Plane verticals include:

The registration model building module is used to establish the visible light image of the target object collected by the visible light camera and the infrared image according to the relative spatial position of the visible light camera and the infrared camera, the conversion parameters, and the horizontal distance between the target object and the visible light camera. The registration model of the coordinate position of the infrared image of the target object collected by the infrared camera;

An image fusion module, configured to perform registration and fusion of the visible light image and the infrared image of the target object according to the registration model.

In the above embodiments of the present invention, the visible light image of the target object collected by the visible light camera and the infrared camera image are established according to the spatial relative positions of the visible light camera and the infrared camera, and the horizontal distance between the target object and the visible light camera. The registration model of the coordinate position of the infrared image of the collected target object, and according to the registration model, the visible light image of the target object and the infrared image are registered and fused, thereby solving the problem of configuring a dual camera that can combine light and infrared The equipment cannot process the fusion of visible light image data and infrared image data.

Description of drawings

FIG. 1 is a flow chart of a registration and fusion method for a visible light image and an infrared image according to an embodiment of the present invention;

Fig. 2 is a structural block diagram of a registration and fusion device for visible light images and infrared images according to an embodiment of the present invention;

Fig. 3 is a schematic diagram of the horizontal positions of a visible light camera and an infrared camera according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of the horizontal positions of a visible light camera and an infrared camera according to another embodiment of the present invention;

Fig. 5 is a flow chart of a registration and fusion method for a visible light image and an infrared image according to an embodiment of the present invention;

Fig. 6 is a flow chart of a registration and fusion method for a visible light image and an infrared image according to another embodiment of the present invention;

Fig. 7 is a flowchart of a registration and fusion method for a visible light image and an infrared image according to an embodiment of the present invention;

Fig. 8 is a structural block diagram of a device for registration and fusion of visible light images and infrared images according to an embodiment of the present invention;

Fig. 9 is a schematic diagram of horizontal positions of a visible light camera and an infrared camera according to an embodiment of the present invention;

Fig. 10 is a schematic diagram of the horizontal positions of a visible light camera and an infrared camera according to another embodiment of the present invention;

Fig. 11 is a flowchart of a method for registration and fusion of visible light images and infrared images according to an embodiment of the present invention;

Fig. 12 is a flowchart of a registration and fusion method for visible light images and infrared images according to another embodiment of the present invention

13 is a schematic diagram of the layout of the camera on the XOZ plane according to the image acquisition and processing method according to the embodiment of the present invention;

Fig. 14 is a schematic diagram of the arrangement of cameras on a YOZ plane according to an image acquisition and processing method according to an embodiment of the present invention;

Fig. 15 is a schematic diagram of the layout of the camera on the XOZ plane according to another embodiment of the present invention;

Fig. 16 is a schematic diagram of the layout of the camera on the YOZ plane according to another embodiment of the present invention;

FIG. 17 is a schematic flowchart of an image acquisition and processing method according to an embodiment of the present invention;

Fig. 18 is a schematic diagram of a partial interface of a device adopting an image acquisition and processing method according to an embodiment of the present invention.

Detailed ways

Hereinafter, the present invention will be described in detail with reference to the drawings and examples. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence.

The present invention provides a method for processing a visible light image and an infrared image, which is applied to a device equipped with a visible light camera and an infrared camera. To implement the present invention, it is necessary to configure the optical axis of one of the optical camera or the infrared light and the plane where the front view of the device is located to be vertical.

First, the optical axis of the visible light camera is perpendicular to the plane of the front view of the device. Fig. 1 is a flow chart of a method for processing visible light images and infrared images according to an embodiment of the present invention. As shown in Fig. 1, the process includes the following steps:

Step S102, establishing the visible light image of the target object collected by the visible light camera and the infrared image of the target object collected by the infrared camera according to the spatial relative positions of the visible light camera and the infrared camera, and the horizontal distance between the target object and the visible light camera. The registration model of the coordinate position of the image;

Step S104, performing registration and fusion of the visible light image and the infrared image of the target object according to the registration model.

Wherein, in step S102 of this embodiment, for example, the following registration model can be established:

Among them, A, B, C, and D are the first, second, third, and fourth conversion parameters, respectively, and m, n, and d are the relative distances between the X, Y, and Z axes of the visible light camera and the infrared camera, respectively,

and γ are the horizontal and vertical angles between the optical axes of the visible light camera and the infrared camera, respectively, L is the horizontal distance between the target and the visible light camera, (x _VR , y _VR ) is the pixel coordinate of the visible light image, (x _IR , y _IR ) is the infrared image pixel coordinates.

In this embodiment, the registration model can be used for fast registration and fusion of coordinate positions of objects in two camera images. From this registration model, it can be known that the non-zero items in the registration model matrix are not fixed values, but are a function of the horizontal distance L between the target object and the visible light camera. Therefore, it is impossible to determine the non-zero items in the registration model matrix by selecting the frame image collected by the scene with a specific horizontal distance from the visible light camera to the target object.

Before the application of the registration model in this embodiment, it is necessary to calibrate the unknown parameters in the registration model, that is, to calibrate the lateral angle between the two optical axes in the registration model of the visible light image and the infrared image

and longitudinal angle γ. For example, in this embodiment, the horizontal angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera can be calibrated in the following two ways:

The first calibration method includes the following steps:

1) Selecting a first reference object whose horizontal distance from the visible light camera is a first set distance;

2) Simultaneously collect images of the first reference object through the visible light camera and the infrared camera, and measure the horizontal length and vertical length of the same position of the first reference object in the visible light image and the infrared image respectively;

3) Calibrate the optical axis of the visible light camera in the registration model according to the first set distance and the horizontal length and vertical length of the same position of the first reference object in the visible light image and the infrared image respectively The horizontal and vertical angles with the optical axis of the infrared camera.

For example, the horizontal angle between the optical axes of the two cameras in the registration model can be calibrated by the following formula

and longitudinal angle γ:

Among them, L _C is the first set distance, L _VR and W _VR are the horizontal length and vertical length of the same position of the first reference object in the visible light image respectively, L _IR and W _IR are the first reference The horizontal length and vertical length of the same position of the object in the infrared image.

The second calibration method includes the following steps:

1) Selecting a second reference object whose horizontal distance from the visible light camera is a second set distance;

2) adjusting the optical axis of the visible light camera so that the same position of the second reference object is located at a specific position of the visible light image and the infrared image;

3) According to the second set distance and the coordinate value of the specific position, calibrate the horizontal angle and the vertical angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model.

For example, assuming that the coordinate values x _VR , y _VR , x _IR , and y _IR of a specific position are all 0, the lateral angle between the two optical axes can be calculated according to the registration model of the visible light image and the infrared image

and the longitudinal intersection angle γ of the two optical axes are respectively:

For example: Assuming that the coordinate values x _IR and y _IR of a specific position are both 0, x _VR is 200, and y _VR is 100, the lateral angle between the two optical axes can be calculated according to the registration model of the visible light image and the infrared image

and the longitudinal intersection angle γ of the two optical axes are respectively:

In this embodiment, after calibrating the horizontal and vertical angles between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model, the following steps may also be included: the optical axis of the visible light camera Substituting the horizontal and vertical angles with the optical axis of the infrared camera into the registration model to establish a height and width mapping model between the visible light image and the infrared image; based on multiple differences between the target object and the visible light camera The horizontal distance is to establish a mapping relationship between the height of the designated area of the target object in multiple groups of visible light images and the horizontal distance between the target object and the visible light camera.

For example, if the registration model of this embodiment is applied to the temperature measurement scene of the human body, the following height and width mapping model between the visible light image and the infrared image is established:

Among them, A, C,

γ, n, L are all known, λ is a configuration parameter, and the range of λ is 0.1<λ≤1. In this embodiment, by adjusting the size of λ, the interference caused by the background temperature in areas other than the face can be avoided.

In this embodiment, step S104 may include: obtaining the center position coordinate value of the specified area of the target object in the visible light image, and the height and width of the specified area of the target object in the visible light image, and according to the The height and width of the specified area of the target object in the visible light image, find the horizontal distance value corresponding to the target object and the visible light camera in the height and width mapping model; combine the corresponding horizontal distance value with the target The center position coordinate value of the designated area of the object is input into the registration model, and the center position coordinate value of the designated area of the target object in the infrared image is calculated and obtained; the height and the height of the designated area of the target object in the visible light image are calculated The width is input into the height and width mapping model, and the height and width of the specified area of the target object in the infrared image are calculated and obtained; according to the center position coordinates of the specified area of the target object in the infrared image, and the The height and width of the specified area of the target object in the infrared image determine the specified area of the target object in the infrared image.

In this embodiment, after determining the specified area of the target object in the infrared image, it may further include: acquiring the highest temperature value in the specified area of the target object in the infrared image; Annotating a specified location of a specified region of the target object in the visible light image.

Fig. 2 is a structural block diagram of a device for registering and fusing visible light images and infrared images according to an embodiment of the present invention. The plane of the front view is vertical, as shown in FIG. 2 , the device includes a registration model building module 210 and an image fusion module 220 .

A registration model establishment module 210, configured to establish the visible light image of the target object collected by the visible light camera and the infrared camera according to the spatial relative position of the visible light camera and the infrared camera, and the horizontal distance between the target object and the visible light camera. A registration model for the coordinate position of the collected infrared image of the target object.

The image fusion module 220 is configured to perform registration and fusion of the visible light image and the infrared image of the target object according to the registration model.

It should be noted that each of the above-mentioned modules can be implemented by software or hardware. For the latter, it can be implemented in the following manner, but not limited to this: the above-mentioned modules are all located in the same processor; or, the above-mentioned modules are respectively located in multiple in the processor.

Second, the optical axis of the infrared camera is perpendicular to the plane where the front view of the device is located.

FIG. 7 is a flow chart of a method for registration and fusion of visible light images and infrared images according to an embodiment of the present invention. As shown in FIG. 7 , the process includes the following steps:

Step S702: Establish the visible light image of the target object collected by the visible light camera and the infrared image of the target object collected by the infrared camera according to the spatial relative positions of the visible light camera and the infrared camera, and the horizontal distance between the target object and the visible light camera. The registration model of the coordinate position of the image;

Step S704, performing registration and fusion of the visible light image and the infrared image of the target object according to the registration model.

Wherein, in step S702 of this embodiment, for example, the following registration model can be established:

Among them, A, B, C, and D are the first, second, third, and fourth conversion parameters, respectively, and m, n, and d are the relative distances between the X, Y, and Z axes of the visible light camera and the infrared camera, respectively,

and γ are the horizontal and vertical angles between the optical axes of the visible light camera and the infrared camera, respectively, L is the horizontal distance between the target and the visible light camera, (x _VR , y _VR ) is the pixel coordinate of the visible light image, (x _IR , y _IR ) is the infrared image pixel coordinates.

In this embodiment, the registration model can be used for fast registration and fusion of coordinate positions of objects in two camera images. From this registration model, it can be known that the non-zero items in the registration model matrix are not fixed values, but are a function of the horizontal distance L between the target object and the visible light camera. Therefore, it is impossible to determine the non-zero items in the registration model matrix by selecting the frame image collected by the scene with a specific horizontal distance from the visible light camera to the target object.

Before the application of the registration model in this embodiment, it is necessary to calibrate the unknown parameters in the registration model, that is, to calibrate the lateral angle between the two optical axes in the registration model of the visible light image and the infrared image

and longitudinal angle γ. For example, in this embodiment, the horizontal angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera can also be calibrated in the following two ways:

The first calibration method includes the following steps:

1) Selecting a first reference object whose horizontal distance from the visible light camera is a first set distance;

2) Simultaneously collect images of the first reference object through the visible light camera and the infrared camera, and obtain the coordinates of the same position of the first reference object in the visible light image and the infrared image respectively;

3) Calibrate the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model according to the first set distance and the coordinates of the same position of the first reference object in the visible light image and the infrared image respectively. The horizontal and vertical angles of the optical axis.

For example, the horizontal angle between the optical axes of the two cameras in the registration model can be calibrated by the following formula

and longitudinal angle γ:

Among them, LC is the first set distance, (x _{VR-C ,} y _VR-C ) is the coordinates of the same position of the first reference object in the visible light image, and the coordinates of the same position of the first reference object in the infrared image (x _IR-C ,y _IR-C ).

The second calibration method includes the following steps:

1) Selecting a second reference object whose horizontal distance from the visible light camera is a second set distance;

2) adjusting the optical axis of the visible light camera so that the same position of the second reference object is located at a specific position of the visible light image and the infrared image;

3) According to the second set distance and the coordinate value of the specific position, calibrate the horizontal angle and the vertical angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model.

For example, assuming that the coordinate values x _VR , y _VR , x _IR , and y _IR of a specific position are all 0, the lateral angle between the two optical axes can be calculated according to the registration model of the visible light image and the infrared image

and the longitudinal intersection angle γ of the two optical axes are respectively:

For example: Assuming that the coordinate values x _IR and y _IR of a specific position are both 0, x _VR is 200, and y _VR is 100, the lateral angle between the two optical axes can be calculated according to the registration model of the visible light image and the infrared image

and the longitudinal intersection angle γ of the two optical axes are respectively:

In this embodiment, after calibrating the horizontal and vertical angles between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model, the following steps may also be included: the optical axis of the visible light camera Substituting the horizontal and vertical angles with the optical axis of the infrared camera into the registration model to establish a height and width mapping model between the visible light image and the infrared image; based on multiple differences between the target object and the visible light camera The horizontal distance is to establish a mapping relationship between the height of the designated area of the target object in multiple groups of visible light images and the horizontal distance between the target object and the visible light camera.

For example, if the registration model of this embodiment is applied to the temperature measurement scene of the human body, the following height and width mapping model between the visible light image and the infrared image is established:

Among them, A, C,

γ, n, L are all known, λ is a configuration parameter, and the range of λ is 0.1<λ≤1. In this embodiment, by adjusting the size of λ, the interference caused by the background temperature in areas other than the face can be avoided.

In this embodiment, step S704 may include: obtaining the center position coordinate value of the specified area of the target object in the visible light image, and the height and width of the specified area of the target object in the visible light image, and according to the The height and width of the specified area of the target object in the visible light image, find the horizontal distance value corresponding to the target object and the visible light camera in the height and width mapping model; combine the corresponding horizontal distance value with the target The center position coordinate value of the designated area of the object is input into the registration model, and the center position coordinate value of the designated area of the target object in the infrared image is calculated and obtained; the height and the height of the designated area of the target object in the visible light image are calculated The width is input into the height and width mapping model, and the height and width of the specified area of the target object in the infrared image are calculated and obtained; according to the center position coordinates of the specified area of the target object in the infrared image, and the The height and width of the specified area of the target object in the infrared image determine the specified area of the target object in the infrared image.

In this embodiment, after determining the specified area of the target object in the infrared image, it may further include: acquiring the highest temperature value in the specified area of the target object in the infrared image; Annotating a specified location of a specified region of the target object in the visible light image.

In this embodiment, a device for registering and fusing visible light images and infrared images is also provided. The device is used to implement the above embodiments and preferred implementation modes, and those that have already been described will not be repeated. As used below, the term "module" may be a combination of software and/or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.

Fig. 8 is a structural block diagram of an apparatus for registration and fusion of a visible light image and an infrared image according to an embodiment of the present invention, the apparatus is located on a device equipped with a visible light camera and an infrared camera, wherein the optical axis of the infrared camera and the device The front view plane is vertical, as shown in FIG. 8 , the device includes a registration model building module 810 and an image fusion module 820 .

The registration model establishment module 810 is configured to establish the visible light image of the target object collected by the visible light camera and the infrared camera according to the spatial relative position of the visible light camera and the infrared camera, and the horizontal distance between the target object and the visible light camera. A registration model for the coordinate position of the collected infrared image of the target object.

The image fusion module 820 is configured to perform registration and fusion of the visible light image and the infrared image of the target object according to the registration model.

It should be noted that each of the above-mentioned modules can be implemented by software or hardware. For the latter, it can be implemented in the following manner, but not limited to this: the above-mentioned modules are all located in the same processor; or, the above-mentioned modules are respectively located in multiple in the processor.

In order to facilitate the understanding of the technical solutions provided by the present invention, the following will describe in detail in conjunction with specific scenario embodiments.

The image processing scheme when the visible light camera and the infrared camera are perpendicular to the front view plane of the device are given above, as an example to explain the scheme more clearly, the following will give multiple implementations of configuring a visible light camera and an infrared camera on a device at the same time Example to illustrate.

An embodiment of the present invention provides a method for registration and fusion of visible light and infrared images. This method is applied to a device configured with a visible light camera and an infrared camera. Figure 3 and Figure 4 are both schematic diagrams of the horizontal position of the visible light camera and the infrared camera on the device according to an embodiment of the present invention, wherein Figure 3 shows the device area 1, the infrared camera 2, the visible light camera 3, and the front view of the device Plane horizontal line 4, infrared camera optical axis 5, visible light camera optical axis 6, and lateral angle 7 between the two optical axes.

As shown in Figure 3, the optical axis 6 of the visible light camera is perpendicular to the front view plane of the device, and the infrared camera 2 is located above the visible light camera 3, and its optical axis is not perpendicular to the front view plane of the device, but intersects with the optical axis 6 of the visible light camera.

Fig. 4 shows the device area 1, the infrared camera 2, the visible light camera 3, the equipment front view plane horizontal line 4, the infrared camera optical axis 5, the visible light camera optical axis 6, and the lateral angle 7 between the two optical axes. As shown in Figure 4, the optical axis 6 of the visible light camera is perpendicular to the plane of the front view of the device, and the infrared camera 2 is located above the visible light camera 3, and its optical axis 5 is not perpendicular to the plane of the front view of the device, but intersects with the optical axis 6 of the visible light camera .

As shown in Figures 3 and 4, in this embodiment, the optical axis of the visible light camera is perpendicular to the plane of the front view of the device, and the optical axis of the infrared camera may not be perpendicular to the plane of the front view of the device. The relative angle of is 0 (that is, the position of the same object in the screen does not rotate).

The following will describe this embodiment in detail in conjunction with a scene where a visible light camera and an infrared camera are used for temperature measurement. Of course, the technical solution provided by this embodiment can also be applied to other scenes that require image fusion. As shown in Figure 5, the registration and fusion of visible light and infrared images provided in this embodiment may include the following steps:

Step S501, establishing a registration model of a visible light image and an infrared image.

Specifically, in this step, according to the relative distances m, d, and n of the X, Y, and Z-axis spatial positions of the visible light and infrared cameras, the horizontal and vertical field angles, the display resolution, and the lateral clamping distance between the two optical axes Angle (that is, the angle between the ZOY planes of the two cameras), the longitudinal intersection angle of the two optical axes (that is, the angle between the ZOX planes of the two cameras), and the horizontal distance L between the target object and the visible light camera to establish a visible light image and an infrared image The following registration model for :

Among them, A, B, C, and D are conversion parameters, m, n, and d are the relative distances of the X, Y, and Z axes of the two cameras, which are known constants.

The horizontal distance L between the target object and the visible light camera is the amount of change. x _VR and y _VR are the pixel coordinate values of the visible light image, and x _IR and y _IR are the pixel coordinate values of the infrared image.

In this embodiment, the conversion parameters A, B, C, and D can be obtained by calculating the horizontal and vertical viewing angles of visible light, the horizontal and vertical viewing angles of infrared light, and the display resolution parameters of infrared and visible light. E.g,

Among them, w _VR is the horizontal display resolution of the visible light camera, h _VR is the vertical display resolution of the visible light camera, w _IR is the horizontal display resolution of the infrared camera, h _IR is the vertical display resolution of the infrared camera, and α is the horizontal field of view of the visible light camera , β is the vertical viewing angle of the visible light camera, θ is the horizontal viewing angle of the infrared camera, and φ is the vertical viewing angle of the infrared camera.

Step S502, calibrate the horizontal angle and vertical angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model. In this embodiment, instead of manually adjusting the visible light and infrared cameras, the horizontal and vertical angles can be calibrated as follows:

1) Select a target object whose horizontal distance from the visible light camera is L _C . For example, the selection range of _LC can be [0.3m～7m];

2) Visible light and infrared cameras simultaneously collect images of the target object (the target object can be a regular cuboid, cube, or a certain part of the human body, such as a human head), and measure the horizontal and vertical lengths of the same position of the object. The measurement method can be automatically calculated and measured by the built-in software of the device, or manually calculated and measured on the collected images. The collected image method can be obtained through the server software connected to the device. The horizontal length of the visible light image of the object is L _VR , The vertical length is W _VR , the horizontal length of the infrared image of the object is L _IR , and the vertical length is W _IR ;

3) According to the registration model of the visible light image and the infrared image, the lateral angle between the two optical axes can be calculated

and the vertical angle γ of the two optical axes:

Step S503, establishing the height and width mapping model of the visible light image and the infrared image as follows:

Among them, A, C,

γ, n, L are all known, λ is a configuration parameter, and the range of λ is 0.1<λ≤1. In this embodiment, by adjusting the size of λ, it is possible to avoid the interference caused by the background temperature brought by areas other than the face in the temperature measurement application.

In this embodiment, the established height and width mapping models can further quickly fuse the size and ratio relationship of objects. For example, for the height H _VR-Face and width W _VR-Face of the face image in the visible light image where the center of the face image at a given distance L is located, H _{ir_face} can be obtained according to the height and width mapping scale model of the visible light image and the infrared image and W _{ir_face} , and further according to x _IR , y _IR , Hi _{ir_face} , and W _{ir_face} , the corresponding collected temperature information of the face area of the infrared image can be obtained.

Step S504, establishing a mapping table of the range intervals corresponding to different face heights in the visible light image and the horizontal distance L between the face and the visible light camera. For example, as shown in Table 1, the H _{vr_face} range interval can be divided into multiple intervals, wherein , H _k+1 >H _k >H _k-1 >...>H ₆ >H ₅ >H ₄ >H ₃ >H ₂ >H ₁ , L ₁ >L ₂ >L ₃ >L ₄ >...>L _k-2 >L _k-1 >L _k .

Table 1

Hvr_face	L
[H 1 ,H 2 ]	L 1
(H 2 ,H 3 ]	L 2
(H 3 ,H 4 ]	L 3
…	…
…	…
…	…
(H k-2 ,H k-1 ]	L k-2
(H k-1 ,H k ]	L k-1
(H k ,H k+1 ]	L k

In step S505, the human face detection module outputs one or more human face center position coordinates (x _VR , y _VR ) in the visible light image, and the height H _{vr_face} and width W _{vr_face} of the human face frame picture, and each detected person Find the corresponding distance L value in the mapping model for the height or width of the face picture, and input the corresponding distance L value and the coordinate value of the face center position (x _VR , y _VR ) into the registration model of the visible light image and the infrared image, Calculate and obtain the coordinate values (x _IR , y _IR ) of the center position of the face picture corresponding to the infrared image, until the calculation is completed until the coordinate values (x _VR , y _VR ) of the center position of the face picture of the infrared images corresponding to the currently detected multiple people are completed.

Step S506, input the height and width of each detected face picture into the height and width mapping model of the visible light image and the infrared image, and calculate and obtain the height H _{ir_face} and width W _{ir_face} of the corresponding face picture.

Step S507, according to the human face picture center position coordinates (x _IR , y _IR ) of the corresponding infrared image of the detected human face, the height H _{ir_face} and the width W _{ir_face} of the human face picture determine the area of the corresponding infrared image, from the corresponding Take the highest temperature in the infrared image area and record it as the face temperature of the corresponding person, and mark the temperature value of each corresponding person around or in the face frame of the visible light image of each corresponding face.

Through the above-mentioned steps of this embodiment, it is solved that the central optical axes of the visible light camera and the infrared camera are not parallel (there is an intersection angle between the two optical axes in the horizontal direction, and there is also an intersection angle in the vertical direction of the two optical axes), and the X-axis, Y-axis, and Z-axis directions of the two cameras are solved. The problem of image registration and fusion in the case of different positions, and further solves the problem of interference with face temperature detection caused by abnormal ambient temperature around the face in the application of temperature measurement scenes.

When the method of this embodiment is applied to the temperature measurement scene, when the position and area range data of multiple faces are detected in the visible light screen, the range of faces corresponding to all detected faces in the infrared screen can be quickly and accurately obtained. Further accurate acquisition of face temperature data corresponding to the face range can completely solve the problem of interference with face temperature detection caused by abnormal ambient temperature around the face, and greatly improve the detection efficiency and accuracy of face temperature.

Another embodiment of the present invention provides another method for registration and fusion of visible light and infrared images, which can be applied to devices equipped with visible light cameras and infrared cameras. In this embodiment, the spatial position relationship between the visible light camera and the infrared camera can be referred to in FIG. 3 and FIG. 4 .

As shown in Figures 3 and 4, in this embodiment, the optical axis of the visible light camera is perpendicular to the plane of the front view of the device, and the optical axis of the infrared camera may not be perpendicular to the plane of the front view of the device. The relative angle of is 0 (that is, the position of the same object in the screen does not rotate).

The following will describe this embodiment in detail in combination with a scene where a visible light camera and an infrared camera are used to measure the temperature of multiple people. Of course, the technical solution provided by this embodiment can also be applied to other scenes that require image fusion. As shown in Figure 6, the shooting scene is that a helmet equipped with a visible light camera and an infrared camera shoots a face in front of the helmet. The registration and fusion of visible light and infrared images provided in this embodiment may include the following steps:

Step S601, establishing a registration model of a visible light image and an infrared image.

Specifically, in this step, according to the relative distances of the X, Y, and Z-axis spatial positions of the two visible light and infrared cameras as m, d, and n, the horizontal and vertical viewing angles, the display resolution, and the lateral direction of the two optical axes The included angle (that is, the angle between the ZOY planes of the two cameras), the longitudinal intersection angle of the two optical axes (that is, the angle between the ZOX planes of the two cameras), and the horizontal distance L between the target object and the visible light camera are used to establish a visible light image and an infrared image. The following registration model for the images:

Among them, A, B, C, and D are conversion parameters, m, n, and d are the relative distances of the X, Y, and Z axes of the two cameras, which are known constants.

and γ are the horizontal and vertical angles between the optical axes of the two cameras, and the horizontal distance L between the target object and the visible light camera is the variation. x _VR and y _VR are the pixel coordinate values of the visible light image, and x _IR and y _IR are the pixel coordinate values of the infrared image.

In this embodiment, the conversion parameters A, B, C, and D can be obtained by calculating the horizontal and vertical viewing angles of visible light, the horizontal and vertical viewing angles of infrared light, and the display resolution parameters of infrared and visible light. E.g,

Among them, w _VR is the horizontal display resolution of the visible light camera, h _VR is the vertical display resolution of the visible light camera, w _IR is the horizontal display resolution of the infrared camera, h _IR is the vertical display resolution of the infrared camera, and α is the horizontal field of view of the visible light camera , β is the vertical viewing angle of the visible light camera, θ is the horizontal viewing angle of the infrared camera, and φ is the vertical viewing angle of the infrared camera.

Step S602, calibrate the horizontal angle and vertical angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model. In this embodiment, another way of calibrating the horizontal angle and the vertical angle in the registration model is provided, specifically, the following steps may be included:

1) Select an object whose horizontal distance from the visible light camera is L _C . For example, the selection range of _LC can be [0.3m～7m];

2) Adjust the optical axis of the visible light camera so that the same position of the object or human body is located at a specific position in the visible light picture and the infrared picture, and the specific position is displayed with a marker on both the visible light picture and the infrared picture (such as crosshairs or other marked images) , for example: x _IR and y _IR are both 0, x _VR is 200, y _VR is 100;

3) According to the registration model of the visible light image and the infrared image, the lateral angle between the two optical axes can be calculated

and the longitudinal intersection angle γ of the two optical axes are respectively:

The specific position of this embodiment can be flexibly selected, for example, in another embodiment, x _VR , y _VR , x _IR , y _IR are all 0, and the two optical axes can be calculated according to the registration model of the visible light image and the infrared image horizontal angle of

and the longitudinal intersection angle γ of the two optical axes are:

Step S603, establishing the height and width mapping model of the visible light image and the infrared image as follows:

Among them, A, C,

γ, n, L are all known, λ is a configuration parameter, and the range of λ is 0.1<λ≤1. By adjusting the size of λ, the interference of the background temperature brought by the area other than the face can be avoided.

In this embodiment, the established height and width mapping models can further quickly fuse the size and ratio relationship of objects. For example, for the height H _VR-Face and width W _VR-Face of the face image in the visible light image where the center of the face image at a given distance L is located, H _{ir_face} can be obtained according to the height and width mapping scale model of the visible light image and the infrared image and W _{ir_face} , and further according to x _IR , y _IR , Hi _{ir_face} and W _{ir_face} can obtain the corresponding collected temperature information of the face area of the infrared image.

Step S604, establish a mapping table of the range intervals corresponding to different face heights in the visible light image and the horizontal distance L between the face and the visible light camera, for example, as shown in Table 2, the H _{vr_face} range interval can be divided into multiple intervals, where , Hk+1>Hk>Hk-1>......>H6>H5>H4>H3>H2>H1, L1>L2>L3>L4>......>Lk-2>Lk -1>Lk.

Table 2

Hvr_face	L
[H1,H2]	L1
(H2,H3]	L2
(H3,H4]	L3
…	…
…	…
…	…
(Hk-2,Hk-1]	Lk-2
(Hk-1,Hk]	Lk-1
(Hk,Hk+1]	Lk

Step S605, the face detection module outputs one or more coordinates of the center position of the face in the visible light image (x _VR , y _VR ), and the height H _{vr_face} and width W _{vr_face} of the face frame picture, and each detected person Find the corresponding distance L value in the mapping model for the height or width of the face picture, and input the corresponding distance L value and the coordinate value of the face center position (x _VR , y _VR ) into the registration model of the visible light image and the infrared image for calculation Obtain the coordinate values (x _IR , y _IR ) of the center position of the face picture corresponding to the infrared image until the calculation is completed until the coordinate values (x _VR , y _VR ) of the center position of the face picture of the infrared images corresponding to the currently detected multiple people are completed.

Step S606, input the height and width of each detected face picture into the height and width mapping model of the visible light image and the infrared image, and calculate and obtain the height H _{ir_face} and width W _{ir_face} of the corresponding face picture.

Step S607, according to the human face picture center position coordinates (x _IR , y _IR ) of the corresponding infrared image of the detected human face, the height H _{ir_face} and the width W _{ir_face} of the human face picture determine the area of the corresponding infrared image, from the corresponding Take the highest temperature in the infrared image area and record it as the face temperature of the corresponding person, and mark the temperature value of each corresponding person around or in the face frame of the visible light image of each corresponding face.

Through the above-mentioned steps of this embodiment, it is solved that the central optical axes of the visible light camera and the infrared camera are not parallel (there is an intersection angle between the two optical axes in the horizontal direction, and there is also an intersection angle in the vertical direction of the two optical axes), and the X-axis, Y-axis, and Z-axis directions of the two cameras are solved. The problem of image registration and fusion under the condition that the positions of the faces are different, and further solves the problem of interference to the temperature detection of the face caused by abnormal ambient temperature around the face in the application of the temperature measurement scene.

The technical solution provided in this embodiment can solve the problem of image registration and fusion in the case where the central optical axes of the visible light camera and the infrared camera are not parallel, and the positions of the two cameras in the X-axis, Y-axis, and Z-axis directions are different. , to solve the problem of interference with face temperature detection caused by abnormal ambient temperature around the face in the temperature measurement scene application. When the technical solution provided by this embodiment is applied to the position and area range data of multiple human faces detected in the visible light screen, the range of faces corresponding to all detected faces in the infrared screen can be obtained quickly and accurately, and the corresponding face ranges can be further accurately obtained. The face temperature data in the face range can completely solve the problem of interference with face temperature detection caused by abnormal ambient temperature around the face, and greatly improve the detection efficiency and accuracy of face temperature. In addition, the calibration process of the image registration and fusion model provided in this embodiment is simple and fast, and avoids a large amount of complicated computing resource consumption. Compared with other image fusion algorithms that require more computing resources to support It requires fewer computing resources and is more efficient.

Example 3

In the above embodiments, when the fusion processing of the visible light image and the infrared light image is performed, the corresponding relationship between each coordinate position of the visible light image and each coordinate position of the infrared image is obtained. According to such a relationship, all coordinate pixels can be traversed to fuse the two images into one, and then the fused image can be displayed or stored.

But in fact, since the field of view of the visible light image and the infrared light image are not consistent, only the images of their respective parts can be fused. If all the images are directly traversed for processing, a lot of computing resources will be wasted. For this reason, in the embodiment provided by the present invention, the fusion range can be calculated, and then image processing is only performed on the fusion range, so as to save computing resources and increase image fusion speed.

FIG. 13 and FIG. 14 show schematic diagrams of arrangement of cameras in an image acquisition and processing method according to an embodiment of the present invention.

13 and 14, in the image acquisition and processing method of the embodiment of the present invention, a visible light camera 01 and an infrared light camera 02 are arranged in the device 03, the optical axis 11 of the visible light camera 01 is perpendicular to the front view plane 31 of the device 03, and the infrared light camera 01 The light camera 02 and the visible light camera 01 are arranged at intervals. In this embodiment, the projections of the infrared camera 02 and the visible light camera 01 on the X, Y, and Z axes are spaced apart from each other, and the projection distance on the Z axis is n.

Among them, the optical axis deviation and spacing of the visible light camera 01 and the infrared light camera 02 should not be too large, so as to ensure that the position of the same object in the picture does not rotate, reduce fusion registration deviation, and then improve the reliability of the obtained fusion area. The image acquisition and processing method of the embodiment of the present invention can guarantee the image acquisition and processing efficiency when the visible light camera 01 and the infrared light camera 02 have a certain deviation, but its specific installation requirements are determined according to the actual situation, and are not specifically limited here.

In this embodiment, the optical axis 21 of the infrared camera 02 intersects with the vertical axis of the front view plane of the device 03, and the projection between the optical axis 21 of the infrared camera 02 and the light 11 axis of the visible light camera on the ZOX plane Angle is

The included angle of the projection on the ZOY plane is γ, and the included angle is pre-tested and calibrated, corresponding to the fact that the optical axis 21 of the infrared camera 02 is parallel to the optical axis 11 of the visible light camera 01, which is consistent with the actual situation of cameras in general application scenarios Matching, no need to adjust the hardware configuration of the device, the application is simple.

In an optional embodiment, the angle between the optical axis 21 of the infrared camera 02 and the projection of the optical axis 11 of the visible light camera 01 on the ZOX plane

The calibration of the included angle γ with the projection on the ZOY plane includes:

Select an object whose horizontal distance from the visible light camera 01 is L _c (distance selection range is 0.5 meters to 7 meters), and simultaneously collect the object (the object is a regular cuboid, cube or human body) through the visible light camera 01 and the infrared light camera 02. part, such as the head of the human body), and measure the horizontal length (projection length on the X-axis) and vertical length (projection length on the Y-axis) of the same position of the object, and obtain the horizontal length of the visible light image of the object The length L _IR and the longitudinal length W _IR , as well as the lateral length L _VR and the longitudinal length W _VR of the infrared image of the object, and then calculate the optical axis 21 of the infrared camera 02 and the visible light The included angle of the projection of the optical axis 11 of the camera 01 on the ZOX plane

and the angle γ of the projection on the ZOY plane. Among them, the measurement of the horizontal length and the vertical length of the same position of the object can be obtained by automatic measurement and calculation by software, or by manual measurement and calculation based on the collected images.

Among them, the registration model of visible light image and infrared light image is:

Among them, parameters A and C refer to the following.

In another optional embodiment, the angle between the optical axis 21 of the infrared camera 02 and the projection of the optical axis 11 of the visible light camera on the ZOX plane

The calibration of the included angle γ with the projection on the ZOY plane includes:

Select an object whose horizontal distance from the visible light camera 01 is L _c (distance selection range is 0.5 meters to 7 meters), adjust the optical axis 11 of the visible light camera 01 (or adjust the optical axis 21 of the infrared light camera 02), so that the object The same position is located at the specific position of the visible light picture and the infrared light picture, so that the included angle of the projection of the optical axis 21 of the infrared light camera 02 and the optical axis 11 of the visible light camera on the ZOX plane

The angle γ between the projection and the projection on the ZOY plane is adjusted to a set value.

Wherein, for example, the pixel coordinates x _IR and y _IR of the corresponding visible light image at the specific position are 0, and the pixel coordinates x _VR and y _VR of the corresponding infrared light image are also 0, and the registration of the corresponding visible light image and the infrared light image The model is:

and

For example, if the pixel coordinates x _IR and y _IR of the corresponding visible light image at a specific position are 0, the pixel coordinates x _VR of the corresponding infrared light image are 200, and y _VR is 100, then the registration of the corresponding visible light image and the infrared light image The model is:

and

Fig. 17 shows a schematic flowchart of an image acquisition and processing method according to an embodiment of the present invention.

Referring to Figure 17, the image acquisition and processing method of the embodiment of the present invention includes:

Step S1701: Obtain an initial model of the fusion area based on the respective parameters and related parameters of the visible light camera and the infrared camera, so as to obtain the transition area according to the initial model of the fusion area.

Among them, the parameters of the initial model of the fusion area include:

Among them, x ₁ ≤ x _VR ≤ x ₂ , y ₁ ≤ y _VR ≤ y ₂ , among which, A, B, C, and D are conversion variables, which are used to simplify the above formulas of x ₁ ~ y ₂ to become larger, specifically:

x _VR and y _VR correspond to the pixel coordinates in the visible light image, and the range obtained according to the initial model of the fusion area is the transition area, and m, n, and d are the X, Z, and Y coordinates of the visible light camera and the infrared light camera, respectively. The distance of the projection on the axis, L _max is the farthest distance that the visible light camera can detect the corresponding object of the image,

is the angle between the optical axis of the infrared camera and the projection of the optical axis of the visible camera on the ZOX plane, and γ is the distance between the optical axis of the infrared camera and the optical axis of the visible camera on the ZOY plane The included angle of projection, the vertical axis of the front view plane of the device is parallel to the Z axis, w _IR is the horizontal display resolution of the infrared camera, h _IR is the vertical display resolution of the infrared camera, and w _VR is the infrared camera Horizontal display resolution, h _VR is the vertical display resolution of the infrared camera, α is the horizontal viewing angle of visible light, β is the vertical viewing angle of visible light, θ is the horizontal viewing angle of infrared light, φ is the vertical viewing angle of infrared light, A, B, C, and D are applicable to the model parameters in the previous calibration.

Among them, the transition area is a square, and x _VR and y _VR correspond to the pixel coordinates in the horizontal direction and the pixel coordinates in the vertical direction respectively. For example, if the image resolution is M*N, the pixel coordinates in the horizontal direction correspond to the M parameter, and the pixels in the vertical direction Coordinates correspond to N parameters.

An embodiment of the present invention provides a method for registration and fusion of visible light and infrared images. This method is applied to a device configured with a visible light camera and an infrared camera. FIG. 9 and FIG. 10 are schematic diagrams of horizontal positions of a visible light camera and an infrared camera on a device according to an embodiment of the present invention. 9 shows the device area 1, the infrared camera 2, the visible light camera 3, the horizontal line 4 of the front view plane of the device, the optical axis 5 of the infrared camera, the optical axis 6 of the visible light camera, and the lateral angle 7 between the two optical axes.

As shown in Figure 9, the optical axis 5 of the infrared camera is perpendicular to the plane of the front view of the device, and the visible light camera 3 is located below the infrared camera 2, and its optical axis 6 is not perpendicular to the plane of the front view of the device, but intersects with the optical axis 5 of the infrared camera.

Figure 10 shows the equipment area 1, the infrared camera 2, the visible light camera 3, the equipment front view plane horizontal line 4, the infrared camera optical axis 5, the visible light camera optical axis 6, and the lateral angle 7 of the two optical axes. As shown in Figure 4, the optical axis 5 of the infrared camera is perpendicular to the front view plane of the device, the visible light camera 3 is located below the infrared camera 2, and the optical axis 6 of the visible light camera is not perpendicular to the front view plane of the device, but intersects with the optical axis 5 of the infrared camera.

As shown in Figures 9 and 10, in this embodiment, the optical axis of the infrared camera is perpendicular to the plane of the front view of the device, and the optical axis of the visible light camera may not be perpendicular to the plane of the front view of the device. The relative angle of is 0 (that is, the position of the same object in the screen does not rotate).

The following will describe this embodiment in detail in conjunction with a scene where a visible light camera and an infrared camera are used for temperature measurement. Of course, the technical solution provided by this embodiment can also be applied to other scenes that require image fusion. As shown in Figure 11, the registration and fusion of visible light and infrared images provided in this embodiment may include the following steps:

Step S1101, establishing a registration model of a visible light image and an infrared image.

Specifically, in this step, according to the relative distances m, d, and n of the X, Y, and Z-axis spatial positions of the visible light and infrared cameras, the horizontal and vertical field angles, the display resolution, and the lateral clamping distance between the two optical axes Angle (that is, the angle between the ZOY planes of the two cameras), the longitudinal intersection angle of the two optical axes (that is, the angle between the ZOX planes of the two cameras), and the horizontal distance L between the target object and the visible light camera to establish a visible light image and an infrared image The following registration model for :

Among them, A, B, C, and D are conversion parameters, m, n, and d are the relative distances of the X, Y, and Z axes of the two cameras, which are known constants.

and γ are the horizontal and vertical angles between the optical axes of the two cameras, and the horizontal distance L between the target object and the visible light camera is the variation. x _VR and y _VR are the pixel coordinate values of the visible light image, and x _IR and y _IR are the pixel coordinate values of the infrared image.

In this embodiment, the conversion parameters A, B, C, and D can be obtained by calculating the horizontal and vertical viewing angles of visible light, the horizontal and vertical viewing angles of infrared light, and the display resolution parameters of infrared and visible light. E.g,

Among them, w _VR is the horizontal display resolution of the visible light camera, h _VR is the vertical display resolution of the visible light camera, w _IR is the horizontal display resolution of the infrared camera, h _IR is the vertical display resolution of the infrared camera, and α is the horizontal field of view of the visible light camera , β is the vertical viewing angle of the visible light camera, θ is the horizontal viewing angle of the infrared camera, and φ is the vertical viewing angle of the infrared camera.

Step S1102, calibrate the horizontal angle and vertical angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model. In this embodiment, instead of manually adjusting the visible light and infrared cameras, the horizontal and vertical angles can be calibrated as follows:

1) 1) Select a target object whose horizontal distance from the visible light camera is L _C . For example, the selection range of _LC can be [0.5m～7m];

2) Visible light and infrared cameras collect images of the target object at the same time (the target object can be a regular cuboid, cube, or a part of the human body, such as the human head, facial features or glasses), and find out the visible light at the same position of the object Coordinates in the image (x _VR-C , y _VR-C ) and coordinates in the infrared image (x _IR-C , y _IR-C );

3) According to the registration model of the visible light image and the infrared image, the lateral angle between the two optical axes can be calculated

and the vertical angle γ of the two optical axes:

Step S1103, establishing the height and width mapping model of the visible light image and the infrared image as follows:

Among them, A, C,

γ, n, L are all known, λ is a configuration parameter, and the range of λ is 0.1<λ≤1. By adjusting the size of λ, it is possible to completely avoid the interference caused by the background temperature brought by areas other than the face.

In this embodiment, the established height and width mapping models can further quickly fuse the size and ratio relationship of objects. For example, for the height H _VR-Face and width W _VR-Face of the face image in the visible light image where the center of the face image at a given distance L is located, H _{ir_face} can be obtained according to the height and width mapping scale model of the visible light image and the infrared image and W _{ir_face} , and further according to x _IR , y _IR , Hi _{ir_face} , and W _{ir_face} , the corresponding collected temperature information of the face area of the infrared image can be obtained.

Step S1104, establish a mapping table of the range intervals corresponding to different face heights in the visible light image and the horizontal distance L between the face and the visible light camera, for example, as shown in Table 3, the H _{vr_face} range interval can be divided into multiple intervals, where , H _k+1 >H _k >H _k-1 >...>H ₆ >H ₅ >H ₄ >H ₃ >H ₂ >H ₁ , L ₁ >L ₂ >L ₃ >L ₄ >...>L _k-2 >L _k-1 >L _k .

table 3

Step S1105, the human face detection module outputs one or more human face center position coordinates (x _VR , y _VR ) in the visible light image, and the height H _{vr_face} and width W _{vr_face} of the human face frame picture, and each detected person Find the corresponding distance L value in the mapping model for the height or width of the face picture, and input the corresponding distance L value and the coordinate value of the face center position (x _VR , y _VR ) into the registration model of the visible light image and the infrared image, Calculate and obtain the coordinate values (x _IR , y _IR ) of the center position of the face picture corresponding to the infrared image, until the calculation is completed until the coordinate values (x _VR , y _VR ) of the center position of the face picture of the infrared images corresponding to the currently detected multiple people are completed.

Step S1106, input the height and width of each detected face picture into the height and width mapping model of the visible light image and the infrared image, and calculate and obtain the height H _{ir_face} and width W _{ir_face} of the corresponding face picture.

Step S1107, according to the human face picture center position coordinates (x _IR , y _IR ) of the infrared image corresponding to each detected human face, the height H _{ir_face} and the width W _{ir_face} of the human face picture determine the area of the corresponding infrared image, from the corresponding Take the highest temperature in the infrared image area and record it as the face temperature of the corresponding person, and mark the temperature value of each corresponding person around or in the face frame of the visible light image of each corresponding face.

Through the above-mentioned steps of this embodiment, it is solved that the central optical axes of the visible light camera and the infrared camera are not parallel (there is an intersection angle between the two optical axes in the horizontal direction, and there is also an intersection angle in the vertical direction of the two optical axes), and the X-axis, Y-axis, and Z-axis directions of the two cameras are solved. The problem of image registration and fusion in the case of different positions, and further solves the problem of interference with face temperature detection caused by abnormal ambient temperature around the face in the application of temperature measurement scenes.

When the method of this embodiment is applied to the temperature measurement scene, when the position and area range data of multiple faces are detected in the visible light screen, the range of faces corresponding to all detected faces in the infrared screen can be quickly and accurately obtained. Further accurate acquisition of face temperature data corresponding to the face range can completely solve the problem of interference with face temperature detection caused by abnormal ambient temperature around the face, and greatly improve the detection efficiency and accuracy of face temperature.

Yet another embodiment of the present invention provides another method for registration and fusion of visible light and infrared images, which can be applied to a device equipped with a visible light camera and an infrared camera. In this embodiment, the spatial positional relationship between the visible light camera and the infrared camera can be referred to FIG. 9 and FIG. 10 .

As shown in Figures 9 and 10, in this embodiment, the optical axis of the infrared camera is perpendicular to the plane of the front view of the device, and the optical axis of the visible light camera may not be perpendicular to the plane of the front view of the device. The relative angle of is 0 (that is, the position of the same object in the screen does not rotate).

The following will describe this embodiment in detail in combination with a scene where a visible light camera and an infrared camera are used to measure the temperature of multiple people. Of course, the technical solution provided by this embodiment can also be applied to other scenes that require image fusion. As shown in Figure 12, the shooting scene is that a helmet equipped with a visible light camera and an infrared camera shoots a face in front of the helmet. The registration and fusion of visible light and infrared images provided in this embodiment may include the following steps:

Step S1201, establishing a registration model of a visible light image and an infrared image.

Specifically, in this step, according to the relative distances of the X, Y, and Z-axis spatial positions of the two visible light and infrared cameras as m, d, and n, the horizontal and vertical viewing angles, the display resolution, and the lateral direction of the two optical axes The included angle (that is, the angle between the ZOY planes of the two cameras), the longitudinal intersection angle of the two optical axes (that is, the angle between the ZOX planes of the two cameras), and the horizontal distance L between the target object and the visible light camera are used to establish a visible light image and an infrared image. The following registration model for the images:

Among them, A, B, C, and D are conversion parameters, m, n, and d are the relative distances of the X, Y, and Z axes of the two cameras, which are known constants.

and γ are the horizontal and vertical angles between the optical axis of the visible light camera and the infrared camera, and the horizontal distance L between the target object and the visible light camera is the variation. x _VR and y _VR are the pixel coordinate values of the visible light image, and x _IR and y _IR are the pixel coordinate values of the infrared image.

In this embodiment, the conversion parameters A, B, C, and D can be obtained by calculating the horizontal and vertical viewing angles of the visible light camera, the horizontal and vertical viewing angles of the infrared camera, and the display resolution parameters of the infrared camera and the visible light camera. E.g,

Among them, w _VR is the horizontal display resolution of the visible light camera, h _VR is the vertical display resolution of the visible light camera, w _IR is the horizontal display resolution of the infrared camera, h _IR is the vertical display resolution of the infrared camera, and α is the horizontal field of view of the visible light camera , β is the vertical viewing angle of the visible light camera, θ is the horizontal viewing angle of the infrared camera, and φ is the vertical viewing angle of the infrared camera.

Step S1202, calibrate the horizontal angle and vertical angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model. In this embodiment, another way of calibrating the horizontal angle and the vertical angle in the registration model is provided, specifically, the following steps may be included:

1) Instead of manually adjusting the visible light and infrared cameras, select an object whose horizontal distance from the visible light camera is L _C. For example, the selection range of _LC can be [0.3m～7m];

2) Adjust the optical axis of the visible light camera so that the same position of the object or human body is located at a specific position in the visible light picture and the infrared picture, and the specific position is displayed with a marker on both the visible light picture and the infrared picture (such as crosshairs or other marked images) , for example: x _IR and y _IR are both 0, x _VR is 200, y _VR is 100;

3) According to the registration model of the visible light image and the infrared image, the lateral angle between the two optical axes can be calculated

and the longitudinal intersection angle γ of the two optical axes are respectively:

The specific position of this embodiment can be flexibly selected, for example, in another embodiment, x _VR , y _VR , x _IR , y _IR are all 0, and the two optical axes can be calculated according to the registration model of the visible light image and the infrared image horizontal angle of

and the longitudinal intersection angle γ of the two optical axes are:

Step S1203, establishing the height and width mapping model of the visible light image and the infrared image as follows:

Among them, A, C,

γ, n, L are all known, λ is a configuration parameter, and the range of λ is 0.1<λ≤1. By adjusting the size of λ, the interference of the background temperature brought by the area other than the face can be avoided.

In this embodiment, the established height and width mapping models can further quickly fuse the size and ratio relationship of objects. For example, for the height H _VR-Face and width W _VR-Face of the face image in the visible light image where the center of the face image at a given distance L is located, H _{ir_face} can be obtained according to the height and width mapping scale model of the visible light image and the infrared image and W _{ir_face} , and further according to x _IR , y _IR , Hi _{ir_face} and W _{ir_face} can obtain the corresponding collected temperature information of the face area of the infrared image.

Step S1204, establish a mapping table of the range intervals corresponding to different face heights in the visible light image and the horizontal distance L between the face and the visible light camera, for example, as shown in Table 4, the H _{vr_face} range interval can be divided into multiple intervals, where , Hk+1>Hk>Hk-1>......>H6>H5>H4>H3>H2>H1, L1>L2>L3>L4>......>Lk-2>Lk -1>Lk.

Table 4

Step S1205, the face detection module outputs one or more coordinate values of the center position of the face in the visible light image (x _VR , y _VR ), and the height H _{vr_face} and width W _{vr_face} of the face frame picture, and each detected person Find the corresponding distance L value in the mapping model for the height or width of the face picture, and input the corresponding distance L value and the coordinate value of the face center position (x _VR , y _VR ) into the registration model of the visible light image and the infrared image for calculation Obtain the coordinate values (x _IR , y _IR ) of the center position of the face picture corresponding to the infrared image until the calculation is completed until the coordinate values (x _VR , y _VR ) of the center position of the face picture of the infrared images corresponding to the currently detected multiple people are completed.

Step S1206, input the height and width of each detected face picture into the height and width mapping model of the visible light image and the infrared image, and calculate and obtain the height H _{ir_face} and width W _{ir_face} of the corresponding face picture.

Step S1207, according to the human face picture center position coordinates (x _IR , y _IR ) of the infrared image corresponding to each detected human face, the height H _{ir_face} and the width W _{ir_face} of the human face picture determine the area of the corresponding infrared image, from the corresponding Take the highest temperature in the infrared image area and record it as the face temperature of the corresponding person, and mark the temperature value of each corresponding person around or in the face frame of the visible light image of each corresponding face.

Through the above-mentioned steps of this embodiment, it is solved that the central optical axes of the visible light camera and the infrared camera are not parallel (there is an intersection angle between the two optical axes in the horizontal direction, and there is also an intersection angle in the vertical direction of the two optical axes), and the X-axis, Y-axis, and Z-axis directions of the two cameras are solved. The problem of image registration and fusion in the case of different positions, and further solves the problem of interference with face temperature detection caused by abnormal ambient temperature around the face in the application of temperature measurement scenes.

In another embodiment shown in FIGS. 15 and 16 , the optical axis 21 of the infrared camera 02 is perpendicular to the plane of the front view of the device, and the optical axis 11 of the visible light camera 01 is not perpendicular to the plane of the front view of the device. Correspondingly, The initial model parameters of its fusion area include:

The transition region (x _VR , y _VR ) satisfies: x ₁ ≤ x _VR ≤ x ₂ , y ₁ ≤ y _VR ≤ y ₂ , where other parameters are the same as those in the foregoing embodiments, and will not be described in detail here. Subsequent processing of the transition region in this embodiment is the same as that of the transition region in the preceding embodiment, and details will not be repeated hereafter.

Fig. 18 shows a schematic diagram of a partial interface of a device adopting an image acquisition and processing method according to an embodiment of the present invention.

Referring to FIG. 18 , the corresponding visible light image area 40 can cover the entire area of the device interface, and the fusion area 41 is smaller than the visible light image area 40. Finally, only the image data within the range of the fusion area 41 is processed to obtain the information of the acquisition object A, and the acquired The information of the object A is displayed separately in the lower left corner of the interface, and the information of the collected object B outside the fusion area 41 is not collected, which effectively reduces the amount of data processing.

Among them, the collection object A and the collection object B are, for example, human faces, and the infrared camera is used to collect the temperature of the face, and only face recognition is performed on the visible light image within the range of the fusion area 41, so that the collection object A can be quickly locked, and then the locked The face temperature of the collected object A is detected, and the test result is displayed separately in the lower left corner of the interface, which can improve the efficiency of face recognition and temperature detection.

The image collection and processing method of the present invention uses a visible light camera and an infrared camera to simultaneously collect images, and obtains the range of the fusion region according to the comparison between the initial model of the fusion region and the resolution of visible light, wherein the visible light image within the fusion region is analyzed to obtain the collected image The feature information can reduce the amount of data processing, save computing resources, and improve the efficiency of image processing. It can effectively improve the processing efficiency in image analysis processing such as face recognition.

The initial model of the fusion area is obtained according to the fixed parameters and related parameters of the visible light camera and the infrared light camera. The fixed parameters and related parameters of the visible light camera and the infrared light camera do not need to be adjusted after calibration, which ensures the convenience of use.

The technical solution provided in this embodiment can solve the problem of image registration and fusion in the case where the central optical axes of the visible light camera and the infrared camera are not parallel, and the positions of the two cameras in the X-axis, Y-axis, and Z-axis directions are different. , to solve the problem of interference with face temperature detection caused by abnormal ambient temperature around the face in the temperature measurement scene application. When the technical solution provided by this embodiment is applied to the position and area range data of multiple human faces detected in the visible light screen, the range of faces corresponding to all detected faces in the infrared screen can be obtained quickly and accurately, and the corresponding face ranges can be further accurately obtained. The face temperature data in the face range can completely solve the problem of interference with face temperature detection caused by abnormal ambient temperature around the face, and greatly improve the detection efficiency and accuracy of face temperature. In addition, the calibration process of the image registration and fusion model provided in this embodiment is simple and fast, and avoids a large amount of complicated computing resource consumption. Compared with other image fusion algorithms that require more computing resources to support It requires fewer computing resources and is more efficient.

Claims (29)

1. A processing method for visible light images and infrared images, which is applied to equipment equipped with visible light cameras and infrared cameras, wherein the optical axis of the visible light camera or the optical axis of the infrared camera is perpendicular to the plane of the front view of the equipment, its features In, including:

   According to the spatial relative position of the visible light camera and the infrared camera, conversion parameters and the horizontal distance between the target object and the visible light camera, establish the visible light image of the target object collected by the visible light camera and the infrared image of the target object collected by the infrared camera. The registration model of the coordinate position of the image;

   The visible light image and the infrared image of the target object are registered and fused according to the registration model.
2. The method according to claim 1, wherein the optical axis of the visible light camera is perpendicular to the front view plane of the device, wherein the registration model is:

   

   Among them, A, B, C, and D are the first, second, third, and fourth conversion parameters, respectively, and m, n, and d are the relative distances between the X, Y, and Z axes of the visible light camera and the infrared camera, respectively,

   

   and γ are the horizontal and vertical angles between the optical axes of the visible light camera and the infrared camera, respectively, L is the horizontal distance between the target and the visible light camera, (x _VR , y _VR ) is the pixel coordinate of the visible light image, (x _IR , y _IR ) is the infrared image pixel coordinates.
3. The method according to claim 2, wherein, in the registration model, the horizontal angle and the vertical angle between the optical axis of the visible light camera and the infrared camera are obtained by the following steps:

   Selecting a first reference object whose horizontal distance from the visible light camera is a first set distance;

   Simultaneously collect images of the first reference object through the visible light camera and the infrared camera, and respectively measure the horizontal length and the vertical length of the same position of the first reference object in the visible light image and the infrared image respectively;

   According to the first set distance, and the horizontal length and vertical length of the same position of the first reference object in the visible light image and the infrared image, respectively, the optical axis and the infrared light axis of the visible light camera are calibrated in the registration model. The horizontal angle and vertical angle of the optical axis of the camera.
4. The method according to claim 3, wherein the lateral angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model is calibrated by the following formula

   

   and longitudinal angle γ:

   

   Among them, L _C is the first set distance, L _VR and W _VR are the horizontal length and vertical length of the same position of the first reference object in the visible light image respectively, L _IR and W _IR are the first reference The horizontal length and vertical length of the same position of the object in the infrared image.
5. The method according to claim 3, wherein, in the registration model, the horizontal angle and the longitudinal angle of the visible light camera and the infrared camera optical axis are obtained by the following steps:

   Selecting a second reference object whose horizontal distance from the visible light camera is a second set distance;

   adjusting the optical axis of the visible light camera so that the same position of the second reference object is located at a specific position of the visible light image and the infrared image;

   According to the second set distance and the coordinate value of the specific position, the horizontal angle and the longitudinal intersection angle between the optical axis of the visible light camera and the optical axis of the infrared camera are calibrated in the registration model.
6. The method according to claim 5, wherein the lateral angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model is calibrated by the following formula

   

   and longitudinal angle γ:

   

   Wherein, _LC is the second set distance, the coordinates of the same position of the second reference object located at a specific position in the visible light image are (0,0), and the same position of the second reference object is located at The coordinates of a specific location are (0,0);

   or,

   

   Wherein, _LC is the second set distance, the coordinates of the same position of the second reference object located at a specific position in the visible light image are (200,100), and the same position of the second reference object is located at a specific position in the visible light image The coordinates of are (0,0).
7. The method according to claim 3 or 5, characterized in that, after calibrating the horizontal angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model, further comprising:

   Substituting the horizontal angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera into the registration model, and establishing a height and width mapping model of the visible light image and the infrared image;

   Based on multiple different horizontal distances between the target object and the visible light camera, establishing a mapping relationship between the height of the designated area of the target object in multiple groups of visible light images and the horizontal distance between the target object and the visible light camera .
8. The method according to claim 7, wherein registering and fusing the visible light image and the infrared image of the target object according to the registration model comprises:

   Acquiring the center position coordinate value of the specified area of the target object in the visible light image, and the height and width of the specified area of the target object in the visible light image, and according to the specified area of the target object in the visible light image height and width, finding the horizontal distance value corresponding to the target object and the visible light camera in the height and width mapping model;

   Inputting the corresponding horizontal distance value and the central position coordinate value of the designated area of the target object into the registration model, and calculating and obtaining the central position coordinate value of the designated area of the target object in the infrared image;

   inputting the height and width of the specified area of the target object in the visible light image into the height and width mapping model, and calculating the height and width of the specified area of the target object in the infrared image;

   Determine the designated area of the target object in the infrared image according to the center position coordinates of the designated area of the target object in the infrared image, and the height and width of the designated area of the target object in the infrared image.
9. The method according to claim 8, further comprising: after determining the designated area of the target object in the infrared image:

   Acquiring the highest temperature value in a specified area of the target object in the infrared image;

   Marking the temperature value at a designated position of a designated area of the target object in the visible light image.
10. The method according to claim 1, wherein the optical axis of the infrared camera is perpendicular to the plane of the front view of the device, wherein the registration model is:

    

    Among them, A, B, C, and D are the first, second, third, and fourth conversion parameters, respectively, and m, n, and d are the relative distances between the X, Y, and Z axes of the visible light camera and the infrared camera, respectively,

    

    and γ are the horizontal and vertical angles between the optical axis of the visible light camera and the infrared camera, respectively, L is the horizontal distance between the target and the visible light camera, (x _VR , y _VR ) is the pixel coordinate of the visible light image, (x _IR , y _IR ) is the pixel coordinates of the infrared image.
11. The method according to claim 10, characterized in that the horizontal angle and the vertical angle between the optical axes of the visible light camera and the infrared camera in the registration model are obtained by the following steps:

    Selecting a first reference object whose horizontal distance from the visible light camera is a first set distance;

    Collecting images of the first reference object simultaneously through a visible light camera and an infrared camera, and obtaining coordinates of the same position of the first reference object in the visible light image and the infrared image respectively;

    Calibrate the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model according to the first set distance and the coordinates of the same position of the first reference object in the visible light image and the infrared image respectively The horizontal and vertical angles of intersection.
12. The method according to claim 3, wherein the lateral angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model

    

    and the longitudinal angle γ is:

    

    

    Among them, LC is the first set distance, (x _{VR-C ,} y _VR-C ) is the coordinates of the same position of the first reference object in the visible light image, and the coordinates of the same position of the first reference object in the infrared image (x _IR-C ,y _IR-C ).
13. The method according to claim 2, wherein the horizontal angle and the vertical angle between the optical axes of the visible light camera and the infrared camera in the registration model are obtained by the following steps:

    Selecting a second reference object whose horizontal distance from the visible light camera is a second set distance;

    adjusting the optical axis of the visible light camera so that the same position of the second reference object is located at a specific position of the visible light image and the infrared image;

    According to the second set distance and the coordinate value of the specific position, the horizontal angle and the longitudinal intersection angle between the optical axis of the visible light camera and the optical axis of the infrared camera are calibrated in the registration model.
14. The method according to claim 5, characterized in that, the lateral angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model

    

    and the longitudinal angle γ is:

    

    Wherein, _LC is the second set distance, the coordinates of the same position of the second reference object located at a specific position in the visible light image are (0,0), and the same position of the second reference object is located at The coordinates of a specific location are (0,0);

    or,

    

    Wherein, _LC is the second set distance, the coordinates of the same position of the second reference object located at a specific position in the visible light image are (200,100), and the same position of the second reference object is located at a specific position in the visible light image The coordinates of are (0,0).
15. The method according to claim 3 or 5, characterized in that, after calibrating the horizontal angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera in the registration model, further comprising:

    Substituting the horizontal angle and longitudinal angle between the optical axis of the visible light camera and the optical axis of the infrared camera into the registration model, and establishing a height and width mapping model of the visible light image and the infrared image;

    Based on multiple different horizontal distances between the target object and the visible light camera, establishing a mapping relationship between the height of the designated area of the target object in multiple groups of visible light images and the horizontal distance between the target object and the visible light camera .
16. The method according to claim 7, wherein registering and fusing the visible light image and the infrared image of the target object according to the registration model comprises:

    Acquiring the center position coordinate value of the specified area of the target object in the visible light image, and the height and width of the specified area of the target object in the visible light image, and according to the specified area of the target object in the visible light image height and width, finding the horizontal distance value corresponding to the target object and the visible light camera in the height and width mapping model;

    Inputting the corresponding horizontal distance value and the central position coordinate value of the designated area of the target object into the registration model, and calculating and obtaining the central position coordinate value of the designated area of the target object in the infrared image;

    inputting the height and width of the specified area of the target object in the visible light image into the height and width mapping model, and calculating the height and width of the specified area of the target object in the infrared image;

    The designated area of the target object in the infrared image is determined according to the center position coordinates of the designated area of the target object in the infrared image, and the height and width of the designated area of the target object in the infrared image.
17. The method according to claim 8, wherein, after determining the designated area of the target object in the infrared image, further comprising:

    Acquiring the highest temperature value in a specified area of the target object in the infrared image;

    Marking the temperature value at a designated position of a designated area of the target object in the visible light image.
18. The method of claim 1, comprising:

    Use the visible light camera and infrared light camera of the equipment to collect images at the same time;

    Obtain the transition region according to the initial model parameters of the fusion region;

    Obtaining a fusion area according to the comparison between the transition area and the pixel resolution of the visible light image;

    The visible light images within the range of the fusion area are analyzed to obtain feature information of the collected images.
19. The method according to claim 1, characterized in that,

    The initial model of the fusion area is obtained according to fixed parameters of the infrared camera and the visible light camera.
20. The method according to claim 2, wherein the optical axis of the visible light camera is perpendicular to the front view plane, and the initial model parameters of the fusion area include:

    

    

    in,

    x ₁ ≤ x _VR ≤ x ₂ , y ₁ ≤ y _VR ≤ y ₂ ,

    x _VR and y _VR correspond to the pixel coordinates of the transition region, m, n, and d are respectively the projection distances between the visible light camera and the infrared light camera on the X, Z, and Y axes, and L _max is the visible light The camera can detect the farthest distance of the image counterpart,

    

    is the angle between the optical axis of the infrared camera and the projection of the optical axis of the visible camera on the ZOX plane, and γ is the distance between the optical axis of the infrared camera and the optical axis of the visible camera on the ZOY plane The included angle of projection, the vertical axis of the front view plane of the device is parallel to the Z axis, w _IR is the horizontal display resolution of the infrared camera, h _IR is the vertical display resolution of the infrared camera, and w _VR is the infrared camera Horizontal display resolution, h _VR is the vertical display resolution of the infrared camera,

    

    α is the horizontal viewing angle of visible light, β is the vertical viewing angle of visible light, θ is the horizontal viewing angle of infrared light, and φ is the vertical viewing angle of infrared light.
21. The method according to claim 2, wherein the optical axis of the infrared camera is perpendicular to the plane of the front view, and the initial model parameters of the fusion area include:

    

    

    in,

    x ₁ ≤ x _VR ≤ x ₂ , y ₁ ≤ y _VR ≤ y ₂ ,

    x _VR and y _VR correspond to the pixel coordinates of the transition region, m, n, and d are respectively the projection distances between the visible light camera and the infrared light camera on the X, Z, and Y axes, and L _max is the visible light The camera can detect the farthest distance of the image counterpart,

    

    is the angle between the optical axis of the infrared camera and the projection of the optical axis of the visible camera on the ZOX plane, and γ is the distance between the optical axis of the infrared camera and the optical axis of the visible camera on the ZOY plane The included angle of projection, the vertical axis of the front view plane of the device is parallel to the Z axis, w _IR is the horizontal display resolution of the infrared camera, h _IR is the vertical display resolution of the infrared camera, and w _VR is the infrared camera Horizontal display resolution, h _VR is the vertical display resolution of the infrared camera,

    

    α is the horizontal viewing angle of visible light, β is the vertical viewing angle of visible light, θ is the horizontal viewing angle of infrared light, and φ is the vertical viewing angle of infrared light.
22. The method according to claim 3 or 4, wherein the step of obtaining the blended area according to the comparison between the transition area and the pixel resolution of the visible light image comprises:

    According to the comparison between the upper limit and the lower limit of the pixel coordinates of the transition area and the resolution of the visible light image, the upper limit and the lower limit of the pixel coordinates of the fusion area are obtained.
23. The method according to claim 5, characterized in that, according to the upper limit and lower limit of the pixel coordinates of the visible light image in the initial model of the fusion area and the parameters related to the visible light resolution, the upper limit and the lower limit of the range of the fusion area are obtained. Steps include:

    When x ₁ >-w _VR , w _min =x ₁ , otherwise, w _min =-w _VR ;

    When x ₂ <w _VR , w _max =x ₂ , otherwise, w _max =w _VR ;

    When y ₁ >-h _VR , h _min =y ₁ , otherwise, h _min =-h _VR ;

    When y ₂ <h _VR , h _max =y ₂ , otherwise, h _max =h _VR ,

    The pixel coordinates of the fusion area satisfy w _min ≤ x _VR ≤ w _max , h _min ≤ y _VR ≤ h _max .
24. The method according to claim 3 or 4, characterized in that,

    The projections on the X, Y, and Z axes of the visible light camera and the infrared light camera are all spaced apart from each other.
25. The method according to claim 3 or 4, further comprising:

    Calibrate the visible light camera and the infrared camera to confirm the angle between the optical axis of the infrared camera and the projection of the optical axis of the visible light camera on the ZOX plane, and confirm the The included angle between the optical axis and the projection of the optical axis of the visible light camera on the ZOY plane.
26. The method according to claim 8, wherein the step of calibrating the visible light camera and the infrared light camera comprises:

    collecting images of the same object through the fixed visible light camera and the infrared light camera, and obtaining a visible light image and an infrared light image of the same object;

    Calculate according to the horizontal length and vertical length of the same position of the same object in the visible light image and the infrared light image of the same object, and obtain the optical axis of the infrared light camera and the optical axis of the visible light camera on the ZOX plane The included angle of the projection of , and the included angle of the projection of the optical axis of the infrared camera and the optical axis of the visible camera on the ZOY plane.
27. The method according to claim 8, wherein the step of calibrating the visible light camera and the infrared camera comprises:

    Adjusting the optical axis of at least one of the visible light camera and the infrared light camera, so that the same position of the same object is located in the respective specific positions of the visible light picture and the infrared light picture, so that the visible light camera and the infrared light camera The angle of the optical axis is adjusted to a preset value to confirm the angle between the optical axis of the infrared camera and the projection of the optical axis of the visible light camera on the ZOX plane, and to confirm that the optical axis of the infrared camera and The included angle of the projection of the optical axis of the visible light camera on the ZOY plane.
28. A processing device for visible light images and infrared images, located on a device equipped with a visible light camera and an infrared camera, wherein the optical axis of the visible light camera or the optical axis of the infrared camera is perpendicular to the plane of the front view of the device, characterized in that ,include:

    The registration model building module is used to establish the visible light image of the target object collected by the visible light camera and the infrared image according to the relative spatial position of the visible light camera and the infrared camera, the conversion parameters, and the horizontal distance between the target object and the visible light camera. The registration model of the coordinate position of the infrared image of the target object collected by the infrared camera;

    An image fusion module, configured to perform registration and fusion of the visible light image and the infrared image of the target object according to the registration model.
29. An image acquisition and processing device, characterized in that it comprises:

    A device, the device comprising a visible light camera and an infrared light camera, the optical axis of the visible light camera is perpendicular to the front view plane of the device;

    The processing unit adopts the image processing method according to any one of claims 1 to 28 to obtain feature information of the collected image.

Images