CN114897947A - A Synchronous Registration Method for Thermal Infrared and Visible Light Images Based on Unity of Time and Space - Google Patents

Abstract

The invention discloses a thermal infrared and visible light image synchronous registration method based on time-space unification, which comprises the steps of building a chessboard calibration plate, and collecting a thermal infrared image and a visible light image which comprise the chessboard calibration plate, wherein the thermal infrared image and the visible light image have unified reference time, and a plurality of groups of thermal infrared and visible light image pairs are obtained based on the collected images and the reference time; processing the thermal infrared and visible light image pair through sub-pixel edge corner detection to obtain the position coordinates of the calibration plate corner; and solving an optimal homography transformation matrix based on the position coordinates of the calibration plate corner points, wherein the optimal homography transformation matrix is used for realizing synchronous registration of the thermal infrared image and the visible light image. The invention can provide good prior work for the fusion processing of the thermal infrared and visible light images and ensure the scene consistency of multi-modal data.

Description

A Synchronous Registration Method for Thermal Infrared and Visible Light Images Based on Unity of Time and Space

technical field

The invention relates to the technical field of thermal infrared and visible light image processing, in particular to a synchronous registration method of thermal infrared and visible light images based on the unity of time and space.

Background technique

Image analysis processing includes various tasks such as image segmentation, stitching, reconstruction, and detection. At present, the method of comparing or fusing effective information between multiple images is gradually becoming the mainstream in the field of image processing. Whether it is single-modal or multi-modal image information, there is inevitably the phenomenon of dislocation of the observation area or object. As an important link in the field of image processing, image registration technology aims to find the best spatial alignment between two or more images in the same scene. The quality of the registration results will directly affect the effect of subsequent work.

The existing image registration algorithms are mainly divided into two categories: methods based on grayscale statistics and methods based on feature matching. Among them, the image registration method based on feature matching has become the most widely used and most frequently used method because of its high operation speed and good robustness. Algorithms like SIFT and SURF have good results in single-modal images and have been widely studied and applied. However, in the actual application process of single-modal image data, especially when the sensor is in motion, it is often easily affected by special environments, resulting in inaccurate and incomplete information content. On the one hand, when the lighting conditions are not good enough, the feature information reflected in the visible light image will be greatly reduced, while the thermal infrared image has the advantage of resisting light interference; on the other hand, the existing thermal infrared images are generally of low resolution and The edge information of the detected object is blurred, while visible light has high resolution and contains rich feature information. It can be seen that the use of the respective imaging characteristics of visible light and thermal infrared sensors to form a unified representation of multi-modal data will greatly promote the subsequent image analysis and processing tasks. However, for the multi-modal data of thermal infrared and visible light images, it is still difficult to actively find the feature relationship that can be correctly matched with the current more advanced algorithms. Moreover, because of the differences in imaging resolution and focal length between sensors, rigid body or affine transformation cannot simply be used as a transformation model, and high-precision registration of the two types of images cannot be achieved.

At the same time, the generalized image registration should also include the unification of time. Since different sensors have different signal sampling frequencies and data storage times, the data collected at the same time may not originate from the same real scene. This will cause the content of multimodal data fusion to be mismatched, resulting in more noise and irrelevant items than single-modal data, which is not conducive to highlighting the advantages of each modal instance.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned technical problems existing in the above-mentioned prior art, the present invention provides a synchronous registration method of thermal infrared and visible light images based on the unity of time and space. A priori works to ensure scene consistency for multimodal data.

In order to realize the above-mentioned technical purpose, the present invention provides the following technical solutions:

A synchronous registration method for thermal infrared and visible light images based on time-space unity, comprising:

Build a checkerboard calibration board and collect thermal infrared images and visible light images including the checkerboard calibration board. The thermal infrared images and visible light images have a unified reference time. Based on the collected images and reference time, several sets of thermal infrared and visible light image pairs are obtained. ;

The thermal infrared and visible light image pairs are processed through sub-pixel edge corner detection to obtain the position coordinates of the corners of the calibration board;

An optimal homography transformation matrix is obtained based on the position coordinates of the corner points of the calibration plate, wherein the optimal homography transformation matrix is used to achieve simultaneous registration of thermal infrared and visible light images.

Optionally, the checkerboard calibration board includes a white grid background and a black grid, wherein the white grid background uses an aluminum plate, and the black grid uses a black rubber cloth.

Optionally, when the thermal infrared image and the visible light image are collected, the reference times of the thermal infrared image and the visible light image are unified through a multi-threaded cross-processing method.

Optionally, the process of unifying the reference time through the multi-thread cross processing method includes:

Collect data streams through infrared thermometer and visible light camera respectively;

Collecting data streams for the infrared thermometer and the visible light camera and assigning threads to obtain a first thread and a second thread, wherein the first thread is the processing thread for the infrared thermometer to collect the data stream, and the second thread is the processing thread for the visible light camera to collect the data stream thread;

When the first thread or the second thread reaches the data storage link, the thread that reaches the data storage link starts to wait until another thread also reaches the data storage link;

When both the first thread and the second thread reach the data storage link, cache the data streams in the first thread and the second thread;

Extract the cached data stream through the third thread and convert it into image data respectively, and release the cache at the same time;

After the cache is released, the real-time collected data streams in the first thread and the second thread are cached again, and the thermal infrared image and the visible light image are obtained through the process of circular cache, extraction, conversion and release of the cache until the thread ends.

Optionally, the process of acquiring several sets of thermal infrared and visible light image pairs includes:

The thermal infrared images and visible light images at the same reference time are correspondingly integrated to obtain several sets of thermal infrared and visible light image pairs.

Optionally, the process of obtaining the position coordinates of the corner points of the calibration board includes:

The thermal infrared and visible light image pairs are processed by the edge detection operator to obtain the edge position of the whole pixel of the checkerboard;

Based on the edge position, select the edge pixel as the center of the sub-pixel point, obtain the normal based on the center of the sub-pixel point, obtain the sub-pixel point based on the normal line and single-pixel distance, and construct the fitting function based on the gray value of the sub-pixel point , where the fitting function is the arctangent function;

The fitting function is solved by the least square method, and the edge function model is obtained;

Calculate the edge function model through quadratic differentiation to obtain sub-pixel edge points;

The edge line is fitted by the sub-pixel edge points in four directions around the thick corner point, and the intersection point is obtained, where the intersection point is used as the position coordinate of the calibration board corner point.

Optionally, the gray value of the sub-pixel point is calculated by linear interpolation in the horizontal direction and the vertical direction.

Optionally, the process of solving the optimal homography transformation matrix includes:

Match the position coordinates of the corner points of the calibration plate in the thermal infrared and visible light image pairs, obtain a coordinate pair set, and randomly select several coordinate pairs as the interior point set;

Based on the interior point set, the initial homography matrix is constructed by the least square method;

Through the initial homography matrix, the projection error is calculated for the coordinate pairs except the interior point set in the coordinate pair set, and the error judgment is performed on the projection error calculation result. When the projection error is less than the threshold, the coordinate pair is added to the interior point set. , if the projection error is greater than the threshold, the coordinate pair will not be added to the interior point set;

Count the number of coordinate pairs in the updated interior point set;

By repeating the process from building a homography matrix to counting the number of coordinate pairs, until the number of iterations is reached, the optimal homography matrix is obtained, where the optimal homography matrix is the homography with the largest number of coordinate pairs in the corresponding interior point set Responsiveness Matrix.

The present invention has the following technical effects:

(1) The present invention adopts multi-thread cross control, introduces identifier and global monitoring mechanism, effectively reduces the acquisition time difference of each sensor, and ensures the synchronization of imaging time stamps within the ultra-low delay range. At the same time, the method is not limited to pre-registration work, and can also be used in real-time applications;

(2) The present invention uses the edge gray distribution of the checkerboard to fit the corner positions obtained by the edge lines, and replaces the feature points to realize the unification of the space scene, effectively reduces the error caused by the feature point extraction and matching, and can improve the registration accuracy and accuracy. The registration speed is simple, and the method is easy to implement.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative labor.

FIG. 1 is a schematic flowchart of a synchronous registration method for thermal infrared and visible light images based on the unified time and space of the present invention;

Fig. 2 is the image acquisition flow chart of the present invention taking infrared thread as an example;

Fig. 3 is the schematic diagram of the installation platform of the infrared thermometer and the visible light camera of the present invention;

4 is a schematic diagram of a 5×5 neighborhood of an edge pixel of the present invention;

5 is a schematic diagram of bi-directional linear grayscale interpolation of sub-pixel points according to the present invention;

FIG. 6 is a visible light image and an infrared image before and after registration according to the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In order to solve the problems existing in the prior art, the present invention provides the following solutions:

As shown in Figure 1, the present invention provides a method for synchronous registration of thermal infrared and visible light images based on the unity of time and space. After preprocessing the formed image, a sub-pixel edge corner detection method is used to precisely locate the checkerboard corners in the two types of images, and finally the optimal homography transformation matrix is solved by random sampling consensus algorithm to complete the matching. allow.

Technical route: First, based on the multi-threaded cross-processing method, unify the reference time of the infrared thermometer and visible light camera to collect data; then build a checkerboard calibration board with different black and white emissivity, adjust and fix the installation position of the infrared thermometer and the visible light camera to ensure that A sensor with a large field of view can completely contain the scene information collected by another sensor; further, two devices are used to collect images of the calibration board at different positions, angles and attitudes, and noise filtering preprocessing is performed; and then the sub-pixel edge is used. Corner detection obtains the position coordinates of the corner points of the calibration plate in each group of images and saves them in sequence. Finally, based on the robust method of random sampling and consistent, the optimal homography transformation matrix is solved by the least square method to complete the registration.

Example 1

A thermal infrared and visible light image synchronous registration method based on time-space unity in this embodiment includes the following steps:

Step 1: As shown in Figure 2, the image acquisition flow chart with infrared threads as an example is used to unify the reference time of the infrared thermometer and the visible light camera based on the multi-thread cross processing method. The specific steps are:

Step 1.1: Start the multi-threading module, and the infrared thermometer and the visible light camera start to collect data streams respectively;

Step 1.2: Allocate and judge the first identifier 1 and the second identifier 2 to the thermal infrared and visible light processing threads, and the initial value is set to False;

Step 1.3: Establish a global monitoring mechanism. When a thread reaches the data stream information storage link, its identifier is changed to True. If another thread identifier is still False, it starts to wait until another thread also reaches the data stream. After the information storage process, convert its identifier to True;

Step 1.4: When the identifiers 1 and 2 are both True, the two threads start to store the frame information in the current data stream into the cache;

Step 1.5: After the frame information is stored, change the corresponding identifier to False again;

Step 1.6: Without affecting the synchronization process, another thread extracts the frame information from the cache, saves it as image data, and releases the cache at the same time;

Step 1.7: Return to step 1.3, store the next data stream information, and run in a loop until the thread ends.

Step 2: Build a checkerboard calibration board with different emissivity of black and white grids, choose a smooth aluminum plate with a lower emissivity as the white grid background, and a rough black rubber cloth with a higher emissivity as the black grid. The visible light camera can still identify the corner points according to the color information; at the same time, the infrared thermometer can also passively detect the checkerboard image with clear corner points at the same temperature, eliminate the corner blur error caused by the active heating operation, and realize two kinds of sensors. effective synchronous acquisition;

Step 3: Connect the two sensor devices to the same industrial computer loaded with the multi-threaded cross-control acquisition method in step 1, and use the infrared thermometer and visible light camera installation platform as shown in Figure 3 to adjust and fix the installation positions of the two sensors , to ensure that the large field of view sensor can fully contain the scene information collected by the other sensor. At the same time, two devices are used to collect images of the calibration board at different positions, angles and attitudes, record multiple sets of thermal infrared and visible light image pairs, and perform noise filtering preprocessing;

Step 4: Use sub-pixel edge corner detection to obtain the position coordinates of the calibration board corners in each group of images and save them in sequence. The specific steps are:

Step 4.1: Use the Canny operator to initially obtain the edge position of the whole pixel of the checkerboard;

Step 4.2: By performing corner detection on the pixel edge position, select any pixel A in the four edge directions of a corner O as the center of the sub-pixel point, and use the outermost two in the 5×5 neighborhood as shown in Figure 4. The connection line of the edge pixel points is used as the tangent line in the neighborhood, and the angle θ in the horizontal direction and the normal line F in the vertical direction are obtained;

Step 4.3: Set two sub-pixel points a ₁ , a ₂ and a ₃ , a ₄ at each end of the center pixel point A at a distance of a single pixel along the normal direction to obtain 5 sub-pixel points in the normal direction;

和在像素(0，1)和(1，1)的灰度在水平方向的线性插值

分别为：Step 4.4: As shown in Figure 5, taking a ₁ as an example, set the coordinates of the sub-pixels close to the center in 4 whole pixel neighborhoods as (x, y), and the distance to the center pixel (0, 0) as 1 , then x=cosθ, y=sinθ. First get the linear interpolation of the gray levels of pixels (0, 0) and (1, 0) in the horizontal direction

and the linear interpolation of the gray levels at pixels (0, 1) and (1, 1) in the horizontal direction

They are:

In the formula, I ₀₀ , I ₀₁ , I ₁₀ , and I ₁₁ represent the grayscale values of pixel coordinates (0, 0), (0, 1), (1, 0), and (1, 1), respectively. Then use the linear interpolation calculation in the vertical direction to obtain the gray value I _xy of the sub-pixel point as:

和在像素(0，2)和(2，2)的灰度在水平方向的线性插值

分别为：Step 4.5: As shown in Figure 5, taking a ₂ as an example, similar to step 4.4, set the coordinates of the sub-pixel points far away from the center in the 9 integer pixel neighborhood as (m, n), to the center pixel point (0, 0 ) is 2, then m=2cosθ, n=2sinθ. First get the linear interpolation of the gray level of the pixels (0, 0) and (2, 0) in the horizontal direction

and the linear interpolation in the horizontal direction of the gray levels at pixels (0, 2) and (2, 2)

They are:

In the formula, I ₀₂ , I ₂₀ , and I ₂₂ represent the grayscale values of pixel coordinates (0, 2), (2, 0), and (2, 2), respectively. Then use the linear interpolation calculation in the vertical direction to obtain the gray value I _mn of the sub-pixel point as:

Step 4.6: Using the gray values of the five sub-pixel points in the normal direction, use the arc tangent function to fit the actual gray distribution of the edge. The fitting function is represented by the following:

y=a ₁ arctan(a ₂ x+a ₃ )+a ₄

In the formula, y represents the gray value of the sub-pixel point, x represents the vector distance from the sub-pixel point to point A, a ₁ represents that the amplitude of the arc tangent function is expanded to the original a ₁ times, and a ₂ and a ₃ represent the inverse The degree of curvature of the tangent function curve, a ₄ represents the offset of the arctangent function in the vertical direction. Use least squares fitting to solve the values of a ₁ , a ₂ , a ₃ , a ₄ to obtain the edge function model;

Step 4.7: Define the sub-pixel edge point E at the coordinate point with the largest slope of the function, by taking the second derivative of the edge function:

为亚像素边缘点E在法线方向的位置，再结合θ即可求得亚像素边缘点相对于A点的完整偏移量，其中水平偏移量为

垂直偏移量为

基于A点坐标已知，因此可得到亚像素边缘点E的最终坐标；Available:

is the position of the sub-pixel edge point E in the normal direction, and combined with θ, the complete offset of the sub-pixel edge point relative to point A can be obtained, where the horizontal offset is

The vertical offset is

Based on the known coordinates of point A, the final coordinates of the sub-pixel edge point E can be obtained;

Step 4.8: Take the other three pixel points in the edge direction of the thick corner point O, obtain the corresponding sub-pixel edge points according to steps 4.2 to 4.7, and use the sub-pixel edge points in the opposite direction to fit two intersecting edge lines , obtain the intersection point as the corner coordinate and save it in sequence.

Step 5: Combine the above-mentioned sub-pixel point coordinate pair set S, use the robust method based on random sampling consistency, and combine with the least squares method to solve the optimal homography transformation matrix, as shown in Figure 6, where (a) in Figure 6 The visible light image and (b) in FIG. 6 and (c) in FIG. 6 are the infrared images before and after registration, respectively, and the specific steps are:

Step 5.1: Randomly select n=4 pairs of matching feature points from the initial matching pair set S as the interior point set S _i , convert the overdetermined equation into a nonlinear optimization problem, and estimate the initial homography by least squares fitting matrix H _i ;

Step 5.2: Use the H _i obtained in Step 5.1 to calculate the remaining matching point pairs in the set S. If the projection error of a feature point is less than the threshold t=5, it will be added to the set S _i , and the others will be treated as for the outside point;

Step 5.3: record the number of matching point _pairs in the set Si;

Step 5.4: Repeat the above steps until the number of iterations is greater than K=200;

Step 5.5: Select the model with the largest number of interior points as the homography matrix H to be solved. Simultaneous registration of thermal infrared and visible light images through an optimal homography transformation matrix.

The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above-mentioned embodiments, and the descriptions in the above-mentioned embodiments and the description are only to illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have Various changes and modifications fall within the scope of the claimed invention. The claimed scope of the present invention is defined by the appended claims and their equivalents.

Claims (8)

1. A thermal infrared and visible light image synchronous registration method based on time-space unification is characterized by comprising the following steps:

the method comprises the steps of building a chessboard calibration plate, collecting thermal infrared images and visible light images comprising the chessboard calibration plate, wherein the thermal infrared images and the visible light images have uniform reference time, and acquiring a plurality of groups of thermal infrared and visible light image pairs based on the collected images and the reference time;

processing the thermal infrared and visible light image pair through sub-pixel edge corner detection to obtain the position coordinates of the calibration plate corner;

and solving an optimal homography transformation matrix based on the position coordinates of the calibration plate corner points, wherein the optimal homography transformation matrix is used for realizing synchronous registration of the thermal infrared image and the visible light image.

2. The synchronous registration method based on temporal-spatial unification of thermal infrared and visible light images according to claim 1, wherein:

the chessboard calibration plate comprises a white lattice background and black lattices, wherein the white lattice background uses an aluminum plate, and the black lattices use black rubber cloth.

3. The synchronous registration method based on temporal-spatial unification of thermal infrared and visible light images according to claim 1, wherein:

when the thermal infrared image and the visible light image are collected, the reference time of the thermal infrared image and the reference time of the visible light image are unified through a multithreading cross processing method.

4. The method for synchronously registering the thermal infrared image and the visible light image based on the time-space unification as claimed in claim 1, wherein:

the process of unifying the reference time by the multithread cross processing method comprises the following steps:

respectively acquiring data streams through an infrared thermodynamics instrument and a visible light camera;

distributing threads to the data streams collected by the infrared thermal instrument and the visible light camera, and acquiring a first thread and a second thread, wherein the first thread is a processing thread for collecting the data streams by the infrared thermal instrument, and the second thread is a processing thread for collecting the data streams by the visible light camera;

when the first thread or the second thread reaches the data storage link, the thread reaching the data storage link starts to wait until the other thread also reaches the data storage link;

when the first thread and the second thread both reach a data storage link, caching data streams in the first thread and the second thread;

extracting the cached data streams through a third thread, respectively converting the data streams into image data, and simultaneously releasing the cache;

after releasing the cache, caching the data streams collected in real time in the first thread and the second thread again, and obtaining the thermal infrared image and the visible light image through the processes of circularly caching, extracting, converting and releasing the cache until the threads are finished.

5. The synchronous registration method based on temporal-spatial unification of thermal infrared and visible light images according to claim 1, wherein:

the process of acquiring sets of pairs of thermal infrared and visible light images includes:

and correspondingly integrating the thermal infrared image and the visible light image at the same reference time to obtain a plurality of groups of thermal infrared and visible light image pairs.

6. The synchronous registration method based on temporal-spatial unification of thermal infrared and visible light images according to claim 1, wherein:

the process of obtaining the position coordinates of the calibration plate corner points comprises the following steps:

processing the thermal infrared and visible light image pair through an edge detection operator to obtain the edge position of the checkerboard whole pixel;

based on the edge position, selecting an edge pixel point as a center of a sub-pixel point, based on the center of the sub-pixel point, obtaining a normal line, based on the normal line and a single-pixel distance, obtaining the sub-pixel point, and based on a gray value of the sub-pixel point, constructing a fitting function, wherein the fitting function is an arc tangent function;

solving the fitting function by a least square method to obtain an edge function model;

calculating the edge function model through secondary differentiation to obtain sub-pixel edge points;

and fitting edge lines through sub-pixel edge points in four directions around the thick angular point to obtain an intersection point, wherein the intersection point is used as the position coordinate of the calibration plate angular point.

7. The synchronous registration method based on temporal-spatial unification of thermal infrared and visible light images according to claim 1, wherein:

the gray values of the sub-pixel points are obtained by calculation through linear interpolation in the horizontal direction and the vertical direction.

8. The synchronous registration method based on temporal-spatial unification of thermal infrared and visible light images according to claim 1, wherein:

the process of solving the optimal homography transformation matrix comprises the following steps:

matching position coordinates of calibration board corner points in the thermal infrared image pair and the visible light image pair to obtain a coordinate pair set, and randomly selecting a plurality of coordinate pairs as an inner point set;

constructing an initial homography matrix by a least square method based on the interior point set;

calculating projection errors of coordinate pairs in the coordinate pair set except for the interior point set through the initial homography matrix, judging errors of projection error calculation results, adding the coordinate pairs into the interior point set when the projection errors are smaller than a threshold value, and not adding the coordinate pairs into the interior point set when the projection errors are larger than the threshold value;

counting the number of coordinate pairs in the updated inner point set;

and repeating the process from the construction of the homography matrix to the statistics of the number of the coordinate pairs until the iteration times are reached, and obtaining the optimal homography matrix, wherein the optimal homography matrix is the homography matrix with the maximum number of the coordinate pairs in the corresponding inner point set.

Images