CN116385495A - Moving target closed-loop detection method of infrared video under dynamic background - Google Patents

Abstract

The invention discloses a method for closed-loop detection of a moving target in an infrared video under a dynamic background, which includes: analyzing the noise type of the infrared image, using a filtering algorithm to remove the noise from the image; and adopting different interference corner point filtering for different frames method, so that the corner points are roughly eliminated and finely eliminated, and the final detected corner points are stored and recorded; the above-mentioned corner points are tracked by using the sparse optical flow method, and the tracked points are filtered by two-way tracking. The homography matrix calculation is performed on the relationship between the frame corner sets, and the background compensation is performed on the current frame image using the homography transformation matrix; the difference between the previous frame infrared image and the current frame image after background compensation is performed, and the difference result is automatically calculated. Adapt to the gray threshold binarization process, perform morphological operations on the binary image to obtain the final target position; form a mask based on the target position detected in the previous frame, and feed back to the corner detection in the next frame to form a complete closed-loop detection circuit.

Description

A closed-loop detection method for moving targets in infrared video under dynamic background

technical field

The invention relates to the field of infrared moving target detection, in particular to a closed-loop detection method for infrared video moving targets under a dynamic background.

Background technique

Moving target detection is one of the core research topics in the field of computer vision. Moving target detection is the basis of target tracking, target recognition, and target behavior understanding. It has broad application prospects in military, security monitoring, industrial automation, intelligent transportation and other fields. According to whether the shooting platform or camera is moving or not, moving object detection can be divided into moving object detection under static background and moving object detection under dynamic background. The moving target detection in the static background is that the camera is still during the shooting process, and the obtained video sequence only contains the movement of the target; the moving target detection in the dynamic background is in the shooting process, the shooting platform or camera and the moving target change at the same time, The changes of the shooting platform and camera include translation, rotation, zooming, etc. The resulting video sequence contains not only the movement of the target, but also the movement of the background. Compared with the target detection in a static background, the difficulty is greatly improved. The main method of moving object detection under dynamic background is background compensation. By calculating the transformation matrix between the previous frame and the current frame, the background compensation is performed on the current frame, and then the moving object is detected using the frame difference method. The accuracy of the background compensation It will directly affect the detection accuracy.

Compared with visible light images, infrared images have the characteristics of low contrast, poor resolution, and more noise, which will lead to fewer corner points or feature points extracted by infrared images than those extracted by visible light during the background compensation process. concentrated. This will affect the accuracy of the calculation of the subsequent transformation matrix and the accuracy of background compensation. In addition, when performing background compensation, it will be found that the corner points or feature points in the background play an active role in the calculation of the transformation matrix, while the corner points or feature points in the moving target object area Feature points can hinder accurate registration of the background. Compared with visible light, the corner points or feature points extracted from infrared images will increase the proportion of feature points in the moving target object area as a whole, which may cause the selected interior points to be moving targets when the PROSAC algorithm performs feature point screening. The increased probability of feature points in the object area will lead to inaccurate registration and affect the final detection result.

Contents of the invention

According to the problems existing in the prior art, the present invention discloses a method for closed-loop detection of a moving target in infrared video under a dynamic background, which specifically includes the following steps:

Analyze the noise types of infrared images, and use filtering algorithms to remove noise from the images;

Different interference corner filtering methods are adopted for different frames, so as to perform coarse and fine elimination of corner points, and store and record the final detected corner points;

Use the sparse optical flow method to track the above corner points, and use the two-way tracking method to filter out the tracking points, determine the corresponding position relationship of the corner points in the previous frame in the current frame, and then use the current tracking points for reverse tracking, and The two groups of corner points are screened to remove the corner points that fail to reverse tracking;

Calculate the homography matrix according to the relationship between the corner sets of the two frames before and after, and use the homography transformation matrix to perform background compensation on the current frame image;

Differentiate the previous frame infrared image from the current frame image after background compensation, perform adaptive gray threshold binarization on the difference result, and perform morphological operations on the binary image to obtain the final target position;

A mask is formed according to the position of the detection target in the previous frame, which is fed back to the corner detection in the next frame to form a complete closed-loop detection circuit.

Further, a Gaussian filter is used to perform noise filtering on the current frame infrared image to remove Gaussian white noise in the infrared image; a median filter is used to filter out random point noise in the infrared image.

Further, judge whether the current frame is the first five frames of infrared images, if it is the first five frames of infrared images, perform coarse elimination, otherwise perform fine elimination;

Divide the infrared image into sub-blocks of the same size, sub-block sequence i ₁ to i ₂₅ ;

Starting from i ₁ , use the Shi-Tomasi algorithm for corner detection on each small block, and sort the number of corner points in each small block from small to large after the detection is completed;

Remove the 5 sub-blocks with the largest number and the 5 sub-blocks with the smallest concentration, and store the final corner points,

If it is judged that it is not the first five frames of images, the detection result of the previous frame will be fed back to the current frame during detection, and the mask generated by the moving target area of the previous frame will be used, and the area with zero mask area will be on the current frame. No corner detection is performed;

Perform Shi-Tomasi corner detection on other areas to complete fine elimination, so that the entire detection method can achieve closed-loop detection;

Store the final corner detection result.

Further, use the LK optical flow pyramid algorithm to track each obtained corner point in the current frame, and obtain the position P _i (x ₁ , y ₀ ) of the corner point P _i (x 0 , y ₀ ) in the previous frame in the current frame y ₁ ), repeat this step until all corner points are calculated, and store all tracking points;

Use the LK optical flow pyramid algorithm to perform reverse tracking on the above-mentioned set of tracking points again, and obtain the position P _i (x ₂ , y ₂ ) of the tracking point P _i (x ₁ , y ₁ ) in the current frame in the previous frame, and repeat In this step, until all tracking points are calculated, all tracking point pairs of reverse tracking are stored;

Remove the forward tracking point pairs according to the output state vector;

If it is judged that the output state vector is 1, the corner point of the previous frame and the corresponding tracking point of the current frame are stored, and if the judged output state vector is 0, the corresponding point pair is removed;

Apply the same removal strategy to the collection of backtracking;

Filter the set of forward tracking point pairs and the set of reverse tracking point pairs, take out point P _i (x ₁ ,y ₁ ) and compare it with P _i (x ₀ ,y ₀ ) and P _i (x ₂ , y ₂ ), if the x-coordinates and y-coordinates of the two points are the same, the two-way tracking is successful, and P _i (x ₀ , y ₀ ) and P _i (x ₁ , y ₁ ) are added to the successful tracking set;

Repeat the above operations until all point pairs are screened.

Further, the PROSAC algorithm is used to calculate the optimal homography transformation matrix H of two frames of infrared images for the obtained two sets of corner points corresponding to the previous frame and the current frame. For the corner points P _i (x ₀ , y ₀ ) and the tracking point P _i (x ₁ ,y ₁ ) corresponding to the current frame should satisfy the following relationship:

Use the optimal homography transformation matrix H to perform background compensation on the current frame, and use bilinear interpolation to perform interpolation in the x-direction and y-direction of pixels to perform image correction.

Perform a difference operation on the previous frame infrared image and the compensated infrared image, and perform Gaussian filtering on the difference image to remove noise;

Use the Otsu algorithm to perform threshold segmentation on the difference image to obtain a binary image suspected to be a moving target;

Corrosion operation is performed on the binary image suspected of being a moving target to remove discrete noise and line noise interference, and then the expansion operation is performed. After expansion, the small area is marked and filtered, and the expansion operation is performed again to obtain the final binary value of the moving target. picture;

Calculate the contour of the moving target according to the binary image of the final moving target, and store the contour;

Traverse each target contour and draw a bounding rectangle on the current frame infrared image according to the contour, and store the position of the rectangle as well as the length and width of the rectangle;

Repeat the above steps until all contours are traversed to obtain the final moving object detection result map.

Further, create a single-channel mask image with the same size and type as the current frame infrared image and set the color to white;

Take an unprocessed circumscribed rectangular frame from the stored rectangular frame set, obtain the position and size of the rectangular frame, expand the length and width of the rectangular frame by m pixels, and map the expanded rectangular frame to the mask mask In the image, set the gray value of the pixels inside the rectangular frame of the mask mask image to 0, add the processed rectangular frame to the processed set, continue to process the next rectangular frame, and repeat the above steps until all the rectangular frames are processed ;

Obtain the mask image of a moving target, and use this mask image as initial information to feed back to the next frame detection to realize closed-loop detection.

Due to the adoption of the above technical solution, the present invention provides a closed-loop detection method for infrared video moving objects under a dynamic background. The method uses corner homogenization processing at the initial stage of detection, and performs optical flow tracking on the homogenized corner points to improve the transformation efficiency. The calculation accuracy of the matrix; in the optical flow tracking stage, use the two-way tracking algorithm to remove the influence of wrong tracking points on background compensation, and improve the accuracy of moving object detection; use the mask of the moving object area in the previous frame to eliminate the moving object in the current frame Influenced by the corner points of the region, only the background points are used to iteratively calculate the optimal homography transformation matrix of the two frames of infrared images, which not only improves the accuracy of the background compensation, but also improves the operating efficiency of the algorithm and improves the real-time performance of the algorithm.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments described in this application. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is the flowchart of the method disclosed in the present invention;

Fig. 2 is a schematic diagram of the mid-infrared image sequence of the present invention;

Fig. 3 is a schematic diagram of corner point homogenization image segmentation in the present invention;

4 is a schematic diagram of sparse optical flow bidirectional tracking in the present invention;

Fig. 5 is a schematic diagram of the difference result without background compensation in the present invention;

Fig. 6 is a schematic diagram of the differential result of not using two-way tracking and "fine elimination" in the present invention;

Fig. 7 is a schematic diagram of the differential results using bidirectional tracking and "fine elimination" in the present invention;

Fig. 8 is a schematic diagram of the threshold segmentation of the difference result without using two-way tracking and "fine elimination" in the present invention;

Fig. 9 is a schematic diagram of threshold segmentation using two-way tracking and "fine elimination" difference results in the present invention;

Figure 10 is a schematic diagram of the final motion region monitoring results;

FIG. 11 is a schematic diagram of a moving object region mask in the present invention.

Detailed ways

In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the drawings in the embodiments of the present invention:

As shown in Figure 1, a method for closed-loop detection of moving targets in infrared video under a dynamic background. "or "fine elimination", and then use optical flow two-way tracking to remove wrong point pairs, then perform background compensation and image difference, and finally generate a mask according to the moving object area, which is fed back to the next frame detection to form a closed-loop detection of infrared moving objects . The concrete steps of the method disclosed by the invention are as follows:

S1: Preprocessing the current frame of the infrared video: According to the analysis of the noise type of the infrared image, use the filtering algorithm to process the image; the specific method is as follows:

S11: Use a 7*7 Gaussian filter to perform noise filtering on the infrared image of the current frame, and remove Gaussian white noise in the infrared image.

S12: Then use a 3*3 median filter to filter out random point noise in the infrared image, the noise of the processed infrared image is significantly reduced, and the detailed information is also well preserved, and the image quality is improved to a certain extent Improvement provides a good foundation for subsequent processing.

S2: Different interference corner filtering strategies are adopted for different frames to achieve "coarse elimination" or "fine elimination" of corner points, and the final detected corner points are stored and recorded. The specific method is as follows:

S21: Determine whether the current frame is the first five frames of images, if it is the first five frames of infrared images, perform "rough elimination", otherwise perform "fine elimination".

S22: If it is determined in S21 that it is the first five frames, the infrared image is divided into 5*5 sub-blocks of the same size, the sub-block sequence is i ₁ to i ₂₅ , and the infrared image is divided into blocks as shown in FIG. 3 .

S23: Starting from i ₁ , use the Shi-Tomasi algorithm to perform corner detection on each small block, and sort the number of corner points in each small block from small to large after the detection is completed.

S24: Remove the 5 sub-blocks with the largest number and the 5 sub-blocks with the smallest concentration, and store the final corner points, so that the detected corner points can be relatively uniform, and complete "rough elimination".

S25: If it is judged in S21 that it is not the first five frames of images, the detection result of the previous frame will be fed back to the current frame during detection, and the mask generated by the moving target area in the previous frame will be used, and the place where the mask area is zero is in the current frame Corner detection is not performed on the frame, and "fine elimination" is completed, and the black area moves to the target area.

S26: Perform Shi-Tomasi corner detection on other areas to complete "fine elimination", so that the entire detection method realizes closed-loop detection.

S27: Store the final corner point detection result.

S3: Use the sparse optical flow method to track the corner points calculated in the previous frame, and use bidirectional tracking to filter out the tracking points, determine the corresponding position relationship of the corner points in the previous frame in the current frame, and then use the current tracking points to reverse Track, and filter the two groups of corner points, and eliminate the corner points that fail to reverse track. The specific method is as follows:

S31: Use the LK optical flow pyramid algorithm to track each corner point obtained in S27 in the current frame, and obtain the position P _i (x ₁ , y ₀ ) of the corner point P _i (x 0 ,y ₀ ) in the previous frame in the current frame y ₁ ), in infrared video, since the video is continuously changing in time, it is reasonable to think that many points in the previous frame can be found in the next frame.

S32: Repeat step S31 until all corner points are calculated, and store all tracking points.

S33: Use the LK optical flow pyramid algorithm to perform reverse tracking on the set of tracking points in S31 again, and obtain the position P _i (x ₂ ,y ₂ ) of the tracking point P _i (x ₁ , y ₁ ) in the current frame in the previous frame ).

S34: Repeat step S33 until all tracking points are calculated, and store all tracking point pairs for reverse tracking.

S35: Remove the forward tracking point pairs according to the output state vector.

S36: If it is judged from S32 that the output state vector is 1, then store the corner point of the previous frame and the corresponding tracking point of the current frame, and if it is judged from S32 that the output state vector is 0, then remove the corresponding point pair.

S37: Perform the same removal strategy on the set of reverse traces.

S38: Filter the set of forward tracking point pairs and the set of reverse tracking point pairs, take out point P _i (x ₁ , y ₁ ) and compare P _i (x ₀ , y ₀ ) and P _i (x ₂ ,y ₂ ), if _the x _- coordinates and y _- coordinates of the two points are the same, the two-way tracking is _successful . The _schematic diagram of the two-way tracking is shown in Figure 4 _. Added to tracking success collection.

S39: Repeat the above operations until the screening of all point pairs is completed.

S4: Calculate the homography matrix according to the relationship between the corner sets of the two frames before and after, and use the homography transformation matrix to perform background compensation on the current frame image. The specific method is as follows:

S41: Using the PROSAC algorithm to calculate the optimal homography transformation matrix H of two frames of infrared images for the two sets of corner points corresponding to the previous frame obtained in S36 and the current frame, for the corner points P _i (x ₀ , y ₀ ) and the tracking point P _i (x ₁ ,y ₁ ) corresponding to the current frame should satisfy the following relationship:

S42: Use the optimal homography transformation matrix H calculated in step S51 to perform background compensation on the current frame. The difference image without background compensation is shown in Figure 5. During the compensation process, in order to eliminate the offset due to pixels as much as possible The error caused by the bilinear interpolation method is used to perform image correction in the x direction and y direction of the pixel point respectively.

S5: Differentiate the previous frame infrared image and the current frame image after background compensation, perform adaptive gray threshold binarization on the difference result, perform morphological operations on the binary image, and obtain the final target position. The specific method is as follows:

S51: Perform differential calculation on the previous frame infrared image and the compensated infrared image, perform Gaussian filtering on the differential image, remove noise, and perform two-way bidirectional tracking on the background compensation differential image without two-way tracking and "fine elimination" as shown in Figure 6 The background compensated difference image of tracking and "fine elimination" is shown in Fig. 7.

S52: Use the Otsu algorithm to perform threshold segmentation on the difference image, and obtain a binary image suspected of being a moving target. The differential threshold segmentation image without bidirectional tracking and "fine elimination" is shown in Figure 8, and performs bidirectional tracking and "fine elimination" difference. The thresholded segmented image is shown in Figure 9.

S53: Carry out erosion operation on the binary image suspected to be a moving target, remove discrete noise and line noise interference, and then perform expansion operation, after expansion, mark and filter out small areas, perform expansion operation again, and obtain the final moving target The binary image and the final motion region detection result are shown in Figure 10.

S54: Calculate the contour of the moving object according to the final binary image of the moving object, and store the contour.

S55: Traverse each target contour, draw a bounding rectangle on the current frame infrared image according to the contour, and store the position of the rectangle and the length and width of the rectangle.

S56: Repeat the above steps until all contours are traversed to obtain a final moving target detection result map.

S6: Form a mask according to the detection target position in the previous frame, and feed back to the corner detection in the next frame to form a complete closed-loop detection system. The specific method is as follows:

S61: Create an initial image of a single-channel mask whose size and type are the same as those of the current frame infrared image and whose color is set to white.

S62: Take an unprocessed circumscribed rectangular frame from the rectangular frame set stored in S55, obtain the position and size of the rectangular frame, and expand the length and width of the rectangular frame by m pixels, m can be adjusted according to the actual situation, Map the expanded rectangular frame to the mask mask image, set the gray value of the pixel inside the mask mask image rectangular frame to 0, add the processed rectangular frame to the processed set, and continue to process the next rectangular frame .

S63: Repeat the above steps until all the rectangular frames are processed.

S64: Complete the above steps to obtain a mask image of a moving target, and feed back the mask image as initial information to the next frame detection to realize closed-loop detection. The mask image is shown in FIG. 11 .

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto, any person familiar with the technical field within the technical scope disclosed in the present invention, according to the technical solution of the present invention Any equivalent replacement or change of the inventive concepts thereof shall fall within the protection scope of the present invention.

Claims (7)

1. A method for detecting a moving target in a closed loop of an infrared video under a dynamic background is characterized by comprising the following steps:

analyzing the noise type of the infrared image, and removing noise from the image by using a filtering algorithm;

different interference corner filtering methods are adopted for different frames, so that coarse elimination and fine elimination of corners are carried out, and final detection corners are stored and recorded;

tracking the angular points by adopting a sparse optical flow method, filtering the tracking points by adopting a bidirectional tracking mode, determining the corresponding position relation of the angular points of the previous frame in the current frame, carrying out backward tracking by using the current tracking points, screening the two groups of angular points, and eliminating angular points with backward tracking failure;

carrying out homography matrix calculation according to the relation between the angular point sets of the front frame and the rear frame, and carrying out background compensation on the current frame image by using the homography transformation matrix;

differentiating the infrared image of the previous frame with the current frame image after background compensation, performing self-adaptive gray threshold binarization processing on the differential result, and performing morphological operation on the binary image to obtain a final target position;

and forming a mask according to the detection target position of the previous frame, and feeding back to the corner detection of the next frame to form a complete closed loop detection circuit.

2. The method for detecting the moving object in the closed loop of the infrared video under the dynamic background according to claim 1, wherein the method comprises the following steps: carrying out noise filtering on the infrared image of the current frame by using a Gaussian filter to remove Gaussian white noise in the infrared image; a median filter is used to filter out random punctiform noise in the infrared image.

3. The method for detecting the moving object in the closed loop of the infrared video under the dynamic background according to claim 1, wherein the method comprises the following steps: judging whether the current frame is the infrared image of the previous five frames, if so, performing coarse elimination, otherwise, performing fine elimination;

dividing the infrared image into subblocks with the same size, and a subblock sequence i ₁ To i ₂₅ ；

From i ₁ Starting to perform corner detection on each small block by using a Shi-Tomasi algorithm, and sorting the number of the corners in each small block from small to large after the detection is completed;

removing 5 sub-blocks with the largest number and 5 sub-blocks with the smallest concentration, storing the final corner points,

if the image is not the previous five frames, the detection result of the previous frame is fed back to the current frame during detection, and the mask generated by the moving target area of the previous frame is used for not detecting the corner point of the area with the mask area of zero on the current frame;

carrying out Shi-Tomasi corner detection on other areas to finish fine elimination, so that the whole detection method realizes closed-loop detection;

and storing the final corner detection result.

4. The method for detecting the moving object in the closed loop of the infrared video under the dynamic background according to claim 2, wherein the method comprises the following steps:

tracking each obtained corner in the current frame by using LK optical flow pyramid algorithm to obtain the corner P of the previous frame _i (x ₀ ,y ₀ ) At the position P of the current frame _i (x ₁ ,y ₁ ) Repeating the steps until all the corner points are calculated, and storing all the tracking points;

and performing back tracking on the tracking point set by using the LK optical flow pyramid algorithm again to obtain a tracking point P of the current frame _i (x ₁ ,y ₁ ) At position P of the previous frame _i (x ₂ ,y ₂ ) Repeating the steps until all tracking points are calculated, and storing all tracking point pairs which are reversely tracked;

removing the forward tracking point pairs according to the output state vector;

if the output state vector is judged to be 1, the corner point of the previous frame and the corresponding tracking point of the current frame are stored, and if the output state vector is judged to be 0, the corresponding point pair is removed;

performing the same removal strategy on the back tracking set;

screening the forward tracking point pair set and the backward tracking point pair set, and taking out the point P _i (x ₁ ,y ₁ ) Compare P in its corresponding set _i (x ₀ ,y ₀ ) And P _i (x ₂ ,y ₂ ) If the x coordinate and the y coordinate of the two points are the same, the bidirectional tracking is successful, and P is calculated _i (x ₀ ,y ₀ ) And P _i (x ₁ ,y ₁ ) Adding the tracking success set;

repeating the above operation until the screening of all the point pairs is completed.

5. The method for detecting the moving object in the closed loop of the infrared video under the dynamic background according to claim 3, wherein the method comprises the following steps: p is adopted for two corner sets corresponding to the obtained previous frame and the current frameThe ROSAC algorithm calculates the optimal homography transformation matrix H of two frames of infrared images, and for the corner P of the previous frame _i (x ₀ ,y ₀ ) Tracking point P corresponding to current frame _i (x ₁ ,y ₁ ) The following relationship should be satisfied:

and performing background compensation on the current frame by using the optimal homography transformation matrix H, and respectively performing interpolation in the x direction and the y direction of the pixel point by adopting a bilinear interpolation mode so as to perform image correction.

6. The method for closed loop detection of a moving object in an infrared video under a dynamic background according to claim 4, wherein the method comprises the following steps:

performing differential operation on the infrared image of the previous frame and the compensated infrared image, and performing Gaussian filtering on the differential image to remove noise;

threshold segmentation is carried out on the differential image by using an Otsu algorithm, and a binary image suspected to be a moving target is obtained;

performing corrosion operation on the binary image suspected to be the moving target, removing discrete noise and linear noise interference, performing expansion operation, marking and filtering small areas after expansion, and performing expansion operation again to obtain a final moving target binary image;

calculating the outline of the moving object according to the final moving object binary image, and storing the outline;

traversing each target contour, drawing an external rectangle on the infrared image of the current frame according to the contour, and storing the position of the rectangle and the length and width of the rectangle;

repeating the steps until all the outlines are traversed to obtain a final moving target detection result diagram.

7. The method for detecting the moving object in the closed loop of the infrared video under the dynamic background according to claim 1, wherein the method comprises the following steps: creating a single-channel mask image with the same size and type as the infrared image of the current frame and set to be white;

taking out an unprocessed external rectangular frame from the stored rectangular frame set, acquiring the position and the size of the rectangular frame, expanding the length and the width of the rectangular frame outwards by m pixel points, mapping the expanded rectangular frame into a mask image, setting the gray value of the pixel points in the mask image rectangular frame to be 0, adding the processed rectangular frame into the processed set, continuing to process the next rectangular frame, and repeating the steps until all the rectangular frames are processed;

and obtaining a mask image of the moving object, and feeding back the mask image as initial information to the next frame detection to realize closed loop detection.

Images