CN114783211B - Scene target monitoring enhancement system and method based on video data fusion - Google Patents
Scene target monitoring enhancement system and method based on video data fusion Download PDFInfo
- Publication number
- CN114783211B CN114783211B CN202210286434.7A CN202210286434A CN114783211B CN 114783211 B CN114783211 B CN 114783211B CN 202210286434 A CN202210286434 A CN 202210286434A CN 114783211 B CN114783211 B CN 114783211B
- Authority
- CN
- China
- Prior art keywords
- target
- data
- scene
- video
- comparison
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G5/00—Traffic control systems for aircraft, e.g. air-traffic control [ATC]
- G08G5/0073—Surveillance aids
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Aviation & Aerospace Engineering (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
The invention discloses a scene target monitoring enhancement system and a scene target monitoring enhancement method based on video data fusion, wherein the scene target monitoring enhancement system comprises the following steps: the advanced scene activity guiding and controlling system is used for carrying out monitoring data fusion processing, receiving the identification comparison result data output by the video detection comparison system and carrying out comprehensive track and video comparison data association fusion processing; and the video detection and comparison system is used for receiving video stream data output by the panoramic video system, identifying airport scene targets, and performing primary and secondary comparison processing on the airport scene targets and the received comprehensive track data output by the advanced scene activity guiding and control system to obtain video identification and comparison result data. The invention adopts the video recognition technology to recognize the scene target, carries out the secondary recognition and confirmation of the cradle head camera on the suspicious target, then fuses and compares the suspicious target with the traditional monitoring data in the A-SMGCS system, and automatically carries out the true and false and increase and decrease processing on the scene target, thereby realizing the elimination of false targets, compensating for lost targets and increasing the reliability and reliability of the monitoring of the scene target.
Description
Technical Field
The invention belongs to the technical field of airport scene target monitoring, and particularly relates to a scene target monitoring enhancement system and method based on video data fusion.
Background
Airport scene target monitoring currently relies on scene monitoring sensors such as scene monitoring radars, multi-point positioning systems (MLAT), broadcast automatic correlation monitoring (ADS-B) and the like to detect airport scene aircrafts and vehicles; the problems of false, split, jump and the like of a monitoring target caused by complex electromagnetic environment of an airport scene and multipath propagation of a monitoring signal affect the stability and reliability of processing and generating a target track by a scene management system such as an advanced scene activity guiding and controlling system (A-SMGCS). At present, research and development developers of the A-SMGCS system at home and abroad try to provide scene target monitoring reliability by adopting methods of multisource monitoring data fusion, kalman filtering algorithm to improve track stability and smoothness, setting false target inhibition areas and the like for improving the stability and reliability of a monitoring target. Researchers have also used video recognition technology to identify airport scene objects, in combination with scene object recognition and visualization systems.
A-SMGCS system adopting methods such as multisource monitoring data fusion and Kalman filtering algorithm to improve track stability and smoothness, setting false target inhibition areas and the like still has the problems of false, split, jump and the like of a large number of targets in the operation of an actual airport system. The method is characterized in that the current scene monitoring sensors such as scene monitoring radars, MLAT, ADS-B and the like have large monitoring source data deviation due to complex airport scene electromagnetic environment, multipath reflection propagation of monitoring signals, interference of GNSS positioning reliability with ADS-B monitoring and other various reasons, the problem of stable and reliable target monitoring can be solved by only relying on simple multi-source monitoring fusion and algorithm processing and not fitting a flight path well, and novel reliable monitoring source signals in the airport scene environment need to be explored to replace or supplement the existing monitoring sources. At present, researchers also adopt a video recognition technology to recognize airport scene aircraft targets and combine the airport scene aircraft targets with a scene monitoring system to provide scene target recognition and visualization, but the method only adopts a plurality of independent cameras for detection, does not adopt an intelligent video recognition technology, has low recognition accuracy, does not carry out detection and confirmation for multiple times, has low recognition reliability, does not carry out full data fusion and comparison processing with a monitoring track of the scene monitoring system, cannot automatically carry out true or false and increase or decrease processing on the scene targets, and cannot comprehensively and correctly enhance the reliability of scene target monitoring.
Disclosure of Invention
Aiming at the defects of the prior art, the invention aims to provide a scene target monitoring enhancement system and a scene target monitoring enhancement method based on video data fusion, so as to solve the problems of false, split and jump of a monitoring target caused by complex scene electromagnetic environment and multipath propagation of the monitoring signal due to the fact that the traditional scene monitoring sensor signals such as a scene monitoring radar, MLAT, ADS-B and the like are processed by the existing A-SMGCS system in a guiding and merging way; according to the invention, intelligent video recognition technology is adopted to recognize targets of scene aircrafts, vehicles and pedestrians, a cradle head camera is used for secondary recognition and confirmation of suspicious targets, and then fusion comparison processing is carried out on the targets and traditional monitoring data in an A-SMGCS system, so that authenticity and increase and decrease processing are automatically carried out on the scene targets, false targets are eliminated, lost targets are compensated, and reliability of scene target monitoring are improved.
In order to achieve the above purpose, the invention adopts the following technical scheme:
the invention provides a scene target monitoring enhancement system based on video data fusion, which comprises the following components: the advanced scene activity guiding and controlling system and the video detection and comparison system;
the advanced scene activity guiding and controlling system is used for performing monitoring data fusion processing, receiving the identification comparison result data output by the video detection comparison system and performing comprehensive track and video comparison data association fusion processing, so that the scene target monitoring reliability is improved;
The video detection contrast system receives video stream data output by the panoramic video system, identifies airport scene targets, performs primary and secondary contrast processing with the received comprehensive track data output by the advanced scene activity guiding and controlling system, and outputs video identification contrast result data to the advanced scene activity guiding and controlling system.
Further, the advanced scene activity guidance and control system comprises: the system comprises a first input/output interface module, a monitoring data processing module, a comprehensive track and video comparison data association processing module and a situation man-machine interface module;
the first input/output interface module receives monitoring data of an input scene monitoring radar, a multi-point positioning (MLAT) and a broadcast automatic correlation monitoring (ADS-B) and outputs comprehensive track data;
the monitoring data processing module is used for carrying out fusion processing on the received monitoring data to form comprehensive track data of scene target monitoring;
the integrated track and video comparison data association processing module is used for receiving primary and secondary video identification comparison result data output by the video detection comparison system, carrying out integrated track and video comparison data association fusion processing, and marking the association fusion processing results respectively; the associated fusion processing results comprise the following categories: the target is confirmed, the target does not exist, the target is to be confirmed, and the target is lost;
The situation man-machine interface display module is used for carrying out differentiated display according to different categories of the associated fusion processing results; the method comprises the steps of identifying a target, wherein the target is not existed, and the target is to be identified, and displaying the identified target on a track label of a situation interface by using marks or colors; and for the target loss category, the target position identified by the received video is adopted, and a target movement track is displayed on a situation interface with a period of 1 second, so that the supplement of detecting the target loss in a scene monitoring mode is realized.
Further, the video detection contrast system includes: the system comprises a second input/output interface module, a panoramic video identification module, a fusion comparison module and a cradle head camera detection module;
the second input/output interface module is used for receiving the video stream data output by the panoramic video system and the comprehensive track data output by the advanced scene activity guiding and controlling system and outputting video identification comparison result data;
the panoramic video identification module is used for identifying video stream data output by a panoramic video system based on a video identification detection method of deep learning of a YOLO v5 network model and outputting panoramic video target identification data comprising target identification, category, position and confidence coefficient data;
The fusion comparison module is used for carrying out primary fusion comparison processing on the comprehensive track data output by the advanced scene activity guiding and controlling system and panoramic video target identification data, and judging whether target comparison is consistent or not; if the fusion comparison results are consistent, outputting a primary fusion comparison processing result to the advanced scene activity guiding and controlling system; if the primary comparison target is inconsistent, the position information of the primary comparison inconsistent target is sent to the tripod head camera detection module, after the secondary detection confirmation analysis is carried out by the tripod head camera detection module, the correlation matching processing is carried out according to the secondary detection target confirmation information, the comprehensive track data and the primary fusion comparison processing result, and the secondary detection target correlation matching result is output to the advanced scene activity guidance and control system;
and the PTZ camera detection module is used for controlling the PTZ camera to be aligned to a target area according to the position information of the primary inconsistent comparison target, performing secondary detection confirmation analysis on the target under the PTZ visual angle by using a video recognition detection method based on the deep learning of the YOLO v5 network model, and outputting secondary detection confirmation information to the fusion comparison module.
The invention also provides a scene target monitoring enhancement method based on video data fusion, which is based on the system and comprises the following steps:
1) Acquiring video stream data of a panoramic video system, and identifying airport scene targets; and receiving comprehensive track data of the advanced scene activity guiding and controlling system;
2) Performing primary fusion comparison processing on the comprehensive track data of the advanced scene activity guiding and controlling system and the airport scene target data identified in the step 1), and judging whether the target comparison is consistent; if the two types are consistent, the step 4) is carried out; if the target is inconsistent, sending the position information of the inconsistent target for the first comparison and entering the step 3);
3) Performing secondary detection confirmation analysis, performing association matching processing on the secondary detection confirmation analysis result, the comprehensive track data of the advanced scene activity guiding and controlling system and the primary fusion comparison processing result in the step 2), and outputting secondary detection and target related matching results;
4) And (3) carrying out association fusion processing and display on the primary fusion comparison processing result in the step (2), the secondary detection and target related matching result data in the step (3) and the comprehensive track data of the advanced scene activity guiding and controlling system.
Further, the step 1) specifically includes: adopting a plurality of cameras, and utilizing a panoramic stitching method to acquire panoramic video pictures covering the scene of the machine; collecting video stream data of a panoramic video system, identifying and detecting airport scene targets by a video identification detection method based on deep learning of a YOLO v5 network model, and outputting target identification, category, position and confidence data; and receives integrated track data formed by the advanced scene activity guiding and controlling system, the scene surveillance radar, the multi-point positioning system (MLAT) and the broadcast automatic correlation monitoring system (ADS-B).
Further, the airport scene object is an aircraft, a vehicle or a pedestrian.
Further, the step 1) specifically further includes:
11 Acquiring panoramic video stream data covering a machine scene detection area of a panoramic video system, and inputting the panoramic video stream data by adopting a video stream RTSP interface protocol;
12 Using a video recognition detection method based on the deep learning of the YOLO v5 network model to recognize panoramic video stream data, recognizing airport scene aircrafts, vehicles and pedestrian targets, carrying out target tracking and confidence calculation assignment, and outputting target identification, category, position and confidence data.
Further, the step 12) specifically includes the following steps:
121 Data acquisition and marking are carried out on the aircraft, the vehicle and the pedestrian targets of each operation scene of the airport scene take-off, landing, taxiing and apron guarantee from the panoramic video stream data to form a data set, wherein the scenes in the data set comprise daytime, overcast, night, rainy, foggy and strong exposure; performing further data enhancement processing on the data set;
122 Learning and training the YOLO v5 network model, updating weight parameters by using a gradient descent method in the training process, and after a plurality of rounds of iterative training, the weight parameters of the network model tend to converge to finish training to obtain a weight file;
123 The YOLO v5 network model uses the weight file to carry out real-time reasoning on panoramic video stream data, identifies the targets of aircrafts, vehicles and pedestrians in a picture, and further tracks the targets by adopting a Kalman filtering algorithm and a Hungary algorithm, so that each target is associated on a time sequence and is assigned with a unique video target identification ID number, and the uniqueness of the targets is ensured;
124 Performing confidence calculation and assignment on the identified target; the confidence coefficient is taken as a target identification confidence coefficient average value of three panoramic video stream data periods (40 ms/period), which is the confidence coefficient of the target in the current frame, the previous first frame and the previous second frame respectively, and the confidence coefficient average value is conf= (conf (t-2) +conf (t-1) +conf (t))/3;
125 Outputting panoramic video scene target identification data comprising an ID number, a category, a position coordinate and a confidence coefficient to a fusion comparison module;
126 Receiving integrated track data formed by advanced scene activity guidance and control system fusion processing scene surveillance radar, a multi-point positioning system (MLAT) and a broadcast automatic correlation surveillance system (ADS-B).
Further, the step 2) specifically includes: converting comprehensive track data of the advanced scene activity guiding and controlling system and panoramic video target identification data on the same coordinate system, and carrying out fusion screening comparison by adopting a Hungary algorithm; if the comparison result is within the consistency threshold range, setting a detection target consistency mark, and feeding back to the advanced scene activity guiding and controlling system; if the comparison result is not in the consistency threshold range, the target is not consistent, and the position information of the target which is inconsistent for the primary comparison is sent to the tripod head camera detection module for further confirmation.
Further, the step 2) specifically further includes:
21 The coordinates of data used by the advanced scene activity guiding and controlling system and the video detection and comparison system are converted into the same coordinate system;
22 The integrated track data of the advanced scene activity guiding and controlling system and the panoramic video target identification data are fused, screened and compared in the same coordinate system by adopting a Hungary algorithm (compared with a one-by-one comparison and matching mode, the repeated calculation amount and the calculation complexity are greatly reduced, and the real-time and rapid matching of the targets is realized); if the comparison result is within the consistency threshold range, setting a detection target consistency mark; and for targets with inconsistent detection results, sending the position information of the targets with inconsistent initial comparison to a holder camera detection system for further confirmation.
Further, the comparison and matching of the comprehensive track data of the advanced scene activity guiding and controlling system and the panoramic video target identification data is specifically as follows: and in the n targets detected and identified by the panoramic video, matching combination when the total weight is minimum is solved by adopting a Hungary algorithm according to the degree of matching difference between the category, confidence and distance between the corresponding matching targets, namely the weight, in the m targets which need to be matched to the comprehensive track data of the advanced scene activity guiding and controlling system, namely the fusion, comparison and matching combination result of the comprehensive track target of the optimized airport scene advanced scene activity guiding and controlling system and the panoramic video target identification data.
Further, the targets with inconsistent detection results have three situations: the comprehensive track targets of the advanced scene activity guiding and controlling system exist, and the panoramic video identification is not detected; identifying and detecting panoramic video, wherein the comprehensive track target of the advanced scene activity guiding and controlling system does not exist; the target is present in both monitoring sources but there is an offset inconsistency in the target position.
Further, the step 22) specifically includes the following steps:
221 Comparing the comprehensive track target of the advanced scene activity guiding and controlling system with the panoramic video target identification data one by one from three attributes of category, confidence level and position to obtain the difference between the two; defining a comprehensive track target of the advanced scene activity guiding and controlling system as S (category, confidence coefficient and position coordinate), and defining a panoramic video recognition target as V (category, confidence coefficient and position coordinate);
222 Comprehensive weight assignment is carried out on three attribute comparison results of category, confidence coefficient and position, wherein the comprehensive weight is equal to category difference multiplied by weight coefficient 1+confidence coefficient multiplied by weight coefficient 2+distance between targets multiplied by weight coefficient 3;
223 A weight matrix of the difference degree between the targets is established;
224 Adopting a Hungary algorithm to solve the matching combination with minimum total weight, namely obtaining the result of fusion, comparison and matching combination of the comprehensive track target and panoramic video target identification data of the optimal airport scene advanced scene activity guiding and controlling system;
225 Classifying the comparison results into two types of consistent and inconsistent targets in the matching combination according to the position relation, the type and the confidence coefficient between the targets in the matching combination;
226 And (3) sending the position data of the target to the pan-tilt camera detection module according to the inconsistent matching combination of the target.
Further, the step 221) specifically includes the following steps:
2211 Calculating the difference value of the two attributes of the target category and the confidence coefficient and taking an absolute value, namely abs (S-V);
2212 Distance calculation between target position attributes, i.e.)In (x) 1 ,y 1 ,z 1 )，(x 2 ,y 2 ,z 2 ) Is the coordinates of two targets.
Further, the step 224) specifically includes the following steps:
2241 Subtracting the row minimum value for each row;
2242 Carrying out matching without weights in a mode of searching an augmented path, wherein only items with the value of 0 are matchable items;
2243 If the maximum matching number is n, ending, and the current matching is the optimal matching; otherwise, executing step 2244);
2244 Covering all zero values with the least horizontal and vertical lines, taking mv = the smallest element of the uncovered matrix block, subtracting mv from the element of all uncovered matrix blocks, adding mv to the element of all matrix blocks covered by horizontal and vertical lines, returning to step 2242), and looping until all targets complete the corresponding matches;
2245 The matching combination resulting in the smallest total weight is (S1, V1), (S2, V3), (S3, V2), … …, (Sm, vn).
Further, the step 225) specifically includes:
when the targets in the matching combination simultaneously meet that the distance between the targets is smaller than a threshold value, the types are the same, and the absolute value of the confidence coefficient difference is smaller than the threshold value, the targets in the matching combination are judged to be consistent, the targets are the same, and the other matching combinations are not consistent.
Further, the step 226) specifically includes:
setting a detection target consistency mark for a matching combination of target consistency, and sending a target (aircraft and vehicle) Identifier (ID), the target consistency mark and a secondary detection mark to an advanced scene activity guiding and control system to participate in subsequent association processing;
and (3) for the matching combination of inconsistent targets, distinguishing three types according to the position relation between the targets and a detection frame (a rectangular frame is arranged at the center point of the panoramic video recognition target detection result as the detection frame): only panoramic video identification targets exist in the detection frame; the comprehensive track targets of the advanced scene activity guiding and controlling system independently exist outside the detection frame; detecting that comprehensive track targets and panoramic video recognition targets of the advanced scene activity guiding and controlling system exist in the frame; and sending the three types of target position data to a holder camera detection module, and providing position information for calling the holder camera to carry out secondary comparison and confirmation.
Further, the step 3) specifically includes the following steps:
31 Calculating the angle of the cradle head, driving the cradle head camera to rotate to align with the target area, and enabling the target to be confirmed to be positioned at the center of the picture of the cradle head camera; when the multi-target to be confirmed appears, the order of target confirmation is selected according to the following rules: from top left to bottom right or bottom right to top left in turn (the direction depends on the current camera position), the target closest to the last confirmed position is preferentially selected;
32 Using a video recognition detection method based on the deep learning of the YOLO v5 network model to carry out secondary detection, confirmation and analysis on the target under the view angle of the holder, and synchronously carrying out video recording in the process;
33 After finishing the secondary detection, confirmation and analysis of the target, carrying out association and matching processing on the target confirmation information of the pan-tilt camera and the primary comparison and detection result of the high-level scene activity guiding and controlling system by adopting a Hungary algorithm to obtain target related matching result data;
34 Outputting the target related matching result data of the step 33) and the video stream recorded by the cradle head camera to an advanced scene activity guiding and controlling system; transmitting the target related matching result data in the step 33) to a panoramic video identification module;
35 When the panoramic video recognition module detects that the video in the target related matching result data detects that the target marks are more than the target marks, the cradle head camera is called to continuously recognize and track the target, a frame of image is formed every 40ms, the position and type data of the target are extracted, and the video is used as a period of 40ms to send the target marks more than the target marks, the target positions and the type data to the advanced scene activity guiding and controlling system.
Further, the step 32) specifically includes the following steps:
321 Performing target identification detection on a target under a view angle of a pan-tilt camera by adopting a video identification detection method based on deep learning of a YOLO v5 network model, and assigning a type and a confidence level to the target of the detection result;
322 If the target is detected and the confidence is within the threshold range, judging that the target exists; if the target is not detected or the confidence coefficient is lower than the threshold value, the zoom lens is used for detecting the magnification of the amplified image again, confidence coefficient assignment is carried out, and if the confidence coefficient is still lower than the threshold value, the target is judged to be absent.
Further, the step 33) specifically includes:
the target related matching result data is: target Identification (ID), video detected multiple target flag, image position data of target, confidence, target present flag (1 if present and 0 if not present), target category, time stamp, whether secondary detection flag is performed.
Further, the step 4) specifically includes:
41 When the advanced scene activity guiding and controlling system receives the target Identification (ID) output in the step 226), the target consistency mark and whether the secondary detection mark is performed (no value at this time), performing the association matching processing with the target Identification (ID) of the integrated track of the advanced scene activity guiding and controlling system, confirming the same target, setting the target confirmed mark on the integrated track data of the target advanced scene activity guiding and controlling system, and distinguishing the confirmed target by using the mark or the color on the track mark of the situation interface of the advanced scene activity guiding and controlling system;
42 When the advanced scene activity guiding and controlling system receives the data output in the step 34), adopting the target Identification (ID), the confidence coefficient, the target existence mark (if the existence is 1, if the existence is not 0) in the data, and whether to perform the association and matching processing of the secondary detection mark (the value is yes at the moment) and the target Identification (ID) of the comprehensive track of the advanced scene activity guiding and controlling system;
when the target existence flag value is 0, which indicates that the target is not existed through video detection, setting a target nonexistence flag on the comprehensive track data of the target high-level scene activity guiding and controlling system, distinguishing the target nonexistence by using a mark or a color on a track label of a situation interface of the high-level scene activity guiding and controlling system, or not displaying the target on the situation interface of the high-level scene activity guiding and controlling system, so as to realize elimination of false targets of scenes;
When the target existence mark value is 1 and the confidence is in the set threshold range, judging that the video detection target and the comprehensive track target of the advanced scene activity guiding and controlling system are the same target, setting a target confirmed mark on the comprehensive track data of the target advanced scene activity guiding and controlling system, and distinguishing the target confirmed by using marks or colors on track labels of a situation interface of the advanced scene activity guiding and controlling system; if the confidence coefficient is not in the set threshold value range, setting a target to-be-confirmed mark on the target A-SMGCS comprehensive track data, and distinguishing the target to-be-confirmed by marks or colors on a track label of a situation interface of the advanced scene activity guiding and controlling system;
43 When the advanced scene activity guiding and controlling system receives the data output in the step 35), tracking, fitting and smoothing the 40ms of video tracking output as a periodic target position point trace to form a track, correlating with the target type data, displaying a target motion track on a situation interface of the advanced scene activity guiding and controlling system with 1 second as a period, and realizing the supplement of detecting the loss of the target in a scene monitoring mode.
The invention has the beneficial effects that:
according to the invention, the airport scene target is identified by adopting an intelligent video identification technology based on deep learning, so that high-accuracy identification of aircrafts, vehicles and pedestrians is realized, and then the airport scene target is subjected to primary fusion comparison with traditional monitoring data in an A-SMGCS system, and secondary identification confirmation is carried out on the suspicious target by adopting a cradle head camera, so that the identification accuracy and reliability are improved; and then fusing and comparing the real-time monitoring data with the traditional monitoring data in the A-SMGCS system again, and automatically performing true and false increase and decrease processing on the scene target, so as to eliminate false targets, make up lost targets, and enhance the accuracy, reliability and comprehensiveness of scene target monitoring. The method solves the problems that the traditional A-SMGCS system is connected with the fusion processing field monitoring radar, MLAT, ADS-B and other traditional field monitoring sensor signals, and the monitoring target is false, split, jumped and the like due to complex field electromagnetic environment and multipath propagation of the monitoring signal. Providing airport surface target monitoring tracks for airport tower controllers with accuracy, stability, reliability and comprehensiveness.
Drawings
Fig. 1 is a schematic diagram of the system of the present invention.
Fig. 2 is a flow chart of the method of the present invention.
FIG. 3 is a schematic diagram of a weight matrix.
Fig. 4 is a schematic diagram of a weight matrix calculation result.
FIG. 5 is a schematic diagram of object detection classification.
Detailed Description
The invention will be further described with reference to examples and drawings, to which reference is made, but which are not intended to limit the scope of the invention.
Referring to fig. 1, the present invention provides a scene target monitoring enhancement system based on video data fusion, comprising: the advanced scene activity guiding and controlling system and the video detection and comparison system;
the advanced scene activity guiding and controlling system is used for performing monitoring data fusion processing, receiving the identification comparison result data output by the video detection comparison system and performing comprehensive track and video comparison data association fusion processing, so that the scene target monitoring reliability is improved;
wherein the advanced scene activity guidance and control system comprises: the system comprises a first input/output interface module, a monitoring data processing module, a comprehensive track and video comparison data association processing module and a situation man-machine interface module;
the first input/output interface module receives monitoring data of an input scene monitoring radar, a multi-point positioning (MLAT) and a broadcast automatic correlation monitoring (ADS-B) and outputs comprehensive track data;
The monitoring data processing module is used for carrying out fusion processing on the received monitoring data to form comprehensive track data of scene target monitoring;
the integrated track and video comparison data association processing module is used for receiving primary and secondary video identification comparison result data output by the video detection comparison system, carrying out integrated track and video comparison data association fusion processing, and marking the association fusion processing results respectively; the associated fusion processing results comprise the following categories: the target is confirmed, the target does not exist, the target is to be confirmed, and the target is lost;
the situation man-machine interface display module is used for carrying out differentiated display according to different categories of the associated fusion processing results; the method comprises the steps of identifying a target, wherein the target is not existed, and the target is to be identified, and displaying the identified target on a track label of a situation interface by using marks or colors; and for the target loss category, the target position identified by the received video is adopted, and a target movement track is displayed on a situation interface with a period of 1 second, so that the supplement of detecting the target loss in a scene monitoring mode is realized.
The video detection contrast system receives video stream data output by the panoramic video system, identifies airport scene targets, performs primary and secondary contrast processing with the received comprehensive track data output by the advanced scene activity guiding and controlling system, and outputs video identification contrast result data to the advanced scene activity guiding and controlling system.
Wherein, the video detection contrast system includes: the system comprises a second input/output interface module, a panoramic video identification module, a fusion comparison module and a cradle head camera detection module;
the second input/output interface module is used for receiving the video stream data output by the panoramic video system and the comprehensive track data output by the advanced scene activity guiding and controlling system and outputting video identification comparison result data;
the panoramic video identification module is used for identifying video stream data output by a panoramic video system based on a video identification detection method of deep learning of a YOLO v5 network model and outputting panoramic video target identification data comprising target identification, category, position and confidence coefficient data;
the fusion comparison module is used for carrying out primary fusion comparison processing on the comprehensive track data output by the advanced scene activity guiding and controlling system and panoramic video target identification data, and judging whether target comparison is consistent or not; if the fusion comparison results are consistent, outputting a primary fusion comparison processing result to the advanced scene activity guiding and controlling system; if the primary comparison target is inconsistent, the position information of the primary comparison inconsistent target is sent to the tripod head camera detection module, after the secondary detection confirmation analysis is carried out by the tripod head camera detection module, the correlation matching processing is carried out according to the secondary detection target confirmation information, the comprehensive track data and the primary fusion comparison processing result, and the secondary detection target correlation matching result is output to the advanced scene activity guidance and control system;
And the PTZ camera detection module is used for controlling the PTZ camera to be aligned to a target area according to the position information of the primary inconsistent comparison target, performing secondary detection confirmation analysis on the target under the PTZ visual angle by using a video recognition detection method based on the deep learning of the YOLO v5 network model, and outputting secondary detection confirmation information to the fusion comparison module.
Referring to fig. 2, the invention also provides a scene target monitoring enhancement method based on video data fusion, which comprises the following steps based on the system:
1) Acquiring video stream data of a panoramic video system, and identifying airport scene targets; and receiving comprehensive track data of the advanced scene activity guiding and controlling system;
wherein, the step 1) specifically includes: adopting a plurality of cameras, and utilizing a panoramic stitching method to acquire panoramic video pictures covering the scene of the machine; collecting video stream data of a panoramic video system, identifying and detecting airport scene targets by a video identification detection method based on deep learning of a YOLO v5 network model, and outputting target identification, category, position and confidence data; and receives integrated track data formed by the advanced scene activity guiding and controlling system, the scene surveillance radar, the multi-point positioning system (MLAT) and the broadcast automatic correlation monitoring system (ADS-B).
In particular, the airport scene object is an aircraft, a vehicle or a pedestrian.
11 Acquiring panoramic video stream data covering a machine scene detection area of a panoramic video system, and inputting the panoramic video stream data by adopting a video stream RTSP interface protocol;
12 Using a video recognition detection method based on the deep learning of the YOLO v5 network model to recognize panoramic video stream data, recognizing airport scene aircrafts, vehicles and pedestrian targets, carrying out target tracking and confidence calculation assignment, and outputting target identification, category, position and confidence data.
Specifically, the step 12) specifically includes the following steps:
121 Data acquisition and marking are carried out on the aircraft, the vehicle and the pedestrian targets of each operation scene of the airport scene take-off, landing, taxiing and apron guarantee from the panoramic video stream data to form a data set, wherein the scenes in the data set comprise daytime, overcast, night, rainy, foggy and strong exposure; performing further data enhancement processing on the data set;
122 Learning and training the YOLO v5 network model, updating weight parameters by using a gradient descent method in the training process, and after a plurality of rounds of iterative training, the weight parameters of the network model tend to converge to finish training to obtain a weight file;
123 The YOLO v5 network model uses the weight file to carry out real-time reasoning on panoramic video stream data, identifies the targets of aircrafts, vehicles and pedestrians in a picture, and further tracks the targets by adopting a Kalman filtering algorithm and a Hungary algorithm, so that each target is associated on a time sequence and is assigned with a unique video target identification ID number, and the uniqueness of the targets is ensured;
124 Performing confidence calculation and assignment on the identified target; the confidence coefficient is taken as a target identification confidence coefficient average value of three panoramic video stream data periods (40 ms/period), which is the confidence coefficient of the target in the current frame, the previous first frame and the previous second frame respectively, and the confidence coefficient average value is conf= (conf (t-2) +conf (t-1) +conf (t))/3;
125 Outputting panoramic video scene target identification data comprising an ID number, a category, position coordinates and confidence level to a fusion comparison module.
126 Receiving integrated track data formed by advanced scene activity guidance and control system fusion processing scene surveillance radar, a multi-point positioning system (MLAT) and a broadcast automatic correlation surveillance system (ADS-B).
2) Performing primary fusion comparison processing on the comprehensive track data of the advanced scene activity guiding and controlling system and the airport scene target data identified in the step 1), and judging whether the target comparison is consistent; if the two types are consistent, the step 4) is carried out; if the target is inconsistent, sending the position information of the inconsistent target for the first comparison and entering the step 3);
Wherein, the step 2) specifically includes: converting comprehensive track data of the advanced scene activity guiding and controlling system and panoramic video target identification data on the same coordinate system, and carrying out fusion screening comparison by adopting a Hungary algorithm; if the comparison result is within the consistency threshold range, setting a detection target consistency mark, and feeding back to the advanced scene activity guiding and controlling system; if the comparison result is not in the consistency threshold range, the target is not consistent, and the position information of the target which is inconsistent for the primary comparison is sent to the tripod head camera detection module for further confirmation;
21 The coordinates of data used by the advanced scene activity guiding and controlling system and the video detection and comparison system are converted into the same coordinate system;
22 The integrated track data of the advanced scene activity guiding and controlling system and the panoramic video target identification data are fused, screened and compared in the same coordinate system by adopting a Hungary algorithm (compared with a one-by-one comparison and matching mode, the repeated calculation amount and the calculation complexity are greatly reduced, and the real-time and rapid matching of the targets is realized); if the comparison result is within the consistency threshold range, setting a detection target consistency mark; and for targets with inconsistent detection results, sending the position information of the targets with inconsistent initial comparison to a holder camera detection system for further confirmation.
The comparison and matching of the comprehensive track data of the advanced scene activity guiding and controlling system and the panoramic video target identification data are specifically as follows: and in the n targets detected and identified by the panoramic video, matching combination when the total weight is minimum is solved by adopting a Hungary algorithm according to the degree of matching difference between the category, confidence and distance between the corresponding matching targets, namely the weight, in the m targets which need to be matched to the comprehensive track data of the advanced scene activity guiding and controlling system, namely the fusion, comparison and matching combination result of the comprehensive track target of the optimized airport scene advanced scene activity guiding and controlling system and the panoramic video target identification data.
The targets with inconsistent detection results have three conditions: the comprehensive track targets of the advanced scene activity guiding and controlling system exist, and the panoramic video identification is not detected; identifying and detecting panoramic video, wherein the comprehensive track target of the advanced scene activity guiding and controlling system does not exist; the target is present in both monitoring sources but there is an offset inconsistency in the target position.
Specifically, the step 22) specifically includes the following steps:
221 Comparing the comprehensive track target of the advanced scene activity guiding and controlling system with the panoramic video target identification data one by one from three attributes of category, confidence level and position to obtain the difference between the two; defining a comprehensive track target of the advanced scene activity guiding and controlling system as S (category, confidence coefficient and position coordinate), and defining a panoramic video recognition target as V (category, confidence coefficient and position coordinate);
222 Comprehensive weight assignment is carried out on three attribute comparison results of category, confidence coefficient and position, wherein the comprehensive weight is equal to category difference multiplied by weight coefficient 1+confidence coefficient multiplied by weight coefficient 2+distance between targets multiplied by weight coefficient 3;
223 A weight matrix of the degree of the gap between the targets is established as shown in fig. 3;
224 Adopting a Hungary algorithm to solve the matching combination with minimum total weight, namely obtaining the result of fusion, comparison and matching combination of the comprehensive track target and panoramic video target identification data of the optimal airport scene advanced scene activity guiding and controlling system;
225 Classifying the comparison results into two types of consistent and inconsistent targets in the matching combination according to the position relation, the type and the confidence coefficient between the targets in the matching combination;
226 And (3) sending the position data of the target to the pan-tilt camera detection module according to the inconsistent matching combination of the target.
Wherein, the step 221) specifically includes the following steps:
2211 Calculating the difference value of the two attributes of the target category and the confidence coefficient and taking an absolute value, namely abs (S-V);
2212 Distance calculation between target position attributes, i.e.)In (x) 1 ,y 1 ,z 1 )，(x 2 ,y 2 ,z 2 ) Is the coordinates of two targets.
The step 224) specifically includes the following steps:
2241 Subtracting the row minimum value for each row;
2242 Carrying out matching without weights in a mode of searching an augmented path, wherein only items with the value of 0 are matchable items;
2243 If the maximum matching number is n, ending, and the current matching is the optimal matching; otherwise, executing step 2244);
2244 Covering all zero values with the least horizontal and vertical lines, taking mv = the smallest element of the uncovered matrix block, subtracting mv from the element of all uncovered matrix blocks, adding mv to the element of all matrix blocks covered by horizontal and vertical lines, returning to step 2242), and looping until all targets complete the corresponding matches;
2245 The matching combination resulting in the smallest total weight is (S1, V1), (S2, V3), (S3, V2), … …, (Sm, vn), see fig. 4.
The step 225) specifically includes:
when the targets in the matching combination simultaneously meet that the distance between the targets is smaller than a threshold value, the types are the same, and the absolute value of the confidence coefficient difference is smaller than the threshold value, the targets in the matching combination are judged to be consistent, the targets are the same, and the other matching combinations are not consistent.
The step 226) specifically includes:
setting a detection target consistency mark for a matching combination of target consistency, and sending a target (aircraft and vehicle) Identifier (ID), the target consistency mark and a secondary detection mark to an advanced scene activity guiding and control system to participate in subsequent association processing;
And (3) for the matching combination of inconsistent targets, distinguishing three types according to the position relation between the targets and a detection frame (a rectangular frame is arranged at the center point of the panoramic video recognition target detection result as the detection frame): (1) only panoramic video identification targets exist in the detection frame; (2) the comprehensive track targets of the advanced scene activity guiding and controlling system independently exist outside the detection frame; (3) detecting that comprehensive track targets and panoramic video recognition targets of the advanced scene activity guiding and controlling system exist in the frame; the three types of target position data are sent to a holder camera detection module, and position information is provided for calling a holder camera to carry out secondary comparison and confirmation; see fig. 5.
3) Performing secondary detection confirmation analysis, performing association matching processing on the secondary detection confirmation analysis result, the comprehensive track data of the advanced scene activity guiding and controlling system and the primary fusion comparison processing result in the step 2), and outputting secondary detection and target related matching results;
31 Calculating the angle of the cradle head, driving the cradle head camera to rotate to align with the target area, and enabling the target to be confirmed to be positioned at the center of the picture of the cradle head camera; when the multi-target to be confirmed appears, the order of target confirmation is selected according to the following rules: from top left to bottom right or bottom right to top left in turn (the direction depends on the current camera position), the target closest to the last confirmed position is preferentially selected;
32 Using a video recognition detection method based on the deep learning of the YOLO v5 network model to carry out secondary detection, confirmation and analysis on the target under the view angle of the holder, and synchronously carrying out video recording in the process;
33 After finishing the secondary detection, confirmation and analysis of the target, carrying out association and matching processing on the target confirmation information of the pan-tilt camera and the primary comparison and detection result of the high-level scene activity guiding and controlling system by adopting a Hungary algorithm to obtain target related matching result data;
34 Outputting the target related matching result data of the step 33) and the video stream recorded by the cradle head camera to an advanced scene activity guiding and controlling system; transmitting the target related matching result data in the step 33) to a panoramic video identification module;
35 When the panoramic video recognition module detects that the video in the target related matching result data detects that the target marks are more than the target marks, the cradle head camera is called to continuously recognize and track the target, a frame of image is formed every 40ms, the position and type data of the target are extracted, and the video is used as a period of 40ms to send the target marks more than the target marks, the target positions and the type data to the advanced scene activity guiding and controlling system.
Specifically, the step 32) specifically includes the following steps:
321 Performing target identification detection on a target under a view angle of a pan-tilt camera by adopting a video identification detection method based on deep learning of a YOLO v5 network model, and assigning a type and a confidence level to the target of the detection result;
322 If the target is detected and the confidence is within the threshold range, judging that the target exists; if the target is not detected or the confidence coefficient is lower than the threshold value, the zoom lens is used for detecting the magnification of the amplified image again, confidence coefficient assignment is carried out, and if the confidence coefficient is still lower than the threshold value, the target is judged to be absent.
The step 33) specifically includes:
the target related matching result data is: target Identification (ID), video detected multiple target flag, image position data of target, confidence, target present flag (1 if present and 0 if not present), target category, time stamp, whether secondary detection flag is performed.
4) Carrying out association fusion processing and display on the primary fusion comparison processing result in the step 2), the secondary detection and target related matching result data in the step 3) and the comprehensive track data of the advanced scene activity guiding and controlling system;
41 When the advanced scene activity guiding and controlling system receives the target Identification (ID) output in the step 226), the target consistency mark and whether the secondary detection mark is performed (no value at this time), performing the association matching processing with the target Identification (ID) of the integrated track of the advanced scene activity guiding and controlling system, confirming the same target, setting the target confirmed mark on the integrated track data of the target advanced scene activity guiding and controlling system, and distinguishing the confirmed target by using the mark or the color on the track mark of the situation interface of the advanced scene activity guiding and controlling system;
42 When the advanced scene activity guiding and controlling system receives the data output in the step 34), adopting the target Identification (ID), the confidence coefficient, the target existence mark (if the existence is 1, if the existence is not 0) in the data, and whether to perform the association and matching processing of the secondary detection mark (the value is yes at the moment) and the target Identification (ID) of the comprehensive track of the advanced scene activity guiding and controlling system;
when the target existence flag value is 0, which indicates that the target is not existed through video detection, setting a target nonexistence flag on the comprehensive track data of the target high-level scene activity guiding and controlling system, distinguishing the target nonexistence by using a mark or a color on a track label of a situation interface of the high-level scene activity guiding and controlling system, or not displaying the target on the situation interface of the high-level scene activity guiding and controlling system, so as to realize elimination of false targets of scenes;
When the target existence mark value is 1 and the confidence is in the set threshold range, judging that the video detection target and the comprehensive track target of the advanced scene activity guiding and controlling system are the same target, setting a target confirmed mark on the comprehensive track data of the target advanced scene activity guiding and controlling system, and distinguishing the target confirmed by using marks or colors on track labels of a situation interface of the advanced scene activity guiding and controlling system; if the confidence coefficient is not in the set threshold value range, setting a target to-be-confirmed mark on the target A-SMGCS comprehensive track data, and distinguishing the target to-be-confirmed by marks or colors on a track label of a situation interface of the advanced scene activity guiding and controlling system;
43 When the advanced scene activity guiding and controlling system receives the data output in the step 35), tracking, fitting and smoothing the 40ms of video tracking output as a periodic target position point trace to form a track, correlating with the target type data, displaying a target motion track on a situation interface of the advanced scene activity guiding and controlling system with 1 second as a period, and realizing the supplement of detecting the loss of the target in a scene monitoring mode.
The present invention has been described in terms of the preferred embodiments thereof, and it should be understood by those skilled in the art that various modifications can be made without departing from the principles of the invention, and such modifications should also be considered as being within the scope of the invention.
Claims (9)
1. A scene target monitoring enhancement system based on video data fusion, comprising: the advanced scene activity guiding and controlling system and the video detection and comparison system;
the advanced scene activity guiding and controlling system is used for performing monitoring data fusion processing, receiving the identification comparison result data output by the video detection comparison system and performing comprehensive track and video comparison data association fusion processing;
the video detection contrast system receives video stream data output by the panoramic video system, recognizes airport scene targets, performs primary and secondary contrast processing on the airport scene targets and the received comprehensive track data output by the advanced scene activity guiding and controlling system, and outputs video recognition contrast result data to the advanced scene activity guiding and controlling system;
the video detection contrast system comprises: the system comprises a second input/output interface module, a panoramic video identification module, a fusion comparison module and a cradle head camera detection module;
The second input/output interface module is used for receiving the video stream data output by the panoramic video system and the comprehensive track data output by the advanced scene activity guiding and controlling system and outputting video identification comparison result data;
the panoramic video identification module is used for identifying video stream data output by a panoramic video system based on a video identification detection method of deep learning of a YOLO v5 network model and outputting panoramic video target identification data comprising target identification, category, position and confidence coefficient data;
the fusion comparison module is used for carrying out primary fusion comparison processing on the comprehensive track data output by the advanced scene activity guiding and controlling system and panoramic video target identification data, and judging whether target comparison is consistent or not; if the fusion comparison results are consistent, outputting a primary fusion comparison processing result to the advanced scene activity guiding and controlling system; if the primary comparison target is inconsistent, the position information of the primary comparison inconsistent target is sent to the tripod head camera detection module, after the secondary detection confirmation analysis is carried out by the tripod head camera detection module, the correlation matching processing is carried out according to the secondary detection target confirmation information, the comprehensive track data and the primary fusion comparison processing result, and the secondary detection target correlation matching result is output to the advanced scene activity guidance and control system;
And the PTZ camera detection module is used for controlling the PTZ camera to be aligned to a target area according to the position information of the primary inconsistent comparison target, performing secondary detection confirmation analysis on the target under the PTZ visual angle by using a video recognition detection method based on the deep learning of the YOLO v5 network model, and outputting secondary detection confirmation information to the fusion comparison module.
2. The video data fusion-based scene target surveillance enhancement system of claim 1, wherein the advanced scene activity guidance and control system comprises: the system comprises a first input/output interface module, a monitoring data processing module, a comprehensive track and video comparison data association processing module and a situation man-machine interface module;
the first input/output interface module receives the monitoring data of the input scene monitoring radar, the multipoint positioning and the broadcasting automatic correlation monitoring and outputs the comprehensive track data;
the monitoring data processing module is used for carrying out fusion processing on the received monitoring data to form comprehensive track data of scene target monitoring;
the integrated track and video comparison data association processing module is used for receiving primary and secondary video identification comparison result data output by the video detection comparison system, carrying out integrated track and video comparison data association fusion processing, and marking the association fusion processing results respectively; the associated fusion processing results comprise the following categories: the target is confirmed, the target does not exist, the target is to be confirmed, and the target is lost;
The situation man-machine interface module is used for distinguishing and displaying according to different categories of the associated fusion processing result; the method comprises the steps of identifying a target, wherein the target is not existed, and the target is to be identified, and displaying the identified target on a track label of a situation interface by using marks or colors; and for the target loss category, the target position identified by the received video is adopted, and a target movement track is displayed on a situation interface with a period of 1 second, so that the supplement of detecting the target loss in a scene monitoring mode is realized.
3. A scene target monitoring enhancement method based on video data fusion, based on the system of any one of claims 1-2, characterized by the steps of:
1) Acquiring video stream data of a panoramic video system, and identifying airport scene targets; and receiving comprehensive track data of the advanced scene activity guiding and controlling system;
2) Performing primary fusion comparison processing on the comprehensive track data of the advanced scene activity guiding and controlling system and the airport scene target data identified in the step 1), and judging whether the target comparison is consistent; if the two types are consistent, the step 4) is carried out; if the target is inconsistent, sending the position information of the inconsistent target for the first comparison and entering the step 3);
3) Performing secondary detection confirmation analysis, performing association matching processing on the secondary detection confirmation analysis result, the comprehensive track data of the advanced scene activity guiding and controlling system and the primary fusion comparison processing result in the step 2), and outputting secondary detection and target related matching results;
4) And (3) carrying out association fusion processing and display on the primary fusion comparison processing result in the step (2), the secondary detection and target related matching result data in the step (3) and the comprehensive track data of the advanced scene activity guiding and controlling system.
4. The scene target monitoring enhancement method based on video data fusion according to claim 3, wherein said step 1) specifically comprises: adopting a plurality of cameras, and utilizing a panoramic stitching method to acquire panoramic video pictures covering the scene of the machine; collecting video stream data of a panoramic video system, identifying and detecting airport scene targets by a video identification detection method based on deep learning of a YOLO v5 network model, and outputting target identification, category, position and confidence data; and receiving the integrated track data formed by the advanced scene activity guiding and controlling system, the scene monitoring radar, the multi-point positioning system and the broadcasting type automatic correlation monitoring system.
5. The scene target monitoring enhancement method based on video data fusion according to claim 3, wherein said step 1) specifically further comprises:
11 Acquiring panoramic video stream data covering a machine scene detection area of a panoramic video system, and inputting the panoramic video stream data by adopting a video stream RTSP interface protocol;
12 Using a video recognition detection method based on the deep learning of the YOLO v5 network model to recognize panoramic video stream data, recognizing airport scene aircrafts, vehicles and pedestrian targets, carrying out target tracking and confidence calculation assignment, and outputting target identification, category, position and confidence data.
6. The scene target monitoring enhancement method based on video data fusion according to claim 5, wherein said step 12) specifically comprises the steps of:
121 Data acquisition and marking are carried out on the aircraft, the vehicle and the pedestrian targets of each operation scene of the airport scene take-off, landing, taxiing and apron guarantee from the panoramic video stream data to form a data set, wherein the scenes in the data set comprise daytime, overcast, night, rainy, foggy and strong exposure; performing further data enhancement processing on the data set;
122 Learning and training the YOLO v5 network model, updating weight parameters by using a gradient descent method in the training process, and after a plurality of rounds of iterative training, the weight parameters of the network model tend to converge to finish training to obtain a weight file;
123 The YOLO v5 network model uses the weight file to carry out real-time reasoning on panoramic video stream data, identifies the targets of aircrafts, vehicles and pedestrians in a picture, and further tracks the targets by adopting a Kalman filtering algorithm and a Hungary algorithm, so that each target is associated on a time sequence and is assigned with a unique video target identification ID number, and the uniqueness of the targets is ensured;
124 Performing confidence calculation and assignment on the identified target; the confidence coefficient is taken as a target identification confidence coefficient average value of three panoramic video stream data periods, which is the confidence coefficient of targets in the current frame, the previous first frame and the previous second frame respectively, and the confidence coefficient average value is conf= (conf (t-2) +conf (t-1) +conf (t))/3;
125 Outputting panoramic video scene target identification data comprising an ID number, a category, a position coordinate and a confidence coefficient to a fusion comparison module;
126 Receiving the integrated track data formed by the advanced scene activity guiding and controlling system, the scene monitoring radar, the multi-point positioning system and the broadcasting type automatic correlation monitoring system.
7. The scene target monitoring enhancement method based on video data fusion according to claim 3, wherein said step 2) specifically further comprises:
21 The coordinates of data used by the advanced scene activity guiding and controlling system and the video detection and comparison system are converted into the same coordinate system;
22 Adopting a Hungary algorithm to perform fusion screening comparison on comprehensive track data of the advanced scene activity guiding and controlling system and panoramic video target identification data in the same coordinate system; if the comparison result is within the consistency threshold range, setting a detection target consistency mark; and for targets with inconsistent detection results, sending the position information of the targets with inconsistent initial comparison to a holder camera detection system for further confirmation.
8. The scene target monitoring enhancement method based on video data fusion according to claim 3, wherein said step 3) specifically comprises the steps of:
31 Calculating the angle of the cradle head, driving the cradle head camera to rotate to align with the target area, and enabling the target to be confirmed to be positioned at the center of the picture of the cradle head camera; when the multi-target to be confirmed appears, the order of target confirmation is selected according to the following rules: selecting the target nearest to the last confirmed position preferentially from top left to bottom right or bottom right to top left in sequence;
32 Using a video recognition detection method based on the deep learning of the YOLO v5 network model to carry out secondary detection, confirmation and analysis on the target under the view angle of the holder, and synchronously carrying out video recording in the process;
33 After finishing the secondary detection, confirmation and analysis of the target, carrying out association and matching processing on the target confirmation information of the pan-tilt camera and the primary comparison and detection result of the high-level scene activity guiding and controlling system by adopting a Hungary algorithm to obtain target related matching result data;
34 Outputting the target related matching result data of the step 33) and the video stream recorded by the cradle head camera to an advanced scene activity guiding and controlling system; transmitting the target related matching result data in the step 33) to a panoramic video identification module;
35 When the panoramic video recognition module detects that the video in the target related matching result data detects that the target marks are more than the target marks, the cradle head camera is called to continuously recognize and track the target, a frame of image is formed every 40ms, the position and type data of the target are extracted, and the video is used as a period of 40ms to send the target marks more than the target marks, the target positions and the type data to the advanced scene activity guiding and controlling system.
9. The scene target monitoring enhancement method based on video data fusion according to claim 8, wherein said step 4) specifically comprises:
41 When the high-level scene activity guiding and controlling system receives the target mark, the target consistency mark and whether the secondary detection mark is carried out, the target mark is associated and matched with the target mark of the comprehensive track of the high-level scene activity guiding and controlling system, the target is confirmed to be the same target, the target confirmed mark is arranged on the comprehensive track data of the target high-level scene activity guiding and controlling system, and the target is confirmed by the mark or the color of the track label of the situation interface of the high-level scene activity guiding and controlling system;
42 When the advanced scene activity guiding and controlling system receives the data output in the step 34), adopting the target mark, the confidence level, the target existence mark and whether the secondary detection mark and the target mark of the comprehensive track of the advanced scene activity guiding and controlling system are associated and matched;
When the target existence flag value is 0, which indicates that the target is not existed through video detection, setting a target nonexistence flag on the comprehensive track data of the target high-level scene activity guiding and controlling system, distinguishing the target nonexistence by using a mark or a color on a track label of a situation interface of the high-level scene activity guiding and controlling system, or not displaying the target on the situation interface of the high-level scene activity guiding and controlling system, so as to realize elimination of false targets of scenes;
when the target existence mark value is 1 and the confidence is in the set threshold range, judging that the video detection target and the comprehensive track target of the advanced scene activity guiding and controlling system are the same target, setting a target confirmed mark on the comprehensive track data of the target advanced scene activity guiding and controlling system, and distinguishing the target confirmed by using marks or colors on track labels of a situation interface of the advanced scene activity guiding and controlling system; if the confidence coefficient is not in the set threshold value range, setting a target to-be-confirmed mark on the target A-SMGCS comprehensive track data, and distinguishing the target to-be-confirmed by marks or colors on a track label of a situation interface of the advanced scene activity guiding and controlling system;
43 When the advanced scene activity guiding and controlling system receives the data output in the step 35), tracking, fitting and smoothing the 40ms of video tracking output as a periodic target position point trace to form a track, correlating with the target type data, displaying a target motion track on a situation interface of the advanced scene activity guiding and controlling system with 1 second as a period, and realizing the supplement of detecting the loss of the target in a scene monitoring mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210286434.7A CN114783211B (en) | 2022-03-22 | 2022-03-22 | Scene target monitoring enhancement system and method based on video data fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210286434.7A CN114783211B (en) | 2022-03-22 | 2022-03-22 | Scene target monitoring enhancement system and method based on video data fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114783211A CN114783211A (en) | 2022-07-22 |
| CN114783211B true CN114783211B (en) | 2023-09-15 |
Family
ID=82424283
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210286434.7A Active CN114783211B (en) | 2022-03-22 | 2022-03-22 | Scene target monitoring enhancement system and method based on video data fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114783211B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116758259A (en) * | 2023-04-26 | 2023-09-15 | 中国公路工程咨询集团有限公司 | Highway asset information identification method and system |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09330499A (en) * | 1996-06-13 | 1997-12-22 | Oki Electric Ind Co Ltd | Method for displaying aircraft location on terminal control console |
| CN1417591A (en) * | 2001-11-09 | 2003-05-14 | 财团法人资讯工业策进会 | Fusion processing method of flight path information in combined radar/automatic echo monitoring |
| KR101228072B1 (en) * | 2012-11-08 | 2013-02-07 | 한국공항공사 | System and method for data recording and playing |
| KR20160007090A (en) * | 2014-07-11 | 2016-01-20 | 엘에스산전 주식회사 | Surface Movement Guidance and Control System for aircraft |
| CN105391975A (en) * | 2015-11-02 | 2016-03-09 | 中国民用航空总局第二研究所 | Video processing method in scene monitoring, device and scene monitoring system |
| CN105812733A (en) * | 2016-03-15 | 2016-07-27 | 中国民用航空总局第二研究所 | Civil aviation air traffic control scene monitoring and guiding system |
| CN109696172A (en) * | 2019-01-17 | 2019-04-30 | 福瑞泰克智能系统有限公司 | A kind of multisensor flight path fusion method, device and vehicle |
| CN109887040A (en) * | 2019-02-18 | 2019-06-14 | 北京航空航天大学 | The moving target actively perceive method and system of facing video monitoring |
| CN111009008A (en) * | 2019-12-06 | 2020-04-14 | 南京莱斯电子设备有限公司 | Self-learning strategy-based automatic airport airplane tagging method |
| CN111967498A (en) * | 2020-07-20 | 2020-11-20 | 重庆大学 | Night target detection and tracking method based on millimeter wave radar and vision fusion |
| CN112927565A (en) * | 2020-03-18 | 2021-06-08 | 中国民用航空总局第二研究所 | Method, device and system for improving accuracy of comprehensive track monitoring data of apron |
| CN113286080A (en) * | 2021-05-18 | 2021-08-20 | 中国民用航空总局第二研究所 | Scene monitoring system and video linkage tracking and enhanced display method and device |
| CN113628479A (en) * | 2021-08-16 | 2021-11-09 | 成都民航空管科技发展有限公司 | Video-based tower control information fusion system and method |
| CN113866758A (en) * | 2021-10-08 | 2021-12-31 | 深圳清航智行科技有限公司 | Scene monitoring method, system, device and readable storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2707104C (en) * | 2007-11-30 | 2018-06-19 | Searidge Technologies Inc. | Airport target tracking system |
| US20120007979A1 (en) * | 2008-04-16 | 2012-01-12 | Elbit Systems Ltd. Advanced Technology Center | Multispectral enhanced vision system and method for aircraft landing in inclement weather conditions |
| CN112859899A (en) * | 2014-10-31 | 2021-05-28 | 深圳市大疆创新科技有限公司 | System and method for monitoring with visual indicia |
| US20160196754A1 (en) * | 2015-01-06 | 2016-07-07 | Honeywell International Inc. | Airport surface monitoring system with wireless network interface to aircraft surface navigation system |
| US10127821B2 (en) * | 2015-06-24 | 2018-11-13 | Honeywell International Inc. | Aircraft systems and methods to improve airport traffic management |
| US10916152B2 (en) * | 2019-07-01 | 2021-02-09 | Honeywell International Inc. | Collision awareness system for ground operations |
-
2022
- 2022-03-22 CN CN202210286434.7A patent/CN114783211B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH09330499A (en) * | 1996-06-13 | 1997-12-22 | Oki Electric Ind Co Ltd | Method for displaying aircraft location on terminal control console |
| CN1417591A (en) * | 2001-11-09 | 2003-05-14 | 财团法人资讯工业策进会 | Fusion processing method of flight path information in combined radar/automatic echo monitoring |
| KR101228072B1 (en) * | 2012-11-08 | 2013-02-07 | 한국공항공사 | System and method for data recording and playing |
| KR20160007090A (en) * | 2014-07-11 | 2016-01-20 | 엘에스산전 주식회사 | Surface Movement Guidance and Control System for aircraft |
| CN105391975A (en) * | 2015-11-02 | 2016-03-09 | 中国民用航空总局第二研究所 | Video processing method in scene monitoring, device and scene monitoring system |
| CN105812733A (en) * | 2016-03-15 | 2016-07-27 | 中国民用航空总局第二研究所 | Civil aviation air traffic control scene monitoring and guiding system |
| CN109696172A (en) * | 2019-01-17 | 2019-04-30 | 福瑞泰克智能系统有限公司 | A kind of multisensor flight path fusion method, device and vehicle |
| CN109887040A (en) * | 2019-02-18 | 2019-06-14 | 北京航空航天大学 | The moving target actively perceive method and system of facing video monitoring |
| CN111009008A (en) * | 2019-12-06 | 2020-04-14 | 南京莱斯电子设备有限公司 | Self-learning strategy-based automatic airport airplane tagging method |
| WO2021109457A1 (en) * | 2019-12-06 | 2021-06-10 | 南京莱斯电子设备有限公司 | Airport airplane automatic labeling method based on self-learning policy |
| CN112927565A (en) * | 2020-03-18 | 2021-06-08 | 中国民用航空总局第二研究所 | Method, device and system for improving accuracy of comprehensive track monitoring data of apron |
| CN111967498A (en) * | 2020-07-20 | 2020-11-20 | 重庆大学 | Night target detection and tracking method based on millimeter wave radar and vision fusion |
| CN113286080A (en) * | 2021-05-18 | 2021-08-20 | 中国民用航空总局第二研究所 | Scene monitoring system and video linkage tracking and enhanced display method and device |
| CN113628479A (en) * | 2021-08-16 | 2021-11-09 | 成都民航空管科技发展有限公司 | Video-based tower control information fusion system and method |
| CN113866758A (en) * | 2021-10-08 | 2021-12-31 | 深圳清航智行科技有限公司 | Scene monitoring method, system, device and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114783211A (en) | 2022-07-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111145545B (en) | Road traffic behavior unmanned aerial vehicle monitoring system and method based on deep learning | |
| CN111523465B (en) | Ship identity recognition system based on camera calibration and deep learning algorithm | |
| CN109143241A (en) | The fusion method and system of radar data and image data | |
| CN102074113B (en) | License tag recognizing and vehicle speed measuring method based on videos | |
| CN108055501A (en) | A kind of target detection and the video monitoring system and method for tracking | |
| KR101999993B1 (en) | Automatic traffic enforcement system using radar and camera | |
| CN111767798B (en) | Intelligent broadcasting guide method and system for indoor networking video monitoring | |
| CN108731587A (en) | A kind of the unmanned plane dynamic target tracking and localization method of view-based access control model | |
| CN114783211B (en) | Scene target monitoring enhancement system and method based on video data fusion | |
| CN111881749A (en) | Bidirectional pedestrian flow statistical method based on RGB-D multi-modal data | |
| CN115035470A (en) | Low, small and slow target identification and positioning method and system based on mixed vision | |
| CN114818819A (en) | Road obstacle detection method based on millimeter wave radar and visual signal | |
| CN110568437A (en) | Precise environment modeling method based on radar assistance | |
| JPH10255057A (en) | Mobile object extracting device | |
| CN113114938A (en) | Target accurate monitoring system based on electronic information | |
| CN109817009A (en) | A method of obtaining unmanned required dynamic information | |
| US20230267753A1 (en) | Learning based system and method for visual docking guidance to detect new approaching aircraft types | |
| Lin et al. | 3d multi-object tracking based on radar-camera fusion | |
| CN116862832A (en) | Three-dimensional live-action model-based operator positioning method | |
| JPH0991439A (en) | Object monitor | |
| CN115457237A (en) | Vehicle target rapid detection method based on radar vision fusion | |
| CN112364854B (en) | Airborne target approaching guidance system and method based on detection, tracking and fusion | |
| CN115565157A (en) | Multi-camera multi-target vehicle tracking method and system | |
| CN114092522A (en) | Intelligent capture tracking method for take-off and landing of airport airplane | |
| CN114708544A (en) | Intelligent violation monitoring helmet based on edge calculation and monitoring method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |