CN114639032A - Vehicle detection tracking method, device and equipment for quasi-real-time digital twin display - Google Patents
Vehicle detection tracking method, device and equipment for quasi-real-time digital twin display Download PDFInfo
- Publication number
- CN114639032A CN114639032A CN202011490014.8A CN202011490014A CN114639032A CN 114639032 A CN114639032 A CN 114639032A CN 202011490014 A CN202011490014 A CN 202011490014A CN 114639032 A CN114639032 A CN 114639032A
- Authority
- CN
- China
- Prior art keywords
- vehicle
- detection
- track
- real
- path sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a vehicle detection tracking method, a device and equipment for quasi-real-time digital twin display, wherein the method comprises the following steps: collecting video stream from a traffic camera, detecting a detection frame of each frame of picture vehicle in the video stream in real time by using a detection classification neural network, and obtaining vehicle type color classification at the same time; matching vehicle positions among different video frames based on the detection frame information of the vehicle so as to track the path sequence of the vehicle and record the vehicle identification serial number; based on the internal and external parameters of the camera, the obtained coordinate projection of the path sequence is converted into a virtual twin space, the projected track is matched, and a uniform vehicle identification serial number is set; and performing smooth filtering and completion on the track in the virtual twin space, and outputting the three-position track, the vehicle type, the color and the vehicle sequence number of the vehicle in constant delay for vehicle animation display of the twin scene. The tracking data is provided for the digital twin system by the sub-second time delay, and the obtained vehicle running track is continuous and stable.
Description
Technical Field
The invention belongs to the technical field of intelligent traffic digital twins, and particularly relates to a vehicle detection tracking method, device and equipment with quasi-real-time digital twins display.
Background
In the field of intelligent traffic, the main purpose of digital twins is to fully utilize traffic data equipment deployed on roads, collect and analyze the behavior of traffic participants, and reproduce the full life cycle process of the traffic participants in a virtual twins scene. In addition, the prediction of traffic flow, the capture and alarm of abnormal behaviors and the like are also pain points and difficulties of intelligent traffic, and the digital twin is taken as the basis to provide scenes and mass real data necessary for realizing the functions. In traffic digital twins, vehicle tracking based on roadside cameras is an important component of its function.
In the conventional vehicle tracking method, a vehicle detector extracts a vehicle detection frame, and then a tracking algorithm analyzes and draws a driving path of a vehicle by using the vehicle detection frame extracted by the detector and combining features and image data.
The digital twin requires good precision of detection and tracking, good reproduction capability for real scenes, and uninterrupted processing of acquired data 24 hours all day long. Due to the performance of the detector and the tracking module, most of the existing methods are difficult to meet the requirement of large-scale and long-time development of the digital twin. The IoUT-based tracker can quickly track the vehicle track in real time, but the tracking algorithms are heavily dependent on the accuracy of the detector, and when the detector generates noise, the IoUT-based tracker is prone to interruption, resulting in fragmented tracks. The tracking mode based on the SORT utilizes Kalman filtering prediction and combines Hungarian algorithm to match vehicle tracks, the method is suitable for tracking pedestrians, but for fast moving vehicles, due to the speed difference generated by the perspective of a camera, the tracking serial numbers of the vehicles are easy to jump. By utilizing a tracking algorithm of image characteristics, a deep neural network is often needed, and the operation efficiency is not high; meanwhile, the vehicle mismatching probability is high due to excessive dependence on image characteristics, and the method is not suitable for being applied to a traffic digital twin system.
In addition, most real-time tracking methods do not adopt data buffering, and obvious missing tracking and re-identification processes are generated during execution, which are mainly represented by sudden disappearance and appearance of target vehicles, so that the display effect is adversely affected, and a feasible new method is urgently needed for a digital twin display system.
Disclosure of Invention
In view of the above drawbacks of the prior art, an object of the present invention is to provide a method, an apparatus and a device for detecting and tracking a vehicle with a quasi-real-time digital twin display, which are used to solve the technical problem of poor vehicle detecting and tracking effect in the prior art.
To achieve the above and other related objects, an embodiment of the present invention provides a vehicle detection and tracking method for quasi-real-time digital twin display, including: vehicle detection classification: collecting video stream from a traffic camera, detecting a detection frame of each frame of picture vehicle in the video stream in real time by using a detection classification neural network, and obtaining vehicle type color classification at the same time; and (3) real-time local matching and tracking: matching vehicle positions among different video frames based on the detection frame information of the vehicle so as to track the path sequence of the vehicle and record the vehicle identification serial number; global matching of traces: based on the internal and external parameters of the camera, the obtained coordinate projection of the path sequence is converted into a virtual twin space, the projected track is matched, and a uniform vehicle identification serial number is set; and (3) time delay output: and performing smooth filtering and completion on the track in the virtual twin space, and outputting the three-position track, the vehicle type, the color and the vehicle sequence number of the vehicle in constant delay for vehicle animation display of the twin scene.
In an embodiment of the present invention, in the vehicle detection classification, the detection classification neural network detects, in real time, detection frames, target reliability and feature vectors of all vehicles appearing in each frame of picture in the video stream; and obtaining the semantic information of the type and the color of the vehicle based on the feature vector.
In one embodiment of the present invention, in the real-time local matching pursuit: taking the detection result of each frame as the input of the detection classification neural network, matching the detection frame in real time by using the continuity information between the detection frame and the path sequence, and adding the detection frame into the path sequence; other detection frames which are not matched are initialized into a path sequence, and a unique vehicle identification serial number is configured; and in the path sequence which is not matched in the local matching, adopting the speed prediction and updating the position of the vehicle detection frame, and continuing the matching process in the next frame.
In an embodiment of the present invention, in the global matching of the tracks: converting the path sequence into a virtual twin space by using an inverse projection method, and detecting each possible path sequence within a certain time after the interruption of the track based on the continuity of the track in a virtual twin space coordinate system and the consistency of vehicle characteristics; if the path sequence meets the continuity consistency check, the two tracks are considered to belong to the same vehicle and are endowed with the same vehicle identification number.
In an embodiment of the present invention, in the delay output: the delay output and the matching process are carried out simultaneously, the buffer of the path sequence before the longest global matching time is detected, and the track and the feature vector under the twin world coordinate system are read, calculated and output; delay output uses smooth filtering to the path sequence, and completes the missing track in an interpolation mode; the output features are incrementally computed as they pass through the tracking process.
The embodiment of the invention also provides a vehicle detection and tracking device for quasi-real-time digital twin display, which comprises: the vehicle detection and classification module is used for collecting video streams from a traffic camera, detecting a detection frame of each frame of image vehicle in the video streams in real time by using a detection and classification neural network, and obtaining vehicle type color classification at the same time; the real-time local matching and tracking module is used for matching vehicle positions among different video frames based on the detection frame information of the vehicle so as to track the path sequence of the vehicle and record the vehicle identification serial number; the global track matching module is used for converting the obtained coordinate projection of the path sequence to a virtual twin space based on the internal and external parameters of the camera, matching the projected track and setting a uniform vehicle identification serial number; and the delay output module is used for performing smooth filtering and completion on the track in the virtual twin space, and outputting the three-position track, the vehicle type, the color and the vehicle sequence number of the vehicle in a constant delay manner so as to display the vehicle animation in the twin scene.
In an embodiment of the present invention, in the real-time local matching and tracking module, the detection result of each frame is used as an input of the detection classification neural network, the detection frame is matched in real time by using continuity information between the detection frame and the path sequence, and is added to the path sequence, the remaining detection frames that are not matched are initialized to a path sequence, and a unique vehicle identification number is configured, and in the path sequence that is not matched in the local matching, the position of the vehicle detection frame is predicted and updated by using a speed, and the matching process is continued in the next frame.
In an embodiment of the present invention, in the global matching module of the track: converting the path sequence into a virtual twin space by using an inverse projection method, and detecting each possible path sequence within a certain time after the interruption of the track based on the continuity of the track in a virtual twin space coordinate system and the consistency of vehicle characteristics; if the path sequence meets the continuity consistency check, the two tracks are considered to belong to the same vehicle and are endowed with the same vehicle identification number.
In an embodiment of the present invention, in the delay output module: the delay output and the matching process are carried out simultaneously, the buffer of the path sequence before the longest global matching time is detected, and the track and the feature vector under the twin world coordinate system are read, calculated and output; delay output uses smooth filtering to the path sequence, and completes the missing track in an interpolation mode; the output features are incrementally computed as they pass through the tracking process.
Embodiments of the present invention also provide an electronic device, comprising a processor and a memory, the memory storing program instructions; the processor executes program instructions to implement the vehicle detection tracking method for the quasi-real time digital twin display as described above.
As described above, the vehicle detection and tracking method, device and apparatus for quasi-real-time digital twin display according to the present invention have the following advantages:
the invention provides tracking data to the digital twin system by sub-second time delay, and the obtained vehicle running track is continuous and stable, and the appearance consistency of the vehicle on the same track is better.
Drawings
Fig. 1 is a flow chart of a vehicle detection and tracking method of the present invention, which is a quasi-real-time digital twin display.
Fig. 2 is a data flow diagram illustrating local matching of map tracks in the vehicle detection and tracking method based on quasi-real-time digital twin display according to the present invention.
Fig. 3 is a data flow diagram illustrating global trajectory matching in the method for detecting and tracking a vehicle according to the present invention.
Fig. 4 is a schematic three-dimensional projection diagram of a traffic camera in the vehicle detecting and tracking method based on the quasi-real-time digital twin display of the present invention.
Fig. 5 is a schematic diagram showing the spatial continuity of local matching in the vehicle detection and tracking method of the quasi-real-time digital twin display according to the present invention.
Fig. 6 is a schematic structural diagram of a vehicle detection and tracking device of the present invention, which is a quasi-real-time digital twin display.
Fig. 7 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Description of the element reference numerals
10 electronic device
1101 processor
1102 memory
100 quasi-real-time digital twin display vehicle detection tracking device
110 vehicle detection and classification module
120 real-time local matching tracking module
130 track global matching module
140 delay output module
S100-S400 steps
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
The embodiment aims to provide a vehicle detection and tracking method, device and equipment for quasi-real-time digital twin display, which are used for solving the technical problem of poor vehicle detection and tracking effect in the prior art.
The principle and implementation of the vehicle detecting and tracking method, device and apparatus for quasi-real-time digital twin display according to the present invention will be described in detail below, so that those skilled in the art can understand the vehicle detecting and tracking method, device and apparatus for quasi-real-time digital twin display according to the present invention without creative work.
Example 1
As shown in fig. 1, the present embodiment provides a vehicle detecting and tracking method for quasi real-time digital twin display, including:
step S100: vehicle detection classification: collecting video stream from a traffic camera, detecting a detection frame of each frame of picture vehicle in the video stream in real time by using a detection classification neural network, and obtaining vehicle type color classification at the same time;
step S200: and (3) real-time local matching and tracking: matching vehicle positions among different video frames based on the detection frame information of the vehicle so as to track the path sequence of the vehicle and record the vehicle identification serial number;
step S300: global matching of traces: based on the internal and external parameters of the camera, the obtained coordinate projection of the path sequence is converted into a virtual twin space, the projected track is matched, and a uniform vehicle identification serial number is set;
step S400: and (3) time delay output: and smoothly filtering and complementing the track in the virtual twin space, and outputting the three-position track, the vehicle type, the color and the vehicle serial number of the vehicle with constant delay for vehicle animation display of the twin scene.
The following describes steps S100 to S400 of the vehicle detection and tracking method for quasi-real-time digital twin display according to the present embodiment in detail.
Step S100: vehicle detection classification: the method comprises the steps of collecting video streams from a traffic camera, detecting a detection frame of each frame of picture vehicle in the video streams in real time by using a detection classification neural network, and obtaining vehicle type color classification at the same time.
In step S100, the detection classification neural network includes, but not limited to, a network having a YOLO structure, an SSD structure, and the like, which can be used for vehicle classification detection, and a neural network capable of providing an operation rate of at least 25 frames per second on a hardware device is preferably selected, and the operation rate is equivalent to a shooting rate of a traffic camera, so that a real-time detection service can be provided.
For example, in the present embodiment, YOLO is used as the vehicle detection neural network, the original version of YOLO describes the type of the target object using one-hot encoding (one-hot), and the type output of the network is calculated using the Sigmoid function. By changing one-hot encoding to multi-hot encoding (multi-hot), one object can be made to belong to multiple categories at the same time.
Through the training of the public data set and the supplementary data set, the detection classification neural network outputs a plurality of detection frames to the picture of each frame, and each detection frame comprises the position, the height, the width, the reliability and the grade of each vehicle type color classification.
In the vehicle detection classification, the detection classification neural network detects detection frames, target credibility and feature vectors of all vehicles appearing in each frame of picture in the video stream in real time; and obtaining the semantic information of the type and the color of the vehicle based on the feature vector.
In step S100, a multitask deep neural network is trained using traffic camera images and an annotation data set, the trained network providing real-time detection and classification tasks for vehicles.
One of the functions of the neural network is to predict each possible vehicle position in the traffic camera picture, and when the reliability of the predicted vehicle frame exceeds a certain set threshold value, the vehicle is judged to be present in the picture.
The second function of the neural network is to extract the characteristics of the vehicle in the detection frame, each dimension of the acquired vehicle characteristic vector is the score of the possibility of a certain vehicle type or color, and the characteristic vectors output by the neural network have macroscopic consistency for the same vehicle in the same video.
And after the detection classification module acquires images from the traffic camera, detecting and outputting a vehicle detection frame, a vehicle type and a color.
Specifically, each video frame of the traffic camera is captured in real time and then submitted to the network classification detection, which predicts the detection frames, target confidence and feature vectors of all vehicles present therein for the traffic camera video frames. The feature vector can obtain semantic information of the vehicle type and color of the vehicle through calculation. The neural network is trained by the traffic camera image and the labeled data set, and has the capability of providing real-time detection service. The predicted feature vectors have macroscopic consistency for the same vehicle of the same video.
Step S200: real-time local matching pursuit: and matching the positions of the vehicles among different video frames based on the detection frame information of the vehicles so as to track the path sequence of the vehicles and record the vehicle identification serial numbers.
In this embodiment, in the real-time local matching pursuit:
taking the detection result of each frame as the input of the detection classification neural network, matching the detection frame in real time by using continuity information between the detection frame and the path sequence, and adding the detection frame into the path sequence;
other detection frames which are not matched are initialized into a path sequence, and a unique vehicle identification serial number is configured;
and in the path sequence which is not matched in the local matching, adopting the speed prediction and updating the position of the vehicle detection frame, and continuing the matching process in the next frame.
Specifically, in this embodiment, the detection result of the neural network for detection and classification in each frame is used as an input, the detection frame is matched in real time by using the continuity information between the detection frame and the path sequence, and the continuity information is added to the path sequence. The remaining unmatched detection boxes are initialized to a path sequence and are accompanied by a unique vehicle identification number. And for the path sequence which is not matched in the local matching, adopting the speed prediction and updating the position of the vehicle detection frame, and continuing the matching process at the next frame. The continuous prediction of the path sequence only lasts for a short time, and after the continuous prediction is stopped due to timeout, the continuous prediction is discarded or enters a global matching process according to the length of the path sequence.
In this embodiment, the input of the matching algorithm in step S200 is the detection result output by the detection classification network, so as to connect the vehicle detection frames into a path sequence, which may also be referred to as a local matching algorithm.
The local matching utilizes continuity information between the path of the vehicle and the detection frame. By scoring parameters such as the size and the distance between detection frames, a matching result with better score is found and added into the path sequence, and a continuous path sequence is obtained. The remaining unmatched detection boxes are initialized to a path sequence and are accompanied by a unique vehicle identification number (hereinafter referred to as a vehicle ID).
Each frame of the local match may need to trace one or more path sequences at the same time, and the local matching algorithm supports matching multiple paths at the same time.
When the path sequence is interrupted due to loss detection or matching failure, the algorithm predicts the driving direction of the path according to the speed during interruption, updates the position of the vehicle frame in real time, and continues the matching process with the next frame data.
The prediction process will last for a short time to avoid excessive errors, if the path sequence stops tracing due to the timeout of the continuous prediction and the path length exceeds a certain threshold, it is considered as a real existing path, and it enters S400 to be recorded as a first track, otherwise, it discards the path.
Specifically, in the real-time local matching pursuit of the present embodiment, DIoU (f) is usedDistance-cross-over ratio) as an important indicator of the continuity of the real-time local matching, as shown in fig. 2, the DIoU calculates the difference between the coincidence degree and the normalized distance square between the two detection frames according to the formula
A and B are two detection frames, and the central points of the two detection frames are cAAnd cBAnd c represents the diagonal distance of the smallest rectangular area capable of containing both a and B. And continuous matching can be carried out on the plurality of paths and the detection box through a greedy algorithm or a Hungarian algorithm to obtain a continuous path sequence. When the continuity is higher than a certain threshold value, the detection box is considered to correspond to a certain path and is added into the path sequence.
After the tracking is lost, a preferred mode is to continuously predict the path by using linear kalman filtering, the kalman filtering continuously updates the position of the detection frame by assuming that the central point of the detection frame makes a uniform motion, and the updated detection frame and a new detection frame of the current frame are used in each local matching.
The predicted path has a lower priority in local matching, and the specific implementation manner is that the matching process is divided into three stages, wherein the first stage uses local matching for the tracked path and a new detection frame of a current frame, the second stage uses local matching for the predicted path and the remaining detection frames, and the last stage initializes the remaining isolated detection frames of the previous two steps to a new path and attaches a vehicle identification number (hereinafter referred to as a vehicle ID). The reason for this is that the predicted path will be less stable and more prone to mismatch and disruption.
In this embodiment, the prediction process in step S200 lasts for about 0.5 second, if the length of the path sequence for terminating the prediction is not greater than the threshold, the path sequence is regarded as an unstable path and discarded, and the rest of the path sequence enters the global matching stage in S300 and is recorded as the first trajectory.
Step S300: global matching of traces: and based on the internal and external parameters of the camera, converting the obtained coordinate projection of the path sequence into a virtual twin space, matching the projected track, and setting a uniform vehicle identification serial number.
In this embodiment, in the global matching of the trajectory:
converting the path sequence into a virtual twin space by using an inverse projection method, and detecting each possible path sequence within a certain time after the interruption of the track based on the continuity of the track in a virtual twin space coordinate system and the consistency of vehicle characteristics;
if the path sequence meets the continuity consistency check, the two tracks are considered to belong to the same vehicle and are endowed with the same vehicle identification number.
And in the global matching, converting the path sequence into a virtual twin space by using an inverse projection method, and checking each possible path sequence within a certain time after the track is interrupted according to the continuity of the track in a virtual twin space coordinate system and the consistency of the vehicle characteristics. If the path sequence satisfies the continuity consistency check, the two tracks are considered to belong to the same vehicle. And the same vehicle identification number is given. The global match does not affect the stage where the second trajectory is, it will continue to match locally or globally and connect to a longer path.
The purpose of step S300 is to connect two possible path sequences, which may also be referred to as a global matching stage, and the algorithm further determines by using the features of the vehicle in addition to using the information of the detection frame, the trajectory, and the like of the vehicle in the upper path.
The global matching stage is also performed in real time, and for the first trajectory that stops tracking in step S300, the global matching converts the screen trajectory into a twin space coordinate system according to the mapping relationship between the traffic camera and the real world, and checks the trajectory continuity of each possible path sequence and the consistency of the vehicle features within a certain time after the trajectory interruption.
And if the path sequence meets the continuity consistency check and is recorded as a second track, the two tracks are considered to belong to the same vehicle. The vehicle ID of the second trajectory will be modified to be consistent with the first trajectory, the global match will not affect the stage where the second trajectory is, it will continue to perform local or global matching and connect to a longer path.
Specifically, in the global matching of the trajectory in this embodiment: within a few seconds after the first trajectory stops tracking, the path whose length reaches the threshold is included in the calculation of the global match with the first trajectory. The global matching utilizes the information of the spatial continuity and the consistency of the vehicle features, as shown in fig. 3, there are respective calculation modes for the spatial continuity and the consistency of the vehicle features, and then the product of their probabilities is used as the matching criterion.
In order to avoid the change of the frame speed caused by the perspective of the camera, the path sequence should be back-projected to the virtual space coordinate system before the calculation of the spatial continuity. The embodiment assumes that the three-dimensional projection mode of the camera is as shown in fig. 4, and the midpoint p' of the vehicle detection frame is taken as [ u v ═]TThe inverse projection mapping can be obtained through the calibration parameters of the camera, and the approximate virtual scene coordinate p of the vehicle is further calculated as [ x, y and z ]]T。
Spatial continuity requires that the ratio of the distance and the speed of the vehicle in the vicinity of the disconnection point is approximately equal to the disconnection time, which is assumed to be based on the vehicle speed remaining substantially constant. The tail velocity and the head velocity can be roughly calculated by the position of the detection frame in the path sequence, the position of a certain point in the path can be respectively estimated by using the tail velocity of the previous path and the head velocity of the next path, and the spatial continuity of the two paths is represented by the estimated normal distribution probability of the two points, as shown in fig. 5.
Meanwhile, since the position of one point cannot reflect the direction of the path, the embodiment adopts the dot product d of the speed directions of the two ends as another index for direction judgment so as to judge the direction
As a function of its probability distribution density.
In addition, the embodiment also utilizes the normal distribution probability of the Euclidean distance between the vehicle feature vectors to calculate the vehicle feature consistency between the tracks, when the vehicle detection frame is tracked, the feature vectors of each track can be accumulated by taking the identification reliability as the weight, and the feature vectors updated by the continuous increment participate in the calculation of the Euclidean distance between the path sequences. And when the two tracks are considered to be matched, further fusing the feature vectors of the two tracks according to the recorded weight, and continuing the tracking process. The process of weighted average is not omissible because the feature vector output by the classification network has obvious noise, and the process of weighted average makes the vehicle type and color output by the algorithm relatively stable, thereby ensuring the effect of quasi-real-time display.
The second trajectory passing through the global matching will be attached with the same vehicle ID as the first trajectory.
Step S400: and (3) time delay output: and performing smooth filtering and completion on the track in the virtual twin space, and outputting the three-position track, the vehicle type, the color and the vehicle sequence number of the vehicle in constant delay for vehicle animation display of the twin scene.
In this embodiment, in the delay output:
the delay output and the matching process are carried out simultaneously, the buffer of the path sequence before the longest global matching time is detected, and the track and the feature vector under the twin world coordinate system are read, calculated and output;
delay output uses smooth filtering to the path sequence, and completes the missing track in an interpolation mode;
the output features are incrementally computed over the course of the trace.
That is, in this embodiment, the delay output is performed simultaneously with the matching process, and the algorithm will check the buffer of the path sequence before the longest global matching time, read, calculate, and output the trajectory and feature vector in the twin world coordinate system, which makes the matching process invisible when the path is interrupted for the user. The delay output uses smooth filtering to the path sequence, the missing track is completed in an interpolation mode, the output characteristic vector is calculated in an excessive mode, and the jump of vehicle models and colors during the step of outputting is reduced. The delayed output mode can provide data required by display for the virtual twin display system in real time, wherein the data comprises three-dimensional twin space coordinates of the vehicle, a vehicle identification serial number, a vehicle type and colors.
Step S400 is performed simultaneously with the trajectory matching. According to the characteristics of the matching algorithm, a certain interruption time exists before the track is endowed with a vehicle identification serial number, is predicted in real time or is matched by global delay, the track or characteristics of the vehicle in the interruption time are unstable, and therefore the track or characteristics cannot be used for twin display, and the purpose of delay output is to cover the instability of the tracking algorithm and achieve the optimal display effect. According to the set parameters, generally, the global delay matching is delayed for the longest time, and the vehicle track is output after being delayed by the same time difference, so that the matching process when the path is interrupted can not be perceived by the user.
Due to the output requirement of step S400, the tracked result needs to be temporarily stored in the buffer, and then read and calculated from the buffer by the output module, and the trajectory and the feature vector under the twin world coordinate system are output. The output track is relatively stable after smooth filtering and interpolation; the output characteristic is subjected to incremental calculation in the tracking process, and jump is not easy to generate.
In this embodiment, the step S400, the step S200 and the step S300 are performed synchronously, preferably, all track sequences of this embodiment maintain a first-in first-out buffer queue inside, the output module checks data at the tail of each buffer queue in each frame, finds a path sequence before the global matching time, and extracts vehicle position information, vehicle type, color and vehicle ID after back projection, filtering and interpolation. Since the algorithm ensures that the path tracking result is determined after a global matching time is over, the delay output path and the off-line processing result should be the same. Semantic features (vehicle type and color) of the vehicle are obtained by incrementally calculating feature vectors, and the probability of generating jump in the middle is low.
The tracking system of the embodiment communicates with the twin display system in the form of transmitting TCP packets, and needs to transmit data packets containing all vehicle information in the time period to the twin system at a constant frequency, and the process of transmitting and receiving the data packets may introduce delays of several tens to several hundreds of milliseconds. But in general, the output delay of the system can be considered to be in the sub-second order with respect to real time.
Example 2
As shown in fig. 6, the vehicle detection and tracking device 100 of the quasi real-time digital twin display according to the present embodiment includes: the system comprises a vehicle detection classification module 110, a real-time local matching tracking module 120, a global matching module 130 of the track and a delay output module 140.
In this embodiment, the vehicle detection and classification module 110 is configured to collect a video stream from a traffic camera, detect a detection frame of a vehicle in each frame of the video stream in real time by using a detection and classification neural network, and obtain a vehicle type color classification.
The technical features of the specific implementation of the vehicle detection and classification module 110 are substantially the same as step S100 in the vehicle detection and tracking method for quasi-real-time digital twin display in embodiment 1, and general technical contents between embodiments may not be repeated.
In this embodiment, the real-time local matching and tracking module 120 is configured to match vehicle positions between different video frames based on the detection frame information of the vehicle, so as to track a path sequence of the vehicle and record a vehicle identification number.
The technical features of the real-time local matching and tracking module 120 are substantially the same as that of step S200 in the vehicle detection and tracking method for quasi-real-time digital twin display in embodiment 1, and general technical contents between embodiments may not be repeated.
In this embodiment, the track global matching module 130 is configured to transform the obtained coordinate projection of the path sequence to a virtual twin space based on internal and external parameters of a camera, match the projected track, and set a uniform vehicle identification number.
The technical features of the specific implementation of the track global matching module 130 are substantially the same as step S300 in the method for detecting and tracking a vehicle with a quasi-real-time digital twin display in embodiment 1, and general technical contents between embodiments may not be repeated.
In this embodiment, the delay output module 140 is configured to perform smooth filtering and completion on the trajectory in the virtual twin space, so as to output the three-dimensional trajectory, the vehicle type, the color, and the vehicle serial number of the vehicle with constant delay, so as to display the vehicle animation in the twin scene.
The technical features of the specific implementation of the delay output module 140 are substantially the same as step S400 in the method for detecting and tracking a vehicle with a quasi-real-time digital twin display in embodiment 1, and the general technical contents in the embodiments are not repeated.
It should be noted that the division of the modules of the above apparatus is only a logical division, and the actual implementation may be wholly or partially integrated into one physical entity, or may be physically separated. And these modules can be realized in the form of software called by processing element; or may be implemented entirely in hardware; and part of the modules can be realized in the form of calling software by the processing element, and part of the modules can be realized in the form of hardware. For example, the vehicle detection and classification module 110 may be a separate processing element, or may be integrated into a chip of an electronic terminal, or may be stored in a memory of the terminal in the form of program code, and a processing element of the terminal calls and executes the functions of the tracking calculation module. Other modules are implemented similarly. In addition, all or part of the modules can be integrated together or can be independently realized. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.
For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. For another example, when one of the above modules is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
Example 3
As shown in fig. 4, the present embodiment further provides an electronic device 10, where the electronic device 10 includes a processor 1101 and a memory 1102.
The electronic device 100 may be, for example, a fixed terminal such as a server, desktop, etc.; the mobile terminal may be a mobile terminal, such as a notebook computer, a smart phone, a tablet computer, or the like, or a vehicle-mounted terminal.
The memory 1102 is connected to the processor 1101 through a system bus and is configured to perform communication with the processor 1101, the memory 1102 is configured to store a computer program, the processor 1101 is coupled to the display 1003 and the memory 1002, and the processor 1101 is configured to run the computer program, so that the electronic device 10 performs the vehicle detection and tracking method of the quasi real-time digital twin display according to embodiment 1. The vehicle detection and tracking method of the quasi real-time digital twin display has been described in detail in embodiment 1, and will not be described herein again.
The vehicle detection and tracking method of the quasi-real-time digital twin display can be applied to various types of electronic devices 10. In an exemplary embodiment, the electronic device 10 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, cameras, or other electronic components for performing the above-described vehicle detection tracking method for the quasi-real-time digital twin display.
The Memory may include a Random Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory.
The Processor 1101 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the above method embodiments may be performed by hardware associated with a computer program. The aforementioned computer program may be stored in a computer readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
In conclusion, the vehicle re-identification method and the vehicle re-identification device can directly perform vehicle re-identification on the traffic monitoring video, and effectively improve the accuracy of vehicle re-identification. Therefore, the invention effectively overcomes various defects in the prior art and has high industrial utilization value.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (10)
1. A vehicle detection tracking method of quasi-real-time digital twin display is characterized in that: the method comprises the following steps:
vehicle detection classification: collecting video streams from a traffic camera, detecting a detection frame of each frame of picture vehicle in the video streams in real time by using a detection classification neural network, and obtaining vehicle type color classification at the same time;
and (3) real-time local matching and tracking: matching vehicle positions among different video frames based on the detection frame information of the vehicle so as to track the path sequence of the vehicle and record the vehicle identification serial number;
global matching of traces: based on the internal and external parameters of the camera, the obtained coordinate projection of the path sequence is converted into a virtual twin space, the projected track is matched, and a uniform vehicle identification serial number is set;
and (3) time delay output: and performing smooth filtering and completion on the track in the virtual twin space, and outputting the three-position track, the vehicle type, the color and the vehicle sequence number of the vehicle in constant delay for vehicle animation display of the twin scene.
2. The method for detecting and tracking vehicles with quasi-real-time digital twin display according to claim 1, wherein: in the vehicle detection classification, the detection classification neural network detects detection frames, target credibility and feature vectors of all vehicles appearing in each frame of picture in the video stream in real time; and obtaining the semantic information of the type and the color of the vehicle based on the feature vector.
3. The vehicle detection tracking method of the quasi-real-time digital twin display according to claim 1 or 2, characterized in that: in the real-time local matching pursuit:
taking the detection result of each frame as the input of the detection classification neural network, matching the detection frame in real time by using the continuity information between the detection frame and the path sequence, and adding the detection frame into the path sequence;
other detection frames which are not matched are initialized into a path sequence, and a unique vehicle identification serial number is configured;
and in the path sequence which is not matched in the local matching, adopting the speed prediction and updating the position of the vehicle detection frame, and continuing the matching process in the next frame.
4. The method for detecting and tracking vehicles with quasi-real-time digital twin display according to claim 1, wherein: in the global matching of the tracks:
converting the path sequence into a virtual twin space by using an inverse projection method, and detecting each possible path sequence within a certain time after the interruption of the track based on the continuity of the track in a virtual twin space coordinate system and the consistency of vehicle characteristics;
if the path sequence meets the continuity consistency check, the two tracks are considered to belong to the same vehicle and are endowed with the same vehicle identification number.
5. The method for detecting and tracking vehicles with quasi-real-time digital twin display according to claim 1, wherein: in the delayed output:
the delay output and the matching process are carried out simultaneously, the buffer of the path sequence before the longest global matching time is detected, and the track and the feature vector under the twin world coordinate system are read, calculated and output;
delay output uses smooth filtering to the path sequence, and completes the missing track in an interpolation mode;
the output features are incrementally computed as they pass through the tracking process.
6. A vehicle detection tracking device of quasi real-time digital twin display is characterized in that: the method comprises the following steps:
the vehicle detection and classification module is used for collecting video streams from a traffic camera, detecting a detection frame of each frame of image vehicle in the video streams in real time by using a detection and classification neural network, and obtaining vehicle type color classification at the same time;
the real-time local matching and tracking module is used for matching vehicle positions among different video frames based on the detection frame information of the vehicle so as to track the path sequence of the vehicle and record the vehicle identification serial number;
the global track matching module is used for converting the obtained coordinate projection of the path sequence to a virtual twin space based on the internal and external parameters of the camera, matching the projected track and setting a uniform vehicle identification serial number;
and the delay output module is used for performing smooth filtering and completion on the track in the virtual twin space, and outputting the three-position track, the vehicle type, the color and the vehicle sequence number of the vehicle in a constant delay manner so as to display the vehicle animation in the twin scene.
7. The vehicle detection tracking apparatus of a quasi-real time digital twin display of claim 6, wherein: in the real-time local matching and tracking module, the detection result of each frame is used as the input of the detection classification neural network, the detection frame is matched in real time by utilizing the continuity information between the detection frame and the path sequence and is added into the path sequence, the rest detection frames which are not matched are initialized into a path sequence, a unique vehicle identification serial number is configured, the path sequence which is not matched in the local matching is adopted to predict and update the position of the vehicle detection frame, and the matching process is continued in the next frame.
8. The vehicle detection tracking apparatus of a quasi-real time digital twin display of claim 6, wherein: in the global matching module of the track: converting the path sequence into a virtual twin space by using an inverse projection method, and detecting each possible path sequence within a certain time after the interruption of the track based on the continuity of the track in a virtual twin space coordinate system and the consistency of vehicle characteristics; if the path sequence meets the continuity consistency check, the two tracks are considered to belong to the same vehicle and are endowed with the same vehicle identification number.
9. The vehicle detection tracking apparatus of a quasi-real time digital twin display of claim 6, wherein: in the delay output module: the delay output and the matching process are carried out simultaneously, the buffer of the path sequence before the longest global matching time is detected, and the track and the feature vector under the twin world coordinate system are read, calculated and output; delay output uses smooth filtering to the path sequence, and completes the missing track in an interpolation mode; the output features are incrementally computed as they pass through the tracking process.
10. An electronic device, characterized in that: comprising a processor and a memory, said memory storing program instructions; the processor executes the program instructions to realize the vehicle detection and tracking method of the quasi real-time digital twin display according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011490014.8A CN114639032A (en) | 2020-12-15 | 2020-12-15 | Vehicle detection tracking method, device and equipment for quasi-real-time digital twin display |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011490014.8A CN114639032A (en) | 2020-12-15 | 2020-12-15 | Vehicle detection tracking method, device and equipment for quasi-real-time digital twin display |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114639032A true CN114639032A (en) | 2022-06-17 |
Family
ID=81945024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011490014.8A Pending CN114639032A (en) | 2020-12-15 | 2020-12-15 | Vehicle detection tracking method, device and equipment for quasi-real-time digital twin display |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114639032A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115391477A (en) * | 2022-10-31 | 2022-11-25 | 智道网联科技(北京)有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN115577511A (en) * | 2022-09-26 | 2023-01-06 | 南京航空航天大学 | Short-term track prediction method, device and system based on unmanned aerial vehicle motion state |
| CN115830075A (en) * | 2023-02-20 | 2023-03-21 | 武汉广银飞科技发展有限公司 | Hierarchical association matching method for pedestrian multi-target tracking |
| CN117079468A (en) * | 2023-10-16 | 2023-11-17 | 深圳市城市交通规划设计研究中心股份有限公司 | Traffic flow track position method for realizing traffic digital twin |
| CN117097430A (en) * | 2023-10-16 | 2023-11-21 | 深圳市城市交通规划设计研究中心股份有限公司 | Method for synchronizing simulation time of vehicle flow track position |
-
2020
- 2020-12-15 CN CN202011490014.8A patent/CN114639032A/en active Pending
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115577511A (en) * | 2022-09-26 | 2023-01-06 | 南京航空航天大学 | Short-term track prediction method, device and system based on unmanned aerial vehicle motion state |
| CN115577511B (en) * | 2022-09-26 | 2023-11-17 | 南京航空航天大学 | Short-term track prediction method, device and system based on unmanned aerial vehicle motion state |
| CN115391477A (en) * | 2022-10-31 | 2022-11-25 | 智道网联科技(北京)有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN115830075A (en) * | 2023-02-20 | 2023-03-21 | 武汉广银飞科技发展有限公司 | Hierarchical association matching method for pedestrian multi-target tracking |
| CN117079468A (en) * | 2023-10-16 | 2023-11-17 | 深圳市城市交通规划设计研究中心股份有限公司 | Traffic flow track position method for realizing traffic digital twin |
| CN117097430A (en) * | 2023-10-16 | 2023-11-21 | 深圳市城市交通规划设计研究中心股份有限公司 | Method for synchronizing simulation time of vehicle flow track position |
| CN117097430B (en) * | 2023-10-16 | 2024-02-27 | 深圳市城市交通规划设计研究中心股份有限公司 | Method for synchronizing simulation time of vehicle flow track position |
| CN117079468B (en) * | 2023-10-16 | 2024-02-27 | 深圳市城市交通规划设计研究中心股份有限公司 | Traffic flow track position method for realizing traffic digital twin |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114639032A (en) | Vehicle detection tracking method, device and equipment for quasi-real-time digital twin display | |
| US10628961B2 (en) | Object tracking for neural network systems | |
| US11030755B2 (en) | Multi-spatial scale analytics | |
| Meimetis et al. | Real-time multiple object tracking using deep learning methods | |
| CN112449152B (en) | Method, system and equipment for synchronizing multi-channel video | |
| EP2709066A1 (en) | Concept for detecting a motion of a moving object | |
| CN107944381B (en) | Face tracking method, face tracking device, terminal and storage medium | |
| JP7329572B2 (en) | TRAFFIC STATE ACQUISITION METHOD AND DEVICE, ROAD SIDE DEVICE, AND CLOUD CONTROL PLATFORM | |
| JPWO2011013281A1 (en) | Moving object detection method and moving object detection apparatus | |
| EP4158528A1 (en) | Tracking multiple objects in a video stream using occlusion-aware single-object tracking | |
| CN113168520A (en) | Method of tracking objects in a scene | |
| WO2020213099A1 (en) | Object detection/tracking device, method, and program recording medium | |
| WO2007147171A2 (en) | Scalable clustered camera system and method for multiple object tracking | |
| KR102110375B1 (en) | Video watch method based on transfer of learning | |
| TWI732374B (en) | Method and apparatus for object recognition | |
| WO2022165675A1 (en) | Gesture recognition method and apparatus, terminal device, and readable storage medium | |
| Jin et al. | 3D multi-object tracking with boosting data association and improved trajectory management mechanism | |
| CN116958873A (en) | Pedestrian tracking method, device, electronic equipment and readable storage medium | |
| EP2709065A1 (en) | Concept for counting moving objects passing a plurality of different areas within a region of interest | |
| JP2007510994A (en) | Object tracking in video images | |
| Ibraheem et al. | FPGA-based vision processing system for automatic online player tracking in indoor sports | |
| Kim et al. | Predicting vehicle collisions using data collected from video games | |
| WO2023184197A1 (en) | Target tracking method and apparatus, system, and storage medium | |
| Sun et al. | Research on Vision-based pedestrian detection and tracking algorithm | |
| Kalirajan et al. | Adaptive visual tracking system using artificial intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |