CN113223060A - Multi-agent cooperative tracking method and device based on data sharing and storage medium - Google Patents
Multi-agent cooperative tracking method and device based on data sharing and storage medium Download PDFInfo
- Publication number
- CN113223060A CN113223060A CN202110411910.9A CN202110411910A CN113223060A CN 113223060 A CN113223060 A CN 113223060A CN 202110411910 A CN202110411910 A CN 202110411910A CN 113223060 A CN113223060 A CN 113223060A
- Authority
- CN
- China
- Prior art keywords
- target
- tracking
- unmanned aerial
- aerial vehicle
- matching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000003860 storage Methods 0.000 title claims abstract description 16
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 53
- 230000000007 visual effect Effects 0.000 claims abstract description 7
- 238000004590 computer program Methods 0.000 claims abstract description 6
- 238000012545 processing Methods 0.000 claims abstract description 6
- 238000002372 labelling Methods 0.000 claims abstract description 4
- 238000012549 training Methods 0.000 description 11
- 238000001514 detection method Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000010009 beating Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000001629 suppression Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000007306 turnover Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/292—Multi-camera tracking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Abstract
The invention discloses a data sharing-based multi-agent cooperative tracking method, a device and a storage medium, wherein the method comprises the following steps: shooting by an unmanned aerial vehicle to obtain data, and performing format adjustment and target labeling processing on the collected data to obtain an MDMT data set; based on the MDMT dataset, the tracker single-machine multi-target tracking algorithm is used for multi-target tracking of a single unmanned aerial vehicle video, cross-camera target association is carried out through information of other cameras, and the same targets appearing in different unmanned aerial vehicle visual field ranges are numbered uniformly. The device comprises: the system comprises an acquisition module, a tracking module and a cross-camera target association module. The medium includes: a computer readable storage medium stores a computer program comprising program instructions. According to the invention, the number of the newly added target in the unmanned aerial vehicle to be tracked is matched through the information of the images in other unmanned aerial vehicles, so that the problem of target shielding can be better solved.
Description
Technical Field
The invention relates to the field of multi-target tracking, in particular to a multi-agent cooperative tracking method and device based on data sharing and a storage medium.
Background
Along with the innovation and development of science and technology, computers have affected people's lives in many fields. Pictures and videos are now widely used in everyday life. The computer vision field developed by detecting and analyzing videos through computer technology has become one of the research fields of computer mainstream. In the process of solving the problems of action recognition, target detection, intelligent monitoring and the like, the multi-target tracking based on computer vision all has important influence on the problems.
Multi-unmanned aerial vehicle multi-target tracking based on computer vision is realized by identifying videos and tracking multiple types of targets, such as: crowd, bird, motor vehicle, etc. Unlike single target tracking, multi-target tracking cannot know the appearance and number of targets in advance. Currently, the mainstream multi-target tracking firstly identifies a target frame, and then associates a target and matches a target number.
The multi-unmanned aerial vehicle multi-target tracking algorithm can not be realized without a large-scale multi-target data set, and the existing multi-target data set is mostly fixed shooting. Therefore, the establishment of a larger multi-unmanned aerial vehicle multi-target tracking data set is an urgent need for multi-unmanned aerial vehicle multi-target tracking research.
For the problems of target shielding, target access to video pictures and target access to video pictures in shooting, researchers carry out global track matching by establishing a hypergraph, and although the method has certain benefit on solving the problems, in practical application, detection and tracking with higher precision and higher speed still need to be explored. In addition, researchers also enhance the detection capability of a single camera through multiple cameras, the problems of shielding and the like are solved by exerting the advantages of data of the multiple cameras, and the problems are difficult to solve in a single-camera environment.
The existing solution is mainly to construct the geometric relationship between cameras, and to supplement information to data through the geometric relationship, and the solution is not suitable for unmanned aerial vehicles. The relative position of the unmanned aerial vehicle changes all the time, and a stable and unchangeable geometric relation cannot be constructed, so that the research of fixing a plurality of cameras in the prior art is difficult to directly transfer and apply to a plurality of unmanned aerial vehicles. In addition, the requirement of the unmanned aerial vehicle on the real-time performance is high, the used tracking algorithm has real-time tracking capability, the existing mature multi-camera multi-target tracking algorithm is an off-line mode, namely, the whole video is detected and globally matched, and the mode cannot meet the real-time performance requirement of the multi-camera of the unmanned aerial vehicle.
Disclosure of Invention
The invention provides a multi-agent cooperative tracking method, a device and a storage medium based on data sharing, which are used for researching multi-target tracking of a plurality of unmanned aerial vehicles, and firstly provide a multi-target tracking data set MDMT of the plurality of unmanned aerial vehicles, wherein the data set has the characteristics of larger scale and more targets compared with the existing data set, and the invention can optimize the performance of the existing algorithm without establishing a geometric relationship, which is described in detail in the following description:
in a first aspect, a method for multi-agent cooperative tracking based on data sharing, the method comprises:
shooting by an unmanned aerial vehicle to obtain data, carrying out format unification and target marking on the collected data, and obtaining an MDMT data set;
based on the MDMT dataset, the tracker single-machine multi-target tracking algorithm is used for multi-target tracking of a single unmanned aerial vehicle video, cross-camera target association is carried out through information of other cameras, and the same targets appearing in different unmanned aerial vehicle visual field ranges are numbered uniformly.
In one embodiment, the MDMT dataset includes:
two unmanned aerial vehicle data set that different angles were shot includes: video sequences of several kinds and several attribute tags;
the double-unmanned aerial vehicle data set comprises two unmanned aerial vehicles which shoot pictures at different angles in the same scene, targets shot by the two unmanned aerial vehicles overlap, and the pictures are not completely the same.
In one embodiment, the cross-camera target association specifically includes:
processing the video sequence of each unmanned aerial vehicle respectively to obtain corresponding regression results R1t and R2t and target frames D1t and D2 t;
matching the target frame D1t of the first unmanned aerial vehicle with the existing tracking frame R2t of the second unmanned aerial vehicle by using cross-camera feature matching;
if the matching is successful, deleting the target frame from the D1t, giving a matching number to the tracking target in the successfully matched target frame, adding the tracking target into the tracking frame R1t, and otherwise, directly entering the next step;
matching the target frame D2t of the second unmanned aerial vehicle with the existing tracking frame R1t of the first unmanned aerial vehicle by using cross-camera feature matching;
if the matching is successful, deleting the target frame from the D2t, giving a matching number to the tracking target in the successfully matched target frame, adding the tracking target into the tracking frame R2t, and otherwise, directly entering the next step;
matching the target frame D1t of the second unmanned aerial vehicle with the target frame D2t of the first unmanned aerial vehicle by using cross-camera feature matching;
if the matching is successful, the target frames of the two unmanned aerial vehicles are endowed with the same number, and the tracking frames R1t and R2t are added respectively, otherwise, the next step is directly carried out;
and assigning new numbers to the remaining target frames, and adding the new numbers into tracking frames R1t and R2t respectively to serve as the tracking result of each unmanned aerial vehicle of the frame.
In one embodiment, the distance between different cameras to the same target is reduced and the distance between different cameras to different targets is increased by the cross-camera feature matching.
In a second aspect, a multi-agent cooperative tracking apparatus based on data sharing, the apparatus comprising:
the acquisition module is used for acquiring data through unmanned aerial vehicle shooting, carrying out format adjustment and target marking on the acquired data and acquiring an MDMT data set;
the tracking module is used for performing multi-target tracking on a single unmanned aerial vehicle video by using a tracker single-machine multi-target tracking algorithm based on the MDMT dataset;
and the cross-camera target association module is used for cross-camera target association through information of other cameras, and unifying serial numbers of the same targets appearing in different unmanned aerial vehicle visual field ranges.
In one embodiment, the cross-camera target association module comprises:
the first matching submodule is used for matching a target frame D1t of the first unmanned aerial vehicle with an existing tracking frame R2t of the second unmanned aerial vehicle by using cross-camera feature matching;
the first judgment sub-module is used for deleting the target frame from the D1t if the matching is successful, giving a matching number to the tracking target in the target frame successfully matched and adding the tracking target into the tracking frame R1t, and otherwise, directly entering the next step;
the second matching submodule is used for matching a target frame D2t of the second unmanned aerial vehicle with an existing tracking frame R1t of the first unmanned aerial vehicle by using cross-camera feature matching;
the second judgment sub-module is used for deleting the target frame from the D2t if the matching is successful, giving a matching number to the tracking target in the successfully matched target frame, adding the matching number into the tracking frame R2t, and otherwise, directly entering the next step;
a third matching submodule, configured to match a target frame D1t of the second drone and a target frame D2t of the first drone by using cross-camera feature matching;
a third judgment submodule, configured to assign the same number to the target frames of the two unmanned aerial vehicles if matching is successful, add the tracking frames R1t and R2t, respectively, and otherwise directly enter the next step;
and the output submodule is used for assigning new numbers to the remaining target frames and adding the new numbers into the tracking frames R1t and R2t respectively to serve as the tracking result of each unmanned aerial vehicle of the frame.
In a third aspect, a multi-agent cooperative tracking apparatus based on data sharing, the apparatus comprising: a processor and a memory, the memory having stored therein program instructions, the processor calling the program instructions stored in the memory to cause the apparatus to perform the method steps of any of the first aspects.
In a fourth aspect, a computer-readable storage medium, storing a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method steps of any of the first aspects.
The technical scheme provided by the invention has the beneficial effects that:
1. the invention provides a first large-scale multi-unmanned aerial vehicle multi-target tracking data set (MDMT), wherein 35 groups of videos are collected in the data set, each group of videos comprises 2 video sequences shot by two unmanned aerial vehicles at different angles and heights, 70 video sequences are counted, 31673 frames of high-definition images are obtained, and each group of video sequences has 153 targets on average. According to the application scenes and the number of targets, dividing the data set into 17 groups of video sequences in a training set and 18 groups of video sequences in a testing set, and using the training and testing for the subsequent multi-target tracking algorithm of the multi-unmanned aerial vehicle;
2. the invention provides a cross-camera target association algorithm, which aims to match the number of a target newly added in an unmanned aerial vehicle to be tracked through the information of images in other unmanned aerial vehicles, so that the problem of target shielding can be better solved;
for example: when the target has lost before several frames among the first unmanned aerial vehicle, but second unmanned aerial vehicle still tracks successfully, gets into first unmanned aerial vehicle's the field of vision again when the target, can resume through second unmanned aerial vehicle's information to reduce beating of serial number, solve the target and shelter from the scheduling and lose the problem.
Drawings
FIG. 1 is a flow chart of a data sharing-based multi-agent cooperative tracking method;
fig. 2 is a display diagram of a multi-drone multi-target tracking data set MDMT;
FIG. 3 is a schematic diagram of data sharing based multi-agent collaborative tracking;
FIG. 4 is a model diagram of a cross-camera feature matching algorithm;
FIG. 5 is a block diagram of a multi-agent cooperative tracking apparatus based on data sharing;
FIG. 6 is a schematic structural diagram of a cross-camera target association module;
fig. 7 is another block diagram of a multi-agent cooperative tracking apparatus based on data sharing.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in further detail below.
In order to solve the problems in the background art, the invention provides a multi-agent cooperative tracking algorithm based on data sharing, and the innovation of the multi-agent cooperative tracking algorithm is to provide a first larger-scale multi-unmanned aerial vehicle multi-target tracking data set (MDMT). Meanwhile, a cross-camera target association algorithm for multi-unmanned aerial vehicle multi-target tracking is designed, the characteristic of multi-unmanned aerial vehicle data sharing is effectively utilized, and the multi-unmanned aerial vehicle multi-target tracking effect and accuracy are improved.
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in further detail below.
Referring to fig. 1-4, an embodiment of the present invention provides a data sharing-based multi-agent cooperative tracking method, which is described in detail below:
first, data preparation
1. Unmanned shooting data
At first, a plurality of 4 generations of UAVs are controlled by a plurality of UAVs, shooting is carried out by the UAVs, and in order to ensure the same video time stamp, each UAV can start and stop synchronization as much as possible. The angles shot by all unmanned aerial vehicles in the same group of videos need to be guaranteed to have an overlapping area, so that the effectiveness of the data set is guaranteed. Each of the flyers photographs a plurality of scenes, for example: parks, lawns, blocks, and the like. Most of the collected data are shot in an area with a large target coverage degree.
The embodiment of the present invention is described by taking the above-mentioned model of the unmanned aerial vehicle as an example, and during specific implementation, the model can be selected according to the needs in practical application, which is not limited in the embodiment of the present invention.
2. Dividing the collected data into a training set and a test set
Only then do the integration and time stamp alignment of the data sets for a total of 70 sets of sequences, 31673 frames of images. The data set was divided into 35 groups, each group consisting of two video segments taken simultaneously by two flyers at different viewing angles, averaging 153 targets per group. According to the number of targets and the classification of shooting scenes, 17 groups of data are used as a training set, 18 groups of data are used as a test set, and the whole data set is shown as a figure 2.
The embodiment of the present invention is described by taking the above data as an example, and when the embodiment of the present invention is specifically implemented, the data may be set according to requirements in practical applications, which is not limited in this respect.
3. Preprocessing the training set data to obtain an MDMT data set
Firstly, extracting the videos in the training set into an image sequence, and then carrying out manual alignment to ensure that the same group of videos have the same start time and end time. The images were then resized to a full 1920 x 1080 high definition image. And then, manually marking the high-definition image, marking a target rectangular frame by using a marking tool VATIC, and classifying the target.
And then carrying out frame-by-frame check on the classification result by using LabelMe, and calibrating the tracking target in a minimum external frame. And numbering the tracking targets and marking id. And (4) repeatedly verifying the label id to ensure the accuracy of the label, thereby providing a large-scale and dense MDMT data set.
Wherein, the MDMT data set includes: the data sets of the double unmanned aerial vehicles shot at different angles have video sequences of a plurality of types and attribute labels. The double-unmanned aerial vehicle data set comprises two unmanned aerial vehicles which shoot pictures at different angles in the same scene at the same time, targets shot by the two unmanned aerial vehicles overlap, and the pictures are not completely the same.
In a specific implementation, other labeling tools may be used, and reduced images with other sizes may also be used, which is not limited in this embodiment of the present invention.
Second, model design
Currently, potential requirements based on multi-target tracking are mined, and more multi-target tracking algorithms appear. Nowadays, multi-target tracking algorithms are divided into two types, one is an online multi-target tracking algorithm, and the other is an offline multi-target tracking algorithm. The off-line multi-target tracking algorithm needs to detect the whole video sequence, establish a network flow graph or hypergraph model, and match a detected target box to obtain a tracking result. And the other is an online multi-target tracking algorithm, and the current frame and the previous frame are used for carrying out numbering distribution and target frame association so as to meet the requirement of real-time property. Because the offline multi-target detection needs to utilize the data of the global target frame, the excellent result is obtained, and meanwhile, the considerable time delay is increased, so that the application of occasions with high real-time performance is not met.
Because the unmanned aerial vehicle tracking has higher requirement on real-time performance, the embodiment of the invention adopts an online multi-target tracking method. The embodiment of the invention adopts an online multi-target tracking algorithm tracker as a basic target tracking algorithm based on a data sharing multi-agent cooperative tracking algorithm.
The current online multi-target tracking algorithm generally uses a target frame based on detection, and the target frame divides multi-target tracking into the following two steps, firstly, target detection is carried out on a current frame, and then the current frame and the target frame are associated.
Thirdly, using a tracker single-machine multi-target tracking algorithm as a basic algorithm, referring to fig. 1, and comprising the following steps:
inputting: a video sequence I with a length T; and (3) outputting: tracking result R ═ { R ═ R1,R2,···,RT}，RTThe tracking result is the Tth frame;
1: initializing model parameters and lambda;
2：for it∈I do
3: tracking the result R for the previous framet-1Performing regression classification on the target frames in the sequence (Rt) to inhibit the scores of the target frames in the sequence (1/10) to form Rt which is a tracking result of the t-th frame;
the above values are adjusted according to the experimental results, and the embodiment of the present invention is described only by taking 1/10 as an example.
4: non-maximum suppression Rt of all target frames;
5: detecting the picture It to obtain a corresponding target frame Dt;
6: deleting any tracking result R in the target frame Dtt-1A target box with IOU (cross-over ratio) greater than λ;
7: numbering the rest target frames in the Dt, and putting the numbered target frames in a tracking result Rt as a tracking result of the current frame;
8：end for。
in specific implementation, a Faster RCNN detector of a feature extraction network ResNet and an FPN network is trained by using Tracktor.
Fourthly, performing cross-camera target association on the basis of the third part, wherein the MTTracktor algorithm is used and is shown in FIG. 3, and the flow is as follows:
inputting: video sequences I1, I2, length T; and (3) outputting: tracking results R1, R2;
1: initializing model parameters;
2：for i1t,i2t∈I1,I2 do
3: the video sequence of each unmanned aerial vehicle respectively executes the steps 3-6 in the third part tracker algorithm to obtain corresponding regression results R1t and R2t and target frames D1t and D2 t;
4: matching the target frame D1t of the first unmanned aerial vehicle with the existing tracking frame R2t of the second unmanned aerial vehicle by using a cross-camera feature matching algorithm;
the cross-camera feature matching algorithm aims to reduce the distance between different cameras from the same target and increase the distance between different cameras from different targets, and is a known algorithm in the art and is not described herein.
5: if the matching is successful, deleting the target frame from the D1t, giving a matching number to the tracking target in the successfully matched target frame, adding the tracking target into the tracking frame R1t, and otherwise, directly entering the next step;
6: matching the target frame D2t of the second unmanned aerial vehicle with the existing tracking frame R1t of the first unmanned aerial vehicle by using a cross-camera feature matching algorithm;
7: if the matching is successful, deleting the target frame from the D2t, giving a matching number to the tracking target in the successfully matched target frame, adding the tracking target into the tracking frame R2t, and otherwise, directly entering the next step;
8: matching the target frame D1t of the second unmanned aerial vehicle with the target frame D2t of the first unmanned aerial vehicle by using a cross-camera feature matching algorithm;
9: if the matching is successful, the target frames of the two unmanned aerial vehicles are endowed with the same number, and the tracking frames R1t and R2t are added respectively, otherwise, the next step is directly carried out;
10: assigning new numbers to the remaining target frames, and adding the new numbers into tracking frames R1t and R2t respectively to serve as the tracking result of each unmanned aerial vehicle of the frame;
11：end for。
during concrete implementation, carry out the multi-target tracking of single unmanned aerial vehicle video through multi-target tracking algorithm, then carry out the cross camera target relevance through the information of other cameras, can carry out the unity of serial number to the same target that appears in different unmanned aerial vehicle field of vision scopes, meanwhile, can carry out better serial number to losing the target of recovering and resume to solve the target better and shelter from, the field of vision business turn over the recovery problem that loses under the scene, improve the precision of algorithm. The distance of the same target between different cameras is reduced through a cross-camera feature matching algorithm, and the distance of different targets between different cameras is increased.
The multi-agent cooperative tracking algorithm MTTracktor provided by the invention is improved in most indexes compared with single unmanned aerial vehicle multi-target tracking, and particularly in IDSW indexes, the number skip condition is greatly reduced, which shows that the multi-agent cooperative tracking algorithm for multi-unmanned aerial vehicles has great improvement on the number retention capability. When the MTTracktor is subjected to target detection and camera matching model training, a geometric relation does not need to be established.
Fifthly, training and testing of models
The experiment used a computer with an Intel (R) core (TM) i9-7900X processor and two NVIDIA 2080Ti display cards to evaluate two reference single-camera multi-target tracking algorithms on the MDMT data set, which are tracker and CTRacker respectively. The reference target tracking algorithm used here is derived from the official code provided by the Tracktor algorithm, and is trained on the 17 MDMT training sets proposed by the present invention, and tested on the 18 MDMT test sets.
The method performs target detection and cross-camera matching algorithm model training on the MTTracktor so as to better evaluate the multi-unmanned aerial vehicle multi-target tracking algorithm MTTracktor. The present invention uniformly sets the picture size to 256 x 128 for better training across camera models. And taking the images of the 2 groups of images of the two cameras as positive samples and the images of the other groups as negative samples, adopting ResNet50 by the backbone network, using SGD optimization, and setting the learning rate to be 1 e-4.
TABLE 1 Performance comparison of MDMT dataset Multi-target tracking algorithms
Table 1 shows the accuracy of the data sharing-based multi-agent cooperation algorithm and the reference multi-target tracking algorithm. As can be seen from Table 1, the precision of Tracktor is higher than that of CTRacker because Tracktor has stronger detection capability and number recall capability. However, in IDSW index, Tracktor is relatively weaker than CTRackter, i.e., CTRackter has a stronger ability to hold a number but has a weaker ability to follow a target.
Through the comparison, the MTTracktor provided by the invention is more excellent than two reference tracking algorithms on more indexes. Particularly, IDSW and MTTracktor enable the number of number jumps to be obviously reduced, which shows that the invention has obvious improvement on the number keeping capability.
The embodiment of the invention has the following two key creation points
1. The first multiple unmanned plane multiple target tracking data set (MDMT)
The technical effects are as follows: the data set contains 35 sets of video sequences, 31673 frames of high-definition images, for a total of 140 million annotation boxes. Meanwhile, the data set has more dense targets, 4599 targets are marked, on average, each group of video sequences contains 153 targets, and compared with the previous data set, the data set has a larger scale and the targets are more dense.
2. Provides a cross-camera target association algorithm
The technical effects are as follows: the invention provides a multi-unmanned aerial vehicle multi-target tracking algorithm without establishing a geometric relation.
The algorithm can match the number of the newly added target in the unmanned aerial vehicle to be tracked through the image information in other unmanned aerial vehicles, so that the jump of the number can be reduced, and the problem of target shielding is better solved.
An embodiment of the present invention provides a multi-agent cooperative tracking apparatus based on data sharing, and referring to fig. 5, the apparatus includes:
the acquisition module 1 is used for acquiring data through unmanned aerial vehicle shooting, performing format adjustment and target marking on the acquired data, and acquiring an MDMT data set;
the tracking module 2 is used for performing multi-target tracking on a single unmanned aerial vehicle video by using a tracker single-machine multi-target tracking algorithm based on the MDMT data set;
and the cross-camera target association module 3 is used for carrying out cross-camera target association through the information of other cameras and unifying the serial numbers of the same targets appearing in the visual field ranges of different unmanned aerial vehicles.
In one embodiment, referring to fig. 6, the cross-camera target association module 3 includes:
the first matching submodule 31 is configured to match a target frame D1t of the first drone with an existing tracking frame R2t of the second drone by using cross-camera feature matching;
the first judgment sub-module 32 is configured to delete the target frame from the D1t if the matching is successful, assign a matching number to the tracking target in the successfully matched target frame, add the tracking target to the tracking frame R1t, and otherwise, directly enter the next step;
the second matching submodule 33 is configured to match the target frame D2t of the second drone with the existing tracking frame R1t of the first drone by using cross-camera feature matching;
the second judgment sub-module 34 is configured to delete the target frame from the D2t if the matching is successful, assign a matching number to the tracking target in the successfully matched target frame, add the tracking target to the tracking frame R2t, and otherwise, directly enter the next step;
the third matching submodule 35 is configured to match the target frame D1t of the second drone with the target frame D2t of the first drone by using cross-camera feature matching;
a third judgment sub-module 36, configured to, if matching is successful, assign the same number to the target frames of the two drones, and add the tracking frames R1t and R2t, respectively, otherwise, directly enter the next step;
and an output sub-module 37, configured to assign new numbers to the remaining target frames, and add the new numbers to the tracking frames R1t and R2t, respectively, as the tracking result of each drone in the frame.
It should be noted that the device description in the above embodiments corresponds to the description of the method embodiments, and the embodiments of the present invention are not described herein again.
The execution main bodies of the modules and units can be devices with calculation functions, such as a computer, a single chip microcomputer and a microcontroller, and in the specific implementation, the execution main bodies are not limited in the embodiment of the invention and are selected according to the requirements in practical application.
Based on the same inventive concept, an embodiment of the present invention further provides a multi-agent cooperative tracking apparatus based on data sharing, referring to fig. 7, the apparatus includes: a processor 4 and a memory 5, the memory 5 having stored therein program instructions, the processor 4 calling the program instructions stored in the memory 5 to cause the apparatus to perform the following method steps in an embodiment:
shooting by an unmanned aerial vehicle to obtain data, performing format adjustment and target marking on the collected data, and obtaining an MDMT data set;
based on the MDMT dataset, the tracker single-machine multi-target tracking algorithm is used for carrying out multi-target tracking on a single unmanned aerial vehicle video, cross-camera target association is carried out through information of other cameras, and the same targets appearing in different unmanned aerial vehicle visual field ranges are numbered uniformly.
In one embodiment, the MDMT dataset includes:
two unmanned aerial vehicle data set that different angles were shot includes: video sequences of several kinds and several attribute tags;
the double-unmanned aerial vehicle data set comprises two unmanned aerial vehicles which shoot pictures at different angles in the same scene, targets shot by the two unmanned aerial vehicles overlap, and the pictures are not completely the same.
In one embodiment, the cross-camera target association specifically includes:
processing the video sequence of each unmanned aerial vehicle respectively to obtain corresponding regression results R1t and R2t and target frames D1t and D2 t;
matching the target frame D1t of the first unmanned aerial vehicle with the existing tracking frame R2t of the second unmanned aerial vehicle by using cross-camera feature matching;
if the matching is successful, deleting the target frame from the D1t, giving a matching number to the tracking target in the successfully matched target frame, adding the tracking target into the tracking frame R1t, and otherwise, directly entering the next step;
matching the target frame D2t of the second unmanned aerial vehicle with the existing tracking frame R1t of the first unmanned aerial vehicle by using cross-camera feature matching;
if the matching is successful, deleting the target frame from the D2t, giving a matching number to the tracking target in the successfully matched target frame, adding the tracking target into the tracking frame R2t, and otherwise, directly entering the next step;
matching the target frame D1t of the second unmanned aerial vehicle with the target frame D2t of the first unmanned aerial vehicle by using cross-camera feature matching;
if the matching is successful, the target frames of the two unmanned aerial vehicles are endowed with the same number, and the tracking frames R1t and R2t are added respectively, otherwise, the next step is directly carried out;
and assigning new numbers to the remaining target frames, and adding the new numbers into tracking frames R1t and R2t respectively to serve as the tracking result of each unmanned aerial vehicle of the frame.
In one embodiment, the distance between different cameras for the same target is made smaller and the distance between different targets is made larger by cross-camera feature matching.
It should be noted that the device description in the above embodiments corresponds to the method description in the embodiments, and the embodiments of the present invention are not described herein again.
The execution main bodies of the processor 4 and the memory 5 may be devices having a calculation function, such as a computer, a single chip, a microcontroller, and the like, and in the specific implementation, the execution main bodies are not limited in the embodiment of the present invention, and are selected according to the needs in the practical application.
The memory 5 and the processor 4 transmit data signals through the bus 6, which is not described in detail in the embodiment of the present invention.
Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, where the storage medium includes a stored program, and when the program runs, the apparatus on which the storage medium is located is controlled to execute the method steps in the foregoing embodiments.
The computer readable storage medium includes, but is not limited to, flash memory, hard disk, solid state disk, and the like.
It should be noted that the descriptions of the readable storage medium in the above embodiments correspond to the descriptions of the method in the embodiments, and the descriptions of the embodiments of the present invention are not repeated here.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the invention are brought about in whole or in part when the computer program instructions are loaded and executed on a computer.
The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on or transmitted over a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium or a semiconductor medium, etc.
Reference documents:
[1]Ristani E,Solera F,Zou R S,et al.Performance Measures and a Data Set for Multi-target,Multi-camera Tracking[C].In European Conference on Computer Vision,2016:17–3
[2]Zhu P,Wen L,Bian X,et al.Vision Meets Drones:A Challenge[J].CoRR,2018,abs/1804.07437.
[3]Wen L,Du D,Cai Z,et al.UA-DETRAC:A new benchmark and protocolfor multi-object detection and tracking[J].Computer Vision and Image Understand-ing,2020,193:102907.
[4]Wu B,Nevatia R.Tracking of Multiple,Partially Occluded Humans based on Static Body Part Detection[C].In IEEE Conference on Computer Vision and Pattern Recognition,2006:951–958.
[5]Bernardin K,Stiefelhagen R.Evaluating Multiple Object Tracking Performance:The CLEAR MOT Metrics[J].EURASIP Journal on Image and Video Process-ing,2008,2008.
[6]Bergmann P,Meinhardt T,Leal-Taixe′L.Tracking Without Bells and Whistles[C].In IEEE International Conference on Computer Vision,2019:941–951.
[7]Peng J,Wang C,Wan F,et al.Chained-Tracker:Chaining Paired Attentive Re-gression Results for End-to-End Joint Multiple-Object Detection and Tracking [C].In European Conference on Computer Vision,2020:145–161.
[8]Xiang Y,Alahi A,Savarese S.Learning to Track:Online Multi-object Tracking by Decision Making[C].In IEEE International Conference on Computer Vision,2015:4705–4713.
[9]Sadeghian A,Alahi A,Savarese S.Tracking the Untrackable:Learning to Track Multiple Cues with Long-Term Dependencies[C].In IEEE International Confer-ence on Computer Vision,2017:300–311.
[10]Chu Q,Ouyang W,Li H,et al.Online Multi-object Tracking Using CNN-Based Single Object Tracker with Spatial-Temporal Attention Mechanism [C].In IEEE International Conference on Computer Vision,2017:4846–4855.
[11]Zhu J,Yang H,Liu N,et al.Online Multi-Object Tracking with Dual Match-ing Attention Networks[C].In European Conference on Computer Vision,2018:379–396.
[12]Hu W,Hu M,Zhou X,et al.Principal Axis-Based Correspondence between Mul-tiple Cameras for People Tracking[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2006,28(4):663–671.
[13]Eshel R,Moses Y.Homography based multiple camera detection and tracking of people in a dense crowd[C].In IEEE Conference on Computer Vision and Pattern Recognition,2008.
[14] vondrick C, Patterson D J, ramann D.E optimal grading up Crowdsourced Video interpretation-A Set of Best Practices for High Quality, ecological Video Labeling [ J ]. International Journal of Computer Vision,2013,101(1): 184-.
[15]Russell B C,Torralba A,Murphy K P,et al.LabelMe:a database and web-based tool for image annotation[J].International Journal of Computer Vision,2008,77(1-3):157–173.
[16]He K,Zhang X,Ren S,et al.Deep residual learning for image recognition[C].In IEEE conference on Computer Vision and Pattern Recognition,2016:770–778.
[17]Lin T-Y,Dolla′r P,Girshick R,et al.Feature pyramid networks for object detec-tion[C].In IEEE conference on Computer Vision and Pattern Recognition,2017:2117–2125.
[18]Ren S,He K,Girshick R,et al.Faster r-cnn:Towards real-time object detection with region proposal networks[C].In Advances in Neural Information Processing Systems,2015:91–99.
[19]Ristani E,Tomasi C.Features for multi-target multi-camera tracking and re-identification[C].In IEEE conference on computer vision and pattern recogni-tion,2018:6036–6046.
[20]Hermans A,Beyer L,Leibe B.In defense of the triplet loss for person re-identification[J].arXiv preprint arXiv:1703.07737,2017.
In the embodiment of the present invention, except for the specific description of the model of each device, the model of other devices is not limited, as long as the device can perform the above functions.
Those skilled in the art will appreciate that the drawings are only schematic illustrations of preferred embodiments, and the above-described embodiments of the present invention are merely provided for description and do not represent the merits of the embodiments.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (8)
1. A multi-agent cooperative tracking method based on data sharing, the method comprising:
shooting by an unmanned aerial vehicle to obtain data, and performing format adjustment and target labeling processing on the collected data to obtain an MDMT data set;
based on the MDMT dataset, the tracker single-machine multi-target tracking algorithm is used for multi-target tracking of a single unmanned aerial vehicle video, cross-camera target association is carried out through information of other cameras, and the same targets appearing in different unmanned aerial vehicle visual field ranges are numbered uniformly.
2. The multi-agent cooperative tracking method based on data sharing according to claim 1, wherein the MDMT dataset comprises:
two unmanned aerial vehicle data set that different angles were shot includes: video sequences of several kinds and several attribute tags;
the double-unmanned aerial vehicle data set comprises two unmanned aerial vehicles which shoot pictures at different angles in the same scene, targets shot by the two unmanned aerial vehicles overlap, and the pictures are not completely the same.
3. The multi-agent cooperative tracking method based on data sharing as claimed in claim 1, wherein the cross-camera target association is specifically:
processing the video sequence of each unmanned aerial vehicle respectively to obtain corresponding regression results R1t and R2t and target frames D1t and D2 t;
matching the target frame D1t of the first unmanned aerial vehicle with the existing tracking frame R2t of the second unmanned aerial vehicle by using cross-camera feature matching;
if the matching is successful, deleting the target frame from the D1t, giving a matching number to the tracking target in the successfully matched target frame, adding the tracking target into the tracking frame R1t, and otherwise, directly entering the next step;
matching the target frame D2t of the second unmanned aerial vehicle with the existing tracking frame R1t of the first unmanned aerial vehicle by using cross-camera feature matching;
if the matching is successful, deleting the target frame from the D2t, giving a matching number to the tracking target in the successfully matched target frame, adding the tracking target into the tracking frame R2t, and otherwise, directly entering the next step;
matching the target frame D1t of the second unmanned aerial vehicle with the target frame D2t of the first unmanned aerial vehicle by using cross-camera feature matching;
if the matching is successful, the target frames of the two unmanned aerial vehicles are endowed with the same number, and the tracking frames R1t and R2t are added respectively, otherwise, the next step is directly carried out;
and assigning new numbers to the remaining target frames, and adding the new numbers into tracking frames R1t and R2t respectively to serve as the tracking result of each unmanned aerial vehicle of the frame.
4. The multi-agent cooperative tracking method based on data sharing as claimed in claim 3, wherein the cross-camera feature matching makes the distance between different cameras from the same target smaller and the distance between different cameras from different targets larger.
5. A multi-agent cooperative tracking apparatus based on data sharing, the apparatus comprising:
the acquisition module is used for acquiring data through unmanned aerial vehicle shooting, performing format adjustment and target labeling processing on the acquired data and acquiring an MDMT data set;
the tracking module is used for performing multi-target tracking on a single unmanned aerial vehicle video by using a tracker single-machine multi-target tracking algorithm based on the MDMT dataset;
and the cross-camera target association module is used for cross-camera target association through information of other cameras, and unifying serial numbers of the same targets appearing in different unmanned aerial vehicle visual field ranges.
6. The multi-agent cooperative tracking device based on data sharing of claim 5, wherein the cross-camera target association module comprises:
the first matching submodule is used for matching a target frame D1t of the first unmanned aerial vehicle with an existing tracking frame R2t of the second unmanned aerial vehicle by using cross-camera feature matching;
the first judgment sub-module is used for deleting the target frame from the D1t if the matching is successful, giving a matching number to the tracking target in the target frame successfully matched and adding the tracking target into the tracking frame R1t, and otherwise, directly entering the next step;
the second matching submodule is used for matching a target frame D2t of the second unmanned aerial vehicle with an existing tracking frame R1t of the first unmanned aerial vehicle by using cross-camera feature matching;
the second judgment sub-module is used for deleting the target frame from the D2t if the matching is successful, giving a matching number to the tracking target in the successfully matched target frame, adding the matching number into the tracking frame R2t, and otherwise, directly entering the next step;
a third matching submodule, configured to match a target frame D1t of the second drone and a target frame D2t of the first drone by using cross-camera feature matching;
a third judgment submodule, configured to assign the same number to the target frames of the two unmanned aerial vehicles if matching is successful, add the tracking frames R1t and R2t, respectively, and otherwise directly enter the next step;
and the output submodule is used for assigning new numbers to the remaining target frames and adding the new numbers into the tracking frames R1t and R2t respectively to serve as the tracking result of each unmanned aerial vehicle of the frame.
7. A multi-agent cooperative tracking apparatus based on data sharing, the apparatus comprising: a processor and a memory, the memory having stored therein program instructions, the processor calling upon the program instructions stored in the memory to cause the apparatus to perform the method steps of any of claims 1-4.
8. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method steps of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110411910.9A CN113223060B (en) | 2021-04-16 | 2021-04-16 | Multi-agent cooperative tracking method and device based on data sharing and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110411910.9A CN113223060B (en) | 2021-04-16 | 2021-04-16 | Multi-agent cooperative tracking method and device based on data sharing and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113223060A true CN113223060A (en) | 2021-08-06 |
| CN113223060B CN113223060B (en) | 2022-04-15 |
Family
ID=77087891
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110411910.9A Expired - Fee Related CN113223060B (en) | 2021-04-16 | 2021-04-16 | Multi-agent cooperative tracking method and device based on data sharing and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113223060B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107240124A (en) * | 2017-05-19 | 2017-10-10 | 清华大学 | Across camera lens multi-object tracking method and device based on space-time restriction |
| CN109584213A (en) * | 2018-11-07 | 2019-04-05 | 复旦大学 | A kind of selected tracking of multiple target number |
| CN110428448A (en) * | 2019-07-31 | 2019-11-08 | 腾讯科技（深圳）有限公司 | Target detection tracking method, device, equipment and storage medium |
| CN110825112A (en) * | 2019-11-22 | 2020-02-21 | 渤海大学 | Oil field dynamic invasion target tracking system and method based on multiple unmanned aerial vehicles |
| US10621461B1 (en) * | 2013-03-13 | 2020-04-14 | Hrl Laboratories, Llc | Graphical display and user-interface for high-speed triage of potential items of interest in imagery |
| CN111145213A (en) * | 2019-12-10 | 2020-05-12 | 中国银联股份有限公司 | Target tracking method, device and system and computer readable storage medium |
| CN111176334A (en) * | 2020-01-16 | 2020-05-19 | 浙江大学 | Multi-unmanned aerial vehicle cooperative target searching method |
| CN111176309A (en) * | 2019-12-31 | 2020-05-19 | 北京理工大学 | Multi-unmanned aerial vehicle self-group mutual inductance understanding method based on spherical imaging |
| CN111290440A (en) * | 2020-04-07 | 2020-06-16 | 中国人民解放军海军航空大学 | Multi-unmanned aerial vehicle formation Standoff tracking control and tracking method based on double virtual structures |
| CN111427379A (en) * | 2020-04-19 | 2020-07-17 | 中国人民解放军海军航空大学 | Observation-driven multi-unmanned aerial vehicle cooperative standoff target tracking method |
| CN111833378A (en) * | 2020-06-09 | 2020-10-27 | 天津大学 | Multi-unmanned aerial vehicle single-target tracking method and device based on proxy sharing network |
| CN112651995A (en) * | 2020-12-21 | 2021-04-13 | 江南大学 | On-line multi-target tracking method based on multifunctional aggregation and tracking simulation training |
-
2021
- 2021-04-16 CN CN202110411910.9A patent/CN113223060B/en not_active Expired - Fee Related
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10621461B1 (en) * | 2013-03-13 | 2020-04-14 | Hrl Laboratories, Llc | Graphical display and user-interface for high-speed triage of potential items of interest in imagery |
| CN107240124A (en) * | 2017-05-19 | 2017-10-10 | 清华大学 | Across camera lens multi-object tracking method and device based on space-time restriction |
| CN109584213A (en) * | 2018-11-07 | 2019-04-05 | 复旦大学 | A kind of selected tracking of multiple target number |
| CN110428448A (en) * | 2019-07-31 | 2019-11-08 | 腾讯科技（深圳）有限公司 | Target detection tracking method, device, equipment and storage medium |
| CN110825112A (en) * | 2019-11-22 | 2020-02-21 | 渤海大学 | Oil field dynamic invasion target tracking system and method based on multiple unmanned aerial vehicles |
| CN111145213A (en) * | 2019-12-10 | 2020-05-12 | 中国银联股份有限公司 | Target tracking method, device and system and computer readable storage medium |
| CN111176309A (en) * | 2019-12-31 | 2020-05-19 | 北京理工大学 | Multi-unmanned aerial vehicle self-group mutual inductance understanding method based on spherical imaging |
| CN111176334A (en) * | 2020-01-16 | 2020-05-19 | 浙江大学 | Multi-unmanned aerial vehicle cooperative target searching method |
| CN111290440A (en) * | 2020-04-07 | 2020-06-16 | 中国人民解放军海军航空大学 | Multi-unmanned aerial vehicle formation Standoff tracking control and tracking method based on double virtual structures |
| CN111427379A (en) * | 2020-04-19 | 2020-07-17 | 中国人民解放军海军航空大学 | Observation-driven multi-unmanned aerial vehicle cooperative standoff target tracking method |
| CN111833378A (en) * | 2020-06-09 | 2020-10-27 | 天津大学 | Multi-unmanned aerial vehicle single-target tracking method and device based on proxy sharing network |
| CN112651995A (en) * | 2020-12-21 | 2021-04-13 | 江南大学 | On-line multi-target tracking method based on multifunctional aggregation and tracking simulation training |
Non-Patent Citations (3)
| Title |
|---|
| PENGFEI ZHU ET AL.: ""Multi-Drone based Single Object Tracking with Agent Sharing Network"", 《ARXIV》 * |
| PENGFEI ZHU ET AL.: ""Vision Meets Drones: A Challenge"", 《ARXIV》 * |
| 肖磊等: ""基于深度学习的视频全量分析方法"", 《软件开发》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113223060B (en) | 2022-04-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhu et al. | Detection and tracking meet drones challenge | |
| Ristani et al. | Features for multi-target multi-camera tracking and re-identification | |
| Girdhar et al. | Detect-and-track: Efficient pose estimation in videos | |
| Jiao et al. | A survey of deep learning-based object detection | |
| Chen et al. | Real-time multiple people tracking with deeply learned candidate selection and person re-identification | |
| Qu et al. | RGBD salient object detection via deep fusion | |
| Wang et al. | Static and moving object detection using flux tensor with split Gaussian models | |
| Ding et al. | Violence detection in video by using 3D convolutional neural networks | |
| Zhu et al. | Multi-drone-based single object tracking with agent sharing network | |
| CN112149762A (en) | Target tracking method, target tracking apparatus, and computer-readable storage medium | |
| Noor et al. | Automatic object tracking and segmentation using unsupervised SiamMask | |
| Grigorev et al. | Deep person re-identification in UAV images | |
| Saribas et al. | TRAT: Tracking by attention using spatio-temporal features | |
| Elharrouss et al. | FSC-set: counting, localization of football supporters crowd in the stadiums | |
| Asadi-Aghbolaghi et al. | Action recognition from RGB-D data: Comparison and fusion of spatio-temporal handcrafted features and deep strategies | |
| Bashar et al. | Multiple object tracking in recent times: A literature review | |
| Zhang et al. | Multi-domain collaborative feature representation for robust visual object tracking | |
| CN113223060B (en) | Multi-agent cooperative tracking method and device based on data sharing and storage medium | |
| Ruan et al. | Object tracking via online trajectory optimization with multi-feature fusion | |
| Park et al. | Intensity classification background model based on the tracing scheme for deep learning based CCTV pedestrian detection | |
| Huang et al. | Motion characteristics estimation of animals in video surveillance | |
| Mishra et al. | Automated detection of fighting styles using localized action features | |
| CN113761263A (en) | Similarity determination method and device and computer readable storage medium | |
| CN110322471B (en) | Method, device and equipment for concentrating panoramic video and storage medium | |
| Zhang et al. | What makes for good multiple object trackers? |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220415 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |