CN111899284B - Planar target tracking method based on parameterized ESM network - Google Patents
Planar target tracking method based on parameterized ESM network Download PDFInfo
- Publication number
- CN111899284B CN111899284B CN202010816457.5A CN202010816457A CN111899284B CN 111899284 B CN111899284 B CN 111899284B CN 202010816457 A CN202010816457 A CN 202010816457A CN 111899284 B CN111899284 B CN 111899284B
- Authority
- CN
- China
- Prior art keywords
- target
- feature
- data
- template
- input image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000001514 detection method Methods 0.000 claims abstract description 29
- 238000000605 extraction Methods 0.000 claims abstract description 22
- 238000005259 measurement Methods 0.000 claims abstract description 19
- 230000007246 mechanism Effects 0.000 claims abstract description 18
- 238000013507 mapping Methods 0.000 claims abstract description 13
- 238000007781 pre-processing Methods 0.000 claims abstract description 8
- 230000000903 blocking effect Effects 0.000 claims abstract description 4
- 238000010606 normalization Methods 0.000 claims abstract description 3
- 238000012549 training Methods 0.000 claims description 30
- 230000009466 transformation Effects 0.000 claims description 25
- 230000008569 process Effects 0.000 claims description 23
- 239000011159 matrix material Substances 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 11
- 239000013598 vector Substances 0.000 claims description 9
- 230000003287 optical effect Effects 0.000 claims description 7
- 238000005457 optimization Methods 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 6
- 238000005286 illumination Methods 0.000 claims description 6
- 238000012795 verification Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 238000000844 transformation Methods 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 claims description 2
- 238000011524 similarity measure Methods 0.000 claims 1
- 230000008859 change Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000013461 design Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000000691 measurement method Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000002994 raw material Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention provides a planar target tracking method based on a parameterized ESM network, which comprises the following steps: s1, acquiring a target template T, an input image of a T frame and initial motion parameters in the T frame, and determining a target area I of the input image by the initial motion parameters t For target template T and target region I t Preprocessing, including image scaling and normalization, the target template T after preprocessing and the target area I of the input image of the T frame are processed by using the feature extraction network t Extracting features to obtain feature map F T And F t I The method comprises the steps of carrying out a first treatment on the surface of the S2, calculating two feature mappings F by using a similarity measurement module T And F t I Differences between; s3, determining and removing the blocked part of the target in the current frame through a blocking detection mechanism, and solving the motion parameters of the target by minimizing the difference of the non-blocked part in the current frame. The method is more suitable for target tracking tasks, and greatly improves the tracking accuracy.
Description
Technical Field
The invention relates to the field of machine vision and pattern recognition, in particular to a planar target tracking method based on a parameterized ESM network.
Background
Planar object tracking refers to a given sequence of video frames and specifies a planar object of interest in a first frame, the object of the planar object tracking algorithm being to calculate the change in pose of the planar object in subsequent video frames. Planar object tracking is a central problem in computer vision and has applications in many fields, such as augmented reality, robotics, unmanned aerial vehicle technology, etc.
Patent document No. 201510147895.6 discloses a moving object tracking method based on a bit plane. The invention obtains a brightness bit plane and a local binary pattern bit plane after smoothing for a tracking target and a searching area; then searching the two appearance planes of the search area for the area closest to the two appearance models corresponding to the tracking target as the tracking target; after tracking is completed, the appearance model is updated according to the established appearance model and the tracking result in the current frame and the preset updating rate. The method has obvious advantages in tracking precision and robustness, and effectively solves the problem of difficult tracking of the moving target under the conditions of illumination condition change, target pose change, apparent change and the like in the video.
The patent document with application number 201910297980.9 discloses a moving target tracking method based on a template matching and depth classification network, which mainly solves the problems that in the prior art, the target detection speed is low, and the tracking is inaccurate when the target is deformed and shielded. The scheme extracts a template network and a detection network from a double-residual depth classification network; extracting template characteristics and detection characteristics on the template and the detection area by using corresponding networks; template matching is carried out on the template characteristics on the detection characteristics, and a template matching diagram is obtained; determining a target position according to the template matching diagram; and tracking the target position and updating the template characteristics. The method has high tracking speed and high accuracy and is used for tracking the video target with severe deformation and illumination change.
For the patent document with the application number of 201510147895.6, the scheme solves the problem that the target tracking is difficult under the conditions of illumination change, appearance change and the like in the video to a certain extent. Although the invention carries out careful design modeling on the brightness and texture of the target, the modeling method of manual design cannot accurately reflect the appearance characteristics of the target. In the patent document 201910297980.9, although a depth network is adopted as a feature extractor, the feature extractor is not embedded into an end-to-end framework constructed in a video tracking task to train and verify, but is trained on a classification task, and a simple sliding window convolution method is adopted in calculating a feature response graph. Indeed, the sliding window convolution method is not necessarily applicable on depth profiles. Furthermore, neither of these inventions takes into account that the target portion is blocked or that the target portion is out of view.
Disclosure of Invention
The embodiment of the invention provides a planar target tracking method based on a parameterized ESM network, which overcomes the defects of the prior art.
In order to achieve the above purpose, the present invention adopts the following technical scheme.
A planar target tracking method based on a parameterized ESM network, a depth planar object tracking model is constructed, the depth planar object tracking model comprising: the method comprises the steps of feature extraction network, similarity measurement module and occlusion detection mechanism, constructing a data set to train the depth plane object tracking model, wherein the plane object tracking method comprises the following steps:
s1, acquiring a target template T, an input image of a T frame and initial motion parameters in the T frame, and determining a target area I of the input image by the initial motion parameters t For target template T and target region I t Preprocessing, including image scaling and normalization, the target template T after preprocessing and the target area I of the input image of the T frame are processed by using the feature extraction network t Extracting features to obtain feature map F T And F t I The dimensions of the template and the target area after pretreatment are h multiplied by l multiplied by 3, and h, l and 3 are the width, length and channel number of the image respectively;
s2, calculating two feature mappings F by using a similarity measurement module T And F t I Differences between;
s3, determining and removing the blocked part of the target in the current frame through a blocking detection mechanism, and solving the motion parameters of the target by minimizing the difference of the non-blocked part in the current frame.
Preferably, the tracking of each frame in the video is divided into two phases, specifically:
the motion parameter of the tracking result of the first stage is used as the initial motion parameter of the second stage, and in the next iteration process, the motion parameter of the tracking result of the last second stage is used as the initial motion parameter of the first stage in the current iteration.
Preferably, the feature extraction network of each stage is composed of 7 convolution layers, each layer is followed by a batch norm layer and an activation function ReLU layer, the convolution kernel number of the first 6 convolution layers is 64, the convolution kernel number of the last convolution layer is 8, in the kth stage, the step size of the first 4-k convolution layers of the 7 convolution layers is 2, and the step size of the remaining convolution layers is 1, k is 1 or 2.
Preferably, the similarity measurement module is an encoder-decoder network based on a u-net framework, the input of which is the feature map F of the target template T T And a target area I of a t-th frame input image t Feature mapping F of (2) t I Outputs as a concatenation of the two feature maps F T And F t I Is a difference tensor of (c).
Preferably, the S3 includes:
target area I of input image of t-th frame t Simplified representation as I, its feature map F t I Is simply denoted as F I L2 normalized feature map F for target region of given template and t-th frame input image T And F I The feature dimension is h 'x l' x d, where h 'and l' correspond to the width and length of the extracted feature image, respectively,k is 1 or 2, d represents the dimension of the feature;
first, F is respectively taken as a unit of each feature T And F I And (c) developing into an m x d matrix along the h ' direction, wherein m=h ' ×l ', denoted asAnd->Feature map representing the template T being expanded, +.>The feature map of the expanded target region is represented, and then a correlation map R is calculated to record the similarity of each pair of features, and the dimension of the correlation map R is m×m, and the formula is as follows:
wherein i, j respectively represent indexes of features in feature mapping of the target template T and the target region, R i,j Representing the similarity between the ith feature in the template feature map and the jth feature in the feature map of the target region, Z is a trainable parameter matrix with Z dimension of d x d, and the confidence vector is formed by selecting the maximum value of each row in RThe formula is as follows:
then, willElement of (C) is normalized to [0,1 ]]Within the interval as final confidence vector +.>
Finally, confidence vectors are usedTaking h ' as a row, arranging the motion parameters into the size of h ' x l ', marking as C, and solving the motion parameters of the target by minimizing the difference of the non-shielded parts, for exampleThe following formula:
wherein p represents the current predicted target motion parameter; x represents the two-dimensional index of the feature in the feature map; c (-) is the contribution degree of the position feature to the optimization after detection by an occlusion detection mechanism, the contribution degree of the occluded part feature to the optimization is 0, and the contribution degree of the non-occluded feature is 1; m (&. Cndot.) measures the variability of each pair of features in the template and target region;a formula representing the coordinate transformation;
solving the formula (3) by adopting an ESM method, wherein the method is as follows:
order the
The increment of the motion parameter is obtained by:
wherein the method comprises the steps ofRepresenting pseudo-inverse of matrix, J T Calculated at U unit transformJacobian matrix, J E (p) represents the jacobian matrix of E (x; p) at p:
the motion parameters are updated in combination with the delta deltap of the motion parameters:
wherein,representing a binary operation.
Preferably, the constructing the dataset trains the depth plane object tracking model, comprising:
constructing two labeled DATA sets GEN-DATA and OCC-DATA, wherein GEN-DATA comprises illumination, deformation and noise factors, OCC-DATA is based on GEN-DATA, and the situation that the target part is blocked and the target part is out of the visual field range is increased, and each sample in the DATA sets GEN-DATA and OCC-DATA is a quadruple (T, Q, p) 0 ,p gt ) Wherein T is a template image, Q is a current input image, and p 0 For initial motion parameters, p gt Is the real motion parameter of the target;
the DATA set GEN-DATA construction process includes: geometric transformations and optical disturbances;
the geometric transformation includes:
given the target template T and the real motion parameters p of the target gt Mapping the pixel points in the target template into the input image Q through a perspective transformation formula, wherein the perspective transformation formula is as follows:
wherein,the pixel coordinate is (u, v) the coordinate of the pixel, (x, y) the coordinate of the pixel after perspective transformation;
the angular point of the target in the input image Q is respectively moved by d pixels along any direction, d is an integer of 0 to 20, and a corresponding transformation matrix, namely an initial motion parameter p, is calculated according to the moved angular point coordinates 0 ；
The optical perturbation includes:
1) Adding motion blur or gaussian blur to the input image;
2) Adding gaussian noise to the input image;
3) The brightness variation of different degrees is implemented on all pixels on the input image along a certain direction;
the DATA set OCC-DATA construction process includes:
for each sample in the GEN-DATA, a point is selected on each side of the object in the input image, forming a size N P Randomly selecting N (N is more than or equal to 0 and less than or equal to N) P ) The points are sequentially connected to divide a target area in a video frame into a plurality of parts, and a part of the target area is randomly selected to be filled with a pattern of another picture so as to simulate the shielding condition;
the DATA sets GEN-DATA and OCC-DATA are each at 8:2 is divided into a training set and a verification set for training the performance of the model and the verification model;
during training, firstly, an occlusion detection mechanism is not added, a GEN-DATA is used for training a feature extraction network and a similarity measurement module, parameters of the feature extraction network and the similarity measurement module are fixed after training is completed, an OCC-DATA is used for training the occlusion detection mechanism, and parameters of the feature extraction network and the similarity measurement module are finely adjusted at the same time;
the loss function formula adopted in the training process is as follows:
wherein,target motion parameters predicted for model, p gt Is the object ofIs a real motion parameter of the robot; n is the number of target corner points, r q Coordinates of corner points; />A formula representing the coordinate transformation.
According to the technical scheme provided by the embodiment of the invention, the planar target tracking method based on the parameterized ESM network is provided, and the trainable feature extraction module and the sufficient training set are adopted, so that the feature extraction module learns the relatively robust feature representation, and the problem that the traditional manual design features cannot accurately reflect the appearance characteristics of the target in the tracking process is solved to a certain extent; the incompatibility problem of the depth features and the traditional similarity measurement method is also solved through the trained feature extraction module and the similarity measurement module; the model solving is assisted by using an occlusion detection mechanism, so that the model is more robust to partial occlusion; meanwhile, the loss function in the logarithmic function form can enable the training process of the model not to be dominated by samples with larger loss. Because the invention uses the COCO data set of Microsoft as the raw material to construct the data set, the model plane object tracking accuracy is far higher than that of the traditional method and the existing depth network-based method, and the model plane object tracking accuracy is sufficient in training samples and end-to-end network construction.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a planar target tracking method based on a parameterized ESM network according to an embodiment of the present invention;
FIG. 2 is a GEN-DATA generation effect diagram of a planar target tracking method based on a parameterized ESM network according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of OCC-DATA generation effect of a planar target tracking method based on a parameterized ESM network according to an embodiment of the present invention;
FIG. 4 is a flowchart of a similarity measurement module of a planar target tracking method based on a parameterized ESM network according to an embodiment of the present invention;
fig. 5 is a schematic diagram of selecting a tracking target in a first frame according to an embodiment of the present invention.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only for explaining the present invention and are not to be construed as limiting the present invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or coupled. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
For the purpose of facilitating an understanding of the embodiments of the invention, reference will now be made to the drawings of several specific embodiments illustrated in the drawings and in no way should be taken to limit the embodiments of the invention.
The embodiment of the invention provides a planar target tracking method based on a parameterized ESM network, which is shown in fig. 1, and comprises the steps of: the feature extraction network, the similarity metric module (ML Layer, metric learning Layer) and the occlusion detection mechanism (CMG, confidence map generator) construct a dataset training depth plane object tracking model.
For training the model, using the MS-COCO dataset as the material, two labeled datasets GEN-DATA and OCC-DATA were constructed to train a deep planar object tracking model. The GEN-DATA mainly comprises factors such as illumination, deformation, noise and the like; OCC-DATA increases the situations that the target portion is blocked and the target portion is out of view based on GEN-DATA. Each sample in the two data sets is a quadruple (T, Q, p 0 ,p gt ) Respectively a template image, a current input image, initial motion parameters and real motion parameters of a target. The template image is from a template pool constructed from MS-COCO, i.e. the picture in MS-COCO is scaled to a picture with a length-width of 80-160 pixels.
The above-mentioned DATA set GEN-DATA construction process mainly includes the method of geometric transformation and optical disturbance, as shown in fig. 2.
The method for the geometric transformation comprises the following steps:
1) Given the target template T and the real motion parameters p of the target gt The pixel points in the target template are mapped into the input image Q through a perspective transformation formula. The perspective transformation formula is as follows:
wherein,the transformation matrix, (u, v) is the coordinates of the pixel, and (x, y) is the coordinates of the pixel after perspective transformation.
2) The corner points of the object in the input image Q are respectively shifted by d pixels in any direction, d being an integer from 0 to 20. Calculating corresponding transformation matrix, namely initial motion parameter p, according to the moved angular point coordinates 0 。
The implementation of the optical disturbance is specifically as follows:
1) Adding motion blur or gaussian blur to the input image;
2) Adding gaussian noise to the input image;
3) Varying degrees of brightness are applied to all pixels on an input image in a certain direction (e.g., top-to-bottom, or left-to-right).
The generation method of the DATA set OCC-DATA specifically comprises the following steps:
for each sample in the GEN-DATA, a point is selected on each side of the object in the input image, forming a size N P Is a set of points. Then randomly selecting N (N is more than or equal to 0 and less than or equal to N) P ) The points are connected in sequence. Thus, the target area in the video frame is divided into several parts. Then randomly selecting a pattern of which a part is filled with another picture to simulate the occlusion situation, as shown in fig. 3.
Both DATA sets GEN-DATA and OCC-DATA are set at 8: the scale of 2 is divided into training and validation sets for training the model and validating the performance of the model.
The characteristics learned by a large amount of data can better reflect the appearance characteristics of the target. During training, the feature detection module is not added first, and the GEN-DATA is used for training the feature extraction network and the similarity measurement module. After the training is completed, we fix the parameters of the two modules, train the occlusion detection module with OCC-DATA, and fine tune the parameters of the feature extraction network and the similarity metric module.
The loss function formula adopted in the training process is as follows:
wherein,target motion parameters predicted for model, p gt Is the real motion parameter of the target. N is the number of target corner points, r q Is the coordinates of the corner points. />A formula representing the coordinate transformation.
The distance sum of the corner points is embedded into a logarithmic function so as to avoid that samples with larger loss dominate the whole training process.
After the training of the depth plane object tracking model is completed, the target tracking process is as follows:
the tracking of each frame is divided into two stages, specifically:
the motion parameter of the tracking result of the first stage is used as the initial motion parameter of the second stage, and in the next iteration process, the motion parameter of the tracking result of the last second stage is used as the initial motion parameter of the first stage in the current iteration.
Taking the first stage as an example:
s1, firstly, acquiring a target template T, an input image of a T frame and initial motion parameters in the T frame, and determining a target area I of the input image according to the initial motion parameters t For target template T and target region I t Preprocessing, including operations such as zooming and normalizing of pictures; use specialTarget region I of input image of feature extraction network to target template T and T frame t Extracting features to obtain feature map F T And F t I The dimensions of the template and the target area after pretreatment are h multiplied by l multiplied by 3, and h, l and 3 are the width, length and channel number of the image respectively.
The feature extraction network for each stage consists of 7 convolutional layers, each followed by a batch norm layer and an activation function (ReLU) layer. The convolution kernels of the first 6 convolution layers are all 64, and the convolution kernel of the last convolution layer is 8. In the kth stage, the step size of the first 4-k convolution layers of the 7 convolution layers is 2, the step size of the remaining convolution layers is 1, and k is 1 or 2. Taking the first phase as an example, k=1.
S2, calculating two feature mappings F by using a similarity measurement module T And F t I Differences between them. Wherein the similarity measurement module is an encoder-decoder network based on a u-net framework, and the input of the similarity measurement module is a feature mapping F of a target template T T And a target area I of a t-th frame input image t Feature mapping F of (2) t I Outputs as a concatenation of the two feature maps F T And F t I As shown in fig. 4.
S3, determining and removing the blocked part of the target in the current frame by using a blocking detection mechanism, and solving the motion parameters of the target by minimizing the difference of the non-blocked part in the current frame.
The detection process of the occlusion detection mechanism is specifically as follows:
to describe this process more clearly, the target area I of the input image of the t-th frame will be t Simplified representation as I, its feature map F t I Is simply denoted as F I . L2 normalized feature map F for target region of input image given template and t-th frame T And F I (the characteristic dimensions thereof are h '. Times.l'. Times.d, h '. Times.l' correspond to the width and length of the extracted characteristic image respectively,k is 1 or 2D represents the dimension of the feature).
First, in units of each feature, F T And F I Expanded in the h ' direction into an m×d matrix (where m=h ' ×l '), denoted asAnd->Feature map representing the template T being expanded, +.>Representing the feature map of the expanded target region, then computing a correlation map R (dimension m×m) to record the similarity of each pair of features, with the formula:
wherein i, j respectively represent indexes of features in feature mapping of the target template T and the target region, R i,j The similarity between the ith feature in the template feature map and the jth feature in the target region feature map is shown, and Z (dimension d x d) is a trainable parameter matrix. Confidence vector is then constructed by selecting the maximum value of each row in RThe formula is as follows:
then, willElement of (C) is normalized to [0,1 ]]Within the interval as final confidence vector +.>
Finally, confidence vectors are usedH ' is taken as a row, and is arranged in the size of h ' ×l ', and is marked as C. The motion parameters of the object are solved by minimizing the differences in the non-occluded parts. See the following formula:
wherein p represents the current predicted target motion parameter; x represents the two-dimensional index of the feature in the feature map; c (-) is the contribution degree of the position feature to the optimization after detection by an occlusion detection mechanism, theoretically, the contribution degree of the occluded part feature to the optimization is 0, and the contribution degree of the non-occluded feature is 1; m (&. Cndot.) measures the variability of each pair of features in the template and target region;a formula representing the coordinate transformation.
The following formula is solved by adopting an ESM method, and the method is specifically as follows:
order the
The increment of the motion parameter can be obtained by:
wherein the method comprises the steps ofRepresenting pseudo-inverse of matrix, J T Calculated at U (unit transform)Jacobian matrix, J E (p) represents the jacobian matrix of E (x; p) at p:
the motion parameters are updated in combination with the delta deltap of the motion parameters:
wherein, the degree represents a binary operation.
The second stage is similar to the first stage, and will not be described again.
The planar target tracking process based on the parameterized ESM network provided by the embodiment of the invention comprises the following steps:
(1) In the first frame, the tracked target area is determined by calibrating the corner points of the target. As shown in fig. 5, the inside of the rectangular frame is the target template.
Taking fig. 5 as an example, when the target is calibrated, the real motion parameters of the first frame of the corresponding target can be calculated through the coordinates of four corner points of the target
The acquisition process of (a) is as follows:
let the width and height of the template be l, h. In fig. 5, the coordinates of point 1 to point 4 are (0, 0), (0,l), (h, l), (h, 0), respectively, by setting up the coordinate system of the template with point 1 as the origin. In the coordinate system of the image with the upper left corner of the frame image as the origin, the coordinates of the points 1 to 4 in the image are (x 1, y 1), (x 2, y 2), (x 3, y 3), (x 4, y 4), a33 is set as 1, and the inverse operation of the following formula is solved
(2) Starting from the second frame, the real motion parameters of the first frameAs an initial motion parameter p of the second frame, the input image is Qpass +.>And obtaining an image block (patch) with the same size as the template, namely the target area. And preprocessing the template and the target area, extracting features, measuring similarity, detecting shielding, and updating p through the solving process of the ESM for a plurality of times. Finally updating the obtained p 1 As a result of the tracking of this frame.
(3) In the subsequent frames, the process is similar to (2).
In summary, the embodiment of the invention provides a planar target tracking method based on a parameterized ESM network, which calculates the difference between depth features by using a trainable metric module and assists the optimization process by using a trainable occlusion detection mechanism. In addition, the invention generates a large number of labeled samples to simulate a real tracking scene, and monitors the training process of the model through a designed loss function, so that the generated target tracking samples are used for training a feature extraction network, a similarity measurement module and an occlusion detection mechanism in an end-to-end mode. Compared with a feature extractor trained on an image classification task and a traditional sliding window convolution method, the training method of the invention enables the model to be more suitable for a target tracking task, and the learned measurement method is more compatible with the deep convolution feature than the traditional sliding window convolution method, so that the tracking accuracy is greatly improved.
Those of ordinary skill in the art will appreciate that: the drawing is a schematic diagram of one embodiment and the modules or flows in the drawing are not necessarily required to practice the invention.
From the above description of embodiments, it will be apparent to those skilled in the art that the present invention may be implemented in software plus a necessary general hardware platform. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments or some parts of the embodiments of the present invention.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for apparatus or system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, with reference to the description of method embodiments in part. The apparatus and system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
The present invention is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present invention are intended to be included in the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.
Claims (5)
1. A planar target tracking method based on a parameterized ESM network, characterized in that a depth planar object tracking model is constructed, the depth planar object tracking model comprising: the method comprises the steps of feature extraction network, similarity measurement module and occlusion detection mechanism, constructing a data set to train the depth plane object tracking model, wherein the plane object tracking method comprises the following steps:
s1, acquiring a target template T, an input image of a T frame and initial motion parameters in the T frame, and determining a target area I of the input image by the initial motion parameters t For target template T and target region I t Preprocessing, including image scaling and normalization, the target template T after preprocessing and the target area I of the input image of the T frame are processed by using the feature extraction network t Extracting features to obtain feature map F T And F t I The dimensions of the template and the target area after pretreatment are h multiplied by l multiplied by 3, and h, l and 3 are the width, length and channel number of the image respectively;
s2, calculating two feature mappings F by using a similarity measurement module T And F t I Differences between;
s3, determining and removing the blocked part of the target in the current frame through a blocking detection mechanism, and solving the motion parameters of the target by minimizing the difference of the non-blocked part in the current frame; the method specifically comprises the following steps:
target area I of input image of t-th frame t Simplified representation as I, its feature map F t I Is simply denoted as F I L2 normalized feature map F for target region of given template and t-th frame input image T And F I The feature dimension is h 'x l' x d, where h 'and l' correspond to the width and length of the extracted feature image, respectively,k is1 or 2, d represents the dimension of the feature;
first, F is respectively taken as a unit of each feature T And F I And (c) developing into an m x d matrix along the h ' direction, wherein m=h ' ×l ', denoted asAnd-> Feature map representing the template T being expanded, +.>The feature map of the expanded target region is represented, and then a correlation map R is calculated to record the similarity of each pair of features, and the dimension of the correlation map R is m×m, and the formula is as follows:
wherein i, j respectively represent indexes of features in feature mapping of the target template T and the target region, R i,j Representing the similarity between the ith feature in the template feature map and the jth feature in the feature map of the target region, Z is a trainable parameter matrix with Z dimension of d x d, and the confidence vector is formed by selecting the maximum value of each row in RThe formula is as follows:
then, willElement of (C) is normalized to [0,1 ]]Within the interval as final confidence vector +.>
Finally, confidence vectors are usedTaking h ' as a row, arranging the motion parameters into the size of h ' x l ', and marking the motion parameters as C, and solving the motion parameters of the target by minimizing the difference of the non-shielded parts, wherein the motion parameters are shown in the following formula:
wherein p represents the current predicted target motion parameter; x represents the two-dimensional index of the feature in the feature map; c (-) is the contribution degree of the position feature to the optimization after detection by an occlusion detection mechanism, the contribution degree of the occluded part feature to the optimization is 0, and the contribution degree of the non-occluded feature is 1; m (&. Cndot.) measures the variability of each pair of features in the template and target region;a formula representing the coordinate transformation;
solving the formula (3) by adopting an ESM method, wherein the method is as follows:
order the
The increment of the motion parameter is obtained by:
wherein the method comprises the steps ofRepresenting pseudo-inverse of matrix, J T Calculated at U unit transform +.>Jacobian matrix, J E (p) represents the jacobian matrix of E (x; p) at p:
the motion parameters are updated in combination with the delta deltap of the motion parameters:
p≡p° Δp … … (7) where ° represents a binary operation.
2. The method according to claim 1, characterized in that the tracking of each frame in the video is divided into two phases, in particular:
the motion parameter of the tracking result of the first stage is used as the initial motion parameter of the second stage, and in the next iteration process, the motion parameter of the tracking result of the last second stage is used as the initial motion parameter of the first stage in the current iteration.
3. The method of claim 2, wherein the feature extraction network of each stage is composed of 7 convolution layers, each layer is followed by a batch norm layer and an activation function ReLU layer, the convolution kernels of the first 6 convolution layers are all 64, the convolution kernel of the last convolution layer is 8, the step size of the first 4-k convolution layers of the 7 convolution layers is 2, and the step size of the remaining convolution layers is 1, k is 1 or 2 in the kth stage.
4. The method according to claim 1, wherein the similarity measure module is an u-net frame based encoder-decoder network whose input is a feature map F of a target template T T And a target area I of a t-th frame input image t Feature mapping F of (2) t I Outputs as a concatenation of the two feature maps F T And F t I Is a difference tensor of (c).
5. The method of claim 1, wherein the constructing a dataset trains the depth plane object tracking model, comprising:
constructing two labeled DATA sets GEN-DATA and OCC-DATA, wherein GEN-DATA comprises illumination, deformation and noise factors, OCC-DATA is based on GEN-DATA, and the situation that the target part is blocked and the target part is out of the visual field range is increased, and each sample in the DATA sets GEN-DATA and OCC-DATA is a quadruple (T, Q, p) 0 ,p gt ) Wherein T is a template image, Q is a current input image, and p 0 For initial motion parameters, p gt Is the real motion parameter of the target;
the DATA set GEN-DATA construction process includes: geometric transformations and optical disturbances;
the geometric transformation includes:
given the target template T and the real motion parameters p of the target gt Mapping the pixel points in the target template into the input image Q through a perspective transformation formula, wherein the perspective transformation formula is as follows:
wherein,the pixel coordinate is (u, v) the coordinate of the pixel, (x, y) the coordinate of the pixel after perspective transformation;
the corner points of the object in the input image Q are respectively shifted by d pixels in any direction,d, taking an integer of 0 to 20, and calculating a corresponding transformation matrix according to the moved corner coordinates, namely an initial motion parameter p 0 ；
The optical perturbation includes:
1) Adding motion blur or gaussian blur to the input image;
2) Adding gaussian noise to the input image;
3) The brightness variation of different degrees is implemented on all pixels on the input image along a certain direction;
the DATA set OCC-DATA construction process includes:
for each sample in the GEN-DATA, a point is selected on each side of the object in the input image, forming a size N P Randomly selecting N (N is more than or equal to 0 and less than or equal to N) P ) The points are sequentially connected to divide a target area in a video frame into a plurality of parts, and a part of the target area is randomly selected to be filled with a pattern of another picture so as to simulate the shielding condition;
the DATA sets GEN-DATA and OCC-DATA are each at 8:2 is divided into a training set and a verification set for training the performance of the model and the verification model;
during training, firstly, an occlusion detection mechanism is not added, a GEN-DATA is used for training a feature extraction network and a similarity measurement module, parameters of the feature extraction network and the similarity measurement module are fixed after training is completed, an OCC-DATA is used for training the occlusion detection mechanism, and parameters of the feature extraction network and the similarity measurement module are finely adjusted at the same time;
the loss function formula adopted in the training process is as follows:
wherein,target motion parameters predicted for model, p gt Is the real motion parameter of the target; n is the number of target corner points, r q Coordinates of corner points; />A formula representing the coordinate transformation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010816457.5A CN111899284B (en) | 2020-08-14 | 2020-08-14 | Planar target tracking method based on parameterized ESM network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010816457.5A CN111899284B (en) | 2020-08-14 | 2020-08-14 | Planar target tracking method based on parameterized ESM network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111899284A CN111899284A (en) | 2020-11-06 |
| CN111899284B true CN111899284B (en) | 2024-04-09 |
Family
ID=73229031
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010816457.5A Active CN111899284B (en) | 2020-08-14 | 2020-08-14 | Planar target tracking method based on parameterized ESM network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111899284B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609316A (en) * | 2021-07-27 | 2021-11-05 | 支付宝(杭州)信息技术有限公司 | Method and device for detecting similarity of media contents |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101324956A (en) * | 2008-07-10 | 2008-12-17 | 上海交通大学 | Method for tracking anti-shield movement object based on average value wander |
| CN103729861A (en) * | 2014-01-03 | 2014-04-16 | 天津大学 | Multiple object tracking method |
| CN106920248A (en) * | 2017-01-19 | 2017-07-04 | 博康智能信息技术有限公司上海分公司 | A kind of method for tracking target and device |
| CN110796680A (en) * | 2019-08-09 | 2020-02-14 | 北京邮电大学 | Target tracking method and device based on similar template updating |
| WO2020155873A1 (en) * | 2019-02-02 | 2020-08-06 | 福州大学 | Deep apparent features and adaptive aggregation network-based multi-face tracking method |
-
2020
- 2020-08-14 CN CN202010816457.5A patent/CN111899284B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101324956A (en) * | 2008-07-10 | 2008-12-17 | 上海交通大学 | Method for tracking anti-shield movement object based on average value wander |
| CN103729861A (en) * | 2014-01-03 | 2014-04-16 | 天津大学 | Multiple object tracking method |
| CN106920248A (en) * | 2017-01-19 | 2017-07-04 | 博康智能信息技术有限公司上海分公司 | A kind of method for tracking target and device |
| WO2020155873A1 (en) * | 2019-02-02 | 2020-08-06 | 福州大学 | Deep apparent features and adaptive aggregation network-based multi-face tracking method |
| CN110796680A (en) * | 2019-08-09 | 2020-02-14 | 北京邮电大学 | Target tracking method and device based on similar template updating |
Non-Patent Citations (3)
| Title |
|---|
| 《Netted Radar Tracking with Multiple Simultaneous Transmissions against Combined PDS Interception》;Wu Zijian;《 Journal of Sensors》;全文 * |
| 《基于时空背景差的带跟踪补偿目标检测方法》;王涛等;《计算机应用》;全文 * |
| 《基于核相关滤波的长时间目标跟踪算法研究》;王涛;《中国优秀硕士学位论文全文数据库（信息科技辑）》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111899284A (en) | 2020-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110533722B (en) | Robot rapid repositioning method and system based on visual dictionary | |
| CN108734723B (en) | Relevant filtering target tracking method based on adaptive weight joint learning | |
| CN107369166B (en) | Target tracking method and system based on multi-resolution neural network | |
| US20160379375A1 (en) | Camera Tracking Method and Apparatus | |
| Zhao et al. | Deep lucas-kanade homography for multimodal image alignment | |
| CN114565655B (en) | Depth estimation method and device based on pyramid segmentation attention | |
| CN109544603B (en) | Target tracking method based on deep migration learning | |
| CN111767960A (en) | Image matching method and system applied to image three-dimensional reconstruction | |
| CN113724379B (en) | Three-dimensional reconstruction method and device for fusing image and laser point cloud | |
| CN104751493A (en) | Sparse tracking method on basis of gradient texture features | |
| Sui et al. | Exploiting the anisotropy of correlation filter learning for visual tracking | |
| CN108335328B (en) | Camera attitude estimation method and camera attitude estimation device | |
| CN111899284B (en) | Planar target tracking method based on parameterized ESM network | |
| CN114463397A (en) | Multi-modal image registration method based on progressive filtering | |
| CN113763274A (en) | Multi-source image matching method combining local phase sharpness orientation description | |
| CN108009272B (en) | Low-altitude moving target searching method based on directed weighted graph | |
| CN116363205A (en) | Space target pose resolving method based on deep learning and computer program product | |
| Yao et al. | Matching wide-baseline stereo images with weak texture using the perspective invariant local feature transformer | |
| CN113160271B (en) | High-precision infrared target tracking method integrating correlation filtering and particle filtering | |
| CN112348847B (en) | Target scale self-adaptive tracking method | |
| CN115345902A (en) | Infrared image dim target detection tracking method and system based on machine learning | |
| CN111160300B (en) | Deep learning hyperspectral image saliency detection algorithm combined with global prior | |
| CN111126198B (en) | Pedestrian re-identification method based on deep representation learning and dynamic matching | |
| CN113705731A (en) | End-to-end image template matching method based on twin network | |
| CN110660079A (en) | Single target tracking method based on space-time context |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |