CN114842045B - Target tracking method and device - Google Patents
Target tracking method and device Download PDFInfo
- Publication number
- CN114842045B CN114842045B CN202210349261.9A CN202210349261A CN114842045B CN 114842045 B CN114842045 B CN 114842045B CN 202210349261 A CN202210349261 A CN 202210349261A CN 114842045 B CN114842045 B CN 114842045B
- Authority
- CN
- China
- Prior art keywords
- cluster
- event
- target
- event stream
- events
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000010586 diagram Methods 0.000 claims abstract description 35
- 238000013528 artificial neural network Methods 0.000 claims abstract description 29
- 239000002245 particle Substances 0.000 claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 20
- 238000001914 filtration Methods 0.000 claims abstract description 18
- 230000008569 process Effects 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 8
- 238000013527 convolutional neural network Methods 0.000 claims description 2
- 238000004422 calculation algorithm Methods 0.000 abstract description 16
- 238000004364 calculation method Methods 0.000 abstract description 16
- 230000000694 effects Effects 0.000 abstract description 10
- 230000006870 function Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 230000006872 improvement Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007621 cluster analysis Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The present invention relates to the field of image recognition, and in particular, to a target tracking method and apparatus. The method comprises the following steps: acquiring an event stream acquired by a sensor, carrying out cluster generation on the event stream to obtain a first time sequence of a target cluster, acquiring an image frame acquired by the sensor, sending the image frame into a neural network, calculating a confidence coefficient thermodynamic diagram reflecting the occurrence probability of the target in an image area through the neural network, and carrying out particle filtering processing on the first time sequence and the confidence coefficient thermodynamic diagram simultaneously to obtain a second time sequence, wherein the second time sequence comprises coordinate data of a single or a plurality of targets. The invention can reduce the hardware calculation force required by the target tracking algorithm under the same condition, or improve the tracking effect under the condition of the same hardware calculation force. And the DVS and the traditional APS mode are combined, so that the tracking precision is improved while the high processing speed is obtained.
Description
Technical Field
The invention relates to the field of image recognition, in particular to a target tracking method.
Background
The problem of slow target tracking speed in the field of image recognition. The slow target tracking speed in the field of image recognition is an important disadvantage in target tracking, which directly causes the reduction of the target tracking accuracy and even causes the failure of target tracking, namely the loss of the target or the confusion of the target. This result is in most cases not recoverable. The event stream acquired by the DVS itself has the characteristic of unequal time stamp intervals, which is difficult to process for the conventional image tracking algorithm. DVS is a new type of image sensor, and is characterized by collecting only the signals of pixels whose values change, and ignoring the signals of pixels whose values are unchanged. In contrast to DVS, APS is known as an advanced photography system, which is a conventional image acquisition system based on image frames.
Disclosure of Invention
The embodiment of the invention provides a target tracking method and a target tracking device, which can reduce the hardware computing force required by a target tracking algorithm under the same condition or improve the tracking effect under the condition of the same hardware computing force.
The embodiment of the invention provides a target tracking method, which comprises the following steps:
acquiring an event stream acquired by a sensor, wherein the event stream comprises a plurality of events which are arranged according to time domain increasing sequence and are based on the motion state change of a target object, and the events are that the pixel values of part or all of the target object are changed;
clustering the event stream to obtain a first time sequence of a target cluster, wherein the first time sequence comprises event center coordinate data of at least one target object;
Acquiring an image frame acquired by a sensor, sending the image frame into a preset neural network, and calculating a confidence coefficient thermodynamic diagram reflecting the occurrence probability of a target in an image area through the preset neural network;
And carrying out particle filtering processing on the first time sequence and the confidence thermodynamic diagram simultaneously to obtain a second time sequence, wherein the second time sequence comprises coordinate data of at least one target object.
In one embodiment, the step of clustering the event stream includes:
If any part of the current cluster exists in the time-space adjacent range of the candidate event, the candidate event is attributed to the current cluster, wherein the candidate event is an event which lags the current moment in the time domain in the event stream, and the current cluster is the cluster existing at the current moment; and/or the number of the groups of groups,
If no portion of the current cluster exists within the contiguous range of the candidate event or no current cluster exists in the system, a new cluster is created that includes the candidate event.
In an embodiment, after the step of clustering the event stream comprises:
if the total number of events in the new cluster is lower than a second value, setting the new cluster to be invisible;
And if the total number of the events in the new cluster is higher than a second value, setting the new cluster to be visible.
In an embodiment, after said creating a new cluster comprising said candidate event, said method further comprises:
Calculating the total number of all clusters in the system;
If the total number is greater than the number N, no new cluster is added, the number N is an integer greater than or equal to 1, and the number N is dynamically adjusted according to the processing capacity of the system.
In one embodiment, the number N is 3.
In an embodiment, the step of clustering the event stream includes:
determining candidate event centers to be tracked from the event stream;
When the motion state of the candidate event center is a preset state within at least one preset time interval threshold, determining that a new event does not exist in the candidate event center;
Discarding the candidate event centers.
In an embodiment, the step of clustering the event stream includes:
Determining a number of events belonging to a first cluster from the event stream;
the first cluster is not visible when the number of events belonging to the first cluster is less than and/or equal to a specified number, otherwise the first cluster is visible.
In one embodiment, the specified number Q is a function of the inter-event space in the event stream, namely Q (Xn-Xm, yn-Ym, tn-tm), where Xn, xm, yn, ym is the X, Y spatial coordinates of the nth and mth events, respectively, and tn, tm is the timestamp of the nth and mth events, respectively.
In one embodiment, the specified number Q is a function of the inter-event space in the event stream, i.e., q=a (Xn-Xm) +b (Yn-Ym) +c (tn-tm), where Xn, xm, yn, ym is the X, Y spatial coordinates of the nth and mth events, tn, tm is the timestamp of the nth and mth events, respectively, and a, b, c are constant coefficients, respectively.
In one embodiment, the step of performing the particle filtering process on the first time series comprises:
Calculating the weight coefficient of each particle in the new cluster:
wherein d (i, j) is the spatial distance between the center of the i-th cluster and the j particles;
And calculating coordinate data of the single or multiple targets according to the weight coefficient.
In an embodiment, the neural network is one of a fully connected neural network or a convolutional neural network.
The embodiment of the invention also provides a target tracking device which comprises an event stream acquisition module, a cluster generation module, a thermodynamic diagram generation module and a filtering module;
The event stream acquisition module is used for acquiring an event stream acquired by a sensor, wherein the event stream comprises a plurality of events which are arranged according to time domain increasing sequence and are based on the motion state change of a target object, and the events are that the pixel values of part or all of the target object are changed;
The cluster generation module is used for generating clusters of the event stream to obtain a first time sequence of a target cluster, wherein the first time sequence comprises event center coordinate data of at least one target object;
The thermodynamic diagram generation module is used for acquiring an image frame acquired by a sensor, sending the image frame into a preset neural network and calculating a confidence thermodynamic diagram reflecting the occurrence probability of a target in an image area through the preset neural network;
The filtering module is configured to perform particle filtering processing on the first time sequence and the confidence thermodynamic diagram simultaneously, so as to obtain a second time sequence, where the second time sequence includes coordinate data of at least one target object.
As can be seen from the foregoing, in the target tracking method provided by the embodiment of the present application, by acquiring an event stream acquired by a sensor, clustering the event stream to obtain a first time sequence of a target cluster, acquiring an image frame acquired by the sensor, sending the image frame into a neural network, calculating a confidence coefficient thermodynamic diagram reflecting the occurrence probability of a target in an image area through the neural network, and performing particle filtering processing on the first time sequence and the confidence coefficient thermodynamic diagram simultaneously to obtain a second time sequence, where the second time sequence includes coordinate data of a single or multiple targets. The application can reduce the hardware calculation force required by the target tracking algorithm under the same condition, or improve the tracking effect under the condition of the same hardware calculation force. And the DVS and the traditional APS mode are combined, so that the tracking precision is improved while the high processing speed is obtained.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a target tracking method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a target tracking method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram showing a first effect of the object tracking method according to the embodiment of the present invention;
FIG. 4 is a schematic diagram showing a second effect of the object tracking method according to the embodiment of the present invention;
FIG. 5 is a schematic diagram of a third effect of the target tracking method according to the embodiment of the present invention;
fig. 6 is a schematic structural diagram of a target tracking apparatus according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the element defined by the phrase "comprising one … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element, and furthermore, elements having the same name in different embodiments of the application may have the same meaning or may have different meanings, the particular meaning of which is to be determined by its interpretation in this particular embodiment or by further combining the context of this particular embodiment.
It should be understood that, although the steps in the flowcharts in the embodiments of the present application are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the figures may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily occurring in sequence, but may be performed alternately or alternately with other steps or at least a portion of the other steps or stages.
It should be noted that, in this document, step numbers such as 101 and 102 are used for the purpose of more clearly and briefly describing the corresponding contents, and not to constitute a substantial limitation on the sequence, and those skilled in the art may execute 102 first and then execute 101 when they are implemented, which is within the scope of the present application.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
The embodiment of the invention provides a target tracking method, and an execution subject of the target tracking method can be the target tracking device provided by the embodiment of the invention or a server integrated with the target tracking device, wherein the target tracking device can be realized in a hardware or software mode.
As shown in fig. 1, fig. 1 is a schematic flow chart of a target tracking method according to an embodiment of the present invention, and a specific flow of the target tracking method may be as follows:
101. Acquiring an event stream acquired by a sensor, wherein the event stream comprises a plurality of events which are arranged according to a time domain increasing sequence and are based on the motion state change of a target object, and the events are the partial or all pixel values of the target object;
102. Cluster generation is carried out on the event stream to obtain a first time sequence of the target cluster, wherein the first time sequence comprises event center coordinate data of at least one target object;
103. acquiring an image frame acquired by a sensor, sending the image frame into a preset neural network, and calculating a confidence thermodynamic diagram reflecting the occurrence probability of a target in an image area through the preset neural network;
104. And carrying out particle filtering processing on the first time sequence and the confidence thermodynamic diagram simultaneously to obtain a second time sequence, wherein the second time sequence comprises coordinate data of at least one target object.
In the embodiment of the present application, referring to fig. 2, the principle of the target tracking method provided by the embodiment of the present application may generate a cluster, so that the amount of data processed in the subsequent step may be greatly reduced, thereby improving the processing efficiency and the processing speed. The improvement can reduce the hardware calculation force required by the target tracking algorithm under the same condition; or to increase the tracking effect with the same hardware computational effort.
The tracking effect generated by the method provided by the application and the improvement of the traditional algorithm can be referred to in table 1:
TABLE 1
Correspondingly, the comparison of the consumption condition of the calculation force compared with the traditional method can be referred to as table 2:
TABLE 2
For example, in robotic navigation, it is desirable to track surrounding obstacle targets very quickly and accurately in real time and continuously. Therefore, high requirements on the accuracy and the speed of the tracking algorithm are required.
The robot encounters an obstacle during the movement, so that a DVS camera that can be mounted on the robot can rapidly acquire a target and generate an event stream. The event stream is sent to a cluster generation module, which processes it as follows:
sequentially processing events received in time sequence, if any part of a current cluster exists in a time-space adjacent range of a candidate event, attributing the candidate event to the cluster, wherein the candidate event is an event which lags the current moment in the time domain in the event stream, and the current cluster is the cluster existing at the current moment;
Or if there is no portion of the current cluster within the contiguous range of the candidate event or there is no current cluster in the system, creating a new cluster comprising the candidate event.
Acquiring an image frame acquired by a sensor, sending the image frame into a neural network, and calculating a confidence thermodynamic diagram reflecting the occurrence probability of a target in an image area through the neural network; specifically, the neural network is trained to output a high confidence value in a region with high target possibility and to output a low confidence value in a region with high target possibility.
Calculating the total number of all clusters in the system;
if the total number of clusters is greater than N, no new clusters are added. In one class of embodiments, this number is 3.
In an embodiment, for the number of clusters, clustering calculation is performed on all the events collected in a given time interval t1 to t2 in a time and space coordinate system, so as to generate m classes, namely m clusters.
Then determining candidate event centers to be tracked from the event stream;
When the motion state of the candidate event center is a preset state within at least one preset time interval threshold, determining that a new event does not exist in the candidate event center;
Discarding the candidate event centers.
Determining the number of events belonging to the cluster from the event stream;
the cluster is not visible when the number of events attributed to the cluster is less than and/or equal to the specified number, otherwise the cluster is visible.
The central coordinates and thermodynamic diagrams of the N classes are input into a module of a particle algorithm. The module outputs the exact position coordinates of the target. I.e. the object of the obstacle. The specific mode is as follows:
The cluster center coordinates are X (k) = [ xp (k) xv (k) yp (k) yv (k) ] T, the position of the target at time k is (xp (k) yp (k)), and the velocity of the target (xv (k) yv (k)) is composed of the component velocities in the horizontal direction and the vertical direction, expressed as vectors: v (k) =xv (k) +yv (k)
The horizontal and vertical directions of the object are decomposed into:
Horizontal position: xp (k+1) =xp (k) +xv (k) ×1+0.5wxp (k) × 1^2
Horizontal speed: xv (k+1) =xv (k) +wxv (k) ×1
Vertical position: yp (k+1) =yp (k) +yv (k) ×1+0.5wyp (k) × 1^2
Vertical speed: yv (k+1) =yv (k) +wyv (k) ×1
The equation of state: x (k+1) =ax (k) +tw (k)
There is a relationship between the location of dvs and the target:
d(k)＝((xp(k)-xs)2+(yp(k)-ys)2)^0.5+v(k)
where d is the distance between dvs and the target, which is affected by the measurement noise v (k). The above equation is expressed as:
Z(k)＝h(X(k))+v(k)
the function h represents a functional relationship between dvs and the target state, which in one class of embodiments may be represented as:
h(X(k))＝((xp(k)-xs)2+(yp(k)-ys)2)^0.5；
The position of the target can be obtained, i.e. the obstacle is tracked. By giving an initial target box, the final target box of the target can be as shown in FIG. 3. Fig. 3, wherein the left graph is a DVS generated event stream and the right graph is a DVS generated event stream superimposed with an APS acquired data frame.
In one class of embodiments, the center of each cluster is obtained by calculation of the position weights of the particles that make up the cluster, i.e., the weight coefficients of the individual particles in the new cluster
Wherein d (i, j) is the spatial distance between the center of the i-th cluster and the j particles;
and calculating coordinate data of the single or multiple targets according to the weight coefficients.
In one embodiment, ball games are illustrated. In ball games, such as high-speed ball games like soccer, it is necessary to judge the speed, path, etc. of the ball, thereby providing effective information to spectators or athletes in the game, training. It is therefore necessary to track the football ball very quickly and accurately. Therefore, high requirements on the accuracy and the speed of the tracking algorithm are required.
As shown in fig. 4, a DVS device is installed at a stationary location, and a target ball can be quickly collected and an event stream can be generated.
The event stream is sent to the cluster generation module. The cluster generation module processes it as follows:
clustering calculation is carried out on all the events acquired in a given time interval t1 to t2 in a time and space coordinate system, and m clusters are generated;
In one class of embodiments, the cluster analysis is performed using one of the following algorithms: k-means, mean shift clustering, density-based clustering methods, aggregation hierarchical clustering, graph group detection.
The method comprises the following specific steps: unlike the conventional method, in the method, the original event stream and time are generated into a series of points in 3-dimensional space (state space), then kk points are randomly selected from the above-mentioned point set as cluster centers according to the following manner, distances between all points and the kk "cluster centers" are calculated, each point is divided into clusters where the "cluster center" closest to it is located, and a new "cluster center" of each cluster is calculated for a new cluster.
Calculating the total number of all clusters in the system;
if the total number of clusters is greater than N, no new clusters are added. In one class of embodiments, this number is 3.
Then determining candidate event centers to be tracked from the event stream;
When the motion state of the candidate event center is a preset state within at least one preset time interval threshold, determining that a new event does not exist in the candidate event center;
Discarding the candidate event centers.
Determining the number of events belonging to the cluster from the event stream;
the cluster is not visible when the number of events attributed to the cluster is less than and/or equal to the specified number, otherwise the cluster is visible.
The central coordinates and confidence maps of the N classes are input into a module of a particle algorithm. The module outputs the exact position coordinates of the target. I.e. the object of the obstacle. The specific mode is as follows:
The cluster center coordinates are X (k) = [ xp (k) xv (k) yp (k) yv (k) ] T, the position of the target at time k is (xp (k) yp (k)), and the velocity of the target (xv (k) yv (k)) is composed of the component velocities in the horizontal direction and the vertical direction, expressed as vectors: v (k) =xv (k) +yv (k).
The horizontal and vertical directions of the object are decomposed into:
Horizontal position: xp (k+1) =xp (k) +xv (k) ×1+0.5wxp (k) × 1^2
Horizontal speed: xv (k+1) =xv (k) +wxv (k) ×1
Vertical position: yp (k+1) =yp (k) +yv (k) ×1+0.5wyp (k) × 1^2
Vertical speed: yv (k+1) =yv (k) +wyv (k) ×1
The equation of state: x (k+1) =ax (k) +tw (k)
There is a relationship between the location of dvs and the target:
d(k)＝((xp(k)-xs)2+(yp(k)-ys)2)^0.5+v(k)
where d is the distance between dvs and the target, which is affected by the measurement noise v (k). The above equation is expressed as:
Z(k)＝h(X(k))+v(k)
the function h represents a functional relationship between dvs and the target state, which in one class of embodiments may be represented as:
h(X(k))＝((xp(k)-xs)2+(yp(k)-ys)2)^0.5；
the position of the target can be obtained, i.e. the obstacle is tracked. By giving an initial target box, the final target box of the target can be as shown in FIG. 4.
In one class of embodiments, the center of each cluster is obtained by calculation of the position weights of the particles that make up the cluster, i.e., the weight coefficients of the individual particles in the new cluster
Wherein d (i, j) is the spatial distance between the center of the i-th cluster and the j particles;
and calculating coordinate data of the single or multiple targets according to the weight coefficients.
In one embodiment, security monitoring is described as an example. In the security monitoring field, for example, the tracking of suspicious personnel in public places, the face and the like of the suspicious personnel need to be tracked at high speed and accurately. Therefore, high requirements on the accuracy and the speed of the tracking algorithm are required.
The DVS device is installed in a fixed place, so that the face of a person can be quickly collected, and an event stream is generated.
The event stream is sent to the cluster generation module. The cluster generation module processes it as follows:
Performing region calculation on all the events acquired in a given time interval t1 to t2 in a time and space coordinate system, and growing m independent regions into m clusters;
the algorithm starts:
Initializing variable pixdist =0;
A seed point is interactively selected and the gray-level average variable reg_mean of the initialization area is the gray-level value of the seed point.
while(pixdist<reg_maxdist)
Adding nine neighborhood pixel points of the current seed point into a linked list neg_list;
Respectively calculating the gray values of all elements in neg_list and the absolute value of reg_mean difference, and obtaining an element i (x, y) with the minimum value, wherein pixdist =abs (neg_list (i, 3) -reg_mean;
update reg_mean= (reg_mean+neg_list (i, 3))/(reg_size+1); (note: reg_size represents the number of pixels in the segmented region)
Marking old seed points as segmented regional pixel points;
Taking i (x, y) as a new seed point, and removing the new seed point i (x, y) from the linked list neg_list;
and (5) ending.
Calculating the total number of all clusters in the system;
if the total number of clusters is greater than N, no new clusters are added. In one class of embodiments, this number is 3.
Then determining candidate event centers to be tracked from the event stream;
When the motion state of the candidate event center is a preset state within at least one preset time interval threshold, determining that a new event does not exist in the candidate event center;
Discarding the candidate event centers.
Determining the number of events belonging to the cluster from the event stream;
the cluster is not visible when the number of events attributed to the cluster is less than and/or equal to the specified number, otherwise the cluster is visible.
The central coordinates of the N classes are input into a module of the particle algorithm. The module outputs the exact position coordinates of the target. I.e. the object of the obstacle. The specific mode is as follows:
The cluster center coordinates are X (k) = [ xp (k) xv (k) yp (k) yv (k) ] T, the position of the target at time k is (xp (k) yp (k)), and the velocity of the target (xv (k) yv (k)) is composed of the component velocities in the horizontal direction and the vertical direction, expressed as vectors: v (k) =xv (k) +yv (k).
The horizontal and vertical directions of the object are decomposed into:
Horizontal position: xp (k+1) =xp (k) +xv (k) ×1+0.5wxp (k) × 1^2
Horizontal speed: xv (k+1) =xv (k) +wxv (k) ×1
Vertical position: yp (k+1) =yp (k) +yv (k) ×1+0.5wyp (k) × 1^2
Vertical speed: yv (k+1) =yv (k) +wyv (k) ×1
The equation of state: x (k+1) =ax (k) +tw (k)
There is a relationship between the location of dvs and the target:
d(k)＝((xp(k)-xs)2+(yp(k)-ys)2)^0.5+v(k)
where d is the distance between dvs and the target, which is affected by the measurement noise v (k). The above equation is expressed as:
Z(k)＝h(X(k))+v(k)
the function h represents a functional relationship between dvs and the target state, which in one class of embodiments may be represented as:
h(X(k))＝((xp(k)-xs)2+(yp(k)-ys)2)^0.5；
The position of the target can be obtained, i.e. the obstacle is tracked. By giving an initial target box, the final target box of the target can be as shown in FIG. 5.
In one class of embodiments, the center of each cluster is obtained by calculation of the position weights of the particles that make up the cluster, i.e., the weight coefficients of the individual particles in the new cluster
Wherein d (i, j) is the spatial distance between the center of the i-th cluster and the j particles+confidence value;
and calculating coordinate data of the single or multiple targets according to the weight coefficients.
As can be seen from the foregoing, the object tracking method provided in this embodiment may obtain an event stream collected by a sensor, cluster the event stream to obtain a first time sequence of an object cluster, obtain an image frame collected by the sensor, send the image frame into a neural network, calculate a confidence coefficient thermodynamic diagram reflecting the occurrence probability of the object in an image area through the neural network, and perform particle filtering processing on the first time sequence and the confidence coefficient thermodynamic diagram at the same time to obtain a second time sequence, where the second time sequence includes coordinate data of a single or multiple objects. The invention can reduce the hardware calculation force required by the target tracking algorithm under the same condition, or improve the tracking effect under the condition of the same hardware calculation force. And the DVS and the traditional APS mode are combined, so that the tracking precision is improved while the high processing speed is obtained.
With continued reference to fig. 6, fig. 6 is a schematic structural diagram of an object tracking device according to an embodiment of the invention. The object tracking device may include:
The event stream obtaining module 301 is configured to obtain an event stream collected by a sensor, where the event stream includes a plurality of events based on a motion state change of a target object, where the events are changes in pixel values of part or all of the target object, and the events are arranged in a time domain increasing order;
the cluster generation module 302 is configured to perform cluster generation on the event stream to obtain a first time sequence of a target cluster, where the first time sequence includes event center coordinate data of at least one target object;
The thermodynamic diagram generating module 303 is configured to obtain an image frame acquired by a sensor, send the image frame into a preset neural network, and calculate a confidence thermodynamic diagram reflecting the probability of occurrence of the target in an image area through the preset neural network;
The filtering module 304 is configured to perform particle filtering processing on the first time sequence and the confidence thermodynamic diagram simultaneously, so as to obtain a second time sequence, where the second time sequence includes coordinate data of at least one target object.
All the above technical solutions may be combined to form an optional embodiment of the present application, and will not be described in detail herein.
As can be seen from the above, the object tracking device provided by the embodiment of the present invention may obtain an event stream collected by a sensor, cluster-generate the event stream to obtain a first time sequence of an object cluster, obtain an image frame collected by the sensor, send the image frame into a neural network, calculate a confidence coefficient thermodynamic diagram reflecting the occurrence probability of the object in an image area through the neural network, and perform particle filtering processing on the first time sequence and the confidence coefficient thermodynamic diagram simultaneously to obtain a second time sequence, where the second time sequence includes coordinate data of a single or multiple objects. The invention can reduce the hardware calculation force required by the target tracking algorithm under the same condition, or improve the tracking effect under the condition of the same hardware calculation force. And the DVS and the traditional APS mode are combined, so that the tracking precision is improved while the high processing speed is obtained.
Those of ordinary skill in the art will appreciate that all or a portion of the steps of the various methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.
To this end, an embodiment of the present application provides a computer readable storage medium having stored therein a plurality of computer programs that can be loaded by a processor to perform the steps of any of the object tracking methods provided by the embodiments of the present application. For example, the computer program may perform the steps of:
Acquiring an event stream acquired by a sensor, wherein the event stream comprises a plurality of events which are arranged according to a time domain increasing sequence and are based on the motion state change of a target object, and the events are that the pixel values of part or all of the target object are changed;
clustering the event stream to obtain a first time sequence of a target cluster, wherein the first time sequence comprises event center coordinate data of at least one target object;
Acquiring an image frame acquired by a sensor, sending the image frame into a preset neural network, and calculating a confidence coefficient thermodynamic diagram reflecting the occurrence probability of a target in an image area through the preset neural network;
And carrying out particle filtering processing on the first time sequence and the confidence thermodynamic diagram simultaneously to obtain a second time sequence, wherein the second time sequence comprises coordinate data of at least one target object.
The specific implementation of each operation above may be referred to the previous embodiments, and will not be described herein.
Wherein the storage medium may include: read Only Memory (ROM), random access memory (RAM, random Access Memory), magnetic or optical disk, and the like.
The steps in any one of the target tracking methods provided by the embodiments of the present application may be executed by the computer program stored in the storage medium, so that the beneficial effects that any one of the target tracking methods provided by the embodiments of the present application may be achieved, which are detailed in the previous embodiments and are not described herein.
The foregoing describes in detail a target tracking method and apparatus provided by embodiments of the present application, and specific examples are applied to illustrate principles and embodiments of the present application, where the foregoing examples are only for helping to understand the method and core idea of the present application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present application, the present description should not be construed as limiting the present application.
Claims (11)
1. A method of target tracking, the method comprising:
acquiring an event stream acquired by a sensor, wherein the event stream comprises a plurality of events which are arranged according to time domain increasing sequence and are based on the motion state change of a target object, and the events are that the pixel values of part or all of the target object are changed;
the method comprises the steps of generating a cluster of an event stream to obtain a first time sequence of a target cluster, wherein the first time sequence comprises event center coordinate data of at least one target object, and the step of generating the cluster of the event stream comprises the steps of attributing a candidate event to a current cluster if any part of the current cluster exists in a time-space adjacent range of the candidate event, wherein the candidate event is an event lagging behind the current moment in a time domain in the event stream, and the current cluster is an existing cluster at the current moment; and/or if there is no part of the current cluster in the adjacent range of the candidate event or there is no current cluster in the system, creating a new cluster comprising the candidate event;
Acquiring an image frame acquired by a sensor, sending the image frame into a preset neural network, and calculating a confidence coefficient thermodynamic diagram reflecting the occurrence probability of a target in an image area through the preset neural network;
And carrying out particle filtering processing on the first time sequence and the confidence thermodynamic diagram simultaneously to obtain a second time sequence, wherein the second time sequence comprises coordinate data of at least one target object.
2. The target tracking method of claim 1, wherein after the step of clustering the event stream comprises:
if the total number of events in the new cluster is lower than a second value, setting the new cluster to be invisible;
And if the total number of the events in the new cluster is higher than a second value, setting the new cluster to be visible.
3. The target tracking method of claim 1, wherein after the creating of the new cluster including the candidate event, the method further comprises:
Calculating the total number of all clusters in the system;
if the total number is greater than the number N, no new cluster is added, the number N is an integer greater than or equal to 1, and the number N is dynamically adjusted according to the processing capacity of the system.
4. A target tracking method as claimed in claim 3, characterized in that the number N is 3.
5. The target tracking method of claim 1, wherein the step of clustering the event stream comprises:
determining candidate event centers to be tracked from the event stream;
When the motion state of the candidate event center is a preset state within at least one preset time interval threshold, determining that a new event does not exist in the candidate event center;
Discarding the candidate event centers.
6. The target tracking method of claim 1, wherein the step of clustering the event stream comprises:
Determining a number of events belonging to a first cluster from the event stream;
And when the number of events belonging to the first cluster is less than or equal to the specified number, the first cluster is invisible, otherwise, the first cluster is visible.
7. The object tracking method of claim 6 wherein the specified number Q is a function of the inter-event space in the event stream, namely Q (Xn-Xm, yn-Ym, tn-tm), wherein Xn, xm, yn, ym is the X, Y spatial coordinates of the nth and mth events, respectively, and tn, tm is the time stamps of the nth and mth events, respectively.
8. The method of claim 6, wherein the specified number Q is a function of inter-event space-time intervals in the event stream, i.e., Q = a (Xn-Xm) +b (Yn-Ym) +c (tn-tm), wherein Xn, xm, yn, ym is X, Y spatial coordinates of the nth and mth events, respectively, tn, tm is time stamps of the nth and mth events, respectively, and a, b, c are constant coefficients, respectively.
9. The target tracking method of claim 1, wherein the step of performing a particle filtering process on the first time series comprises:
Calculating the weight coefficient of each particle in the new cluster:
wherein d (i, j) is the spatial distance between the center of the i-th cluster and the j particles;
and calculating coordinate data of the single or multiple targets according to the weight coefficients.
10. The target tracking method of claim 1, wherein the neural network is a fully connected neural network or a convolutional neural network.
11. The target tracking device is characterized by comprising an event stream acquisition module, a cluster generation module, a thermodynamic diagram generation module and a filtering module;
The event stream acquisition module is used for acquiring an event stream acquired by a sensor, wherein the event stream comprises a plurality of events which are arranged according to time domain increasing sequence and are based on the motion state change of a target object, and the events are that the pixel values of part or all of the target object are changed;
The cluster generation module is configured to perform cluster generation on the event stream to obtain a first time sequence of a target cluster, where the first time sequence includes event center coordinate data of at least one target object, and the step of performing cluster generation on the event stream includes, if any part of a current cluster already exists in a time-space adjacent range of a candidate event, attributing the candidate event to the current cluster, where the candidate event is an event that lags a current time in a time domain in the event stream, and the current cluster is an existing cluster at the current time; and/or if there is no part of the current cluster in the adjacent range of the candidate event or there is no current cluster in the system, creating a new cluster comprising the candidate event;
The thermodynamic diagram generation module is used for acquiring an image frame acquired by a sensor, sending the image frame into a preset neural network and calculating a confidence thermodynamic diagram reflecting the occurrence probability of a target in an image area through the preset neural network;
The filtering module is configured to perform particle filtering processing on the first time sequence and the confidence thermodynamic diagram simultaneously, so as to obtain a second time sequence, where the second time sequence includes coordinate data of at least one target object.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210349261.9A CN114842045B (en) | 2022-04-01 | 2022-04-01 | Target tracking method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210349261.9A CN114842045B (en) | 2022-04-01 | 2022-04-01 | Target tracking method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114842045A CN114842045A (en) | 2022-08-02 |
| CN114842045B true CN114842045B (en) | 2024-04-16 |
Family
ID=82564185
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210349261.9A Active CN114842045B (en) | 2022-04-01 | 2022-04-01 | Target tracking method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114842045B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019032305A2 (en) * | 2017-08-07 | 2019-02-14 | Standard Cognition, Corp. | Item put and take detection using image recognition |
| CN110569793A (en) * | 2019-09-09 | 2019-12-13 | 西南交通大学 | Target tracking method for unsupervised similarity discrimination learning |
| WO2020107524A1 (en) * | 2018-11-27 | 2020-06-04 | 上海芯仑光电科技有限公司 | Target tracking method and computing device |
| KR20210096473A (en) * | 2020-01-28 | 2021-08-05 | 인하대학교 산학협력단 | Robust visual object tracking based on global and local search with confidence estimation |
| CN113724297A (en) * | 2021-08-31 | 2021-11-30 | 中国科学院长春光学精密机械与物理研究所 | Event camera-based tracking method |
| CN113870322A (en) * | 2021-08-23 | 2021-12-31 | 首都师范大学 | Event camera-based multi-target tracking method and device and computer equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200005468A1 (en) * | 2019-09-09 | 2020-01-02 | Intel Corporation | Method and system of event-driven object segmentation for image processing |
-
2022
- 2022-04-01 CN CN202210349261.9A patent/CN114842045B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019032305A2 (en) * | 2017-08-07 | 2019-02-14 | Standard Cognition, Corp. | Item put and take detection using image recognition |
| WO2020107524A1 (en) * | 2018-11-27 | 2020-06-04 | 上海芯仑光电科技有限公司 | Target tracking method and computing device |
| CN110569793A (en) * | 2019-09-09 | 2019-12-13 | 西南交通大学 | Target tracking method for unsupervised similarity discrimination learning |
| KR20210096473A (en) * | 2020-01-28 | 2021-08-05 | 인하대학교 산학협력단 | Robust visual object tracking based on global and local search with confidence estimation |
| CN113870322A (en) * | 2021-08-23 | 2021-12-31 | 首都师范大学 | Event camera-based multi-target tracking method and device and computer equipment |
| CN113724297A (en) * | 2021-08-31 | 2021-11-30 | 中国科学院长春光学精密机械与物理研究所 | Event camera-based tracking method |
Non-Patent Citations (3)
| Title |
|---|
| 基于簇相似度的实时多尺度目标跟踪算法;李康;何发智;陈晓;潘一腾;于海平;;模式识别与人工智能(03);全文 * |
| 杨帅 ; 薛永江 ; 王瑞昆 ; .基于异步时域事件的高速运动目标的跟踪算法.计算机时代.2016,(06),全文. * |
| 网络模糊信息数据集合多目标优化跟踪仿真;陈强;张小勇;张锋;;计算机仿真(11);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114842045A (en) | 2022-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Senst et al. | Detecting people carrying objects based on an optical flow motion model | |
| US10636190B2 (en) | Methods and systems for exploiting per-pixel motion conflicts to extract primary and secondary motions in augmented reality systems | |
| Ren et al. | Multi-camera video surveillance for real-time analysis and reconstruction of soccer games | |
| de Almeida et al. | Change detection in human crowds | |
| CN112861808B (en) | Dynamic gesture recognition method, device, computer equipment and readable storage medium | |
| CN110827320B (en) | Target tracking method and device based on time sequence prediction | |
| Win et al. | Real-time human motion detection, tracking and activity recognition with skeletal model | |
| Barbu et al. | Simultaneous object detection, tracking, and event recognition | |
| CN116740539A (en) | Visual SLAM method and system based on lightweight target detection network | |
| CN114373194A (en) | Human behavior identification method based on key frame and attention mechanism | |
| Kim et al. | Interactive crowd content generation and analysis using trajectory-level behavior learning | |
| Amir Sjarif et al. | Crowd analysis and its applications | |
| CN114842045B (en) | Target tracking method and device | |
| Chibloun et al. | Abnormal crowd behavior detection using speed and direction models | |
| CN114723776B (en) | Target tracking method and device | |
| Nallaivarothayan et al. | Anomalous event detection using a semi-two dimensional hidden markov model | |
| CN107067411B (en) | Mean-shift tracking method combined with dense features | |
| CN114882429A (en) | Queue counting method and system based on fusion of multiple information characteristics | |
| CN111191524A (en) | Sports people counting method | |
| Kim et al. | Directional pedestrian counting with a hybrid map-based model | |
| CN108346158B (en) | Multi-target tracking method and system based on main block data association | |
| Chakraborty et al. | Learning to track object position through occlusion | |
| Bouraya et al. | A WSM-based Comparative Study of Vision Tracking Methodologies | |
| Wang et al. | A GM-HMM based abnormal pedestrian behavior detection method | |
| Revathi et al. | Hybridisation of feed forward neural network and self-adaptive PSO with diverse of features for anomaly detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |