CN117874529A - Motion trail prediction method, model training method, device, equipment and medium - Google Patents
Motion trail prediction method, model training method, device, equipment and medium Download PDFInfo
- Publication number
- CN117874529A CN117874529A CN202410275992.2A CN202410275992A CN117874529A CN 117874529 A CN117874529 A CN 117874529A CN 202410275992 A CN202410275992 A CN 202410275992A CN 117874529 A CN117874529 A CN 117874529A
- Authority
- CN
- China
- Prior art keywords
- motion
- position information
- track
- target
- displacement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000033001 locomotion Effects 0.000 title claims abstract description 632
- 238000012549 training Methods 0.000 title claims abstract description 88
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000006073 displacement reaction Methods 0.000 claims abstract description 195
- 238000004364 calculation method Methods 0.000 claims abstract description 7
- 238000004590 computer program Methods 0.000 claims description 56
- 238000012360 testing method Methods 0.000 claims description 23
- 238000004422 calculation algorithm Methods 0.000 claims description 22
- 238000003860 storage Methods 0.000 claims description 21
- 230000002123 temporal effect Effects 0.000 claims description 11
- 238000010276 construction Methods 0.000 claims description 3
- 238000004088 simulation Methods 0.000 abstract description 10
- 238000012545 processing Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 8
- 238000012937 correction Methods 0.000 description 7
- 239000011159 matrix material Substances 0.000 description 7
- 238000009826 distribution Methods 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000011273 social behavior Effects 0.000 description 2
- 230000001052 transient effect Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses a motion trail prediction method, a model training method, a device, equipment and a medium in the technical field of model training. According to the method and the device, model input data are composed of predicted position information of the next moment of the target motion track, fitting position information of the next moment of the target motion track, displacement simulation information obtained by calculation of the adjacent motion track of the target motion track, similarity between the predicted position information and the displacement simulation information and movement constraint of the predicted position information determined based on the adjacent motion track, so that the second track prediction model can comprehensively consider the relation between the target motion track and other tracks in the space where the second track prediction model is located, two prediction results of the same target motion track are combined, and accuracy of track prediction can be improved.
Description
Technical Field
The present disclosure relates to the field of model training technologies, and in particular, to a motion trail prediction method, a model training method, a device, equipment, and a medium.
Background
Trajectory prediction refers to the task of knowing the trajectory of an object (pedestrian, vehicle, etc.) over a period of time, predicting the coordinates of that object at several moments in the future. The technology is often used in scenes such as target tracking, automatic driving, social behavior analysis and the like. However, the motion trail of the object is limited by factors such as space, and interference may exist between different motion trail, so that the accuracy of the trail prediction scheme is limited.
Therefore, how to improve the accuracy of trajectory prediction is a problem that needs to be solved by those skilled in the art.
Disclosure of Invention
In view of the foregoing, an object of the present application is to provide a motion trajectory prediction method, a model training method, a device, equipment and a medium, so as to improve accuracy of trajectory prediction. The specific scheme is as follows:
in one aspect, the present application provides a motion trajectory prediction method, including:
obtaining predicted position information and fitting position information of the target motion trail at the next moment; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm;
calculating displacement imitation information according to the adjacent motion trail of the target motion trail;
calculating a degree of similarity between the predicted position information and the displacement mimicking information;
determining a movement constraint of the predicted position information based on the adjacent motion trajectories;
and inputting the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into a second track prediction model so that the second track prediction model predicts again to obtain new predicted position information of the target movement track at the next moment.
On the other hand, obtaining the predicted position information of the next moment of the target motion trail includes:
and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as the prediction position information.
In another aspect, the generating of the prediction result includes:
calculating the position deviation of the target motion trail and the adjacent motion trail;
and inputting the position deviation, the target motion trail and the adjacent motion trail into the first trail prediction model so that the first trail prediction model generates the prediction result.
In another aspect, the calculating the position deviation of the target motion trajectory and the adjacent motion trajectory includes:
constructing the position points at the same time in the target motion track and the adjacent motion track as position combinations;
respectively calculating the distance between two position points in each position combination;
and taking the average value of each distance as the position deviation.
On the other hand, obtaining fitting position information of the next moment of the target motion trail includes:
and fitting the target motion trail by using at least one curve fitting algorithm to obtain the fitting position information.
In another aspect, before calculating the displacement imitation information according to the adjacent motion trail of the target motion trail, the method further includes:
setting a proximity range according to the predicted position information;
the motion trajectories within the proximity range are taken as the proximity motion trajectories based on temporal and/or spatial correlation.
In another aspect, the determining the motion trajectory within the proximity range as the proximity motion trajectory based on the temporal correlation and/or the spatial correlation includes:
taking a motion track existing in the adjacent range at the same time as the target motion track as the adjacent motion track; and/or taking each historical motion trail existing in the adjacent range as the adjacent motion trail.
In another aspect, the calculating displacement imitation information according to the adjacent motion trail of the target motion trail includes:
selecting a similar motion track with a similar motion direction with the target motion track from the adjacent motion tracks;
calculating the displacement of the target position in the similar motion trail;
and calculating the displacement imitation information according to the displacement, the average step length of the similar motion trail and the average step length of the target motion trail.
In another aspect, the selecting a similar motion trajectory having a similar motion direction to the target motion trajectory from the adjacent motion trajectories includes:
calculating a direction included angle between the target motion trail and the adjacent motion trail;
if the direction included angle is smaller than a preset angle threshold, determining the adjacent motion trail as the similar motion trail; otherwise, the adjacent motion trail is determined to be a dissimilar motion trail.
In another aspect, the calculating the displacement of the target position in the similar motion trail includes:
taking a position point at the tail of the track in the similar motion track as the target position;
and obtaining the displacement according to the difference between the target position and the position before the target position.
In another aspect, the calculating the displacement imitation information according to the displacement, the average step length of the similar motion track and the average step length of the target motion track includes:
calculating the displacement mimicking information using a first formula; the first formula is:
；D M for the displacement to be modeled as information,p k is the movement track of the targettr cur Is provided with a track tail portion of the track,p s a track tail for the adjacent motion track The location point of the portion,Step(tr cur ) Is the movement track of the targettr cur Is used for the average step size of (a),Step(tr k ) Is similar to the motion trailtr k Is used for the average step size of (a),Tr neigh for the set of the adjacent motion trajectories,Tr orient a set of the similar motion trajectories;d argmin (dist(p k ,p s [-1]) Is) isp k And a plurality ofp s Index of minimum distance between.
In another aspect, the calculating the degree of similarity between the predicted position information and the displacement emulation information includes:
and taking the included angle between the displacement of the predicted position information and the displacement of the displacement imitation information as the similarity degree.
In another aspect, the determining the movement constraint of the predicted position information based on the neighboring motion trajectories includes:
taking the position point of the predicted position information as a circle center, and determining a constraint ring by combining a preset radius;
taking a position point in the adjacent motion trail positioned in the constraint ring as a target point;
and determining each directional line segment pointing to the destination point from the position point where the predicted position information is located as the movement constraint.
In another aspect, before calculating the displacement imitation information according to the adjacent motion trail of the target motion trail, the method further includes:
judging whether a neighboring motion trail determined based on spatial association exists in the set of neighboring motion trail;
If so, executing the step of calculating displacement imitation information according to the adjacent motion trail of the target motion trail and the subsequent step;
and if the target motion trail does not exist, determining new predicted position information of the next moment of the target motion trail based on the predicted position information and the fitting position information.
In another aspect, the present application provides a model training method, including:
acquiring an initial sample set comprising a plurality of motion trajectories;
downsampling the initial sample set to obtain an updated sample set;
taking the motion trail in the updated sample set as a target motion trail, and acquiring predicted position information and fitting position information of the target motion trail at the next moment; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm;
calculating displacement imitation information according to the adjacent motion trail of the target motion trail;
calculating a degree of similarity between the predicted position information and the displacement mimicking information;
determining a movement constraint of the predicted position information based on the adjacent motion trajectories;
combining the predicted position information, the fitted position information, the displacement mimicking information, the degree of similarity, and the movement constraint into first training data;
And training by using the first training data to obtain a first prediction model.
On the other hand, obtaining the predicted position information of the next moment of the target motion trail includes:
and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as the prediction position information.
In another aspect, the method further comprises:
calculating the position deviation of the target motion trail and the adjacent motion trail;
the position deviation, the target motion trail and the adjacent motion trail are combined into second training data;
and training by using the second training data to obtain a second prediction model.
In another aspect, the downsampling the initial sample set to obtain an updated sample set includes:
and enabling the track sparseness of the initial sample set to be equivalent to the track sparseness of a preset test scene.
On the other hand, the application provides a motion trail prediction device, which comprises:
the acquisition module is used for acquiring predicted position information and fitting position information of the target motion trail at the next moment;
the first calculation module is used for calculating displacement imitation information according to the adjacent motion trail of the target motion trail;
A second calculation module for calculating a degree of similarity between the predicted position information and the displacement mimicking information;
a determining module for determining a movement constraint of the predicted position information based on the adjacent motion trajectories;
and the prediction module is used for inputting the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into a second track prediction model so that the second track prediction model predicts again to obtain new predicted position information of the next moment of the target movement track.
In another aspect, the present application provides a model training apparatus, comprising:
the sample construction module is used for acquiring an initial sample set comprising a plurality of motion tracks; downsampling the initial sample set to obtain an updated sample set; taking the motion trail in the updated sample set as a target motion trail, and acquiring predicted position information and fitting position information of the target motion trail at the next moment; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm; calculating displacement imitation information according to the adjacent motion trail of the target motion trail; calculating a degree of similarity between the predicted position information and the displacement mimicking information; determining a movement constraint of the predicted position information based on the adjacent motion trajectories; combining the predicted position information, the fitted position information, the displacement mimicking information, the degree of similarity, and the movement constraint into first training data; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm;
And the model training module is used for training by utilizing the first training data to obtain a first prediction model.
In another aspect, the present application provides an electronic device, including:
a memory for storing a computer program;
and a processor for executing the computer program to implement the previously disclosed motion trail prediction method.
In a sixth aspect, the present application provides a readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the previously disclosed motion trajectory prediction method.
According to the scheme, the application provides a motion trail prediction method, which comprises the following steps: obtaining predicted position information and fitting position information of the target motion trail at the next moment; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm; calculating displacement imitation information according to the adjacent motion trail of the target motion trail; calculating a degree of similarity between the predicted position information and the displacement mimicking information; determining a movement constraint of the predicted position information based on the adjacent motion trajectories; and inputting the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into a second track prediction model so that the second track prediction model predicts again to obtain new predicted position information of the target movement track at the next moment.
Therefore, the beneficial effects of this application lie in: the model input data is composed of predicted position information of the next moment of the target motion track, fitting position information of the next moment of the target motion track, displacement simulation information calculated by the adjacent motion track of the target motion track, similarity between the predicted position information and the displacement simulation information and movement constraint of the predicted position information determined based on the adjacent motion track, so that the second track prediction model can combine two predicted results (the predicted position information and the fitting position information) of the target motion track, correlation information of the adjacent motion track to the target motion track (the similarity between the displacement simulation information, the predicted position information and the displacement simulation information and the movement constraint of the predicted position information determined based on the adjacent motion track), and new predicted position information of the next moment of the target motion track is predicted again, thereby comprehensively considering the relation between the motion track of an object (namely the target) and other tracks in the space where the object is located, and combining the two predicted results of the same object can improve the accuracy of track prediction.
Correspondingly, the motion trail prediction and model training device, the device and the readable storage medium have the technical effects.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings may be obtained according to the provided drawings without inventive effort to a person skilled in the art.
FIG. 1 is a flow chart of a motion trail prediction method disclosed in the present application;
FIG. 2 is a flow chart of a model training method disclosed in the present application;
FIG. 3 is a schematic view of a model structure disclosed in the present application;
FIG. 4 is a schematic diagram of a motion profile fitting result disclosed in the present application; wherein, (a) is a graph in the case where the first-order square fitting and the reference length are 3, (b) is a graph in the case where the first-order square fitting and the reference length are 9, and (c) is a graph in the case where the second-order square fitting and the reference length are 9;
FIG. 5 is a schematic diagram of a trajectory prediction process disclosed herein;
FIG. 6 is a schematic diagram of a movement constraint disclosed herein;
fig. 7 is a schematic diagram of a motion trail prediction device disclosed in the present application;
FIG. 8 is a schematic diagram of a model training apparatus disclosed herein;
FIG. 9 is a block diagram of a server provided herein;
fig. 10 is a schematic diagram of a terminal provided in the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other examples, which a person of ordinary skill in the art would obtain without making any inventive effort, are within the scope of the present application based on the examples herein.
Currently, trajectory prediction refers to the task of knowing the trajectory of an object (pedestrian, vehicle, etc.) over a period of time, and predicting the coordinates of that object at several moments in the future. The technology is often used in scenes such as target tracking, automatic driving, social behavior analysis and the like. However, the motion trail of the object is limited by factors such as space, and interference may exist between different motion trail, so that the accuracy of the trail prediction scheme is limited. Therefore, the motion trail prediction scheme provided by the application can improve the accuracy of trail prediction.
Referring to fig. 1, an embodiment of the present application discloses a motion trajectory prediction method, including:
s101, obtaining predicted position information and fitting position information of the next moment of the target motion trail.
It should be noted that, the predicted position information of the next moment of the target motion track may be predicted by other prediction models (such as the second prediction model), and the input data of these models may be: the known coordinate point sequence forming the target motion trail can further comprise: the model predicts the predicted position information based on the input data, and the known coordinate point sequence of the adjacent motion track of the target motion track and the position deviation value of the known coordinate point sequence of the adjacent motion track of the target motion track. Thus in one example, obtaining predicted location information for a next time of a target motion trajectory includes: and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as prediction position information. In one example, the generation of the prediction result includes: calculating the position deviation of the target motion track and the adjacent motion track; and inputting the position deviation, the target motion track and the adjacent motion track into a first track prediction model so that the first track prediction model generates a prediction result. The predicted position information of the next moment of the target motion trail is: and the displacement of the coordinate point newly generated by the target motion trail at the next moment.
The calculating of the position deviation of the target motion track and the adjacent motion track comprises the following steps: constructing position points at the same time in the target motion track and the adjacent motion track as position combinations; respectively calculating the distance between two position points in each position combination; taking the average value of each distance as the position deviation. Namely: and calculating the position relation between one coordinate point in the target motion track and one coordinate point in the adjacent motion track at the same time, wherein the position relation is represented by the distance between the two points of the directed line segment, and the average value of the distances of all the directed line segments is the position deviation of the target motion track and the adjacent motion track.
In one example, obtaining fitting position information for a next time of a target motion trajectory includes: and fitting the target motion trail by using at least one curve fitting algorithm to obtain fitting position information. Curve fitting algorithms such as: least squares algorithms, and the like. And (3) processing the known coordinate point sequence forming the target motion track by at least one curve fitting algorithm to obtain fitting position information of the next moment of the target motion track. The fitting position information of the next moment of the target motion trail is also: and the displacement of the coordinate point newly generated by the target motion track at the next moment, and the fitting position information and the prediction position information are predicted by adopting different methods aiming at the same motion track to obtain the same kind of data information. That is, the predicted position information is predicted by the first trajectory prediction model; and obtaining the fitting position information through a curve fitting algorithm.
In one example, before calculating the displacement emulation information according to the adjacent motion trajectories of the target motion trajectories, the method further includes: setting a proximity range according to the predicted position information; the motion trajectories within the proximity range are treated as proximity motion trajectories based on temporal and/or spatial correlation. In one example, regarding a motion trajectory within a proximity range as a proximity motion trajectory based on a temporal correlation and/or a spatial correlation includes: taking a motion track existing in the adjacent range at the same time as the motion track of the target as an adjacent motion track; and/or each historical motion trail existing in the adjacent range (i.e., motion trail which appears in the adjacent range in the past) is used as the adjacent motion trail. The adjacent range may be a regular shape such as a rectangle, a circle, or an irregular shape, and the area of the adjacent range may be a preset value.
S102, calculating displacement simulation information according to the adjacent motion trail of the target motion trail.
The displacement imitation information can represent the correlation between the target motion trail and the adjacent motion trail to a certain extent. In one example, calculating displacement emulation information from adjacent motion trajectories of a target motion trajectory includes: selecting a similar motion track with a similar motion direction with the motion track of the target from the adjacent motion tracks; calculating the displacement of the target position in the similar motion trail; and calculating according to the displacement, the average step length of the similar motion track and the average step length of the target motion track to obtain the displacement imitation information. In one example, selecting a similar motion trajectory having a similar motion direction as the target motion trajectory from the adjacent motion trajectories includes: calculating the direction included angle between the target motion track and the adjacent motion track; if the direction included angle is smaller than a preset angle threshold value, determining the adjacent motion trail as a similar motion trail; otherwise, the adjacent motion trajectories are determined to be dissimilar motion trajectories. In one example, calculating the displacement of the target position in the similar motion trajectories includes: taking a position point at the tail of a track in a similar motion track as a target position; and obtaining displacement according to the difference between the target position and the position before the target position.
In one example, the calculating the displacement imitation information according to the displacement, the average step length of the similar motion track and the average step length of the target motion track includes:
calculating the displacement mimicking information using a first formula; the first formula is:
；D M for the displacement to be modeled as information,p k is the movement track of the targettr cur Is provided with a track tail portion of the track,p s as the position point adjacent to the track tail of the motion track,Step(tr cur ) Is the movement track of the targettr cur Is used for the average step size of (a),Step(tr k ) Is similar to the motion trailtr k Is used for the average step size of (a),Tr neigh for the set of the adjacent motion trajectories,Tr orient a set of the similar motion trajectories;d argmin (dist(p k ,p s [-1]) Is) isp k And a plurality ofp s Index of minimum distance between.
S103, calculating the similarity degree between the predicted position information and the displacement imitation information.
The degree of similarity can also represent the correlation between the target motion trajectory and the adjacent motion trajectory to a certain extent. In one example, calculating the degree of similarity between the predicted position information and the displacement mimicking information includes: the angle between the displacement of the predicted position information and the displacement of the displacement mimicking information is taken as the degree of similarity.
S104, determining movement constraint of the predicted position information based on the adjacent motion trail.
The movement constraint can also represent the correlation between the target motion trajectory and the adjacent motion trajectory to some extent. In one example, determining movement constraints for predicted location information based on neighboring motion trajectories includes: taking a position point where the predicted position information is located (namely a coordinate point which is newly generated by the target motion track at the next moment) as a circle center, and determining a constraint circle by combining a preset radius; taking a position point in an adjacent motion track in the restriction ring as a target point; and determining each directional line segment pointing to the destination point from the position point where the predicted position information is located as a movement constraint.
S105, inputting the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into the second track prediction model so that the second track prediction model predicts again to obtain new predicted position information of the next moment of the target movement track.
In one example, before calculating the displacement emulation information according to the adjacent motion trajectories of the target motion trajectories, the method further includes: judging whether the adjacent motion trail determined based on spatial association exists in the set of adjacent motion trail; if so, executing the step of calculating displacement imitation information according to the adjacent motion trail of the target motion trail and the subsequent steps; if the target motion trail does not exist, new predicted position information of the next moment of the target motion trail is determined based on the predicted position information and the fitting position information.
It can be seen that, in this embodiment, the model input data is composed of predicted position information of the next time of the target motion track, fitting position information of the next time of the target motion track, displacement mimicking information calculated from the adjacent motion track of the target motion track, a degree of similarity between the predicted position information and the displacement mimicking information, and a movement constraint of the predicted position information determined based on the adjacent motion track, so that the second track prediction model may combine two prediction results (the predicted position information and the fitting position information) of the target motion track, correlation information of the adjacent motion track to the target motion track (the degree of similarity between the displacement mimicking information, the predicted position information and the displacement mimicking information, and the movement constraint of the predicted position information determined based on the adjacent motion track), and predict again new predicted position information of the next time of the target motion track, thereby comprehensively considering a relationship between the motion track of an object (i.e., the target) and other tracks in a space where the object is located, and combining the two prediction results of the same object, so as to be able to improve the accuracy of track prediction.
A model training method provided in the embodiments of the present application is described below, and a model training method described below may be referred to with other embodiments described herein.
Referring to fig. 2, an embodiment of the present application discloses a model training method, including:
s201, acquiring an initial sample set comprising a plurality of motion trajectories.
S202, dividing each motion track in the initial sample set into equal-length sub-motion tracks.
S203, adding each sub-motion track to different motion scenes, and enabling the track sparseness of each motion scene to be equivalent to the track sparseness of a preset test scene.
S204, taking the sub-motion trail of each motion scene as a target motion trail, and acquiring predicted position information and fitting position information of the next moment of the target motion trail.
S205, calculating displacement simulation information according to the adjacent motion trail of the target motion trail.
S206, calculating the similarity degree between the predicted position information and the displacement imitation information.
S207, determining movement constraint of the predicted position information based on the adjacent motion trail.
S208, combining the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into first training data.
S209, training by using the first training data to obtain a first prediction model.
In this embodiment, the initial sample set is downsampled through S202 and S203 to obtain an updated sample set. The updated sample set is composed of trajectories of arbitrary motion scenes. Namely: downsampling the initial sample set to obtain an updated sample set, comprising: the track sparseness of the initial sample set is equivalent to the track sparseness of a preset test scene (namely, the actual use scene of the model). Specifically, each motion trail in the initial sample set is segmented into equal-length sub-motion trail. And adding each sub-motion track to different motion scenes, enabling the track sparseness degree of each motion scene to be equivalent to the track sparseness degree of a preset test scene, and enabling the track of any motion scene to form an updated sample set.
In one example, the proximity range is set according to the predicted position information; the motion trajectories within the proximity range are treated as proximity motion trajectories based on temporal and/or spatial correlation.
In one example, a motion trajectory existing in the proximity range at the same time as the target motion trajectory is taken as a proximity motion trajectory; and/or taking each historical motion trail existing in the adjacent range as an adjacent motion trail.
In one example, a similar motion trajectory having a similar motion direction to the target motion trajectory is selected from the adjacent motion trajectories; calculating the displacement of the target position in the similar motion trail; and calculating according to the displacement, the average step length of the similar motion track and the average step length of the target motion track to obtain the displacement imitation information.
In one example, the direction included angle between the target motion trail and the adjacent motion trail is calculated; if the direction included angle is smaller than a preset angle threshold value, determining the adjacent motion trail as a similar motion trail; otherwise, the adjacent motion trajectories are determined to be dissimilar motion trajectories.
In one example, a position point of a track tail in a similar motion track is taken as a target position; and obtaining displacement according to the difference between the target position and the position before the target position.
In one example, the degree of similarity is taken as the angle between the displacement of the predicted position information and the displacement of the displacement mimicking information.
In one example, the constraint circle is determined by taking the position point where the predicted position information is located as the circle center and combining with a preset radius; taking a position point in an adjacent motion track in the restriction ring as a target point; and determining each directional line segment pointing to the destination point from the position point where the predicted position information is located as a movement constraint.
In one example, it is determined whether there is a neighboring motion trajectory in the set of neighboring motion trajectories that is determined based on spatial correlation; if so, executing the step of calculating displacement imitation information according to the adjacent motion trail of the target motion trail and the subsequent steps; if the target motion trail does not exist, new predicted position information of the next moment of the target motion trail is determined based on the predicted position information and the fitting position information.
In one example, obtaining predicted location information for a next time of a target motion trajectory includes: and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as prediction position information. Specifically, calculating the position deviation of the target motion track and the adjacent motion track; combining the position deviation, the target motion trail and the adjacent motion trail into second training data; and training by using the second training data to obtain a second prediction model.
In one example, equating the trace sparsity of the initial sample set to the trace sparsity of the preset test scene, comprising: and downsampling the initial sample set according to the track sparseness of a preset test scene.
The more specific working process in this embodiment may refer to the corresponding content disclosed in other embodiments, and will not be described herein.
Therefore, the embodiment provides a model training method, which can be used for constructing a training sample with equivalent sparsity according to the track sparsity of a test scene, and the training sample comprises a plurality of pieces of information, so that a prediction model with high performance and accuracy can be obtained through training, and the accuracy of track prediction can be improved.
It should be noted that, the motion scene includes: street, office, etc., where there may be a path of movement of a pedestrian, a path of movement of a vehicle, etc. Wherein the range of pedestrian activity is relatively limited, such as: in offices, pedestrians typically have a range of action from 0-20 meters. Therefore, in such a scene, the motion trail distribution of different pedestrians is not greatly different, and almost all possible position coordinates in the current scene can be covered under a sufficient sample size. However, for ultra-large scenes, the range of different scenes may be up to kilometers, so that it cannot be guaranteed that the training set covers the real test sample, and errors are caused by modeling the position coordinates.
To this end, the present application proposes a displacement prediction network (Displacement Prediction Network, DPN). As shown in fig. 3, the displacement prediction network is composed of 4 fully connected layers and 1 nonlinear layer. The input and output of the displacement prediction network are displacement (non-position). The input is a displacement sequence of a plurality of frames before the motion trail of the pedestrian, and the output is the displacement predicted at the next moment. The output of the trajectory prediction is the sum of the several frame prediction displacements of the pedestrian. Since different pedestrians may have different directions of action in the future, for example: a pedestrian may either travel straight or turn left or right at an intersection, so the output of trajectory prediction may also be a combination of multiple trajectories.
And, can also utilize the least square estimation to carry on the orbit fitting, can obtain three predicting the orbit finally, the input of the least square is the known sequence. Fig. 4 shows the effect of square fitting of different reference lengths, different orders. Fig. 4 (a) is a graph in which the first-order square fitting and the reference length are 3; fig. 4 (b) is a curve in the case where the first-order square fitting and the reference length are 9; fig. 4 (c) is a curve in the case where the second order square fitting and the reference length are 9.
As can be seen from fig. 4, the result of the least squares fitting is sufficient to meet the requirements in terms of visual effect, without considering the problems of accompanying, collision, boundary, etc., and in addition, the least squares method is not constrained by the training set, and has natural robustness and generalization performance. Therefore, the track fitting can be performed by combining a least square method, so that the predicted track of the least square method is fused in the final result.
In order to construct a more robust prediction model and take account of the characteristics of object relevance, accessibility constraints and the like in ultra-large scenes, a displacement prediction correction network (Displacement Prediction and Modification, DPM) can be used. The displacement prediction correction network is realized by comprehensively considering the limitation of the motion space and the relevance of adjacent tracks on the basis of DPN. Specifically, the proximity range (set range) is first determined according to the coordinate position (predicted latest point) of the current predicted frame predicted from the input motion trajectory, the trajectory within the range is taken as the proximity trajectory, and the proximity trajectories are classified into two types according to the occurrence time of the proximity trajectories. The first type is a time-associated trajectory representing a trajectory that moves in the vicinity of a target pedestrian while it is active (a neighboring trajectory that exists at the same time as the current trajectory); the second type is a spatially-correlated trajectory, representing a trajectory around its location at the current moment (historically all trajectories that occur within close proximity) with the target pedestrian.
According to the two basic models of DPN and DPM, the embodiment provides a joint displacement prediction model (Joint Displacement Prediction Network, JDP, namely a second prediction model) for solving the problem of adjacent track relevance; a position correction module is provided to solve the problem of limited movement space. Referring to fig. 5, the jdp output is sent to the input of the position correction module (PositionModification, PM, i.e., the first prediction model), and the PM integrates the predicted displacement with other mined features to implement correction of the prediction.
Specifically, the structure of the joint displacement prediction model is the same as the DPN structure, except that the input features are different. Unlike DPN, the input of JDP includes three parts: a first, displacement sequence of the current trajectory (i.e., the target motion trajectory, the same portion as DPN); a displacement sequence of the second, nearest neighbor trajectory (i.e., the neighboring motion trajectory); and thirdly, the position deviation of the nearest neighbor track from the current track is used for describing the distance between the nearest neighbor track and the current track (representing the correlation strength). For the third part, the position relation between each frame of the adjacent track and each frame of the current track at the same time is calculated, and the base is obtained: a directed line segment between the two points, each distance being represented by [ x, y ], the final third portion having a size of [ N,2], N representing the number of frames. The three parts are spliced together and input into the JDP.
The output of JDP is the same as DPN, the next frame coordinate of the current track. By combining nearest neighbor tracks, modeling the accompanying phenomena in the adjacent tracks, and carrying out feature mining on the adjacent tracks through experimental verification, the comprehensive performance of the model is improved.
The structure of the position correction module is the same as that of DPN, except that the input features are different. Unlike DPN and JDP, the input of PM includes two parts: a first set of candidate displacements; and second, an included angle matrix.
The candidate displacement set D is composed of three parts. D (D) S Output results (i.e., predicted position information) representing JDP; d (D) LSE Representing least square fitting results (i.e., fitting position information) of different orders; d (D) M Representing the displacement mimicking result (i.e., displacement mimicking information). The final three parts are pieced together to be the candidate displacement set. D (D) M As shown in the formula (4), the displacement of the nearest point (the displacement formed by the latest actual point and the previous point) in the similar track in the nearby similar direction (the latest actual point of the similar track) is extracted, the average step length of the track is divided to obtain the displacement direction, and the average step length of the current track is multiplied to obtain the magnitude of the simulated displacement.
Wherein d=d S ∪D LSE ∪D M (1) Namely: d (D) S 、D LSE And D M The spelling is the candidate displacement set.
Sets of adjacent tracksTr neigh ={tr k ∈TrⅠmin(dist(p k ,p s [-1]))＜th dist -2) for selecting a minimum value of the distance differences between the latest point of the current track and all points of other neighboring tracks to be less than a threshold valueth dist Is a neighbor track of (c).
Aggregation of similarly oriented trajectoriesTr orient ={tr k ∈TrⅠmin(dist(d k ,d s [-1]))＜th angle -3) for selecting the difference between the direction of the latest point displacement of the current track and the direction of all point displacements of other neighboring tracks, the minimum value of which is smaller than the threshold valueth angle Is a neighbor track of (c).
Displacement simulation results
(4)。
For the angle matrix, the formula is as follows: a=a M ∪A N (5) Namely: a is that M And A N The spelling is the matrix of included angles. The angle A between the displacement imitation result and the result of model prediction (i.e. predicted position information) M ={cossim(d k ,d s )Ⅰd k ∈D M -6), which is the sum of the direction of the latest point displacement of the current trackD M Cosine similarity of each result in (c). A is that N ={dist(p s [-1],p i )Ⅰp i ∈tr k Ⅰtr k ∈Tr neigh ∩Tr orient }(7)。
Neighbor simulation included angle A M The included angle between the displacement representing the predicted position information and the displacement of the displacement imitation information is represented by calculating cosine similarity of the displacement and the displacement imitation information, and is used for extracting whether the value between the simulated displacement and the current displacement is learned or not; potential direction of movement A N (i.e., movement constraint) referring to fig. 6, a directional line segment representing the current predicted position to all points meeting the condition (the constraint circle is set by taking the predicted point as the center of a circle, and points falling into the constraint circle in all adjacent tracks) is used for constraining the maximum movement range of the current track.
The characteristics of the candidate displacement set and the included angle matrix after combination are input into PM, and corrected displacement is output. The merging process comprises the following steps: and splicing the candidate displacement set feature D and the included angle matrix A, wherein the spliced feature F=A U-D. Before splicing, processing the data dimensions of the candidate displacement set feature D and the included angle matrix A so as to keep the data dimensions of the candidate displacement set feature D and the included angle matrix A consistent.
Accordingly, the training process of the model includes:
in the first step, after training set preprocessing (preparation operation) and normalization, training data downsampling is performed.
This is because if modeling of spatial and temporal correlations is considered, the model is required for crowd crowding. Because the population in the training set is denser, the given track is longer, and the model can acquire more priori information. The anti-observation test set, because it is sampled more sparsely, dilutes much of the neighbor information. The model derived directly from the training set is likely to force the algorithm to rely too much on adjacent tracks, resulting in reduced accuracy. We therefore do data downsampling. Firstly, defining the track length of a training sample, and dividing all tracks into a plurality of sub-track combinations with the length of k frames (suggested as 18 frames and simulated test set), so that the training set has more track numbers; these trajectories are then split until each scene after splitting is guaranteed to have about M trajectories (suggested as 1500, simulated test sets). Finally, we split the R training scenarios into R more sparse sub-scenarios and develop training based on these sub-scenarios.
And secondly, constructing the model input features according to the above.
Thirdly, training by using the constructed model input characteristics to obtain JDP and PM.
Because serious differences may exist between the training data set and the actual use scene, the prediction performance is affected, and the training data is generally denser through data statistics, and the test set for the actual scene is sparser, so that the training data can simulate the distribution condition of the test set by carrying out downsampling processing on the training set. The problem of big difference of distribution of the training set and the testing set in the ultra-large scene track prediction task is solved.
In the test process, in order to cope with the situation that some tracks have no adjacent tracks or adjacent space limitation, a judging step can be added in the prediction process: it is assumed that all tracks need to undergo DPN to achieve initial position prediction. Then, confirming a neighboring range according to the position predicted by DPN, and performing JDP prediction on samples with time-related tracks in the range; otherwise, JDP is skipped. Then, if a space adjacent track exists, inputting the JDP (or DPN) result into PM for correction; otherwise, directly outputting the result of JDP (or DPN). And finally, autoregressing until the coordinates of all the moments to be determined are predicted.
Therefore, in this embodiment, aiming at the problem of large distribution difference between the training set and the test set, the training set is downsampled to simulate the distribution of the test set; aiming at the problem of limited movement space, the method uses the characteristics of the history track in the space to represent; aiming at the problem that different tracks are strongly influenced by the relevance of adjacent tracks, joint modeling is carried out by utilizing the accompanying adjacent tracks; since the motion trajectories exhibit obvious polynomial properties, a polynomial least squares fit strategy is tried to assist in prediction or to initialize prediction. Therefore, the embodiment can predict the track with low energy consumption and high precision aiming at the oversized scene. Experiments prove that the prediction time of the embodiment for a single track only consumes 0.02 seconds. The proposal provided by the application also obtains the army in the international artificial intelligence challenge game of CICAI 2023.
The following describes a motion trajectory prediction device provided in the embodiments of the present application, and the motion trajectory prediction device described below may refer to other embodiments described herein.
Referring to fig. 7, an embodiment of the present application discloses a motion trajectory prediction apparatus, including:
the acquisition module is used for acquiring predicted position information and fitting position information of the target motion trail at the next moment; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm;
The first calculation module is used for calculating displacement imitation information according to the adjacent motion trail of the target motion trail;
the second calculation module is used for calculating the similarity degree between the predicted position information and the displacement imitation information;
a determining module for determining a movement constraint of the predicted position information based on the neighboring motion trajectories;
and the prediction module is used for inputting the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into the second track prediction model so that the second track prediction model predicts again to obtain new predicted position information of the target motion track at the next moment.
In one example, the acquisition module is specifically configured to:
and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as prediction position information.
In one example, the generation of the prediction result includes:
calculating the position deviation of the target motion track and the adjacent motion track;
and inputting the position deviation, the target motion track and the adjacent motion track into a first track prediction model so that the first track prediction model generates a prediction result.
In one example, calculating a position deviation of a target motion trajectory and a neighboring motion trajectory includes:
Constructing position points at the same time in the target motion track and the adjacent motion track as position combinations;
respectively calculating the distance between two position points in each position combination;
taking the average value of each distance as the position deviation.
In one example, the acquisition module is specifically configured to:
and fitting the target motion trail by using at least one curve fitting algorithm to obtain fitting position information.
In one example, before calculating the displacement emulation information according to the adjacent motion trajectories of the target motion trajectories, the method further includes:
the adjacent track determining module is used for setting an adjacent range according to the predicted position information; the motion trajectories within the proximity range are treated as proximity motion trajectories based on temporal and/or spatial correlation.
In one example, the proximity trajectory determination module is specifically configured to:
taking a motion track existing in the adjacent range at the same time as the motion track of the target as an adjacent motion track; and/or taking each historical motion trail existing in the adjacent range as an adjacent motion trail.
In one example, the first computing module is specifically configured to:
selecting a similar motion track with a similar motion direction with the motion track of the target from the adjacent motion tracks;
Calculating the displacement of the target position in the similar motion trail;
and calculating according to the displacement, the average step length of the similar motion track and the average step length of the target motion track to obtain the displacement imitation information.
In one example, the first computing module is specifically configured to:
calculating the direction included angle between the target motion track and the adjacent motion track;
if the direction included angle is smaller than a preset angle threshold value, determining the adjacent motion trail as a similar motion trail; otherwise, the adjacent motion trajectories are determined to be dissimilar motion trajectories.
In one example, the first computing module is specifically configured to:
taking a position point at the tail of a track in a similar motion track as a target position;
and obtaining displacement according to the difference between the target position and the position before the target position.
In one example, the first computing module is specifically configured to:
calculating the displacement mimicking information using a first formula; the first formula is:
；D M for the displacement to be modeled as information,p k is the movement track of the targettr cur Is provided with a track tail portion of the track,p s as the position point adjacent to the track tail of the motion track,Step(tr cur ) Is the movement track of the targettr cur Is used for the average step size of (a),Step(tr k ) Is similar to the motion trailtr k Is used for the average step size of (a),Tr neigh for the set of the adjacent motion trajectories, Tr orient A set of the similar motion trajectories;d argmin (dist(p k ,p s [-1]) Is) isp k And a plurality ofp s Index of minimum distance between.
In one example, the second computing module is specifically configured to:
the angle between the displacement of the predicted position information and the displacement of the displacement mimicking information is taken as the degree of similarity.
In one example, the determination module is specifically configured to:
taking the position point where the predicted position information is positioned as a circle center, and determining a constraint ring by combining with a preset radius;
taking a position point in an adjacent motion track in the restriction ring as a target point;
and determining each directional line segment pointing to the destination point from the position point where the predicted position information is located as a movement constraint.
In one example, before calculating the displacement emulation information according to the adjacent motion trajectories of the target motion trajectories, the method further includes:
the judging module is used for judging whether the adjacent motion trail determined based on the spatial association exists in the set of the adjacent motion trail; if so, executing the step of calculating displacement imitation information according to the adjacent motion trail of the target motion trail and the subsequent steps; if the target motion trail does not exist, new predicted position information of the next moment of the target motion trail is determined based on the predicted position information and the fitting position information.
The more specific working process of each module and unit in this embodiment may refer to the corresponding content disclosed in the foregoing embodiment, and will not be described herein.
Therefore, the embodiment provides a motion trail prediction device, comprehensively considers the relation between the motion trail and other trails in the space where the motion trail is located, combines two prediction results of the same object, and can improve the accuracy of trail prediction.
A model training apparatus provided in the embodiments of the present application is described below, and a model training apparatus described below may be referred to with other embodiments described herein.
Referring to fig. 8, an embodiment of the present application discloses a model training apparatus, including:
the sample construction module is used for acquiring an initial sample set comprising a plurality of motion tracks; downsampling the initial sample set to obtain an updated sample set; taking the motion trail in the updated sample set as a target motion trail, and acquiring predicted position information and fitting position information of the target motion trail at the next moment; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm; calculating displacement imitation information according to the adjacent motion trail of the target motion trail; calculating a degree of similarity between the predicted position information and the displacement mimicking information; determining a movement constraint of the predicted position information based on the adjacent motion trajectories; combining the predicted position information, the fitted position information, the displacement mimicking information, the degree of similarity and the movement constraint into first training data;
And the model training module is used for training by utilizing the first training data to obtain a first prediction model.
The more specific working process of each module and unit in this embodiment may refer to the corresponding content disclosed in the foregoing embodiment, and will not be described herein.
Therefore, the embodiment provides the motion trail prediction device, which can be attached to the trail sparseness of the test scene to construct the training sample with equivalent sparseness, and the training sample comprises a plurality of pieces of information, so that a prediction model with high performance and accuracy can be obtained through training, and the accuracy of trail prediction can be improved.
An electronic device provided in an embodiment of the present application is described below, and an electronic device described below may refer to other embodiments described herein.
The embodiment of the application discloses electronic equipment, which comprises:
a memory for storing a computer program;
and a processor for executing the computer program to implement the method disclosed in any of the above embodiments.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: obtaining predicted position information and fitting position information of the target motion trail at the next moment; calculating displacement imitation information according to the adjacent motion trail of the target motion trail; calculating a degree of similarity between the predicted position information and the displacement mimicking information; determining a movement constraint of the predicted position information based on the adjacent motion trajectories; and inputting the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into a second track prediction model so that the second track prediction model predicts again to obtain new predicted position information of the target motion track at the next moment.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as prediction position information.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: calculating the position deviation of the target motion track and the adjacent motion track; and inputting the position deviation, the target motion track and the adjacent motion track into a first track prediction model so that the first track prediction model generates a prediction result.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: constructing position points at the same time in the target motion track and the adjacent motion track as position combinations; respectively calculating the distance between two position points in each position combination; taking the average value of each distance as the position deviation.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: and fitting the target motion trail by using at least one curve fitting algorithm to obtain fitting position information.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: setting a proximity range according to the predicted position information; the motion trajectories within the proximity range are treated as proximity motion trajectories based on temporal and/or spatial correlation.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: taking a motion track existing in the adjacent range at the same time as the motion track of the target as an adjacent motion track; and/or taking each historical motion trail existing in the adjacent range as an adjacent motion trail.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: selecting a similar motion track with a similar motion direction with the motion track of the target from the adjacent motion tracks; calculating the displacement of the target position in the similar motion trail; and calculating according to the displacement, the average step length of the similar motion track and the average step length of the target motion track to obtain the displacement imitation information.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: calculating the direction included angle between the target motion track and the adjacent motion track; if the direction included angle is smaller than a preset angle threshold value, determining the adjacent motion trail as a similar motion trail; otherwise, the adjacent motion trajectories are determined to be dissimilar motion trajectories.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: taking a position point at the tail of a track in a similar motion track as a target position; and obtaining displacement according to the difference between the target position and the position before the target position.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: the angle between the displacement of the predicted position information and the displacement of the displacement mimicking information is taken as the degree of similarity.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: taking the position point where the predicted position information is positioned as a circle center, and determining a constraint ring by combining with a preset radius; taking a position point in an adjacent motion track in the restriction ring as a target point; and determining each directional line segment pointing to the destination point from the position point where the predicted position information is located as a movement constraint.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: judging whether the adjacent motion trail determined based on spatial association exists in the set of adjacent motion trail; if so, executing the step of calculating displacement imitation information according to the adjacent motion trail of the target motion trail and the subsequent steps; if the target motion trail does not exist, new predicted position information of the next moment of the target motion trail is determined based on the predicted position information and the fitting position information.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: acquiring an initial sample set comprising a plurality of motion trajectories; dividing each motion track in an initial sample set into equal-length sub-motion tracks; adding each sub-motion track to different motion scenes, and enabling the track sparseness of each motion scene to be equivalent to the track sparseness of a preset test scene; taking the sub-motion trail of each motion scene as a target motion trail, and acquiring predicted position information and fitting position information of the next moment of the target motion trail; calculating displacement imitation information according to the adjacent motion trail of the target motion trail; calculating a degree of similarity between the predicted position information and the displacement mimicking information; determining a movement constraint of the predicted position information based on the adjacent motion trajectories; combining the predicted position information, the fitted position information, the displacement mimicking information, the degree of similarity and the movement constraint into first training data; and training by using the first training data to obtain a first prediction model.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as prediction position information.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: calculating the position deviation of the target motion track and the adjacent motion track; combining the position deviation, the target motion trail and the adjacent motion trail into second training data; and training by using the second training data to obtain a second prediction model.
In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: and sampling each motion scene according to the track sparseness degree of the preset test scene.
Further, the embodiment of the application also provides electronic equipment. The electronic device may be a server as shown in fig. 9 or a terminal as shown in fig. 10. Fig. 9 and 10 are each a block diagram of an electronic device according to an exemplary embodiment, and the contents of the drawings should not be construed as any limitation on the scope of use of the present application.
Fig. 9 is a schematic structural diagram of a server according to an embodiment of the present application. The server specifically may include: at least one processor, at least one memory, a power supply, a communication interface, an input-output interface, and a communication bus. The memory is used for storing a computer program, and the computer program is loaded and executed by the processor to realize the relevant steps in the motion trail prediction disclosed in any one of the previous embodiments.
In this embodiment, the power supply is configured to provide a working voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external equipment, and the communication protocol to be followed by the communication interface is any communication protocol applicable to the technical scheme of the application, and is not particularly limited herein; the input/output interface is used for acquiring external input data or outputting data to the external, and the specific interface type can be selected according to the specific application requirement, and is not limited in detail herein.
In addition, the memory may be a read-only memory, a random access memory, a magnetic disk, an optical disk, or the like as a carrier for storing resources, where the resources stored include an operating system, a computer program, data, and the like, and the storage mode may be transient storage or permanent storage.
The operating system is used for managing and controlling each hardware device and computer program on the Server to realize the operation and processing of the processor on the data in the memory, and the operation and processing can be Windows Server, netware, unix, linux and the like. The computer program may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the motion trajectory prediction method disclosed in any of the foregoing embodiments. The data may include data such as information on a developer of the application program in addition to data such as update information of the application program.
Fig. 10 is a schematic structural diagram of a terminal according to an embodiment of the present application, where the terminal may specifically include, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like.
Generally, the terminal in this embodiment includes: a processor and a memory.
The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor may also include a main processor, which is a processor for processing data in an awake state, also called a CPU (Central Processing Unit ), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor may incorporate a GPU (Graphics Processing Unit, image processor) for rendering and rendering of content required to be displayed by the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.
The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory is at least used to store a computer program, where the computer program, after being loaded and executed by the processor, can implement relevant steps in the motion trajectory prediction method performed by the terminal side disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory can also comprise an operating system, data and the like, and the storage mode can be short-term storage or permanent storage. The operating system may include Windows, unix, linux, among others. The data may include, but is not limited to, update information for the application.
In some embodiments, the terminal may further include a display screen, an input-output interface, a communication interface, a sensor, a power supply, and a communication bus.
Those skilled in the art will appreciate that the structure shown in fig. 10 is not limiting of the terminal and may include more or fewer components than shown.
A readable storage medium provided in embodiments of the present application is described below, and the readable storage medium described below may be referred to with respect to other embodiments described herein.
A readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the motion trajectory prediction method disclosed in the foregoing embodiment. The readable storage medium is a computer readable storage medium, and can be used as a carrier for storing resources, such as read-only memory, random access memory, magnetic disk or optical disk, wherein the resources stored on the readable storage medium comprise an operating system, a computer program, data and the like, and the storage mode can be transient storage or permanent storage.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: obtaining predicted position information and fitting position information of the target motion trail at the next moment; calculating displacement imitation information according to the adjacent motion trail of the target motion trail; calculating a degree of similarity between the predicted position information and the displacement mimicking information; determining a movement constraint of the predicted position information based on the adjacent motion trajectories; and inputting the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into a second track prediction model so that the second track prediction model predicts again to obtain new predicted position information of the target motion track at the next moment.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as prediction position information.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: calculating the position deviation of the target motion track and the adjacent motion track; and inputting the position deviation, the target motion track and the adjacent motion track into a first track prediction model so that the first track prediction model generates a prediction result.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: constructing position points at the same time in the target motion track and the adjacent motion track as position combinations; respectively calculating the distance between two position points in each position combination; taking the average value of each distance as the position deviation.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: and fitting the target motion trail by using at least one curve fitting algorithm to obtain fitting position information.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: setting a proximity range according to the predicted position information; the motion trajectories within the proximity range are treated as proximity motion trajectories based on temporal and/or spatial correlation.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: taking a motion track existing in the adjacent range at the same time as the motion track of the target as an adjacent motion track; and/or taking each historical motion trail existing in the adjacent range as an adjacent motion trail.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: selecting a similar motion track with a similar motion direction with the motion track of the target from the adjacent motion tracks; calculating the displacement of the target position in the similar motion trail; and calculating according to the displacement, the average step length of the similar motion track and the average step length of the target motion track to obtain the displacement imitation information.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: calculating the direction included angle between the target motion track and the adjacent motion track; if the direction included angle is smaller than a preset angle threshold value, determining the adjacent motion trail as a similar motion trail; otherwise, the adjacent motion trajectories are determined to be dissimilar motion trajectories.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: taking a position point at the tail of a track in a similar motion track as a target position; and obtaining displacement according to the difference between the target position and the position before the target position.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: the angle between the displacement of the predicted position information and the displacement of the displacement mimicking information is taken as the degree of similarity.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: taking the position point where the predicted position information is positioned as a circle center, and determining a constraint ring by combining with a preset radius; taking a position point in an adjacent motion track in the restriction ring as a target point; and determining each directional line segment pointing to the destination point from the position point where the predicted position information is located as a movement constraint.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: judging whether the adjacent motion trail determined based on spatial association exists in the set of adjacent motion trail; if so, executing the step of calculating displacement imitation information according to the adjacent motion trail of the target motion trail and the subsequent steps; if the target motion trail does not exist, new predicted position information of the next moment of the target motion trail is determined based on the predicted position information and the fitting position information.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: acquiring an initial sample set comprising a plurality of motion trajectories; dividing each motion track in an initial sample set into equal-length sub-motion tracks; adding each sub-motion track to different motion scenes, and enabling the track sparseness of each motion scene to be equivalent to the track sparseness of a preset test scene; taking the sub-motion trail of each motion scene as a target motion trail, and acquiring predicted position information and fitting position information of the next moment of the target motion trail; calculating displacement imitation information according to the adjacent motion trail of the target motion trail; calculating a degree of similarity between the predicted position information and the displacement mimicking information; determining a movement constraint of the predicted position information based on the adjacent motion trajectories; combining the predicted position information, the fitted position information, the displacement mimicking information, the degree of similarity and the movement constraint into first training data; and training by using the first training data to obtain a first prediction model.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as prediction position information.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: calculating the position deviation of the target motion track and the adjacent motion track; combining the position deviation, the target motion trail and the adjacent motion trail into second training data; and training by using the second training data to obtain a second prediction model.
In this embodiment, when the processor executes the computer program, the following steps may be specifically implemented: and sampling each motion scene according to the track sparseness degree of the preset test scene.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of readable storage medium known in the art.
The principles and embodiments of the present application are described herein with specific examples, the above examples being provided only to assist in understanding the methods of the present application and their core ideas; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (22)
1. The motion trail prediction method is characterized by comprising the following steps of:
obtaining predicted position information and fitting position information of the target motion trail at the next moment; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm;
calculating displacement imitation information according to the adjacent motion trail of the target motion trail;
calculating a degree of similarity between the predicted position information and the displacement mimicking information;
determining a movement constraint of the predicted position information based on the adjacent motion trajectories;
and inputting the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into a second track prediction model so that the second track prediction model predicts again to obtain new predicted position information of the target movement track at the next moment.
2. The method of claim 1, wherein obtaining predicted location information for a next time of the target motion profile comprises:
and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as the prediction position information.
3. The method of claim 2, wherein the generating of the prediction result comprises:
calculating the position deviation of the target motion trail and the adjacent motion trail;
and inputting the position deviation, the target motion trail and the adjacent motion trail into the first trail prediction model so that the first trail prediction model generates the prediction result.
4. A method according to claim 3, wherein said calculating a positional deviation of said target motion profile and said adjacent motion profile comprises:
constructing the position points at the same time in the target motion track and the adjacent motion track as position combinations;
respectively calculating the distance between two position points in each position combination;
and taking the average value of each distance as the position deviation.
5. The method of claim 1, wherein obtaining fitting location information for a next time of the target motion profile comprises:
And fitting the target motion trail by using at least one curve fitting algorithm to obtain the fitting position information.
6. The method of claim 1, wherein before calculating displacement emulation information from adjacent motion trajectories of the target motion trajectory, further comprising:
setting a proximity range according to the predicted position information;
the motion trajectories within the proximity range are taken as the proximity motion trajectories based on temporal and/or spatial correlation.
7. The method according to claim 6, wherein the regarding the motion trajectories within the proximity range as the proximity motion trajectories based on temporal and/or spatial correlation comprises:
taking a motion track existing in the adjacent range at the same time as the target motion track as the adjacent motion track; and/or taking each historical motion trail existing in the adjacent range as the adjacent motion trail.
8. The method of claim 1, wherein calculating displacement emulation information from adjacent motion trajectories of the target motion trajectory comprises:
selecting a similar motion track with a similar motion direction with the target motion track from the adjacent motion tracks;
Calculating the displacement of the target position in the similar motion trail;
and calculating the displacement imitation information according to the displacement, the average step length of the similar motion trail and the average step length of the target motion trail.
9. The method of claim 8, wherein selecting a similar motion profile in the adjacent motion profile having a similar motion direction as the target motion profile comprises:
calculating a direction included angle between the target motion trail and the adjacent motion trail;
if the direction included angle is smaller than a preset angle threshold, determining the adjacent motion trail as the similar motion trail; otherwise, the adjacent motion trail is determined to be a dissimilar motion trail.
10. The method of claim 8, wherein said calculating the displacement of the target position in the similar motion profile comprises:
taking a position point at the tail of the track in the similar motion track as the target position;
and obtaining the displacement according to the difference between the target position and the position before the target position.
11. The method of claim 8, wherein the calculating the displacement emulation information from the displacement, the average step size of the similar motion trajectories, and the average step size of the target motion trajectories comprises:
Calculating the displacement mimicking information using a first formula; the first formula is:
；D M for the displacement to be modeled as information,p k is the movement track of the targettr cur Is provided with a track tail portion of the track,p s as the position point adjacent to the track tail of the motion track,Step(tr cur ) Is the movement track of the targettr cur Is used for the average step size of (a),Step(tr k ) Is similar to the motion trailtr k Is used for the average step size of (a),tr neigh for the set of the adjacent motion trajectories,tr orient a set of the similar motion trajectories;d argmin (dist(p k ,p s [-1]) Is) isp k And a plurality ofp s Index of minimum distance between.
12. The method of claim 1, wherein said calculating a degree of similarity between said predicted position information and said displacement mimicking information comprises:
and taking the included angle between the displacement of the predicted position information and the displacement of the displacement imitation information as the similarity degree.
13. The method of claim 1, wherein the determining movement constraints of the predicted position information based on the adjacent motion trajectories comprises:
taking the position point of the predicted position information as a circle center, and determining a constraint ring by combining a preset radius;
taking a position point in the adjacent motion trail positioned in the constraint ring as a target point;
And determining each directional line segment pointing to the destination point from the position point where the predicted position information is located as the movement constraint.
14. The method according to any one of claims 1 to 13, further comprising, before calculating displacement mimicking information from adjacent motion trajectories of the target motion trajectories:
judging whether a neighboring motion trail determined based on spatial association exists in the set of neighboring motion trail;
if so, executing the step of calculating displacement imitation information according to the adjacent motion trail of the target motion trail and the subsequent step;
and if the target motion trail does not exist, determining new predicted position information of the next moment of the target motion trail based on the predicted position information and the fitting position information.
15. A method of model training, comprising:
acquiring an initial sample set comprising a plurality of motion trajectories;
downsampling the initial sample set to obtain an updated sample set;
taking the motion trail in the updated sample set as a target motion trail, and acquiring predicted position information and fitting position information of the target motion trail at the next moment; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm;
Calculating displacement imitation information according to the adjacent motion trail of the target motion trail;
calculating a degree of similarity between the predicted position information and the displacement mimicking information;
determining a movement constraint of the predicted position information based on the adjacent motion trajectories;
combining the predicted position information, the fitted position information, the displacement mimicking information, the degree of similarity, and the movement constraint into first training data;
and training by using the first training data to obtain a first prediction model.
16. The method of claim 15, wherein obtaining predicted location information for a next time of the target motion profile comprises:
and obtaining a prediction result generated by the first track prediction model aiming at the target motion track as the prediction position information.
17. The method as recited in claim 16, further comprising:
calculating the position deviation of the target motion trail and the adjacent motion trail;
the position deviation, the target motion trail and the adjacent motion trail are combined into second training data;
and training by using the second training data to obtain a second prediction model.
18. The method of claim 15, wherein downsampling the initial sample set to obtain an updated sample set comprises:
and enabling the track sparseness of the initial sample set to be equivalent to the track sparseness of a preset test scene.
19. A motion trajectory prediction apparatus, comprising:
the acquisition module is used for acquiring predicted position information and fitting position information of the target motion trail at the next moment;
the first calculation module is used for calculating displacement imitation information according to the adjacent motion trail of the target motion trail;
a second calculation module for calculating a degree of similarity between the predicted position information and the displacement mimicking information;
a determining module for determining a movement constraint of the predicted position information based on the adjacent motion trajectories;
and the prediction module is used for inputting the predicted position information, the fitting position information, the displacement imitation information, the similarity degree and the movement constraint into a second track prediction model so that the second track prediction model predicts again to obtain new predicted position information of the next moment of the target movement track.
20. A model training device, comprising:
the sample construction module is used for acquiring an initial sample set comprising a plurality of motion tracks; downsampling the initial sample set to obtain an updated sample set; taking the motion trail in the updated sample set as a target motion trail, and acquiring predicted position information and fitting position information of the target motion trail at the next moment; calculating displacement imitation information according to the adjacent motion trail of the target motion trail; calculating a degree of similarity between the predicted position information and the displacement mimicking information; determining a movement constraint of the predicted position information based on the adjacent motion trajectories; combining the predicted position information, the fitted position information, the displacement mimicking information, the degree of similarity, and the movement constraint into first training data; the prediction position information is obtained through prediction of a first track prediction model; obtaining fitting position information through a curve fitting algorithm;
and the model training module is used for training by utilizing the first training data to obtain a first prediction model.
21. An electronic device, comprising:
A memory for storing a computer program;
a processor for executing the computer program to implement the method of any one of claims 1 to 18.
22. A readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the method of any one of claims 1 to 18.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410275992.2A CN117874529B (en) | 2024-03-12 | Motion trail prediction method, model training method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410275992.2A CN117874529B (en) | 2024-03-12 | Motion trail prediction method, model training method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117874529A true CN117874529A (en) | 2024-04-12 |
| CN117874529B CN117874529B (en) | 2024-05-28 |
Family
ID=
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140002617A1 (en) * | 2012-06-27 | 2014-01-02 | The Board Of Trustees Of The University Of Illinois | Particle tracking system and method |
| CN114022847A (en) * | 2021-11-23 | 2022-02-08 | 清华大学 | Intelligent agent trajectory prediction method, system, equipment and storage medium |
| US20220144265A1 (en) * | 2019-07-27 | 2022-05-12 | Huawei Technologies Co., Ltd. | Moving Track Prediction Method and Apparatus |
| CN114559951A (en) * | 2022-03-16 | 2022-05-31 | 北京三快在线科技有限公司 | Obstacle trajectory prediction method, obstacle trajectory prediction device, storage medium, and electronic device |
| CN114967465A (en) * | 2022-06-02 | 2022-08-30 | 乐聚(深圳)机器人技术有限公司 | Trajectory planning method and device, electronic equipment and storage medium |
| CN115204044A (en) * | 2022-07-11 | 2022-10-18 | 阿里巴巴（中国）有限公司 | Method, apparatus and medium for generating trajectory prediction model and processing trajectory information |
| WO2022222095A1 (en) * | 2021-04-22 | 2022-10-27 | 深圳元戎启行科技有限公司 | Trajectory prediction method and apparatus, and computer device and storage medium |
| CN115375887A (en) * | 2022-08-25 | 2022-11-22 | 苏州浪潮智能科技有限公司 | Moving target trajectory prediction method, device, equipment and medium |
| WO2023010827A1 (en) * | 2021-08-02 | 2023-02-09 | 北京三快在线科技有限公司 | Migration scene-based trajectory prediction model training method and apparatus |
| CN115905793A (en) * | 2022-10-27 | 2023-04-04 | 山东浪潮新基建科技有限公司 | Track reproduction method based on Internet of things and curve fitting |
| WO2023051390A1 (en) * | 2021-09-30 | 2023-04-06 | 华为技术有限公司 | Trajectory estimation model training method, trajectory estimation method, and device |
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140002617A1 (en) * | 2012-06-27 | 2014-01-02 | The Board Of Trustees Of The University Of Illinois | Particle tracking system and method |
| US20220144265A1 (en) * | 2019-07-27 | 2022-05-12 | Huawei Technologies Co., Ltd. | Moving Track Prediction Method and Apparatus |
| WO2022222095A1 (en) * | 2021-04-22 | 2022-10-27 | 深圳元戎启行科技有限公司 | Trajectory prediction method and apparatus, and computer device and storage medium |
| WO2023010827A1 (en) * | 2021-08-02 | 2023-02-09 | 北京三快在线科技有限公司 | Migration scene-based trajectory prediction model training method and apparatus |
| WO2023051390A1 (en) * | 2021-09-30 | 2023-04-06 | 华为技术有限公司 | Trajectory estimation model training method, trajectory estimation method, and device |
| CN114022847A (en) * | 2021-11-23 | 2022-02-08 | 清华大学 | Intelligent agent trajectory prediction method, system, equipment and storage medium |
| CN114559951A (en) * | 2022-03-16 | 2022-05-31 | 北京三快在线科技有限公司 | Obstacle trajectory prediction method, obstacle trajectory prediction device, storage medium, and electronic device |
| CN114967465A (en) * | 2022-06-02 | 2022-08-30 | 乐聚(深圳)机器人技术有限公司 | Trajectory planning method and device, electronic equipment and storage medium |
| CN115204044A (en) * | 2022-07-11 | 2022-10-18 | 阿里巴巴（中国）有限公司 | Method, apparatus and medium for generating trajectory prediction model and processing trajectory information |
| CN115375887A (en) * | 2022-08-25 | 2022-11-22 | 苏州浪潮智能科技有限公司 | Moving target trajectory prediction method, device, equipment and medium |
| CN115905793A (en) * | 2022-10-27 | 2023-04-04 | 山东浪潮新基建科技有限公司 | Track reproduction method based on Internet of things and curve fitting |
Non-Patent Citations (1)
| Title |
|---|
| 方岚;于凤芹;: "自适应在线判别外观学习的分层关联多目标跟踪", 中国图象图形学报, no. 04, 15 April 2020 (2020-04-15) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Angah et al. | Tracking multiple construction workers through deep learning and the gradient based method with re-matching based on multi-object tracking accuracy | |
| CN113264066B (en) | Obstacle track prediction method and device, automatic driving vehicle and road side equipment | |
| US20210148727A1 (en) | Simulating diverse long-term future trajectories in road scenes | |
| CN110221600B (en) | Path planning method and device, computer equipment and storage medium | |
| CN110610510A (en) | Target tracking method and device, electronic equipment and storage medium | |
| US11783588B2 (en) | Method for acquiring traffic state, relevant apparatus, roadside device and cloud control platform | |
| KR20210076886A (en) | Parking model generation method and device, electronic device and storage medium | |
| CN109598744A (en) | A kind of method, apparatus of video tracking, equipment and storage medium | |
| CN113221677A (en) | Track abnormity detection method and device, road side equipment and cloud control platform | |
| CN111931720B (en) | Method, apparatus, computer device and storage medium for tracking image feature points | |
| CN114139637B (en) | Multi-agent information fusion method and device, electronic equipment and readable storage medium | |
| JP2022028870A (en) | Lane detection method, apparatus, electronic device, storage medium, and vehicle | |
| CN111652181B (en) | Target tracking method and device and electronic equipment | |
| CN103679792A (en) | Rendering method and system for three-dimensional models | |
| CN114913386A (en) | Training method of multi-target tracking model and multi-target tracking method | |
| Zhang et al. | Extended social force model‐based mean shift for pedestrian tracking under obstacle avoidance | |
| CN114987546A (en) | Training method, device and equipment of trajectory prediction model and storage medium | |
| CN110807379A (en) | Semantic recognition method and device and computer storage medium | |
| US20230245429A1 (en) | Method and apparatus for training lane line detection model, electronic device and storage medium | |
| Qi et al. | A DNN-based object detection system on mobile cloud computing | |
| Kong et al. | A multi-context representation approach with multi-task learning for object counting | |
| CN117874529B (en) | Motion trail prediction method, model training method, device, equipment and medium | |
| CN109446437B (en) | Information mining method, device, server and storage medium | |
| CN117874529A (en) | Motion trail prediction method, model training method, device, equipment and medium | |
| US20220390256A1 (en) | Crowd-driven mapping, localization and social-friendly navigation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |