CN114817430A - Trajectory data processing method, model training method and device and automatic driving vehicle - Google Patents

Abstract

The disclosure provides a track data processing method, which relates to the technical field of artificial intelligence, in particular to the technical field of automatic driving, deep learning and computer vision. The specific implementation scheme is as follows: according to the current track information of the target obstacle, determining intention information and semantic relation information of the target obstacle, wherein the semantic relation information is used for representing the relation between the target obstacle and at least one object; and determining target track information of the target obstacle according to the current track information, the intention information and the semantic relation information. The disclosure also provides a training method and device of the deep learning model, electronic equipment, storage medium and an automatic driving vehicle.

Description

Trajectory data processing method, model training method and device and automatic driving vehicle

Technical Field

The present disclosure relates to the field of artificial intelligence technology, and more particularly, to the field of automated driving, deep learning, and computer vision technology. More specifically, the disclosure provides a trajectory data processing method, a training method and device of a deep learning model, an electronic device, a storage medium and an automatic driving vehicle.

Background

The autonomous vehicle may sense the surroundings through a sensing component (e.g., a sensor) to obtain the surrounding environment data. And combining the surrounding environment data with the map navigation data, and processing the data based on an artificial intelligence technology to make a driving decision. And finally, completing automatic driving of the automatic driving vehicle according to the driving decision through a control and execution system.

Disclosure of Invention

The disclosure provides a trajectory data processing method, a training method and device of a deep learning model, electronic equipment, a storage medium and an automatic driving vehicle.

According to an aspect of the present disclosure, there is provided a trajectory data processing method, including: according to current track information of a target obstacle, determining intention information and semantic relation information of the target obstacle, wherein the semantic relation information is used for representing the relation between the target obstacle and at least one object; and determining target track information of the target obstacle according to the current track information, the intention information and the semantic relation information.

According to an aspect of the present disclosure, there is provided a training method of a deep learning model, the deep learning model including a first sub-model and a second sub-model, the method including: according to sample track information of a target obstacle and the first sub-model, determining intention information and semantic relation information of the target obstacle, wherein the semantic relation information is used for representing the relation between the target obstacle and at least one object; inputting the sample track information, the intention information and the semantic relation information into the second submodel, and determining output track information of the target obstacle; and training the deep learning model according to the output track information and the track label of the sample track information.

According to another aspect of the present disclosure, there is provided a trajectory data processing apparatus including: the system comprises a first determination module, a second determination module and a third determination module, wherein the first determination module is used for determining intention information and semantic relation information of a target obstacle according to current track information of the target obstacle, and the semantic relation information is used for representing the relation between the target obstacle and at least one object; and a second determining module, configured to determine target trajectory information of the target obstacle according to the current trajectory information, the intention information, and the semantic relationship information.

According to another aspect of the present disclosure, there is provided a training apparatus for a deep learning model, the deep learning model including a first sub-model and a second sub-model, the apparatus including: a third determining module, configured to determine intention information and semantic relationship information of a target obstacle according to sample trajectory information of the target obstacle and the first sub-model, where the semantic relationship information is used to represent a relationship between the target obstacle and at least one object; a fourth determining module, configured to input the sample trajectory information, the intention information, and the semantic relationship information into the second submodel, and determine output trajectory information of the target obstacle; and the training module is used for training the deep learning model according to the output track information and the track label of the sample track information.

According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method provided in accordance with the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a method provided according to the present disclosure.

According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements a method provided according to the present disclosure.

According to another aspect of the present disclosure, an autonomous vehicle is provided that includes an electronic device provided by the present disclosure.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become readily apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram of an exemplary system architecture to which the trajectory data processing method and apparatus may be applied, according to one embodiment of the present disclosure;

FIG. 2 is a flow diagram of a trajectory data processing method according to one embodiment of the present disclosure;

FIG. 3 is a flow diagram of a trajectory data processing method according to another embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a trajectory data processing method according to one embodiment of the present disclosure;

FIG. 5 is a flow diagram of a method of training a deep learning model according to one embodiment of the present disclosure;

FIG. 6 is a schematic diagram of a training method of a deep learning model according to one embodiment of the present disclosure;

FIG. 7 is a schematic diagram of a second submodel according to one embodiment of the present disclosure;

FIG. 8 is a schematic diagram of a first submodel according to one embodiment of the present disclosure;

FIG. 9 is a block diagram of a trajectory data processing device according to one embodiment of the present disclosure;

FIG. 10 is a block diagram of a training apparatus for deep learning models according to one embodiment of the present disclosure; and

FIG. 11 is a block diagram of an electronic device to which a trajectory data processing method and/or a training method of a deep learning model may be applied, according to one embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The driving track of the obstacle can be determined by utilizing a large-scale deep learning model. After the deep learning model is trained based on the training data set of the scene A, the trained deep learning model can accurately determine the driving track of the obstacle under the scene A. However, in scenario B, the trained deep learning model needs to be trained again to improve the accuracy of the model in scenario B.

FIG. 1 is a schematic diagram of an exemplary system architecture to which the trajectory data processing method and apparatus may be applied, according to one embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, a system architecture 100 according to this embodiment may include sensors 101, 102, 103, a network 104, and a server 105. Network 104 is used to provide a medium for communication links between sensors 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired and/or wireless communication links, and so forth.

The sensors 101, 102, 103 may interact with a server 105 over a network 104 to receive or send messages, etc.

The sensors 101, 102, 103 may be functional elements integrated on the autonomous vehicle 106, such as infrared sensors, ultrasonic sensors, millimeter wave radar, information acquisition devices, and the like. The sensors 101, 102, 103 may be used to collect status data of obstacles around the autonomous vehicle 106 and surrounding road data.

The server 105 may also be integrated on the autonomous vehicle 106, but is not limited to this, and may also be disposed at a remote end capable of establishing communication with a vehicle-mounted terminal, and may be embodied as a distributed server cluster composed of a plurality of servers, or may be embodied as a single server.

The server 105 may be a server that provides various services. For example, a map application, a data processing application, and the like may be installed on the server 105. Taking the server 105 running the data processing class application as an example: the state data of the obstacle and the road data transmitted from the sensors 101, 102, 103 are received via the network 104. One or more of the state data of the obstacle and the road data may be used as the data to be processed. And processing the data to be processed to obtain target data.

It should be noted that the trajectory data processing method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the trajectory data processing device provided by the embodiment of the present disclosure may also be disposed in the server 105. But is not limited thereto. The trajectory data processing method provided by the embodiments of the present disclosure may also be generally performed by the sensor 101, 102, or 103. Accordingly, the trajectory data processing device provided by the present disclosure may also be disposed in the sensor 101, 102, or 103.

It should be understood that the number of sensors, networks, and servers in fig. 1 is merely illustrative. There may be any number of sensors, networks, and servers, as desired for implementation.

It should be noted that the sequence numbers of the respective operations in the following methods are merely used as a representation of the operations for description, and should not be construed as representing the execution order of the respective operations. The method need not be performed in the exact order shown, unless explicitly stated.

FIG. 2 is a flow diagram of a trajectory data processing method according to one embodiment of the present disclosure.

As shown in fig. 2, the method 200 may include operations S210 to S220.

In operation S210, intention information and semantic relationship information of the target obstacle are determined according to current trajectory information of the target obstacle.

For example, the semantic relationship information is used to characterize a relationship between the target obstacle and the at least one object.

For example, the target obstacle may be an obstacle that is sensed by any sensor on the autonomous vehicle 106. In one example, the obstacle may include a vehicle, a pedestrian, or the like.

For example, the object may include an obstacle. In one example, as described above, the obstacle may include a vehicle, a pedestrian, or the like.

Also for example, the object may include a feature. In one example, the surface features may include lane lines, signs, and the like.

For example, the current trajectory information may include geographic coordinates of the target obstacle at the current time. For another example, the current trajectory information may further correspond to a plurality of geographic coordinates of the target obstacle within a preset time period, and each geographic coordinate may correspond to a time within the preset time period.

For example, the intention information and semantic relationship information of the target obstacle may be determined in various ways. In one example, the intention information Inten _1 of the vehicle C _1 may be, for example, a cut-in, taking as an example that the target obstacle is the vehicle C _1 perceived by the autonomous vehicle 106. In one example, the semantic relationship information Sr _1 of the vehicle C _1 may be that the vehicle C _1 will overtake from the left side of the vehicle C _2, for example.

In operation S220, target trajectory information of the target obstacle is determined according to the current trajectory information, the intention information, and the semantic relationship information.

For example, the target trajectory information may include the geographical coordinates of the target obstacle at the next time instant. In one example, from the current trajectory information, intent information Inten _1, and semantic relationship information Sr _1 of vehicle C _1, the geographic coordinates of vehicle C _1 at the next time may be determined so that autonomous vehicle 106, which is aware of vehicle C _1, determines whether to adjust the travel route based on the geographic coordinates of vehicle C _1 at the next time.

Through the embodiment of the disclosure, the trajectory information of the target obstacle can be accurately determined based on the current trajectory information, the intention information and the semantic relation information.

In some embodiments, the target obstacle described above may also be the autonomous vehicle 106 itself.

In some embodiments, determining intent information and semantic relationship information for the target obstacle from the current trajectory information for the target obstacle comprises: extracting the characteristics of the current track information to obtain semantic characteristic information; and determining intention information and semantic relation information of the target obstacle according to the semantic feature information.

For example, various deep learning models may be utilized to perform feature extraction on the current trajectory information. For another example, the various deep learning models may be, for example, a semantic segmentation model, an object detection model, and so on, and the disclosure is not limited herein.

For example, intention information and semantic relationship information of the target obstacle may be determined from semantic feature information using a GNN (Graph Neural Network) model. For example, intention information and semantic relationship information of the target obstacle may be specified from the semantic feature information using a Graph Transformer model or a GCN (Graph Convolutional Network) model.

One way of determining the adjacency matrix for the GNN model, GCN model, or graph Transformer model is described below in conjunction with fig. 3.

Fig. 3 is a flowchart of a trajectory data processing method according to another embodiment of the present disclosure.

As shown in fig. 3, the method 310 may determine intention information and semantic relationship information of the target obstacle according to the current trajectory information of the target obstacle, which will be described in detail with reference to operations S311 to S314.

In operation S311, at least one target object having a distance from the target obstacle less than or equal to a preset distance threshold is determined according to the position information of the target obstacle.

For example, the position information of the target obstacle may be the geographical coordinates of the target obstacle at the current time described above.

As another example, the preset distance threshold may be, for example, 20 meters. In one example, objects within 20 meters of the vicinity of the target obstacle may be targeted.

In operation S312, at least one piece of edge relation information is obtained according to the target obstacle and the at least one target object.

For example, an edge connecting the target obstacle with each target object may be established, resulting in each edge relationship information.

In operation S313, an adjacency matrix is obtained according to the at least one edge relation information.

For example, an adjacency matrix may be determined based on the target obstacle, the at least one target object, and the at least one edge relation information.

In operation S314, intention information and semantic relationship information are determined according to the current trajectory information and the adjacency matrix.

For example, the current trajectory information is input to the GNN model that performs data processing based on the adjacency matrix, and intention information and semantic relationship information are determined.

In some embodiments, determining the target trajectory information from the current trajectory information, the intent information, and the semantic relationship information comprises: determining hidden layer characteristic information according to the current track information; determining input characteristic information according to the intention information and the semantic relation information; and determining target track information according to the hidden layer characteristic information and the input characteristic information. As will be described in detail below with reference to fig. 4.

FIG. 4 is a schematic diagram of a trajectory data processing method according to another embodiment of the present disclosure.

As shown in fig. 4, semantic feature information 402 may be input into a first sub-model 410, resulting in intent information and semantic relationship information for a target obstacle. In one example, the semantic feature information 402 is obtained by feature extraction of the current trajectory information 401.

As shown in fig. 4, the current trajectory information 401, the intention information, and the semantic relationship information may be input into the second submodel 420 to determine target trajectory information 403 of the target obstacle.

For example, the first submodel 410 may be, for example, the GNN model described above.

For example, the second sub-model 420 may be, for example, a LSTM (Long Short-Term Memory) model.

In one example, the LSTM model includes a plurality of LSTM cells. The current track information 401 may be taken as input X _ i to the ith LSTM unit. The ith LSTM unit may determine the ith hidden layer feature information h _ i according to the current track information 401. i is an integer greater than or equal to. In this embodiment, i may be 1.

Next, the intent information and semantic relationships may be fused to determine the input feature information X _ i + 1. The (i + 1) th LSTM unit can determine the (i + 1) th hidden layer feature information according to the input feature information X _ i +1 and the (i) th hidden layer feature information h _ i. In one example, the target trajectory information 403 may be determined based on the (i + 1) th hidden layer feature information.

FIG. 5 is a flow diagram of a method of training a deep learning model according to another embodiment of the present disclosure.

In embodiments of the present disclosure, the deep learning model may include a first sub-model and a second sub-model.

For example, the first sub-model may be one of a GNN model, a GCN model, or a graph Transformer model.

As another example, the second sub-model may be an LSTM model.

As shown in fig. 5, the method 500 includes operations S510 to S530.

In operation S510, intention information and semantic relationship information of the target obstacle are determined according to the sample trajectory information of the target obstacle and the first submodel.

For example, the semantic relationship information is used to characterize a relationship between the target obstacle and the at least one object.

In operation S520, the sample trajectory information, the intention information, and the semantic relationship information are input to the second submodel, and output trajectory information of the target obstacle is determined.

It is understood that operations S510 and S520 in the method 500 are the same as or similar to operations S210 and S220 in the method 200, and the detailed description of the disclosure is omitted here.

In operation S530, a deep learning model is trained according to the output trajectory information and the trajectory labels of the sample trajectory information.

For example, the difference between the output trajectory information and the trajectory labels may be calculated using a MSE (Mean Square Error) loss function. And adjusting parameters of the first sub-model and the second sub-model according to the difference so as to train the deep learning model.

Through the embodiment of the disclosure, the track information of the target obstacle can be accurately determined by the deep learning model obtained through training.

Through the embodiment of the disclosure, the deep learning model comprises two sub-models, and the track information of the target obstacle can be rapidly and accurately determined in different scenes. For example, after the deep learning model is trained based on the training data set of scene a, the trained deep learning model can accurately determine the driving track of the obstacle in the scene a.

In the scene B, the parameters of the first sub-model in the trained deep learning model can be kept unchanged, and the second sub-model is trained by using the training data based on the scene B, so as to improve the accuracy of the deep learning model in the scene B and reduce the time cost required for training the model.

In some embodiments, determining intent information and semantic relationship information for the target obstacle from the sample trajectory information and the first submodel for the target obstacle comprises: carrying out feature extraction on the sample track information to obtain semantic feature information; and inputting the semantic feature information into the first sub-model, and determining the ideogram information and the semantic relation information.

For example, various deep learning models may be utilized to perform feature extraction on the current trajectory information. For another example, the various deep learning models may be, for example, a semantic segmentation model, an object detection model, and so on, and the disclosure is not limited herein.

For example, the GNN model may be used to determine the semantic information and semantic relationship information of the target obstacle from the semantic feature information. For another example, intention information and semantic relation information of the target obstacle may be determined from the semantic feature information using a graph Transformer model or a GCN model.

In some embodiments, determining intent information and semantic relationship information for the target obstacle from the sample trajectory information and the first submodel for the target obstacle comprises: determining at least one target object with the distance to the target obstacle smaller than or equal to a preset distance threshold according to the position information of the target obstacle; obtaining at least one piece of side relation information according to the target barrier and at least one target object; obtaining an adjacency matrix according to at least one edge relation information; and determining intention information and semantic relation information according to the sample track information and a first sub-model for data processing based on the adjacent matrix. It is understood that the manner of determining the intention information and the semantic relationship information in the present embodiment is the same as or similar to that of the method 310, and the details of the present disclosure are not repeated herein.

In some embodiments, the second submodel includes a plurality of deep learning units, the sample trajectory information, the intent information, and the semantic relationship information are input to the second submodel, and determining the output trajectory information includes: inputting sample track information into an ith deep learning unit of a second submodel, and determining hidden layer characteristic information, wherein i is an integer greater than or equal to 1; determining input characteristic information according to the intention information and the semantic relation information; and inputting the hidden layer feature information and the input feature information into the (i + 1) th deep learning unit to determine output track information.

In some embodiments, training the deep learning model according to the trajectory labels of the output trajectory information and the sample trajectory information includes: and adjusting parameters of the first sub-model and the second sub-model according to the output track information and the track label so as to train the deep learning model. As will be described in detail below with reference to fig. 6.

FIG. 6 is a schematic diagram of a trajectory data processing method according to another embodiment of the present disclosure.

As shown in fig. 6, the deep learning model 600 may include a first sub-model 610 and a second sub-model 620.

The semantic feature information 602 may be input into the first sub-model 610, resulting in the semantic information and semantic relationship information of the target obstacle. In one example, the semantic feature information 602 may be obtained by performing feature extraction on the sample trajectory information 601.

As shown in fig. 6, the sample trajectory information 601, the intent information, and the semantic relationship information may be input into a second submodel 620 to determine output trajectory information 603 for the target obstacle.

For example, the first submodel 610 may be, for example, a GNN model.

For example, the second sub-model 620 may be, for example, an LSTM model.

In one example, a difference value 605 between the trajectory information 603 and the trajectory tag 604 may be output using an MSE loss function. Parameters of the first sub-model 610 and the second sub-model 620 are adjusted according to the difference value 605 to train the deep learning model 600.

For example, the second submodel 620 includes a plurality of deep learning units. In one example, as described above, the second sub-model 620 may be, for example, an LSTM model, and accordingly, the deep learning unit may be, for example, an LSTM unit.

In one example, the sample track information 601 may be taken as the input X _ i of the ith LSTM unit. The ith LSTM unit determines the ith hidden layer characteristic information h _ i according to the sample track information 601. i is an integer greater than or equal to 1. In this embodiment, i may be 1.

Next, the intent information and semantic relationships may be fused to determine the input feature information X _ i + 1. The (i + 1) th LSTM unit determines the (i + 1) th hidden layer feature information according to the input feature information X _ i +1 and the ith hidden layer feature information h _ i. In one example, the output trajectory information 603 may be determined based on the (i + 1) th hidden layer feature information.

FIG. 7 is a schematic diagram of a second submodel according to another embodiment of the present disclosure

As shown in FIG. 7, the second submodel 720 may include I LSTM cells. In one example, the LSTM unit is the deep learning unit described above. I is an integer greater than or equal to I. In this embodiment, I may be 4.

For example, the sample trajectory information may correspond to a plurality of geographic coordinates within a preset sample period. The sample trajectory information may include I sample trajectory sub-information, each sample trajectory sub-information corresponding to a plurality of geographic coordinates within one sample sub-period. Accordingly, for a target obstacle within each sample sub-period, the following operations may be performed: and determining at least one target object with the distance to the target obstacle smaller than or equal to a preset distance threshold according to the position information of the target obstacle. And obtaining at least one piece of side relation information according to the target obstacle and the at least one target object. And obtaining an adjacency matrix corresponding to each sample sub-period according to the at least one edge relation information. In one example, for each sample subinterval, the position information of a moment is selected from each sample subinterval as the position information of the target obstacle in each sample subinterval.

In addition, feature extraction can be performed on each sample sub-information to obtain each semantic feature information. In one example, feature extraction may be performed on the 1 st sample sub-information to obtain the 1 st semantic feature information. Similarly, the 2 nd semantic feature information to the I th semantic feature information may be obtained.

The first sub-model may process each semantic feature information based on an adjacency matrix corresponding to each sample sub-period, determining each intent information and each semantic relationship information. In one example, the first sub-model may process the 1 st semantic feature information based on an adjacency matrix corresponding to the 1 st sample sub-period, determining the 1 st intention information and the 1 st semantic relationship information. Similarly, 2 nd intention information through I th intention information may be determined. Similarly, the 2 nd semantic relation information to the I th semantic relation information may also be determined.

For another example, the sample track information may be used as the input feature information X _1 of the LSTM unit 721 to determine the 1 st hidden layer feature information h _ 1. The 1 st intention information and the 1 st semantic relation information are fused, and the obtained 1 st fusion information can be used as the input characteristic information X _2 of the LSTM unit 722. The input feature information X _2 and the 1 st hidden layer feature information h _1 are input to the LSTM unit 722 to determine the 2 nd hidden layer feature information h _ 2.

Next, the 2 nd intention information and the 2 nd semantic relation information are fused, and the obtained 2 nd fused information may be used as the input feature information X _3 of the LSTM unit 723. The input feature information X _3 and the 2 nd hidden layer feature information h _2 are input to the LSTM unit 723 to determine the 3 rd hidden layer feature information h _ 3. Similarly, the I-1 st intention information and the I-1 st semantic relation information can be fused, and the obtained I-1 st fusion information can be used as the input characteristic information X _ I of the LSTM unit 724. The input feature information X _ I and the I-1 th hidden layer feature information are input to the LSTM unit 724 to determine the I-th hidden layer feature information h _ I.

The output track information may be determined according to the I-th hidden layer feature information h _ I.

In another example, the output trajectory information may be determined according to the 1 st to I-th hidden layer feature information h _1 to h _ I.

In some embodiments, the semantic feature information is input into the first sub-model, and determining the intent information and the semantic relationship information further comprises: inputting semantic feature information into a first sub-model, and determining sample intention information and sample semantic relation information, wherein the sample track information has intention labels and semantic relation labels; determining a first loss value according to the sample intention information and the intention label; determining a second loss value according to the sample semantic relation information and the semantic relation label; and training the first sub-model according to the first loss value and the second loss value to obtain a pre-trained first sub-model. This will be described in detail below with reference to fig. 8.

FIG. 8 is a schematic diagram of a first submodel according to one embodiment of the present disclosure.

As shown in fig. 8, semantic feature information 802 is input into a first sub-model 810, and sample intent information 811 and sample semantic relationship information 812 are determined. For example, the semantic feature information 802 is obtained by extracting features from the sample trajectory information. The sample trajectory information has an intent tag 813 and a semantic relationship tag 814.

From the sample intent information 811 and the intent tag 813, a first loss value 815 is determined. From the sample semantic relationship information 812 and the semantic relationship labels 814, a second penalty value 816 is determined. The first submodel 810 is trained based on the first loss value 815 and the second loss value 816 to obtain a pre-trained first submodel. In one example, the first loss value 815 and the second loss value 816 may be added to obtain an overall loss value, and the first sub-model 810 may be pre-trained according to the overall loss value.

FIG. 9 is a block diagram of a trajectory data processing device according to one embodiment of the present disclosure.

As shown in fig. 9, the apparatus 900 may include a first determination module 910 and a second determination module 920.

A first determining module 910, configured to determine intention information and semantic relationship information of a target obstacle according to current trajectory information of the target obstacle. For example, the semantic relationship information is used to characterize a relationship between the target obstacle and at least one object.

A second determining module 920, configured to determine target trajectory information of the target obstacle according to the current trajectory information, the intention information, and the semantic relationship information.

In some embodiments, the first determining module comprises: the first feature extraction submodule is used for extracting features of the current track information to obtain semantic feature information; and the first determining submodule is used for determining intention information and semantic relation information of the target obstacle according to the semantic feature information.

In some embodiments, the first determining module comprises: the second determining submodule is used for determining at least one target object of which the distance to the target obstacle is smaller than or equal to a preset distance threshold according to the position information of the target obstacle; the first obtaining submodule is used for obtaining at least one piece of side relation information according to the target barrier and the at least one target object; a second obtaining submodule, configured to obtain an adjacency matrix according to the at least one piece of edge relation information; and a third determining submodule, configured to determine the intention information and semantic relationship information according to the current trajectory information and the adjacency matrix.

In some embodiments, the second determining module comprises: the fourth determining submodule is used for determining hidden layer characteristic information according to the current track information; a fifth determining submodule, configured to determine input feature information according to the intention information and the semantic relationship information; and the sixth determining submodule is used for determining target track information according to the hidden layer characteristic information and the input characteristic information.

FIG. 10 is a block diagram of a training apparatus for deep learning models according to another embodiment of the present disclosure.

As shown in fig. 10, the apparatus 1000 may include a third determination module 1010, a fourth determination module 1020, and a training module 1030.

The deep learning model includes a first sub-model and a second sub-model.

A third determining module 1010, configured to determine intention information and semantic relationship information of a target obstacle according to the sample trajectory information of the target obstacle and the first submodel. For example, the semantic relationship information is used to characterize a relationship between the target obstacle and at least one object.

A fourth determining module 1020, configured to input the sample trajectory information, the intention information, and the semantic relationship information into the second sub-model, and determine output trajectory information of the target obstacle.

A training module 1030, configured to train the deep learning model according to the output trajectory information and the trajectory labels of the sample trajectory information.

In some embodiments, the third determining module comprises: the second feature extraction submodule is used for extracting features of the sample track information to obtain semantic feature information; and a seventh determining submodule for inputting the semantic feature information into the first submodel and determining the intention information and the semantic relation information.

In some embodiments, the third determining module comprises: the eighth determining submodule is used for determining at least one target object of which the distance to the target obstacle is smaller than or equal to a preset distance threshold according to the position information of the target obstacle; a third obtaining submodule, configured to obtain at least one piece of edge relation information according to the target obstacle and the at least one target object; a fourth obtaining submodule, configured to obtain an adjacency matrix according to the at least one piece of edge relation information; and a ninth determining submodule, configured to determine the intention information and the semantic relationship information according to the sample trajectory information and the first submodel that performs data processing based on the adjacency matrix.

In some embodiments, the second submodel comprises a plurality of deep learning units, the fourth determining module comprising: a tenth determining submodule, configured to input the sample trajectory information into an ith deep learning unit of the second submodel, and determine hidden layer feature information, where i is an integer greater than or equal to 1; an eleventh determining submodule, configured to determine input feature information according to the intention information and the semantic relationship information; and a twelfth determining submodule, configured to input the hidden layer feature information and the input feature information into an (i + 1) th deep learning unit, and determine output trajectory information.

In some embodiments, the training module comprises: and the adjusting submodule is used for adjusting the parameters of the first submodel and the second submodel according to the output track information and the track label so as to train the deep learning model.

In some embodiments, the seventh determination sub-module further comprises: a first determining unit, configured to input the semantic feature information into the first sub-model, and determine sample intention information and sample semantic relation information, where the sample trajectory information has an intention tag and a semantic relation tag; a second determining unit for determining a first loss value according to the sample intention information and the intention label; a third determining unit, configured to determine a second loss value according to the sample semantic relationship information and the semantic relationship label; and the training unit is used for training the first submodel according to the first loss value and the second loss value to obtain a pre-trained first submodel.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 11 illustrates a schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 11, the device 1100 comprises a computing unit 1101, which may perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)1102 or a computer program loaded from a storage unit 1108 into a Random Access Memory (RAM) 1103. In the RAM 1103, various programs and data necessary for the operation of the device 1100 may also be stored. The calculation unit 1101, the ROM 1102, and the RAM 1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.

A number of components in device 1100 connect to I/O interface 1105, including: an input unit 1106 such as a keyboard, a mouse, and the like; an output unit 1107 such as various types of displays, speakers, and the like; a storage unit 1108 such as a magnetic disk, optical disk, or the like; and a communication unit 1109 such as a network card, a modem, a wireless communication transceiver, and the like. A communication unit 1109 allows the device 1100 to exchange information/data with other devices via a computer network, such as an internet, and/or various telecommunication networks.

The computing unit 1101 can be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and the like. The calculation unit 1101 performs the respective methods and processes described above, such as a trajectory data processing method and/or a training method of a deep learning model. For example, in some embodiments, the trajectory data processing method and/or the training method of the deep learning model may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 1100 via ROM 1102 and/or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the trajectory data processing method and/or the training method of the deep learning model described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the trajectory data processing method and/or the training method of the deep learning model in any other suitable manner (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/acts specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

In some embodiments, the present disclosure also provides an autonomous vehicle comprising an electronic device provided by the present disclosure. For example, the autonomous vehicle may include an electronic device 1100, for example.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims (24)

1. A trajectory data processing method, comprising:

determining intention information and semantic relation information of a target obstacle according to current track information of the target obstacle, wherein the semantic relation information is used for representing the relation between the target obstacle and at least one object; and

and determining target track information of the target obstacle according to the current track information, the intention information and the semantic relation information.

2. The method of claim 1, wherein the determining intent information and semantic relationship information of the target obstacle from the current trajectory information of the target obstacle comprises:

extracting the characteristics of the current track information to obtain semantic characteristic information; and

and determining intention information and semantic relation information of the target obstacle according to the semantic feature information.

3. The method of claim 1, wherein the determining intent information and semantic relationship information of a target obstacle from current trajectory information of the target obstacle comprises:

determining at least one target object with the distance to the target obstacle smaller than or equal to a preset distance threshold according to the position information of the target obstacle;

obtaining at least one piece of side relation information according to the target obstacle and the at least one target object;

obtaining an adjacency matrix according to the at least one edge relation information; and

and determining the intention information and semantic relation information according to the current track information and the adjacency matrix.

4. The method of claim 1, wherein the determining target trajectory information from the current trajectory information, the intent information, and the semantic relationship information comprises:

determining hidden layer characteristic information according to the current track information;

determining input characteristic information according to the intention information and the semantic relation information; and

and determining target track information according to the hidden layer characteristic information and the input characteristic information.

5. A method of training a deep learning model, the deep learning model comprising a first sub-model and a second sub-model, the method comprising:

according to sample track information of a target obstacle and the first sub-model, determining intention information and semantic relation information of the target obstacle, wherein the semantic relation information is used for representing the relation between the target obstacle and at least one object;

inputting the sample track information, the intention information and the semantic relation information into the second submodel, and determining output track information of the target obstacle; and

and training the deep learning model according to the output track information and the track label of the sample track information.

6. The method of claim 5, wherein the determining intent information and semantic relationship information of a target obstacle from sample trajectory information of the target obstacle and the first submodel comprises:

extracting the characteristics of the sample track information to obtain semantic characteristic information; and

and inputting the semantic feature information into the first sub-model, and determining the intention information and the semantic relation information.

7. The method of claim 5, wherein the determining intent information and semantic relationship information of a target obstacle from sample trajectory information of the target obstacle and the first submodel comprises:

determining at least one target object with the distance to the target obstacle smaller than or equal to a preset distance threshold according to the position information of the target obstacle;

obtaining at least one piece of side relation information according to the target obstacle and the at least one target object;

obtaining an adjacency matrix according to the at least one edge relation information; and

and determining the intention information and the semantic relation information according to the sample track information and the first sub-model for data processing based on the adjacency matrix.

8. The method of claim 5, wherein the second submodel includes a plurality of deep learning units,

inputting the sample trajectory information, the intent information, and the semantic relationship information into the second submodel, and determining output trajectory information includes:

inputting the sample track information into an ith deep learning unit of the second submodel, and determining hidden layer feature information, wherein i is an integer greater than or equal to 1;

determining input characteristic information according to the intention information and the semantic relation information; and

and inputting the hidden layer feature information and the input feature information into an (i + 1) th deep learning unit to determine output track information.

9. The method of claim 5, wherein the training the deep learning model according to the trajectory labels of the output trajectory information and the sample trajectory information comprises:

and adjusting parameters of the first sub-model and the second sub-model according to the output track information and the track label so as to train the deep learning model.

10. The method of claim 6, wherein said entering the semantic feature information into the first submodel, determining the intent information and the semantic relationship information further comprises:

inputting the semantic feature information into the first sub-model, and determining sample intention information and sample semantic relation information, wherein the sample track information has intention labels and semantic relation labels;

determining a first loss value based on the sample intent information and the intent tag;

determining a second loss value according to the sample semantic relation information and the semantic relation label; and

and training the first submodel according to the first loss value and the second loss value to obtain a pre-trained first submodel.

11. A trajectory data processing device comprising:

the system comprises a first determination module, a second determination module and a third determination module, wherein the first determination module is used for determining intention information and semantic relation information of a target obstacle according to current track information of the target obstacle, and the semantic relation information is used for representing the relation between the target obstacle and at least one object; and

a second determining module, configured to determine target trajectory information of the target obstacle according to the current trajectory information, the intention information, and the semantic relationship information.

12. The apparatus of claim 11, wherein the first determining means comprises:

the first feature extraction submodule is used for extracting features of the current track information to obtain semantic feature information; and

and the first determining submodule is used for determining intention information and semantic relation information of the target obstacle according to the semantic feature information.

13. The apparatus of claim 11, wherein the first determining means comprises:

the second determining submodule is used for determining at least one target object of which the distance to the target obstacle is smaller than or equal to a preset distance threshold according to the position information of the target obstacle;

the first obtaining submodule is used for obtaining at least one piece of side relation information according to the target barrier and the at least one target object;

the second obtaining submodule is used for obtaining an adjacency matrix according to the at least one piece of edge relation information; and

and the third determining submodule is used for determining the intention information and the semantic relation information according to the current track information and the adjacency matrix.

14. The apparatus of claim 11, wherein the second determining means comprises:

the fourth determining submodule is used for determining hidden layer characteristic information according to the current track information;

a fifth determining submodule, configured to determine input feature information according to the intention information and the semantic relationship information; and

and the sixth determining submodule is used for determining target track information according to the hidden layer characteristic information and the input characteristic information.

15. An apparatus for training a deep learning model, the deep learning model comprising a first sub-model and a second sub-model, the apparatus comprising:

a third determining module, configured to determine intention information and semantic relationship information of a target obstacle according to sample trajectory information of the target obstacle and the first sub-model, where the semantic relationship information is used to represent a relationship between the target obstacle and at least one object;

a fourth determining module, configured to input the sample trajectory information, the intention information, and the semantic relationship information into the second submodel, and determine output trajectory information of the target obstacle; and

and the training module is used for training the deep learning model according to the output track information and the track label of the sample track information.

16. The apparatus of claim 15, wherein the third determining means comprises:

the second feature extraction submodule is used for extracting features of the sample track information to obtain semantic feature information; and

and the seventh determining submodule is used for inputting the semantic feature information into the first submodel and determining the intention information and the semantic relation information.

17. The apparatus of claim 15, wherein the third determining means comprises:

the eighth determining submodule is used for determining at least one target object of which the distance to the target obstacle is smaller than or equal to a preset distance threshold according to the position information of the target obstacle;

a third obtaining submodule, configured to obtain at least one piece of edge relation information according to the target obstacle and the at least one target object;

a fourth obtaining submodule, configured to obtain an adjacency matrix according to the at least one edge relation information; and

a ninth determining submodule, configured to determine the intention information and the semantic relationship information according to the sample trajectory information and the first sub-model that performs data processing based on the adjacency matrix.

18. The apparatus of claim 15, wherein the second submodel includes a plurality of deep learning units,

the fourth determining module includes:

a tenth determining submodule, configured to input the sample trajectory information into an ith deep learning unit of the second submodel, and determine hidden layer feature information, where i is an integer greater than or equal to 1;

an eleventh determining submodule, configured to determine input feature information according to the intention information and the semantic relationship information; and

and the twelfth determining submodule is used for inputting the hidden layer feature information and the input feature information into the (i + 1) th deep learning unit and determining output track information.

19. The apparatus of claim 15, wherein the training module comprises:

and the adjusting submodule is used for adjusting the parameters of the first submodel and the second submodel according to the output track information and the track label so as to train the deep learning model.

20. The apparatus of claim 16, wherein the seventh determination submodule further comprises:

the first determining unit is used for inputting the semantic feature information into the first sub-model and determining sample intention information and sample semantic relation information, wherein the sample track information has intention labels and semantic relation labels;

a second determining unit for determining a first loss value according to the sample intention information and the intention label;

a third determining unit, configured to determine a second loss value according to the sample semantic relationship information and the semantic relationship label; and

and the training unit is used for training the first sub-model according to the first loss value and the second loss value to obtain a pre-trained first sub-model.

21. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

22. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1 to 10.

23. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

24. An autonomous vehicle comprising the electronic device of claim 21.

Images