CN115984812A - Multitask prediction model training method, mobile equipment control method and device - Google Patents
Multitask prediction model training method, mobile equipment control method and device Download PDFInfo
- Publication number
- CN115984812A CN115984812A CN202211656583.4A CN202211656583A CN115984812A CN 115984812 A CN115984812 A CN 115984812A CN 202211656583 A CN202211656583 A CN 202211656583A CN 115984812 A CN115984812 A CN 115984812A
- Authority
- CN
- China
- Prior art keywords
- prediction
- task
- obstacle
- sequence
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
A multi-task prediction model training method, a mobile device control method and a mobile device control device are disclosed. The method comprises the following steps: acquiring a first sequence and a second sequence; generating obstacle prediction data corresponding to each of the plurality of prediction tasks via a multitask prediction network based on the first sequence and the second sequence; for each prediction task in a plurality of prediction tasks, determining a task loss value corresponding to the prediction task based on the obstacle prediction data corresponding to the prediction task and the corresponding loss calculation mode; determining a multitask loss value based on task loss values corresponding to the plurality of predicted tasks respectively; training a multi-task prediction network based on the multi-task loss value; and determining the trained multi-task prediction network as a multi-task prediction model in response to the fact that the trained multi-task prediction network meets the preset training end condition. The embodiment of the disclosure can improve the reliability of automatic driving.
Description
Technical Field
The disclosure relates to driving technologies, and in particular, to a multitask prediction model training method, and a control method and device for a mobile device.
Background
The application of the automatic driving technology to mobile devices such as vehicles is becoming more and more widespread, and in general, a vehicle performs driving control based on data collected by sensors such as a camera and a radar provided in the vehicle, thereby realizing automatic driving.
Disclosure of Invention
The automatic driving control method and the automatic driving control device are provided for solving the technical problem that the reliability of automatic driving is low at present. The embodiment of the disclosure provides a multi-task prediction model training method, a mobile device control method and a mobile device control device.
According to an aspect of the embodiments of the present disclosure, there is provided a method for training a multi-task prediction model, including:
obtaining a first sequence and a second sequence, the first sequence comprising: a local map of a local area corresponding to the first mobile device at each of a plurality of time instants, the second sequence comprising: at each of the plurality of time instants, obstacle information around the first mobile device;
generating obstacle prediction data corresponding to each of a plurality of prediction tasks via a multitask prediction network based on the first sequence and the second sequence;
determining a task loss value corresponding to each of the plurality of prediction tasks based on the obstacle prediction data corresponding to the prediction task and the corresponding loss calculation manner;
determining a multitask loss value based on task loss values corresponding to the plurality of predicted tasks respectively;
training the multitask prediction network based on the multitask loss value;
and determining the trained multi-task prediction network as a multi-task prediction model in response to the fact that the trained multi-task prediction network meets a preset training end condition.
According to another aspect of the embodiments of the present disclosure, there is provided a control method of a mobile device, including:
obtaining a third sequence and a fourth sequence, wherein the third sequence comprises: a local map of a local area corresponding to the second mobile device at each of a plurality of time instants, the fourth sequence comprising: at each of the plurality of time instants, obstacle information around the second mobile device;
generating obstacle prediction data corresponding to each of a plurality of prediction tasks via a multi-task prediction model based on the third sequence and the fourth sequence;
and performing travel control on the second mobile device based on obstacle prediction data corresponding to each of the plurality of prediction tasks.
According to still another aspect of the embodiments of the present disclosure, there is provided a multitask predictive model training device including:
a first obtaining module, configured to obtain a first sequence and a second sequence, where the first sequence includes: a local map of a local area corresponding to the first mobile device at each of a plurality of time instants, the second sequence comprising: at each of the plurality of time instants, obstacle information around the first mobile device;
a first generation module, configured to generate obstacle prediction data corresponding to each of the plurality of prediction tasks via a multi-task prediction network based on the first sequence and the second sequence acquired by the first acquisition module;
a first determining module, configured to determine, for each of the plurality of predicted tasks, a task loss value corresponding to the predicted task based on the obstacle prediction data corresponding to the predicted task and the corresponding loss calculation manner, which are generated by the first generation module;
a second determining module, configured to determine a multitask loss value based on the task loss values corresponding to the plurality of predicted tasks determined by the first determining module;
a training module, configured to train the multi-task prediction network based on the multi-task loss value determined by the second determining module;
and the third determining module is used for responding to the fact that the multi-task prediction network trained by the training module meets a preset training ending condition, and determining the trained multi-task prediction network as a multi-task prediction model.
According to still another aspect of the embodiments of the present disclosure, there is provided a control apparatus of a mobile device, including:
a second obtaining module, configured to obtain a third sequence and a fourth sequence, where the third sequence includes: a local map of a local area corresponding to a second mobile device at each of a plurality of time instants, the second sequence comprising: at each of the plurality of time instants, obstacle information around the second mobile device;
a second generation module, configured to generate obstacle prediction data corresponding to each of the plurality of prediction tasks via a multi-task prediction model based on the third sequence and the fourth sequence acquired by the second acquisition module;
and a control module configured to perform travel control on the second mobile device based on obstacle prediction data corresponding to each of the plurality of prediction tasks generated by the second generation module.
According to still another aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium storing a computer program for executing the above-described multitask predictive model training method or the above-described control method of a mobile device.
According to still another aspect of an embodiment of the present disclosure, there is provided an electronic apparatus including:
a processor;
a memory for storing the processor-executable instructions;
the processor is configured to read the executable instructions from the memory and execute the instructions to implement the multitask predictive model training method or the control method of the mobile device.
Based on the multitask prediction model training method, the mobile device control method, the device, the computer-readable storage medium and the electronic device provided by the embodiments of the disclosure, in the model training stage, obstacle prediction data corresponding to a plurality of prediction tasks may be generated via a multitask prediction network based on a first sequence composed of local maps corresponding to a plurality of times and a second sequence composed of obstacle information corresponding to a plurality of times, a task loss value corresponding to a plurality of prediction tasks may be obtained from the obstacle prediction data corresponding to the plurality of prediction tasks according to a loss calculation method corresponding to the plurality of prediction tasks, a multitask loss value may be determined according to the task loss value corresponding to the plurality of prediction tasks, and a multitask prediction model using a multitask framework may be obtained by using the multitask loss value in the training of the multitask prediction network. Therefore, in the use stage of the model, the multi-task prediction model can predict the relevant information of the obstacle from different dimensions through the execution of a plurality of prediction tasks, the prediction results of the different dimensions are used for the driving control of the mobile equipment, the data referred by the driving control of the mobile equipment are enriched, the prediction results of the different dimensions can complement each other, and therefore the reliability of automatic driving can be improved.
The technical solution of the present disclosure is further described in detail by the accompanying drawings and examples.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description of the embodiments of the present disclosure when taken in conjunction with the accompanying drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the principles of the disclosure and not to limit the disclosure. In the drawings, like reference numbers generally indicate like parts or steps.
Fig. 1 is a flowchart illustrating a multi-task predictive model training method according to an exemplary embodiment of the disclosure.
Fig. 2 is a flowchart illustrating a method for training a multi-task predictive model according to another exemplary embodiment of the disclosure.
Fig. 3 is a flowchart illustrating a multi-tasking predictive model training method according to still another exemplary embodiment of the disclosure.
FIG. 4 is a flowchart illustrating a method for training a multi-tasking predictive model according to yet another exemplary embodiment of the disclosure.
Fig. 5 is a flowchart illustrating a multi-tasking predictive model training method according to yet another exemplary embodiment of the disclosure.
Fig. 6 is a flowchart illustrating a multi-tasking predictive model training method according to yet another exemplary embodiment of the disclosure.
Fig. 7 is a flowchart illustrating a multi-tasking predictive model training method according to yet another exemplary embodiment of the disclosure.
Fig. 8 is a flowchart illustrating a control method of a mobile device according to an exemplary embodiment of the present disclosure.
Fig. 9 is a flowchart illustrating a control method of a mobile device according to another exemplary embodiment of the present disclosure.
Fig. 10 is a schematic structural diagram of a multi-task prediction model training apparatus according to an exemplary embodiment of the present disclosure.
Fig. 11 is a schematic structural diagram of a multitask predictive model training device according to another exemplary embodiment of the present disclosure.
Fig. 12 is a schematic structural diagram of a multi-task prediction model training device according to still another exemplary embodiment of the disclosure.
Fig. 13 is a schematic structural diagram of a multitask predictive model training device according to still another exemplary embodiment of the present disclosure.
Fig. 14 is a schematic structural diagram of a multitask predictive model training device according to still another exemplary embodiment of the present disclosure.
FIG. 15 is a schematic structural diagram of a device for training a multi-tasking predictive model according to still another exemplary embodiment of the disclosure.
Fig. 16 is a schematic structural diagram of a control apparatus of a mobile device according to an exemplary embodiment of the present disclosure.
Fig. 17 is a schematic structural diagram of a control apparatus of a mobile device according to another exemplary embodiment of the present disclosure.
Fig. 18 is a block diagram of an electronic device provided in an exemplary embodiment of the present disclosure.
Detailed Description
Hereinafter, example embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. It is to be understood that the described embodiments are merely a subset of the embodiments of the present disclosure and not all embodiments of the present disclosure, with the understanding that the present disclosure is not limited to the example embodiments described herein.
It should be noted that: the relative arrangement of parts and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
It will be understood by those of skill in the art that the terms "first," "second," and the like in the embodiments of the present disclosure are used merely to distinguish one element from another, and are not intended to imply any particular technical meaning, nor is the necessary logical order between them.
It is also understood that in embodiments of the present disclosure, "a plurality" may refer to two or more and "at least one" may refer to one, two or more.
It is also to be understood that any reference to any component, data, or structure in the embodiments of the disclosure, may be generally understood as one or more, unless explicitly defined otherwise or stated otherwise.
In addition, the term "and/or" in the present disclosure is only one kind of association relationship describing an associated object, and means that three kinds of relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" in the present disclosure generally indicates that the former and latter associated objects are in an "or" relationship.
It should also be understood that the description of the various embodiments of the present disclosure emphasizes the differences between the various embodiments, and the same or similar parts may be referred to each other, so that the descriptions thereof are omitted for brevity.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
The disclosed embodiments may be applied to electronic devices such as terminal devices, computer systems, servers, etc., which are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known terminal devices, computing systems, environments, and/or configurations that may be suitable for use with electronic devices, such as terminal devices, computer systems, servers, and the like, include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set top boxes, programmable consumer electronics, network pcs, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above systems, and the like.
Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.
Summary of the application
The automatic driving prediction system of the vehicle is an important component for realizing automatic driving of the vehicle, the automatic driving prediction system can predict the track of an obstacle on a road according to data collected by sensors such as a camera and a radar arranged on the vehicle, and the track prediction result can be used for driving control of the vehicle.
In the course of implementing the present disclosure, the inventors found that the trajectory prediction is reliable in the short-term prediction, but the error is very large in the medium-and long-term prediction, and therefore, the reliability of automatic driving is difficult to ensure by using the trajectory prediction result for the travel control of the vehicle.
Exemplary method
Fig. 1 is a flowchart illustrating a method for training a multi-task predictive model according to an exemplary embodiment of the disclosure. The method shown in fig. 1 includes step 110, step 120, step 130, step 140, step 150, and step 160, which are described below.
It should be noted that, in the embodiments of the present disclosure, the mobile device may be a vehicle, a train, or another device with a mobile function, and for convenience of understanding, the embodiments of the present disclosure are all described by taking the mobile device as a vehicle as an example.
It should be noted that the plurality of time instants involved in the embodiments of the present disclosure may be represented as N time instants; wherein N may be 5, 8, 10, 15 or other values, which are not listed here.
At each moment in N moments, the sensor arranged on the first mobile device can acquire data to obtain N acquired data sets in one-to-one correspondence with the N moments.
Optionally, the sensor disposed on the first mobile device includes, but is not limited to, a camera, a radar, a Global Positioning System (GPS), an Inertial Measurement Unit (IMU), a wheel speed meter, and the like.
After the acquisition data group corresponding to any moment is obtained, the positioning information acquired by the GPS can be extracted from the acquisition data group corresponding to the moment, a map area with a preset area size and with the position corresponding to the positioning information as the center is intercepted from the constructed high-precision map, and the intercepted map area can be used as a local map corresponding to the moment.
After the acquired data set corresponding to any time is obtained, the acquired data set corresponding to the time may be further processed by using an obstacle identification algorithm to obtain obstacle information around the first mobile device, where the obstacle information includes, but is not limited to, an obstacle position, an obstacle speed, an obstacle orientation, an obstacle distance (which may be a distance between an obstacle and the first mobile device), and the like.
In the above, the manner of obtaining the local map and the obstacle information corresponding to any time is discussed, and in this manner, for each time of the N times, the corresponding local map and the corresponding obstacle information may be determined, respectively, so as to obtain N local maps corresponding to the N times one by one and N obstacle information corresponding to the N times one by one. Arranging the N local maps according to the sequence of the corresponding time to form a first sequence in step 110; the second sequence in step 110 may be formed by arranging the N pieces of obstacle information in the order of the corresponding times.
It should be noted that, besides being obtained by using the GPS, the positioning information may also be obtained by calculating information acquired by a wheel speed meter or other sensors; the local map can be obtained by intercepting the high-precision map, and can also be obtained by real-time construction by utilizing a Simultaneous Localization and mapping (SLAM) technology.
Step 120 is to generate obstacle prediction data corresponding to each of the plurality of prediction tasks via the multitask prediction network based on the first sequence and the second sequence.
It should be noted that the multitask prediction network may be a prediction network to be trained, the multitask prediction network may have a plurality of prediction tasks, the plurality of prediction tasks may be represented as M prediction tasks, and each of the M prediction tasks may be an obstacle prediction task related to automatic driving; wherein M may be 2, 3, 4 or other values, which are not listed here.
In step 120, the first sequence and the second sequence may be provided as input data to a multitask prediction network, respectively, and the multitask prediction network may operate accordingly to generate M obstacle prediction data in one-to-one correspondence with the M prediction tasks.
Of course, in step 120, a certain fusion algorithm may also be adopted to fuse the information carried by the first sequence and the information carried by the second sequence, and provide the fusion result as input data to the multitask prediction network, where the multitask prediction network may perform operations according to the input data to generate M obstacle prediction data corresponding to the M prediction tasks one to one.
Optionally, a loss calculation method corresponding to each of the M prediction tasks may be preset; the loss calculation modes corresponding to different prediction tasks can be the same or different; the loss calculation mode corresponding to each prediction task can be characterized in the form of a loss function.
In step 130, for each of the M prediction tasks, a loss function for characterizing a loss calculation manner corresponding to the prediction task may be determined, obstacle prediction data corresponding to the prediction task is substituted into the loss function for calculation, and an obtained calculation result may be used as a task loss value corresponding to the prediction task.
In step 140, M task loss values corresponding to the M prediction tasks one to one may be summed, and the resulting summation result is used as a multitask loss value; alternatively, weights may be given to the M prediction tasks, respectively, to obtain M weights corresponding to the M prediction tasks one to one, and the obtained M weights may be used to weight (for example, weighted sum, weighted average, or the like) the M task loss values corresponding to the M prediction tasks one to one, and the obtained weighting result may be used as the multitask loss value.
And 150, training the multitask prediction network based on the multitask loss value.
In step 150, with reference to the multitask loss value, a gradient descent method (e.g., a random gradient descent method, a steepest gradient descent method, etc.) may be used to perform parameter update on the multitask prediction network to minimize the multitask loss value, so as to optimize parameters of the multitask prediction network, thereby implementing training on the multitask prediction network.
And step 160, in response to that the trained multi-task prediction network meets the preset training end condition, determining the trained multi-task prediction network as a multi-task prediction model.
It should be noted that, when training the multitask prediction network, a large number of sample data may be utilized, each sample data includes a first sequence and a second sequence, so that, for each sample data, the above step 110 to step 150 may be performed, and the process of performing the above step 110 to step 150 for each sample data may be regarded as an iterative process.
After a plurality of times of iterative processing, if the convergence of the trained multi-task prediction network is detected at a certain time, the trained multi-task prediction network can be judged to accord with the preset training end condition, and the trained multi-task prediction network can be directly determined as a multi-task prediction model at the moment.
Of course, the preset training end condition is not limited to this, for example, the trained multi-task prediction network may be determined to meet the preset training end condition when the number of iterations reaches the preset number.
Based on the multi-task prediction model training method provided by the above embodiments of the present disclosure, in the model training stage, the obstacle prediction data corresponding to each of the plurality of prediction tasks may be generated via the multi-task prediction network based on the first sequence composed of the local maps corresponding to each of the plurality of times and the second sequence composed of the obstacle information corresponding to each of the plurality of times, the task loss values corresponding to each of the plurality of prediction tasks may be obtained from the obstacle prediction data corresponding to each of the plurality of prediction tasks according to the loss calculation manner corresponding to each of the plurality of prediction tasks, the multi-task loss values may be determined according to the task loss values corresponding to each of the plurality of prediction tasks, and the multi-task prediction model using the multi-task loss values may be obtained by using the multi-task prediction network for training. Therefore, in the use stage of the model, the multi-task prediction model can predict the relevant information of the obstacle from different dimensions through the execution of a plurality of prediction tasks, the prediction results of the different dimensions are used for the driving control of the mobile equipment, the data referred by the driving control of the mobile equipment are enriched, the prediction results of the different dimensions can complement each other, and therefore the reliability of automatic driving can be improved.
On the basis of the embodiment shown in fig. 1, as shown in fig. 2, step 120 includes step 1201, step 1203, step 1205, step 1207, and step 1209.
Optionally, the first sequence may include: m local maps which are arranged in sequence can correspond to the same map coordinate system; the second sequence may include: n pieces of obstacle information are sequentially arranged, and the N pieces of obstacle information can correspond to the same image coordinate system; the transformation relationship between the map coordinate system and the image coordinate system may be predetermined, and the transformation relationship may be in the form of a transformation matrix.
In step 1201, for any obstacle information in the N obstacle information, the image coordinate system may be converted into the map coordinate system by using the conversion relationship to obtain an obstacle distribution map in the map coordinate system, where the obstacle distribution map may be used to represent the obstacle distribution in the local map corresponding to the obstacle information at the same time, for example, to represent which positions in the local map have obstacles distributed, and in this way, N obstacle distribution maps corresponding to the N local maps one to one may be obtained.
Optionally, the sizes of the local maps and the obstacle distribution maps may be uniform, for example, the heights and widths of the local maps and the obstacle distribution maps may be preset heights and widths; wherein the preset height can be represented as H1, and the preset width can be represented as W1.
In an alternative embodiment, step 1203, includes:
superposing the plurality of local maps in the first sequence and the barrier distribution maps corresponding to the plurality of local maps in the first sequence along the channel direction to obtain a first characteristic map;
carrying out down-sampling processing on the first feature map to obtain a second feature map;
the second feature map is converted into a first feature vector.
Assuming that the height and the width of each local map and each obstacle distribution map are H1 and W1, the height, the width and the number of dimensions of the first feature map obtained by the superposition processing of the N local maps and the N obstacle distribution maps along the channel direction may be H1, W1 and 2N, respectively.
Optionally, the N local maps may be located in the first N dimensions of the first feature map, and the N obstacle distribution maps may be located in the last N dimensions of the first feature map; alternatively, the N obstacle distribution maps may be located in the first N dimensions of the first feature map, and the N local maps may be located in the last N dimensions of the first feature map; alternatively, the N local maps and the N obstacle distribution maps may be distributed in a staggered manner in the dimension direction of the first feature map, for example, according to a rule of "local map-obstacle distribution map-local map-obstacle distribution map".
The multitasking predictive network may include: and a downsampling layer, wherein the downsampling layer processes the first feature map to realize feature map dimension reduction, so that a second feature map is obtained, the height of the second feature map can be H1, the width of the second feature map can be W1, and the number of dimensions of the second feature map can be 1.
The multitasking predictive network may further include: and the full connection layer is used for processing the second characteristic diagram to form a characteristic sequence with a certain length, the characteristic sequence can be in a one-dimensional vector form, and the characteristic sequence can be used as a first characteristic vector.
In this embodiment, the information carried by the first sequence and the information carried by the N obstacle maps can be fused efficiently and reliably by a simple processing method such as superimposition processing or downsampling processing, so that the first feature vector carrying the fusion result can be obtained.
Of course, in step 1203, fusion of the information carried by the first sequence and the information carried by the N obstacle maps may be implemented by using another information fusion algorithm.
In step 1205, the N pieces of obstacle information in the second sequence may be converted into feature vectors, and then the feature vectors are spliced by using a feature splicing algorithm, and the splicing result may be used as a second feature vector; alternatively, the second sequence may be directly converted into a third feature map (the conversion method may refer to the method for obtaining the first feature map mentioned above), and then the third feature map is converted into a feature vector, which may be used as the second feature vector.
Alternatively, the second feature vector may be a one-dimensional vector, and the length of the second feature vector may be the same as that of the first feature vector.
In step 1207, a feature stitching algorithm may be adopted to stitch the first feature vector and the second feature vector, and a stitching result may be used as a third feature vector.
Optionally, the multitask prediction network may include: an information fusion part and a prediction part, the information fusion part may include: in the downsampling layer and the fully-connected layer, the information fusion part may perform steps 1201 to 1207 to obtain a third feature vector, and provide the third feature vector to the prediction part, and the prediction part may predict the obstacle related information from different dimensions according to the third feature vector to generate M obstacle data corresponding to the M prediction tasks one to one.
In the embodiment of the disclosure, according to the first sequence and the second sequence, a feature vector (i.e., a first feature vector) with a certain length can be formed by encoding, and a native feature vector (i.e., a second feature vector) can be obtained at the same time, and a third feature vector capable of effectively embodying information carried by the first sequence and information carried by the second sequence can be generated by fusion splicing of the feature vector formed by encoding and the native feature vector, so that the multitask prediction network can refer to environmental information, obstacle information and the like around the first mobile device to predict from different dimensions, thereby obtaining rich prediction results.
In one optional example, the prediction part in the multitask prediction network may include: the method comprises a trajectory prediction network, a behavior prediction network and an effectiveness prediction network, wherein the plurality of prediction tasks can comprise: a track prediction task, a behavior prediction task and an effectiveness prediction task; the trajectory prediction network may be configured to execute the trajectory prediction task, the behavior prediction network may be configured to execute the behavior prediction task, and the effectiveness prediction network may be configured to execute the effectiveness prediction task.
Alternatively, the trajectory prediction network may also be referred to as a trajectory prediction head, the behavior prediction network may also be referred to as a behavior prediction head, and the effectiveness prediction network may also be referred to as an effectiveness prediction head.
Optionally, when calculating the task loss value, the loss calculation manners corresponding to the trajectory prediction task, the behavior prediction task, and the effectiveness prediction task may be different.
Based on the embodiment shown in fig. 1, as shown in fig. 3, step 120 includes step 1211, step 1213 and step 1215.
After the information fusion portion generates the third feature vector based on the first sequence and the second sequence, the information fusion portion may provide the third feature vector to the trajectory prediction network, and the trajectory prediction network may perform an operation based on this to generate obstacle prediction data corresponding to the trajectory prediction task.
Optionally, the obstacle data corresponding to the trajectory prediction task may include: the predicted track of the obstacle around the first mobile device and the confidence corresponding to the predicted track; the confidence corresponding to the predicted track can be used to characterize the credibility of the predicted track.
After the information fusion portion generates the third feature vector based on the first sequence and the second sequence, the information fusion portion may provide the third feature vector to the behavior prediction network, and the behavior prediction network may perform an operation based on this to generate obstacle data corresponding to the behavior prediction task.
In an alternative embodiment, step 1213 comprises at least one of:
generating, based on the first sequence and the second sequence, obstacle prediction data corresponding to a shift behavior prediction task of the plurality of prediction tasks via a shift behavior prediction network of the multitask prediction network;
and generating obstacle prediction data corresponding to a lane change behavior prediction task of the plurality of prediction tasks via a lane change behavior prediction network of the multitask prediction network based on the first sequence and the second sequence.
Here, the number of the behavior prediction networks may be two, and the behavior prediction networks are a speed change behavior prediction network and a lane change behavior prediction network, respectively, and accordingly, the number of the behavior prediction tasks may also be two, and the behavior prediction tasks are a speed change behavior prediction task and a lane change behavior prediction task, respectively, the speed change behavior prediction network may be configured to execute the speed change behavior prediction task, and the lane change behavior prediction network may be configured to execute the lane change behavior prediction task.
Alternatively, the obstacle prediction data corresponding to the shift behavior prediction task may include: probability vectors corresponding to the at least one speed change behavior respectively; wherein the at least one shift behavior may comprise at least one of: uniform velocity behavior, acceleration behavior, and deceleration behavior; the probability vector corresponding to any one of the gear shifting actions may be used to represent the probability of the gear shifting action occurring with an obstacle around the first mobile device.
Optionally, the obstacle data corresponding to the lane change behavior prediction task may include: probability vectors corresponding to the at least one lane change behavior respectively; wherein the at least one lane change behavior may comprise at least one of: a straight-ahead behavior, a left lane-changing behavior, and a right lane-changing behavior; the probability vector corresponding to any lane change behavior may be used to represent the probability of the lane change behavior occurring with an obstacle around the first mobile device.
Alternatively, in the calculation of the mission loss value, the loss calculation manner corresponding to the shifting behavior prediction mission and the loss calculation manner corresponding to the lane change behavior prediction mission may be the same.
In the implementation mode, the model training stage relates to two behavior prediction tasks, and in the model using stage, the multi-task prediction model can simultaneously predict the speed change behavior and the lane change behavior of the obstacle, and the prediction results of the speed change behavior and the lane change behavior are used for driving control, so that the reliability of automatic driving is improved.
In a specific implementation, the number of the behavior prediction networks may be only one, for example, only the speed change behavior prediction network or the lane change behavior prediction network, which is also feasible.
After the information fusion part generates the third feature vector based on the first sequence and the second sequence, the information fusion part may provide the third feature vector to the validity prediction network, and the validity prediction network may perform an operation based on this to generate obstacle prediction data corresponding to the validity prediction task.
Optionally, the obstacle prediction data corresponding to the effectiveness prediction task may include: for indicating a confidence level of whether an obstacle around the first mobile device is valid.
In the embodiment of the disclosure, the multi-task prediction network simultaneously comprises a track prediction network for executing the track prediction task, a behavior prediction network for executing the behavior prediction task and an effectiveness prediction network for executing the effectiveness prediction task, so that in the use stage of the model, the multi-task prediction model can reliably predict the track, the behavior and the effectiveness of the obstacle at the same time, and the prediction results of the dimensions are all used for driving control, thereby being beneficial to improving the reliability of automatic driving.
In one optional example, the obstacle prediction data corresponding to the trajectory prediction task includes: the mobile device comprises a plurality of predicted tracks of a target obstacle around a first mobile device, confidence degrees corresponding to the predicted tracks and probability distribution parameters corresponding to the predicted tracks, wherein the predicted tracks have correspondence with preset tracks of the target obstacle.
It should be noted that the target obstacle may be any obstacle around the first mobile device, and since the correlation calculation manner for each obstacle around the first mobile device is similar, only the correlation calculation manner for the target obstacle is described in detail herein.
For convenience of description, the plurality of predicted trajectories referred to in the embodiments of the present disclosure may be subsequently represented as K predicted trajectories, and accordingly, the plurality of preset trajectories referred to in the embodiments of the present disclosure may be represented as K preset trajectories.
Optionally, the probability distribution parameter corresponding to any trajectory may be a gaussian distribution parameter, such that the probability distribution parameter includes, but is not limited to, a mean, a variance, and the like; the K predicted trajectories may correspond to the K preset trajectories one to one.
On the basis of the embodiment shown in fig. 3, as shown in fig. 4, step 130 includes step 1301, step 1303, step 1305, step 1306, step 1307, step 1309, and step 1311.
Alternatively, the first preset time period after the plurality of time instants may be 3 seconds, 5 seconds, 10 seconds, 15 seconds, and the like after the N time instants, which are not listed again.
It should be noted that, in the model training phase, information labeling related to the obstacle may be performed in advance, for example, labeling a real track and a real behavior of the obstacle. In this way, in step 1301, the real trajectory of the target obstacle can be determined efficiently and reliably according to the labeled information.
And step 1303, selecting a first predicted track with the highest corresponding confidence coefficient from the multiple predicted tracks.
In step 1303, two confidences of the K confidences corresponding to the K predicted tracks one to one may be compared in magnitude to select a confidence with a maximum value from the K confidences, and the predicted track corresponding to the confidence may be used as the first predicted track.
In step 1305, similarity between each of the K preset trajectories and the real trajectory may be calculated to obtain K similarities corresponding to the K preset trajectories one to one, and a similarity with a maximum value is selected from the K similarities, where the preset trajectory corresponding to the similarity may be used as the target preset trajectory.
In step 1306, a second predicted trajectory corresponding to the target preset trajectory is selected from the plurality of predicted trajectories.
Due to the fact that the K predicted tracks and the K preset tracks have one-to-one correspondence, the predicted track corresponding to the target preset track can be determined, and the predicted track can be used as a second predicted track.
Optionally, in the model training stage, when information related to the obstacle is labeled, a corresponding probability distribution parameter may be determined for the real trajectory, and the probability distribution parameter corresponding to the real trajectory may be stored.
In step 1307, a probability distribution parameter corresponding to the first predicted trajectory may be extracted from the obstacle prediction data corresponding to the trajectory prediction task, a stored probability distribution parameter corresponding to the real trajectory is obtained, a difference between the two probability distribution parameters is evaluated by calculating a relative entropy (KL divergence) and the like with respect to the two probability distribution parameters, and the first loss value is determined according to the evaluated difference.
Alternatively, the first loss value may be positively correlated with the assessed difference. Assuming that the estimated difference is characterized by a value, the value can be directly used as a first loss value; or, a logarithm operation with a base number of a natural number e may be performed on the numerical value, and an obtained operation result is used as the first loss value; alternatively, the value may be multiplied by a preset coefficient greater than 0, and the resulting multiplication result may be used as the first loss value.
Alternatively, a negative correlation may be made between the second loss value and the confidence level corresponding to the second predicted trajectory. For example, the opposite of the confidence corresponding to the second predicted trajectory may be used as the second loss value; alternatively, a logarithm operation whose base is a natural number e may be performed on the confidence corresponding to the second predicted trajectory, and the inverse of the obtained operation result may be used as the second loss value.
In step 1311, the first loss value and the second loss value may be summed, and the obtained summation result is used as a task loss value corresponding to the trajectory prediction task; alternatively, the first loss value and the second loss value may be given weights, respectively, and the first loss value and the second loss value may be weighted (for example, weighted sum or weighted average) by the given weights, and the obtained weighting result may be used as the task loss value corresponding to the trajectory prediction task.
It should be noted that, the first loss value in the foregoing may be regarded as a trajectory regression loss, and the second loss value in the foregoing may be regarded as a trajectory confidence loss, so that the task loss value = trajectory regression loss + trajectory confidence loss corresponding to the trajectory prediction task.
In the embodiment of the disclosure, in the model training stage, the trajectory regression loss and the trajectory confidence loss can be efficiently and reliably calculated according to the obstacle prediction data corresponding to the trajectory prediction task, and by using the trajectory regression loss and the trajectory confidence loss, the task loss value corresponding to the trajectory prediction task can be efficiently and reliably calculated, so that the task loss value corresponding to the trajectory prediction task is used for training the multi-task prediction network.
In an alternative example, the obstacle prediction data corresponding to the behavior prediction task includes: the target obstacles around the first mobile device correspond to a plurality of predicted probability values of a plurality of preset behaviors.
For convenience of description, the multiple preset behaviors involved in the embodiments of the present disclosure may be subsequently represented as R preset behaviors, and accordingly, the multiple predicted probability values of the target obstacle corresponding to the multiple preset behaviors may be represented as R probability values, and the R probability values and the R preset behaviors may have a one-to-one correspondence relationship.
Based on the embodiment shown in fig. 3, as shown in fig. 5, step 130 includes step 1313, step 1315, step 1317 and step 1319.
And 1313, determining the real behaviors of the target obstacle in a second preset time period after the plurality of moments.
Alternatively, the second preset time period may be the same time period as the first preset time period described above.
It should be noted that, in the model training phase, information labeling related to the obstacle may be performed in advance, for example, labeling a real track and a real behavior of the obstacle. In this way, in step 1313, the actual behavior of the target obstacle can be determined efficiently and reliably based on the labeled information.
It should be noted that the real behavior may be a certain preset behavior among the R preset behaviors, in step 1315, a predicted probability value corresponding to the preset behavior may be selected from the R predicted probability values, and the predicted probability value may be used as a target predicted probability value.
Assume that a target prediction probability value among the R prediction probability values is denoted by x i Then, the following formula may be adopted to calculate the normalized value G corresponding to the target predicted probability value:
where exp () denotes an exponential function with the base natural number e, x i Representing the ith probability value, x, of the R prediction probability values j Representing the j-th probability value of the R predicted probability values.
Of course, the method for calculating the normalized value corresponding to the target predicted probability value is not limited to this, and those skilled in the art may also use other feasible normalization methods according to actual requirements to realize normalization processing of the target predicted probability value, so as to obtain the normalized value corresponding to the target predicted probability value.
In step 1319, the normalized value may be directly used as a task loss value corresponding to the behavior prediction task; or, a logarithm operation whose base is a natural number e may be performed on the normalized numerical value, and an obtained operation result may be used as a task loss value corresponding to the behavior prediction task.
In the embodiment of the disclosure, the task loss value corresponding to the behavior prediction task can be determined efficiently and reliably by screening the target prediction probability value matched with the real behavior and combining normalization processing.
In the case where the number of behavior prediction tasks is two, and the shift behavior prediction task and the lane change behavior prediction task are provided separately, the task loss value corresponding to the shift behavior prediction task and the task loss value corresponding to the lane change behavior prediction task may be calculated separately in the above manner.
In an alternative example, the obstacle prediction data corresponding to the effectiveness prediction task includes: a confidence level of a target obstacle around the first mobile device;
based on the embodiment shown in fig. 3, as shown in fig. 6, step 130 includes step 1321, step 1323, step 1325 and step 1327.
Alternatively, the designated numerical range may be (0, 1), and of course, the designated numerical range may also be (0, 5), (0, 10) or other numerical ranges, and for easy understanding, the case where the designated numerical range is (0, 1) is exemplified in the embodiments of the present disclosure.
Alternatively, after the confidence of the target obstacle is mapped to the designated value range, the mapped value may be compared with a preset value (e.g., 0.6, 0.7, or other set value), if the mapped value is greater than the preset value, the obstacle attribute of the target obstacle may be determined to be a valid attribute, and if the mapped value is less than or equal to the preset value, the obstacle attribute of the target obstacle may be determined to be an invalid attribute.
In specific implementation, the ratio of the mapping value to a preset value may also be calculated, the obtained ratio is compared with another preset value, and the obstacle attribute of the target obstacle is determined to be a valid attribute or an invalid attribute with reference to the comparison result.
And step 1325, in response to the obstacle attribute being the valid attribute, determining a task loss value corresponding to the validity prediction task based on the mapping value.
If the obstacle attribute is an effective attribute, the mapping value is referred, a task loss value corresponding to the effectiveness prediction task can be determined in a certain mode, and the task loss value corresponding to the effectiveness prediction task and the mapping value can be in positive correlation.
Optionally, the mapping value may be directly used as a task loss value corresponding to the validity prediction task; or, a logarithm operation with a base e may be performed on the mapping value, and an obtained operation result may be used as a task loss value corresponding to the validity prediction task.
Alternatively, the preset value may be 1.
If the obstacle attribute is an effective attribute, a task loss value corresponding to the effectiveness prediction task can be determined in a certain mode by referring to a difference value between a preset numerical value and a mapping value, and the task loss value corresponding to the effectiveness prediction task and the mapping value can be in negative correlation.
Optionally, a difference value between the preset numerical value and the mapping value may be directly used as a task loss value corresponding to the validity prediction task; or, a logarithm operation with a base e may be performed on the difference value between the preset value and the mapping value, and the obtained operation result may be used as a task loss value corresponding to the validity prediction task.
In the embodiment of the disclosure, the confidence of the target obstacle is mapped to the designated numerical interval, and the obstacle attribute of the target obstacle can be efficiently and reliably determined according to the obtained mapping value, so that the obstacle attribute can be calculated in an appropriate manner for two cases, namely an effective attribute and an invalid attribute, respectively, and thus the task loss value corresponding to the validity prediction task can be efficiently and reliably obtained.
In the above, various calculation manners of the task loss values are introduced, and it is assumed that the task loss value corresponding to the trajectory prediction task is referred to as the trajectory loss value for short, the task loss value corresponding to the behavior prediction task is referred to as the behavior loss value for short, and the task loss value corresponding to the validity prediction task is referred to as the validity loss value for short, then: multitask loss value = trajectory loss value + behavior loss value + effectiveness loss value.
In an alternative example, as shown in fig. 7, in the model training phase, a historical frame local map (equivalent to the first sequence in the above) and historical frame obstacle information (equivalent to the second sequence in the above) may be obtained first.
Next, the information carried by the local map of the historical frame and the information carried by the obstacle information of the historical frame may be fused by an information fusion part in the multitask prediction network to obtain the third feature vector above.
Then, the information fusion part can respectively provide the third feature vector to a track prediction network, a speed change behavior prediction network, a lane change behavior prediction network and an effectiveness prediction network, wherein the track prediction network can output a multi-mode track (corresponding to a plurality of predicted tracks in the above) and corresponding confidence coefficients and the like according to the third feature vector, the speed change behavior prediction network can output a speed change probability vector according to the third feature vector, the lane change behavior prediction network can output a lane change probability vector according to the third feature vector, and the effectiveness prediction network can output confidence coefficients representing whether the obstacle is effective or not according to the third feature vector. According to the output result of the track prediction network, a track loss value can be calculated, according to the output results of the speed change behavior prediction network and the lane change behavior prediction network, a behavior loss value can be calculated, according to the output result of the effectiveness prediction network, an effectiveness loss value can be calculated, according to the track loss value, the behavior loss value and the effectiveness loss value, a multitask loss value can be calculated, and the multitask loss value can be used for training of the multitask prediction network, so that a trained multitask prediction model can be obtained.
Any of the multi-tasking predictive model training methods provided by embodiments of the present disclosure may be performed by any suitable device having data processing capabilities, including but not limited to: terminal equipment, a server and the like. Alternatively, any of the multi-tasking predictive model training methods provided by the embodiments of the present disclosure may be executed by a processor, such as a processor that executes any of the multi-tasking predictive model training methods mentioned by the embodiments of the present disclosure by calling corresponding instructions stored in a memory. Which will not be described in detail below.
Fig. 8 is a flowchart illustrating a control method of a mobile device according to an exemplary embodiment of the present disclosure. The method shown in fig. 8 includes steps 810, 820 and 830, which are described below.
And a step 820 of generating obstacle prediction data corresponding to each of the plurality of prediction tasks via the multi-task prediction model based on the third sequence and the fourth sequence.
It should be noted that steps 810 to 820 are similar to steps 110 to 120 described above, and the difference is mainly that steps 810 to 820 are performed in the model using phase, and steps 110 to 120 are performed in the model training phase, and the specific implementation of steps 810 to 820 is as described above with reference to the description of steps 110 to 120, and will not be described herein again.
Step 830 is performed to control the second mobile device to travel based on the obstacle prediction data corresponding to each of the plurality of prediction tasks.
Optionally, the obstacle data corresponding to each of the plurality of prediction tasks may include at least two of: the trajectory prediction task comprises obstacle prediction data corresponding to the trajectory prediction task, obstacle prediction data corresponding to the speed change behavior prediction task, obstacle prediction data corresponding to the lane change behavior prediction task, and obstacle prediction data corresponding to the effectiveness prediction task.
In step 830, a driving strategy can be reasonably planned for the second mobile device by referring to the obstacle prediction data corresponding to each of the plurality of prediction tasks, and driving control can be performed on the second mobile device according to the planning result.
In one example, the second mobile device travels in a leftmost lane of the three lanes, and the second mobile device predicts that the right-front vehicle has an intention to lane change to the leftmost lane using the obstacle prediction data corresponding to the lane change behavior prediction task, and then the speed of the second mobile device may be reduced by controlling the second mobile device to avoid the second mobile device colliding with the right-front left-lane-changed vehicle.
In another example, the second mobile device travels in a middle lane of the three lanes, the second mobile device needs to change lane to the left to the leftmost lane, and the second mobile device predicts that the vehicle located at the rear left has an intention to accelerate using the obstacle prediction data corresponding to the shifting behavior prediction task, and then the second mobile device may be caused to temporarily change lane to the left by controlling the second mobile device so as to avoid the second mobile device colliding with the vehicle accelerated at the rear left.
In the embodiment of the disclosure, in the stage of using the model, through the execution of the plurality of prediction tasks, the multi-task prediction model can predict the relevant information of the obstacle from different dimensions, and the prediction results of the different dimensions are all used for the driving control of the mobile device, so that the data referred by the driving control of the mobile device is richer, and the prediction results of the different dimensions can be mutually supplemented, thereby improving the reliability of automatic driving.
In one optional example, one of the plurality of prediction tasks is an effectiveness prediction task;
based on the embodiment shown in fig. 8, as shown in fig. 9, step 830 includes step 8301, step 8303, step 8305 and step 8307.
In step 8301, the obstacle prediction data corresponding to each of the plurality of prediction tasks may be decoded by the decoding module to obtain an obstacle prediction result.
In one example, the obstacle prediction data corresponding to the trajectory prediction task includes: the predicted trajectories of an obstacle around the second mobile device and the confidence degrees corresponding to the predicted trajectories may be selected from the predicted trajectories, where the predicted trajectory may be a final predicted trajectory of the obstacle, and the obstacle prediction result may include: the predicted trajectory.
In another example, the obstacle prediction data corresponding to the shifting behavior prediction task includes: a first probability value corresponding to the uniform speed behavior, a second probability value corresponding to the acceleration behavior, and a third probability value corresponding to the deceleration behavior of an obstacle around the second mobile device may be obtained by selecting a probability value with a maximum value from the first probability value, the second probability value, and the third probability value, where the speed change behavior corresponding to the probability value may be used as a final speed change behavior of the obstacle, and the obstacle prediction result may include: this shifting behavior.
In yet another example, the obstacle prediction data corresponding to the effectiveness prediction task includes: the confidence of an obstacle around the second mobile device may be mapped to a designated value range, the obtained mapped value is compared with a preset value (e.g., 0.6, 0.7, or other set value), and an obstacle attribute of the obstacle is determined with reference to the comparison result, where the obstacle prediction result may include: the barrier property.
And 8303, post-processing the obstacle prediction result to obtain a post-processing result.
In step 8303, a driving strategy may be planned for the second mobile device with reference to the obstacle prediction result, and the planned driving strategy may be used as a post-processing result.
In the case where the second mobile device predicts the presence of an intention of the right front vehicle to lane change to the left-most lane using the obstacle prediction data corresponding to the lane change behavior prediction task in correspondence with the above that the second mobile device travels the leftmost lane among the three lanes, the travel strategy as a result of the post-processing may include: and (4) decelerating. If the obstacle attribute corresponding to the vehicle located in the front right direction in the obstacle prediction result is an invalid attribute, the deceleration in the driving maneuver may be filtered out, for example, the deceleration may be deleted from the driving maneuver.
Corresponding to the above case where the second mobile device travels in the middle lane of the three lanes, the second mobile device needs to change lane to the left-most lane, and there is an intention of acceleration of the vehicle located behind the left, the travel strategy as the post-processing result may include: the lane change to the left is suspended. If the obstacle attribute corresponding to the vehicle located at the left rear in the obstacle prediction result is an invalid attribute, the tentative left lane change in the driving strategy may be filtered out, and for example, the tentative left lane change may be deleted from the driving strategy.
Assuming that the deceleration in the driving strategy as a result of the post-processing is filtered out in step 8305, the second mobile device may be accelerated or maintained at a constant speed without performing a deceleration operation by the control of the second mobile device. Assuming that the suspension left lane change in the driving strategy as a result of the post-processing is filtered out in step 8305, the second mobile device may normally left lane change by the control of the second mobile device.
In the embodiment of the disclosure, through decoding processing, the obstacle prediction result can be efficiently and reliably obtained from the obstacle prediction data corresponding to each of the plurality of prediction tasks, after the post-processing result of the obstacle prediction result is obtained, the result corresponding to the effective prediction task in the obstacle prediction result can be referred to, and the effective part and the ineffective part in the post-processing result can be distinguished, so that the ineffective part can be filtered out, and only the effective part is used for driving control of the second mobile device, which is beneficial to reasonable driving control of the second mobile device.
Any of the control methods of the mobile device provided by the embodiments of the present disclosure may be performed by any suitable device having data processing capability, including but not limited to: terminal equipment, a server and the like. Alternatively, the control method of any mobile device provided by the embodiments of the present disclosure may be executed by a processor, for example, the processor may execute the control method of any mobile device mentioned in the embodiments of the present disclosure by calling a corresponding instruction stored in a memory. Which will not be described in detail below.
Exemplary devices
Fig. 10 is a schematic structural diagram of a multi-task prediction model training apparatus according to an exemplary embodiment of the present disclosure. The apparatus shown in fig. 10 includes a first acquisition module 1010, a first generation module 1020, a first determination module 1030, a second determination module 1040, a training module 1050, and a third determination module 1060.
A first obtaining module 1010, configured to obtain a first sequence and a second sequence, where the first sequence includes: a local map of a local area corresponding to the first mobile device at each of a plurality of time instants, the second sequence comprising: at each of a plurality of times, obstacle information around the first mobile device;
a first generating module 1020, configured to generate obstacle prediction data corresponding to each of the plurality of prediction tasks via a multitask prediction network based on the first sequence and the second sequence acquired by the first acquiring module 1010;
a first determining module 1030, configured to determine, for each of a plurality of predicted tasks, a task loss value corresponding to the predicted task based on the obstacle prediction data corresponding to the predicted task and the corresponding loss calculation manner, which are generated by the first generation model;
a second determining module 1040, configured to determine a multitask loss value based on a task loss value corresponding to each of the multiple predicted tasks determined by the first determining module 1030;
a training module 1050 configured to train the multitask prediction network based on the multitask loss value determined by the second determining module 1040;
the third determining module 1060 is configured to determine the trained multi-task prediction network as the multi-task prediction model in response to that the multi-task prediction network trained by the training module 1050 meets a preset training end condition.
In an alternative example, based on the embodiment shown in fig. 10, as shown in fig. 11, the first generating module 1020 includes:
a first generating sub-module 10201, configured to generate obstacle prediction data corresponding to a trajectory prediction task of the multiple prediction tasks via a trajectory prediction network of the multitask prediction network based on the first sequence and the second sequence acquired by the first acquiring module 1010;
a second generating sub-module 10203, configured to generate obstacle prediction data corresponding to a behavior prediction task of the multiple prediction tasks via a behavior prediction network of the multitask prediction network based on the first sequence and the second sequence acquired by the first acquiring module 1010;
a third generating sub-module 10205, configured to generate obstacle prediction data corresponding to a validity prediction task of the multiple prediction tasks via a validity prediction network of the multitask prediction network based on the first sequence and the second sequence acquired by the first acquiring module 1010.
In an optional example, the second generation submodule 10203 includes at least one of:
a first generation unit configured to generate obstacle prediction data corresponding to a shift behavior prediction task of the plurality of prediction tasks via a shift behavior prediction network of the multitask prediction network based on the first sequence and the second sequence acquired by the first acquisition module 1010;
a second generating unit, configured to generate obstacle prediction data corresponding to a lane change behavior prediction task of the multiple prediction tasks via a lane change behavior prediction network of the multitask prediction network based on the first sequence and the second sequence acquired by the first acquiring module 1010.
In an alternative example, the obstacle prediction data corresponding to the trajectory prediction task includes: the method comprises the steps that multiple predicted tracks of a target obstacle around first mobile equipment, confidence degrees corresponding to the multiple predicted tracks and probability distribution parameters corresponding to the multiple predicted tracks, wherein the multiple predicted tracks have correspondences with multiple preset tracks of the target obstacle;
on the basis of the embodiment shown in fig. 11, as shown in fig. 12, the first determining module 1030 includes:
a first determining submodule 10301 for determining a real trajectory of the target obstacle within a first preset time period after the plurality of time instants;
a first selecting sub-module 10303 configured to select, from the plurality of predicted tracks, a first predicted track with a maximum corresponding confidence;
a second selection sub-module 10305 configured to select, from the plurality of preset trajectories, a target preset trajectory closest to the real trajectory determined by the first determination sub-module 10301;
a third selecting submodule 10306 for selecting a second predicted trajectory corresponding to the target preset trajectory from the plurality of predicted trajectories;
a second determining submodule 10307, configured to determine a first loss value based on the probability distribution parameter corresponding to the first predicted track selected by the first selecting submodule 10303 and the probability distribution parameter corresponding to the real track determined by the first determining submodule 10301;
a third determining sub-module 10309, configured to determine a second loss value based on the confidence corresponding to the second predicted track selected by the third selecting sub-module 10306;
a fourth determining sub-module 10311, configured to determine a task loss value corresponding to the trajectory prediction task based on the first loss value determined by the second determining sub-module 10307 and the second loss value determined by the third determining sub-module 10309.
In an alternative example, the obstacle prediction data corresponding to the behavior prediction task includes: a plurality of predicted probability values of a plurality of preset behaviors corresponding to target obstacles around the first mobile device;
on the basis of the embodiment shown in fig. 11, as shown in fig. 13, the first determining module 1030 includes:
a fifth determining submodule 10313, configured to determine a true behavior of the target obstacle in a second preset time period after the multiple times;
a fourth selection submodule 10315, configured to select, from the multiple predicted probability values, a target predicted probability value that corresponds to the preset behavior and is matched with the real behavior determined by the fifth determination submodule 10313;
a sixth determining submodule 10317, configured to determine, based on the multiple predicted probability values, a normalized value corresponding to the target predicted probability value selected by the fourth selecting submodule 10315;
a seventh determining submodule 10319, configured to determine a task loss value corresponding to the behavior prediction task based on the normalized value determined by the sixth determining submodule 10317.
In an alternative example, the obstacle prediction data corresponding to the effectiveness prediction task includes: a confidence level of a target obstacle around the first mobile device;
on the basis of the embodiment shown in fig. 11, as shown in fig. 14, the first determining module 1030 includes:
a mapping submodule 10321, configured to map the confidence of the target obstacle to an assigned numerical value interval, so as to obtain a mapping value;
an eighth determining submodule 10323, configured to determine an obstacle attribute of the target obstacle based on the mapping value obtained by the mapping submodule 10321;
a ninth determining sub-module 10325, configured to determine, in response to the obstacle attribute determined by the eighth determining sub-module 10323 being a valid attribute, a task loss value corresponding to the validity prediction task based on the mapping value obtained by the mapping sub-module 10321;
a tenth determining sub-module 10327, configured to, in response to the obstacle attribute determined by the eighth determining sub-module 10323 being an invalid attribute, determine a task loss value corresponding to the validity prediction task based on a difference between the preset numerical value and the mapping value obtained by the mapping sub-module 10321.
In an alternative example, based on the embodiment shown in fig. 10, as shown in fig. 15, the first generating module 1020 includes:
an eleventh determining sub-module 10207, configured to determine, based on the second sequence acquired by the first acquiring module 1010, an obstacle distribution map corresponding to each of the multiple local maps in the first sequence acquired by the first acquiring module 1010;
a twelfth determining sub-module 10209, configured to determine a first feature vector based on the first sequence acquired by the first acquiring module 1010 and the obstacle distribution map corresponding to each of the multiple local maps in the first sequence determined by the eleventh determining sub-module 10207;
a thirteenth determining sub-module 10211, configured to determine a second feature vector based on the second sequence acquired by the first acquiring module 1010;
the splicing submodule 10213 is configured to splice the first eigenvector determined by the twelfth determining submodule 10209 and the second eigenvector determined by the thirteenth determining submodule 10211 to obtain a third eigenvector;
the fourth generating sub-module 10215 is configured to generate obstacle prediction data corresponding to each of the plurality of prediction tasks via the multi-task prediction network based on the third eigenvector obtained by the stitching sub-module 10213.
In an alternative example, the twelfth determination submodule 10209 includes:
the overlapping unit is configured to overlap, along the channel direction, the plurality of local maps in the first sequence acquired by the first acquiring module 1010 and the obstacle distribution maps corresponding to the plurality of local maps in the first sequence determined by the eleventh determining sub-module 10207, so as to obtain a first feature map;
the processing unit is used for carrying out downsampling processing on the first feature map obtained by the superposition unit to obtain a second feature map;
and the conversion unit is used for converting the second feature map into a first feature vector.
Fig. 16 is a schematic structural diagram of a control apparatus of a mobile device according to an exemplary embodiment of the present disclosure. The apparatus shown in fig. 16 includes a second obtaining module 1610, a second generating module 1620, and a control module 1630.
A second obtaining module 1610 configured to obtain a third sequence and a fourth sequence, where the third sequence includes: a local map of a local area corresponding to the second mobile device at each of a plurality of time instants, the second sequence comprising: at each of the plurality of times, obstacle information around the second mobile device;
a second generating module 1620, configured to generate, based on the third sequence and the fourth sequence acquired by the second acquiring module 1610, obstacle prediction data corresponding to each of the plurality of prediction tasks via a multi-task prediction model;
a control module 1630 configured to perform travel control on the second mobile device based on the obstacle prediction data corresponding to each of the plurality of prediction tasks generated by the second generation module 1620.
In an alternative example, one of the plurality of prediction tasks is a validity prediction task;
on the basis of the embodiment shown in fig. 16, as shown in fig. 17, the control module 1630 includes:
a decoding submodule 16301, configured to decode obstacle prediction data corresponding to each of the multiple prediction tasks generated by the second generation module 1620, to obtain an obstacle prediction result;
a processing submodule 16303, configured to perform post-processing on the obstacle prediction result obtained by the decoding submodule 16301 to obtain a post-processing result;
a filtering submodule 16305, configured to filter, based on a result corresponding to the validity prediction task in the obstacle prediction result obtained by the decoding submodule 16301, a result associated with an obstacle whose obstacle attribute is an invalid attribute in the post-processing result obtained by the processing submodule 16303;
the control sub-module 16307 is configured to perform driving control on the second mobile apparatus based on the filtered post-processing result obtained by the filtering sub-module 16305.
Exemplary electronic device
Next, an electronic apparatus according to an embodiment of the present disclosure is described with reference to fig. 18. The electronic device may be either or both of the first device and the second device, or a stand-alone device separate from them, which stand-alone device may communicate with the first device and the second device to receive the acquired input signals therefrom.
FIG. 18 illustrates a block diagram of an electronic device in accordance with an embodiment of the disclosure.
As shown in fig. 18, the electronic device 1800 includes one or more processors 1810 and memory 1820.
The processor 1810 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 1800 to perform desired functions.
In one example, the electronic device 1800 may further include: an input device 1830 and an output device 1840, which are interconnected by a bus system and/or other form of connection mechanism (not shown).
For example, when the electronic device is a first device or a second device, the input device 1830 may be the microphone or the microphone array described above for capturing an input signal of a sound source. When the electronic device is a stand-alone device, the input means 1830 may be a communication network connector for receiving the acquired input signals from the first device and the second device.
The input device 1830 may also include, for example, a keyboard, a mouse, and the like.
The output device 1840 may output various information including the determined distance information, direction information, and the like to the outside. The output devices 1840 may include, for example, a display, speakers, printer, and communication network and remote output devices connected thereto, among others.
Of course, for simplicity, only some of the components of the electronic device 1800 relevant to the present disclosure are shown in fig. 18, omitting components such as buses, input/output interfaces, and the like. In addition, the electronic device 1800 may include any other suitable components depending on the particular application.
Exemplary computer program product and computer-readable storage Medium
In addition to the above-described methods and apparatus, embodiments of the present disclosure may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the multi-tasking predictive model training method according to various embodiments of the present disclosure, or the steps in the control method of a mobile device according to various embodiments of the present disclosure, described in the "exemplary methods" section above of this specification.
The computer program product may write program code for carrying out operations for embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform steps in a multitask predictive model training method described in the "exemplary methods" section above in this specification, or steps in a control method for a mobile device according to various embodiments of the present disclosure.
The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The foregoing describes the general principles of the present disclosure in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present disclosure are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present disclosure. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the disclosure will be described in detail with reference to specific details.
The block diagrams of devices, apparatuses, devices, systems involved in the present disclosure are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
It is also noted that in the apparatus, devices, and methods of the present disclosure, various components or steps may be broken down and/or re-combined. These decompositions and/or recombinations are to be considered equivalents of the present disclosure.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the disclosure to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (14)
1. A multi-tasking predictive model training method, comprising:
obtaining a first sequence and a second sequence, the first sequence comprising: a local map of a local area corresponding to the first mobile device at each of a plurality of time instants, the second sequence comprising: at each of the plurality of times, obstacle information around the first mobile device;
generating obstacle prediction data corresponding to each of a plurality of prediction tasks via a multitask prediction network based on the first sequence and the second sequence;
determining a task loss value corresponding to each of the plurality of prediction tasks based on the obstacle prediction data corresponding to the prediction task and the corresponding loss calculation manner;
determining a multitask loss value based on task loss values corresponding to the plurality of prediction tasks respectively;
training the multitask prediction network based on the multitask loss value;
and determining the trained multi-task prediction network as a multi-task prediction model in response to the fact that the trained multi-task prediction network meets a preset training end condition.
2. The method of claim 1, wherein the generating, via a multitask prediction network, obstacle prediction data corresponding to each of a plurality of prediction tasks based on the first sequence and the second sequence comprises:
generating, via a trajectory prediction network of the multi-task prediction network, obstacle prediction data corresponding to a trajectory prediction task of the plurality of prediction tasks based on the first sequence and the second sequence;
generating, based on the first sequence and the second sequence, obstacle prediction data corresponding to behavior prediction tasks of the plurality of prediction tasks via a behavior prediction network of the multitask prediction network;
generating, based on the first sequence and the second sequence, obstacle prediction data corresponding to a validity prediction task of the plurality of prediction tasks via a validity prediction network of the multitask prediction network.
3. The method of claim 2, wherein the generating, via a behavior prediction network of the multi-tasking network, obstacle prediction data corresponding to behavior prediction tasks of the plurality of prediction tasks based on the first and second sequences comprises at least one of:
generating, based on the first sequence and the second sequence, obstacle prediction data corresponding to a shift behavior prediction task of the plurality of prediction tasks via a shift behavior prediction network of the multitask prediction network;
generating, based on the first sequence and the second sequence, obstacle prediction data corresponding to a lane change behavior prediction task of the plurality of prediction tasks via a lane change behavior prediction network of the multitask prediction network.
4. The method of claim 2, wherein the obstacle prediction data corresponding to the trajectory prediction task comprises: a plurality of predicted tracks of a target obstacle around the first mobile device, confidence degrees corresponding to the plurality of predicted tracks, and probability distribution parameters corresponding to the plurality of predicted tracks, wherein the plurality of predicted tracks have correspondence with a plurality of preset tracks of the target obstacle;
the determining, for each of the plurality of prediction tasks, a task loss value corresponding to the prediction task based on the obstacle prediction data corresponding to the prediction task and the corresponding loss calculation manner includes:
determining a real track of the target obstacle within a first preset time period after the plurality of moments;
selecting a first predicted track with the maximum corresponding confidence coefficient from the plurality of predicted tracks;
selecting a target preset track closest to the real track from the plurality of preset tracks;
selecting a second predicted track corresponding to the target preset track from the plurality of predicted tracks;
determining a first loss value based on the probability distribution parameter corresponding to the first predicted track and the probability distribution parameter corresponding to the real track;
determining a second loss value based on the confidence corresponding to the second predicted track;
and determining a task loss value corresponding to the track prediction task based on the first loss value and the second loss value.
5. The method of claim 2, wherein the obstacle prediction data corresponding to the behavior prediction task comprises: target obstacles around the first mobile device correspond to a plurality of predicted probability values of a plurality of preset behaviors;
the determining, for each of the plurality of prediction tasks, a task loss value corresponding to the prediction task based on the obstacle prediction data corresponding to the prediction task and the corresponding loss calculation manner includes:
determining a true behavior of the target obstacle within a second preset time period after the plurality of moments;
selecting a target prediction probability value matched with the real behavior corresponding to a preset behavior from the plurality of prediction probability values;
determining a normalized value corresponding to the target predicted probability value based on the plurality of predicted probability values;
and determining a task loss value corresponding to the behavior prediction task based on the normalized numerical value.
6. The method of claim 2, wherein the obstacle prediction data corresponding to the effectiveness prediction task comprises: a confidence level of a target obstacle around the first mobile device;
the determining, for each of the plurality of prediction tasks, a task loss value corresponding to the prediction task based on the obstacle prediction data corresponding to the prediction task and the corresponding loss calculation manner includes:
mapping the confidence coefficient of the target obstacle to a designated numerical value interval to obtain a mapping value;
determining an obstacle attribute of the target obstacle based on the mapped value;
in response to the obstacle attribute being a valid attribute, determining a task loss value corresponding to the validity prediction task based on the mapping value;
and in response to the fact that the obstacle attribute is an invalid attribute, determining a task loss value corresponding to the effectiveness prediction task based on a difference value between a preset numerical value and the mapping value.
7. The method of claim 1, wherein the generating, via a multitask prediction network, obstacle prediction data corresponding to each of a plurality of prediction tasks based on the first sequence and the second sequence comprises:
determining an obstacle distribution map corresponding to each of the plurality of local maps in the first sequence based on the second sequence;
determining a first feature vector based on the first sequence and the obstacle distribution maps corresponding to the local maps in the first sequence;
determining a second feature vector based on the second sequence;
splicing the first feature vector and the second feature vector to obtain a third feature vector;
and generating obstacle prediction data corresponding to each of the plurality of prediction tasks via a multitask prediction network based on the third feature vector.
8. The method of claim 7, wherein the determining a first feature vector based on the first sequence and the obstacle profiles corresponding to each of the plurality of local maps in the first sequence comprises:
superposing the plurality of local maps in the first sequence and the barrier distribution maps corresponding to the plurality of local maps in the first sequence along the channel direction to obtain a first characteristic map;
performing downsampling processing on the first feature map to obtain a second feature map;
and converting the second feature map into a first feature vector.
9. A control method of a mobile device, comprising:
obtaining a third sequence and a fourth sequence, wherein the third sequence comprises: a local map of a local area corresponding to the second mobile device at each of a plurality of time instants, the fourth sequence comprising: at each of the plurality of times, obstacle information around the second mobile device;
generating obstacle prediction data corresponding to each of a plurality of prediction tasks via a multi-task prediction model based on the third sequence and the fourth sequence;
and performing travel control on the second mobile device based on obstacle prediction data corresponding to each of the plurality of prediction tasks.
10. The method of claim 9, wherein one of the plurality of prediction tasks is a validity prediction task;
the driving control of the second mobile device based on the obstacle prediction data corresponding to each of the plurality of prediction tasks includes:
decoding the obstacle prediction data corresponding to the plurality of prediction tasks to obtain an obstacle prediction result;
post-processing the obstacle prediction result to obtain a post-processing result;
based on a result corresponding to the effectiveness prediction task in the obstacle prediction result, filtering a result associated with an obstacle of which the obstacle attribute is an ineffective attribute in the post-processing result;
and performing driving control on the second mobile equipment based on the filtered post-processing result.
11. A multitask predictive model training device comprising:
a first obtaining module, configured to obtain a first sequence and a second sequence, where the first sequence includes: a local map of a local area corresponding to the first mobile device at each of a plurality of time instants, the second sequence comprising: at each of the plurality of time instants, obstacle information around the first mobile device;
a first generation module, configured to generate obstacle prediction data corresponding to each of a plurality of prediction tasks via a multitask prediction network based on the first sequence and the second sequence acquired by the first acquisition module;
a first determining module, configured to determine, for each of the plurality of predicted tasks, a task loss value corresponding to the predicted task based on the obstacle prediction data corresponding to the predicted task and the corresponding loss calculation manner, which are generated by the first generation module;
a second determining module, configured to determine a multitask loss value based on the task loss values corresponding to the plurality of predicted tasks determined by the first determining module;
a training module, configured to train the multitask prediction network based on the multitask loss value determined by the second determining module;
and the third determining module is used for responding to the fact that the multi-task prediction network trained by the training module meets a preset training ending condition, and determining the trained multi-task prediction network as a multi-task prediction model.
12. A control apparatus of a mobile device, comprising:
a second obtaining module, configured to obtain a third sequence and a fourth sequence, where the third sequence includes: at each of a plurality of time instants, a local map of a local area corresponding to a second mobile device, the second sequence comprising: at each of the plurality of times, obstacle information around the second mobile device;
a second generation module, configured to generate obstacle prediction data corresponding to each of the plurality of prediction tasks via a multi-task prediction model based on the third sequence and the fourth sequence acquired by the second acquisition module;
and a control module configured to perform travel control on the second mobile device based on obstacle prediction data corresponding to each of the plurality of prediction tasks generated by the second generation module.
13. A computer-readable storage medium storing a computer program for executing the multitask predictive model training method according to any one of claims 1-8 or executing the control method for a mobile device according to any one of claims 9-10.
14. An electronic device, the electronic device comprising:
a processor;
a memory for storing the processor-executable instructions;
the processor is configured to read the executable instructions from the memory and execute the instructions to implement the multitask predictive model training method according to any one of claims 1-8, or execute the control method of the mobile device according to any one of claims 9-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211656583.4A CN115984812A (en) | 2022-12-22 | 2022-12-22 | Multitask prediction model training method, mobile equipment control method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211656583.4A CN115984812A (en) | 2022-12-22 | 2022-12-22 | Multitask prediction model training method, mobile equipment control method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115984812A true CN115984812A (en) | 2023-04-18 |
Family
ID=85975358
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211656583.4A Pending CN115984812A (en) | 2022-12-22 | 2022-12-22 | Multitask prediction model training method, mobile equipment control method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115984812A (en) |
-
2022
- 2022-12-22 CN CN202211656583.4A patent/CN115984812A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Grigorescu et al. | A survey of deep learning techniques for autonomous driving | |
| Zeng et al. | Dsdnet: Deep structured self-driving network | |
| CN113264066B (en) | Obstacle track prediction method and device, automatic driving vehicle and road side equipment | |
| US11100646B2 (en) | Future semantic segmentation prediction using 3D structure | |
| US20210148727A1 (en) | Simulating diverse long-term future trajectories in road scenes | |
| US11756309B2 (en) | Contrastive learning for object detection | |
| US20210103744A1 (en) | Spatio-temporal embeddings | |
| CN113537445B (en) | Track prediction method, device, equipment and storage medium | |
| CN111626219A (en) | Trajectory prediction model generation method and device, readable storage medium and electronic equipment | |
| CN112308006A (en) | Sight line area prediction model generation method and device, storage medium and electronic equipment | |
| US20200160060A1 (en) | System and method for multiple object tracking | |
| CN111428805A (en) | Method and device for detecting salient object, storage medium and electronic equipment | |
| CN111639591A (en) | Trajectory prediction model generation method and device, readable storage medium and electronic equipment | |
| Lange et al. | Lopr: Latent occupancy prediction using generative models | |
| WO2023051398A1 (en) | Security compensation method and apparatus, and storage medium and electronic device | |
| CN115984812A (en) | Multitask prediction model training method, mobile equipment control method and device | |
| CN116343169A (en) | Path planning method, target object motion control device and electronic equipment | |
| CN114332509B (en) | Image processing method, model training method, electronic device and automatic driving vehicle | |
| WO2022187251A1 (en) | Generating scene flow labels from point clouds using object labels | |
| Yun et al. | Self-configurable stabilized real-time detection learning for autonomous driving applications | |
| CN108944945B (en) | State prediction method and device for driving assistance, electronic equipment and vehicle | |
| CN116654022B (en) | Pedestrian track prediction method, system, equipment and medium based on multiple interactions | |
| Luo et al. | Dynamic simplex: Balancing safety and performance in autonomous cyber physical systems | |
| CN112462764B (en) | Vehicle constraint graph construction method and device | |
| Jeanpierre et al. | COACHES: An assistance Multi-Robot System in public areas |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |