WO2021146905A1

WO2021146905A1 - Deep learning-based scene simulator construction method and apparatus, and computer device

Info

Publication number: WO2021146905A1
Application number: PCT/CN2020/073469
Authority: WO
Inventors: 葛相辰
Original assignee: 深圳元戎启行科技有限公司
Priority date: 2020-01-21
Filing date: 2020-01-21
Publication date: 2021-07-29
Also published as: CN113490940A

Abstract

A deep learning-based scene simulator construction method, comprising: acquiring driving scene data and historical dangerous road condition scene data; extracting a plurality of road condition scene features from the driving scene data, and performing deep learning by using a deep learning algorithm according to the plurality of road condition scene features to obtain a scene simulation layer; extracting a plurality of dangerous road condition features from the historical dangerous road condition scene data, and training an initial adversarial network according to the plurality of dangerous road condition features to obtain a dangerous road condition layer; and continuously training a scene simulator by using the scene simulation layer and the dangerous road condition layer until a preset condition is satisfied, so as to obtain a trained scene simulator. The scene simulator is configured to generate a simulated driving scene when a simulation test is performed.

Description

Method, device and computer equipment for constructing scene simulator based on deep learning

Technical field

This application relates to a method, device and computer equipment for constructing a scene simulator based on deep learning.

Background technique

With the rapid development of Internet technology, driverless technology has also developed rapidly. In the process of autonomous driving development, it is usually necessary to build a simulation environment to simulate driving under various driving conditions. Through the automatic driving simulator, some road condition information can be randomly generated, such as simulation information such as lanes, vehicles, and pedestrians.

In the traditional way, some pre-set scenes are usually used to generate simulation test scenes, which are simulators for virtual simulation test of vehicles through road simulation tests or vehicle simulation software. However, the simulation test scenes generated by such simulators cannot truly simulate The real reaction of the vehicle in the corresponding environment and the low authenticity and reliability of the generated simulated driving scene result in low simulation test efficiency and low accuracy of test results.

Summary of the invention

According to various embodiments disclosed in the present application, a method, device and computer device for constructing a scene simulator based on deep learning are provided.

A method for constructing a scene simulator based on deep learning includes:

Obtain driving scene data and historical dangerous road condition data;

Extracting multiple road condition and scene features in the driving scene data, and using a deep learning algorithm to perform deep learning according to the multiple road condition and scene features to obtain a scene simulation layer;

Extracting multiple dangerous road condition features from historical dangerous road condition scene data, and training the initial confrontation network according to the multiple dangerous road condition features to obtain a dangerous road condition layer; and

The scene simulation layer and the dangerous road condition layer are used to continuously train the scene simulator until the preset conditions are met, and the trained scene simulator is obtained; the scene simulator is used to generate a simulated driving scene when performing a simulation test.

A scene simulator construction device based on deep learning, including:

Data acquisition module, used to acquire driving scene data and historical dangerous road condition data;

The scene simulation training module is used to extract a variety of road condition and scene features in the driving scene data, and use a deep learning algorithm to perform deep learning according to the multiple road condition and scene features to obtain a scene simulation layer;

The dangerous road condition training module is used to extract multiple dangerous road condition features from historical dangerous road condition scene data, and train the initial confrontation network according to the multiple dangerous road condition features to obtain the dangerous road condition layer; and

The scene simulator building module is used to use the scene simulation layer and the dangerous road condition layer to continuously train the scene simulator until the preset conditions are met, and the trained scene simulator is obtained; the scene simulator is used for simulation testing When generating simulated driving scenes.

A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the remote takeover-based vehicle control method provided in any one of the embodiments of the present application when the computer program is executed.

One or more non-volatile computer-readable storage media storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, one or more processors execute the readable storage medium to realize the present invention. Apply for the steps of the vehicle control method based on remote takeover provided in any of the embodiments.

The details of one or more embodiments of the present application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the description, drawings and claims.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. A person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

Fig. 1 is an application scene diagram of a method for constructing a scene simulator based on deep learning in one or more embodiments.

Fig. 2 is a schematic flowchart of a method for constructing a scene simulator based on deep learning in one or more embodiments.

Fig. 3 is a schematic flow chart of the steps of the simulation layer of the training scene according to one or more embodiments.

Fig. 4 is a schematic flowchart of the steps of training a dangerous road condition layer according to one or more embodiments.

Fig. 5 is a schematic flowchart of the steps of using a scene simulator to perform a test in another embodiment.

Fig. 6 is a block diagram of an apparatus for constructing a scene simulator based on deep learning in accordance with one or more embodiments.

Fig. 7 is a block diagram of a device for constructing a scene simulator based on deep learning in another embodiment.

Figure 8 is a block diagram of a computer device according to one or more embodiments.

Detailed ways

In order to make the technical solutions and advantages of the present application clearer, the following further describes the present application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

The method for constructing a scene simulator based on deep learning provided in this application can be applied to the application environment as shown in FIG. 1. The server 102 and the vehicle 104 communicate through a network. After the server 102 obtains the driving scene data and historical dangerous road condition scene data, it extracts various road condition scene features in the driving scene data, and uses deep learning algorithms to perform deep learning based on the various road condition scene characteristics to obtain a scene simulation layer; extract historical dangerous road condition scenes According to the characteristics of multiple dangerous road conditions in the data, the initial confrontation network is trained according to the characteristics of multiple dangerous road conditions to obtain the dangerous road condition layer. The server 102 uses the scene simulation layer and the dangerous road condition layer to continuously train the scene simulator until the preset conditions are met, and then the trained scene simulator is obtained. The server 102 then uses the scene simulator to generate a simulated driving scene when performing a simulation test on the vehicle 104. The server 102 may be implemented as an independent server or a server cluster composed of multiple servers, and the vehicle 104 may be various self-driving vehicles.

In one of the embodiments, as shown in FIG. 2, a method for constructing a scene simulator based on deep learning is provided. Taking the method applied to the server in FIG. 1 as an example for description, the method includes the following steps:

Step 202: Acquire driving scene data and historical dangerous road condition scene data.

The driving scene data may be a variety of road environment data collected in advance, for example, may include historical driving record data of the vehicle, such as road condition data collected by a driving recorder of the vehicle. Driving scene data includes a variety of road types and driving environment factors. For example, road types can include urban roads, dedicated roads, and rural roads; driving environment factors can include weather, air quality, temperature, noise, and lighting brightness. . The driving scene data also includes a variety of scene information, such as ground roads, lane lines, signal lights, landmarks, and traffic participants. Traffic participants can include passing vehicles, pedestrians, and moving paths.

The historical dangerous road condition scene data may be historical data of multiple types of dangerous road conditions collected from one or more platforms, and the historical dangerous road condition data may be road condition scene data in a real dangerous scene. The types of risk factors can include a variety of factors, such as roadblock factors, traffic rules factors, lane vehicle factors, pedestrian factors, environmental factors and other factors.

The server may obtain a large amount of driving scene data and historical dangerous road condition scene data from a local database or a third-party database in advance, so as to construct and train the scene simulator. Among them, the scene simulator may be a neural network model based on deep learning. The use of deep learning methods can form more abstract high-level representation attribute categories or features by combining low-level features to discover distributed feature representations of data, to build a neural network that simulates the human brain for analysis and learning, and explain it by imitating the mechanism of the human brain Data, such as images, sounds, text, etc. The scene simulator can also simulate the function of the hardware processor through software, so that the computer can simulate the environment of the hardware processor.

Step 204: Extract multiple road condition scene features in the driving scene data, and perform deep learning by using a deep learning algorithm according to the multiple road condition scene features to obtain a scene simulation layer.

After the server obtains a large amount of driving scene data and historical dangerous road condition scene data, it can first perform feature extraction on multiple road condition scenes in the driving scene data, and extract multiple road condition scene features in the driving scene data.

Specifically, the server may input driving scene data and historical dangerous road condition scene data into a pre-built initial neural network model, which may be constructed using a preset deep learning algorithm and neural network structure. The initial neural network includes multiple levels, such as a scene simulation level, a dangerous road condition learning level, and a dangerous road condition generation level.

The server uses the scene simulation layer to extract features from a variety of road conditions in the driving scene data, and extract a variety of road features, lane features, signal light features, landmark building features, pedestrian features, traffic vehicle features, and weather features. Traffic scene characteristics. The neural network corresponding to the scene simulation level then learns various road condition and scene features, and generates a scene simulation layer according to the learned various road condition and scene features. The scene simulation layer can then use the learned various road condition scene features to randomly generate a variety of corresponding models. For example, the scene simulation layer can automatically generate road models, models and other models, vehicle models, and pedestrian models that are included in the road condition scenes. Scene model.

Step 206: Extract a variety of dangerous road condition features in the historical dangerous road condition scene data, and train the initial confrontation network according to the multiple dangerous road condition features to obtain a dangerous road condition layer.

The server can also perform feature extraction on a large amount of historical dangerous road condition scene data, extract a variety of dangerous road condition features in the historical dangerous road condition scene data, and train the initial confrontation network according to the multiple dangerous road condition characteristics to obtain a dangerous road condition layer.

Specifically, the server may learn a variety of dangerous road condition features through a neural network corresponding to the dangerous road condition level. Among them, the neural network corresponding to the dangerous road condition level can be a confrontation network, for example, it can be a generative confrontation network (GAN, Generative Adversarial Networks, deep learning model), and the generative confrontation network model can include a generative model (Generative Model) and a discriminant model (Discriminative Model) mutual game learning to generate data and data enhancement with better output effects.

The dangerous road condition layer can include a dangerous road condition learning layer and a dangerous road condition generation layer. The server then uses the extracted various dangerous road condition features to learn the initial confrontation network to obtain the dangerous road condition learning level. The server further learns the initial confrontation network and dangerous road conditions. Learning and training are carried out at different levels, so as to obtain the dangerous road condition generation level. The dangerous road condition generation layer can then randomly generate a variety of dangerous road condition scene models, for example, various types of dangerous road condition scenes can be automatically and randomly generated through the dangerous road condition generation layer.

Step 208: Use the scene simulation layer and the dangerous road condition layer to continuously train the scene simulator until the preset conditions are met, and the trained scene simulator is obtained; the scene simulator is used to generate a simulated driving scene when the vehicle is simulated and tested.

After the server training obtains the scene simulation layer and the dangerous road condition layer, it further conducts combined training on the scene simulation layer and the dangerous road condition layer. Specifically, the server may further use a generative confrontation network algorithm to learn and train the scene simulation layer and the dangerous road condition layer, so that the model can randomly generate the dangerous road condition scene in the current simulation scene when generating multiple road condition scenes. In the process of training the model, until the obtained scene simulation satisfies the preset conditions, the trained scene simulator is generated. The server can then use the trained scene simulator to generate a simulated driving scene when performing a simulation test on the unmanned vehicle.

In the above-mentioned deep learning-based scene simulator construction method, after the server obtains driving scene data and historical dangerous road condition scene data, it extracts various road condition scene features from the driving scene data, and uses deep learning algorithms to perform deep learning based on various road condition scene features. Through learning, the scene simulation layer can be effectively trained; the server can further extract various dangerous road condition features from historical dangerous road condition scene data, and train the initial confrontation network according to the various dangerous road condition features, so as to obtain the dangerous road condition layer. Training the dangerous road condition layer through the confrontation network can make the dangerous road condition layer obtained by training randomly and effectively generate the dangerous road condition scene. The server then uses the scene simulation layer and the dangerous road condition layer to continuously train the scene simulator until the preset conditions are met, thereby obtaining the trained scene simulator; the scene simulator is used to generate a simulated driving scene when the vehicle is simulated and tested. After separately training the scene simulation layer and the dangerous road condition layer, the generative confrontation network is further used for combined training, which can effectively construct a realistic scene simulator, which can effectively generate a realistic and reliable simulation Driving scene.

In one of the embodiments, the scene simulator includes a scene simulation layer and a dangerous road condition layer. The scene simulation layer is used to perform in-depth learning of a variety of road condition scenes using a deep learning algorithm, and use the learned characteristics of a variety of road conditions to randomly generate correspondences. A variety of traffic scenes. The scene simulation layer may also include a scene element simulation layer, a scene signal simulation layer, and a driving scene simulation layer. Among them, the scene element simulation layer is used to extract various scene element information features in the driving scene data, learn and train various scene object information features, and use the learned multiple scene element information features to generate simulated scene elements. The scene signal simulation layer is used to extract a variety of scene signal features in the driving scene data, learn and train a variety of scene signal features, and use the learned multiple scene signal features to generate an analog scene signal. The driving scene simulation layer is used to extract a variety of audio and video signal features in the driving scene data, learn and train a variety of audio and video signal features, and obtain and use the learned audio and video signal features to combine analog scene elements and analog scene signals to generate Simulate driving scenes.

The dangerous road condition layer includes a dangerous road condition learning layer and a dangerous road condition generation layer. The dangerous road condition learning layer is used to learn and train a variety of dangerous road condition scenarios according to the confrontation network. The dangerous road condition generation layer is used to use the learned characteristics of a variety of dangerous road conditions randomly. Generate a variety of dangerous road conditions.

By using deep learning algorithms for in-depth learning based on various road condition scene characteristics, the scene simulation layer can be effectively trained; by training the dangerous road condition layer through the confrontation network, the dangerous road condition layer obtained by the training can be randomly and effectively generated the dangerous road condition scene, so as to be able to Effectively construct a realistic scene simulator.

In one of the embodiments, extracting a variety of road condition scene features in the driving scene data, using a deep learning algorithm for deep learning according to the multiple road condition scene features, to obtain a scene simulation layer, including: multi-level feature extraction of the driving scene data, Extract scene element information features, scene signal features, and audio and video signal features from the driving scene data; use deep learning algorithms to learn and train the initial network model according to the scene element information features, scene signal features, and audio and video signal features to obtain the trained scene Simulation layer.

The driving scene data includes a variety of road conditions and scene information, such as road information, lane information, vehicle information, signal light information, pedestrian information, and feature information. The feature information refers to various tangible objects on the ground, such as mountains, forests, etc. Buildings, etc., as well as intangibles, such as provinces, county boundaries, and other types of feature information.

After the server acquires a large amount of driving scene data and historical dangerous road condition scene data, it can first perform multiple levels of feature extraction on multiple road condition scenes in the driving scene data, and extract multiple road condition scene features in the driving scene data. The road condition scene characteristics may include various road condition scene characteristics such as scene element information characteristics, scene signal characteristics, and audio and video signal characteristics. The scene element information feature is the information feature corresponding to the multiple entity elements contained in the driving scene, for example, it may include specific road element, vehicle element, pedestrian element, signal light element and other corresponding element information features. The scene signal feature can be the sensor signal information corresponding to the deeper road information, vehicle information, pedestrian information, signal light information and other information. The audio and video signal characteristics are the audio and video signals corresponding to road information, vehicle information, pedestrian information, signal light information and other information respectively in the audio and video visualization scene.

Specifically, the server may input driving scene data and historical dangerous road condition scene data into a pre-built initial neural network model. The initial neural network includes multiple levels, for example, a scene simulation level, a dangerous road condition level, and the like. The server performs multi-level feature extraction on multiple road conditions in the driving scene data through the scene simulation layer, and extracts multiple scene element information features including road element features, vehicle element features, pedestrian element features, signal element features, etc.; further Extracting scene signal features in the driving scene data according to multiple scene element information features; further extracting audio and video signal features in the driving scene data according to multiple scene element information features and scene signal features.

The server then uses the deep learning algorithm to learn the corresponding neural network according to the scene element information characteristics, the scene signal characteristics and the audio and video signal characteristics, and generates a scene simulation layer according to the learned various road condition scene characteristics, and further trains the scene simulation layer. Thereby, the scene simulation layer after training is obtained. The scene simulation layer can then use the learned various road condition scene features to randomly generate a variety of corresponding models. For example, the scene simulation layer can automatically generate road models, models and other models, vehicle models, and pedestrian models that are included in the road condition scenes. Scene model. After multi-level feature extraction is performed on the driving scene data, the scene simulation layer is trained using the extracted multi-level features, so that the scene simulation layer can be effectively constructed.

In one of the embodiments, as shown in FIG. 3, the scene simulation layer includes a scene element simulation layer, a scene signal simulation layer, and a driving scene simulation layer. According to various road conditions and scene characteristics, deep learning algorithms are used for deep learning to obtain the scene simulation layer. The steps include the following:

Step 302: Extract multiple types of scene element information features in the driving scene data, and train multiple types of scene object information features to obtain a completed scene element simulation layer.

Step 304: Extract various scene signal features in the driving scene data, and train the various scene signal features to obtain a scene signal simulation layer that has been trained.

Step 306: Extract a variety of audio and video signal features from the driving scene data, and train the multiple audio and video signal features to obtain a completed driving scene simulation layer.

The scene element information includes a variety of environmental role elements, such as specific road entity information, vehicle entity information, pedestrian entity information, signal light entity information, and other environmental role information.

After the server obtains a large amount of driving scene data and historical dangerous road condition scene data, it inputs the driving scene data and historical dangerous road condition scene data into the pre-built initial neural network model. The initial neural network model can be constructed using preset deep learning algorithms and neural network structures. The initial neural network includes multiple levels, such as a scene simulation level, a dangerous road condition learning level, and a dangerous road condition generation level. The scene simulation level may also include multiple neural network layers, and specifically may include a scene element simulation layer, a scene signal simulation layer, and a driving scene simulation layer. The server further performs multi-level feature extraction on multiple road condition scenes in the driving scene data through multiple neural network layers in the scene simulation level, and extracts multiple road condition scene features in the driving scene data.

Specifically, the server extracts various scene element information features in the driving scene data through the scene element simulation layer, uses preset deep learning algorithms to learn various scene object information features, and compares the scene element simulation layer with the learned features. The neural network is trained until the training condition threshold is met, and the trained scene element simulation layer is obtained. The trained scene element simulation layer can be used to generate simulated scene element information.

The server further extracts multiple scene signal features in the driving scene data through the scene signal simulation layer, uses preset deep learning algorithms to learn multiple scene signal features, and trains the neural network corresponding to the scene signal simulation layer according to the learned features , Until the training condition threshold is met, the scene signal simulation layer where the training is completed is obtained. The server can further combine various scene element information features and various scene signal features to learn and train the neural network, so as to obtain a completed scene signal simulation layer. The trained scene signal simulation layer can be used to generate a variety of scene elements and scene signal information.

The server extracts a variety of comprehensive audio and video signal features in the driving scene data through the driving scene simulation layer. The multiple comprehensive audio and video signal features include dynamic multiple scene element information features and multiple scene signal features. The server then uses preset deep learning algorithms to learn a variety of audio and video signal features, and trains the neural network corresponding to the driving scene simulation layer according to the learned features, until the training condition threshold is met, and the trained driving scene simulation layer is obtained . The server can further combine various scene element information features, various scene signal features, and various audio and video signal features to learn and train the neural network, so as to obtain a completed driving scene simulation layer. The trained driving scene simulation layer can be used to generate a variety of dynamic driving scene information.

By performing multi-level feature extraction on the driving scene data, and using the extracted multi-level features to train multiple scene simulation layers, the scene simulation layer can be effectively constructed.

In one of the embodiments, as shown in FIG. 4, the dangerous road condition layer includes a dangerous road condition learning layer, and the step of extracting multiple dangerous road condition features from historical dangerous road condition scene data includes the following contents:

Step 402: Extract multiple dangerous road condition features from historical dangerous road condition scene data through the dangerous road condition learning layer.

Step 404: Identify the risk factors and risk levels of each dangerous road condition feature.

In step 406, a set of dangerous scene factors is generated by using multiple dangerous road condition features according to the risk factors and the degree of risk.

The server can obtain a large amount of historical dangerous road condition scene data, and the historical dangerous road condition scene data includes dangerous road condition scene information of various risk factors and degree of danger.

The server can input a large amount of historical dangerous road condition scene data into the preset neural network corresponding to the dangerous road condition layer. Specifically, the server inputs historical dangerous road condition scene data to the dangerous road condition learning layer in the dangerous road condition layer, and extracts features of historical dangerous road condition scene data through the dangerous road condition learning layer, and extracts multiple dangerous road conditions in the historical dangerous road condition scene data. feature. The server then identifies the risk factors and the degree of risk for each type of dangerous road condition feature. Dangerous factors can be various environmental factors that cause dangerous road conditions, such as the number of lanes, lane vehicles, pedestrians, road obstacles, bad weather, vehicle failures, illegal driving, and other dangerous factors that cause vehicles to fall into a dangerous state. Risk factors can also be multiple factor types corresponding to multiple risk types. The degree of danger can be calculated based on the damage level, risk complexity and probability of occurrence.

After the server extracts a variety of dangerous road condition features in the historical dangerous road condition scene data, it extracts the dangerous factors in each dangerous road condition feature, as well as the damage level, risk complexity, and probability of occurrence of each dangerous road condition feature, according to the damage level, Dangerous complexity and probability of occurrence calculate the degree of danger of each dangerous road condition feature. The server then generates a set of dangerous scene factors by using multiple dangerous road condition features according to the risk factors and the degree of risk. The set of dangerous scene factors is the characteristics of dangerous road conditions learned by the dangerous road condition learning layer, including multiple risk factors. Through feature extraction and deep learning of various historical dangerous road condition scene data, it is possible to effectively learn a variety of dangerous road condition features.

In one of the embodiments, the dangerous road condition layer includes a dangerous road condition generation layer, and the initial confrontation network is trained according to a variety of dangerous road condition features to obtain the dangerous road condition layer, including: generating risk factors based on the probability values of the risk factors in the dangerous scene element set Random domain: Use the generative confrontation network to train various dangerous road condition features in the set of dangerous scene factors according to the random domain of risk factors, and obtain the dangerous road condition generation layer after the training; the dangerous road condition generation layer is used to randomly generate dangerous road condition information.

The server extracts a variety of dangerous road condition features in historical dangerous road condition scene data through the dangerous road condition learning layer, identifies the risk factors and degree of danger of each dangerous road condition feature, and generates a set of dangerous scene factors based on the risk factors and degree of danger. Later, further use the set of dangerous scene factors to train the dangerous road condition generation layer.

Specifically, the server may also calculate the probability value of each type of dangerous factor in the historical dangerous road condition scene data through the dangerous road condition learning layer, that is, the probability of occurrence of the dangerous factor. Among them, the initial confrontation network can be a generative confrontation network, and the generative confrontation network can include a discriminant model and a generative model. That is, the dangerous road condition learning layer can be a discriminant model, and the dangerous road condition generation layer can be a generative model.

The server then uses the generative confrontation network to train a variety of dangerous road conditions in the set of dangerous scene factors according to the random domain of risk factors, so that it can effectively train the dangerous road condition generation layer, and the dangerous road condition generation layer can then randomly generate a variety of dangerous road conditions. The scene model, for example, can automatically and randomly generate various types of dangerous road condition scenes through the dangerous road condition generation layer. By using the generative confrontation network algorithm to train the dangerous road condition learning layer and the dangerous road condition generating layer, it is possible to effectively construct the dangerous road condition layer with higher validity and authenticity.

In one of the embodiments, the scene simulation layer and the dangerous road condition layer are used to continuously train the scene simulator until the preset conditions are met, and the trained scene simulator is obtained, including: constructing a scene based on the scene simulation layer and the dangerous road condition layer Simulator; use the generative confrontation network algorithm to continuously train the scene simulation layer and the dangerous road condition layer; until the driving scene generated by the scene simulator meets the condition threshold and the generated dangerous road condition meets the probability threshold, the trained scene simulator is obtained .

After the server trains to obtain the scene simulation layer and the dangerous road condition layer, a scene simulator is constructed according to the scene simulation layer and the dangerous road condition layer. The server further conducts combined training on the scene simulation layer and the dangerous road condition layer. Specifically, the server can further use the generative confrontation network algorithm to continuously train the scene simulation layer and the dangerous road condition layer. In the process of training the model, the simulator can generate multiple road condition scenarios based on multiple dangerous road conditions. The probability value of is randomly generated dangerous road scenes in the current simulation scene. Until the driving scene generated by the scene simulator meets the condition threshold and the generated dangerous road condition meets the probability threshold, the training is stopped, thereby obtaining a completed scene simulator. The server can then use the trained scene simulator to generate a simulated driving scene when performing a simulation test on the unmanned vehicle.

For example, when a simulated driving scene is generated through a simulator, a road map of the corresponding road type can be constructed. The road map includes various types of ground, lane lines, signal lights, landmarks and other information, and traffic participants are added to the road map The information, for example, may include vehicles and pedestrians and their respective movement paths. The simulator can also randomly generate dangerous road conditions with various dangerous factors in the simulated driving scene, such as dangerous pedestrian trajectories, dangerous vehicle trajectories, roadblock information, harsh environments and other dangerous road conditions.

After separately training the scene simulation layer and the dangerous road condition layer, the generative confrontation network is further used for combined training, which can effectively construct a realistic scene simulator, which can effectively generate a realistic and reliable simulation Driving scene.

In one of the embodiments, as shown in FIG. 5, the method further includes the step of using a scene simulator to perform a test, which specifically includes the following contents:

Step 502: Obtain a simulation test instruction, and call the scene simulator according to the simulated driving instruction.

Step 504: Use the scene simulator to generate driving scene information, and randomly generate dangerous road condition information in the driving scene information; make the vehicle perform simulated driving in the generated simulated driving scene.

Step 506: Obtain vehicle driving data of the vehicle during the simulated driving process.

Step 508: Generate vehicle simulation test information according to the driving scene information and the vehicle driving data.

After the server uses the driving scene data and historical dangerous road condition scene data to construct and train a scene simulator including a scene simulation layer and a dangerous road condition layer, it can then use the scene simulator to generate a simulated driving scene to perform simulation tests on unmanned vehicles.

Specifically, the vehicle may send a simulation test request to the server, or the vehicle monitoring platform may directly send a simulation test instruction to the server. After the server obtains the simulation test instruction, it calls the scene simulator according to the simulated driving instruction. The server further generates driving scene information through the scene simulation layer of the scene simulator, and randomly generates dangerous road condition information in the driving scene information through the dangerous road condition layer of the scene simulator. Thus, the vehicle can perform simulated driving in the generated simulated driving scene. Specifically, the vehicle is equipped with corresponding sensors, so that the vehicle can effectively test the sensor data of the vehicle in a simulated driving.

The server can obtain the vehicle driving data of the vehicle in the simulation driving process. The vehicle driving data includes vehicle state data and road image data collected by the vehicle. For example, the vehicle state data may include vehicle operating state data, sensor data, and energy consumption data. The image data includes multiple road image data collected by the collection equipment. The server then generates vehicle simulation test information of the vehicle according to the driving scene information and vehicle driving data, and the vehicle simulation test information can be used to analyze various performance indicators of the unmanned vehicle. By using the constructed scene simulator to generate driving simulation scenes and perform simulation tests on unmanned vehicles, it is possible to effectively construct simulated driving scenes with high authenticity and reliability, and to effectively conduct simulation tests on unmanned vehicles. Effectively improve the validity and reliability of the test.

It should be understood that although the various steps in the flowcharts of FIGS. 2-5 are displayed in sequence as indicated by the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless specifically stated in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least part of the steps in Figures 2-5 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. These sub-steps or stages The execution order of is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

In one embodiment, as shown in FIG. 6, a device for building a scene simulator based on deep learning is provided, including: a data acquisition module 602, a scene simulation training module 604, a dangerous road condition training module 606, and a scene simulator building module 608, of which:

The data acquisition module 602 is used to acquire driving scene data and historical dangerous road condition data;

The scene simulation training module 604 is used to extract various road condition and scene features from the driving scene data, and use a deep learning algorithm to perform deep learning according to the various road condition and scene features to obtain a scene simulation layer;

The dangerous road condition training module 606 is used to extract multiple dangerous road condition features from historical dangerous road condition scene data, and train the initial countermeasure network according to the multiple dangerous road condition features to obtain the dangerous road condition layer; and

The scene simulator building module 608 is used to continuously train the scene simulator using the scene simulation layer and the dangerous road condition layer, until the preset conditions are met, and the trained scene simulator is obtained; the scene simulator is used to generate simulations when the vehicle is simulated and tested Driving scene.

In one of the embodiments, the scene simulator includes a scene simulation layer and a dangerous road condition layer, and the scene simulation layer is used for deep learning of the various road condition scenes using a deep learning algorithm, and using the learned multiple road condition scenes Features randomly generate corresponding multiple road condition scenarios; the dangerous road condition layer includes a dangerous road condition learning layer and a dangerous road condition generation layer, and the dangerous road condition learning layer is used to learn and train multiple dangerous road condition scenarios according to the confrontation network. The dangerous road condition generation layer is used to randomly generate a variety of dangerous road condition scenarios using the learned characteristics of a variety of dangerous road conditions.

In one of the embodiments, the scene simulation training module 604 is also used to perform multi-level feature extraction on the driving scene data, extracting scene element information features, scene signal features, and audio and video signal features in the driving scene data; and according to the scene element information features , Scene signal characteristics and audio and video signal characteristics Use deep learning algorithms to learn and train the initial network model to obtain the trained scene simulation layer.

In one of the embodiments, the scene simulation layer includes a scene element simulation layer, a scene signal simulation layer, and a driving scene simulation layer. The scene simulation training module 604 is also used to extract information features of various scene elements in the driving scene data, and to Train the scene element information characteristics to obtain the trained scene element simulation layer; extract multiple scene signal features in the driving scene data, train multiple scene signal features, and obtain the trained scene signal simulation layer; and extract the driving scene A variety of audio and video signal features in the data are trained on multiple audio and video signal features to obtain a completed driving scene simulation layer.

In one of the embodiments, the dangerous road condition layer includes a dangerous road condition learning layer, and the dangerous road condition training module 606 is also used to extract multiple dangerous road condition features in historical dangerous road condition scene data through the dangerous road condition learning layer; Risk factors and degree of risk; and generate a set of risk scene factors based on risk factors and risk complexity using multiple dangerous road conditions.

In one of the embodiments, the dangerous road condition layer includes a dangerous road condition generation layer, and the dangerous road condition training module 606 is further configured to generate a risk factor random domain according to the probability value of the risk factor in the dangerous scene element set; and use a generative confrontation network according to the risk factor The random domain conducts multiple dangerous road condition feature training on the set of dangerous scene factors, and obtains the completed dangerous road condition generation layer; the generated dangerous road condition layer is used to randomly generate dangerous road condition information.

In one of the embodiments, the scene simulator construction module 608 is also used to construct a scene simulator based on the scene simulation layer and the dangerous road condition layer; use the generative confrontation network algorithm to continuously train the scene simulation layer and the dangerous road condition layer; and When the driving scene generated by the scene simulator meets the condition threshold and the generated dangerous road condition meets the probability threshold, the trained scene simulator is obtained.

In one of the embodiments, as shown in FIG. 7, the device further includes a simulation test module 610, which is used to obtain simulation test instructions, call the scene simulator according to the simulated driving instructions; use the scene simulator to generate driving scene information, and randomly Dangerous road condition information is generated from the driving scene information; the vehicle is simulated driving in the generated simulated driving scene; the vehicle driving data of the vehicle in the simulated driving process is obtained; and the vehicle simulation test information is generated based on the driving scene information and the vehicle driving data.

For the specific definition of the device for constructing a scene simulator based on deep learning, please refer to the above definition of the method for constructing a scene simulator based on deep learning, which will not be repeated here. The various modules in the device for constructing a scene simulator based on deep learning can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 8. The computer equipment includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The database of the computer equipment is used to store data such as driving scene data and historical dangerous road condition scene data. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer readable instructions are executed by the processor to implement an 8 method.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may Including more or fewer parts than shown in the figure, or combining some parts, or having a different arrangement of parts.

A computer device includes a memory and one or more processors. The memory stores computer readable instructions. When the computer readable instructions are executed by the processor, the one or more processors execute the above method embodiments. step.

One or more non-volatile computer-readable storage media storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute A step of.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile computer. In a readable storage medium, when the computer-readable instructions are executed, they may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered as the range described in this specification.

The above-mentioned embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can be made, and these all fall within the protection scope of this application. Therefore, the scope of protection of the patent of this application shall be subject to the appended claims.

Claims

A method for constructing a scene simulator based on deep learning includes:

Obtain driving scene data and historical dangerous road condition data;

Extracting multiple road condition and scene features in the driving scene data, and using a deep learning algorithm to perform deep learning according to the multiple road condition and scene features to obtain a scene simulation layer;

Extracting multiple dangerous road condition features from historical dangerous road condition scene data, and training the initial confrontation network according to the multiple dangerous road condition features to obtain a dangerous road condition layer; and

The scene simulation layer and the dangerous road condition layer are used to continuously train the scene simulator until the preset conditions are met, and the trained scene simulator is obtained; the scene simulator is used to generate a simulated driving scene when the vehicle is simulated and tested.
The method according to claim 1, wherein the scene simulator comprises a scene simulation layer and a dangerous road condition layer, and the scene simulation layer is used for deep learning of the multiple road condition scenes by using a deep learning algorithm, Using the learned characteristics of multiple road conditions to randomly generate corresponding multiple road condition scenarios; the dangerous road condition layer includes a dangerous road condition learning layer and a dangerous road condition generation layer. The dangerous road condition scenes are learned and trained, and the dangerous road condition generation layer is used to randomly generate multiple dangerous road condition scenes by using the learned characteristics of the various dangerous road conditions.
The method according to claim 1, wherein said extracting multiple types of road condition and scene features in said driving scene data, and using a deep learning algorithm to perform deep learning according to said multiple types of road condition and scene features to obtain a scene simulation layer, include:

Performing multi-level feature extraction on the driving scene data, extracting scene element information features, scene signal features, and audio and video signal features in the driving scene data; and

A deep learning algorithm is used to learn and train the initial network model according to the features of the scene element information, the features of the scene signal, and the features of the audio and video signals, to obtain a trained scene simulation layer.
The method according to claim 3, wherein the scene simulation layer includes a scene element simulation layer, a scene signal simulation layer, and a driving scene simulation layer, and the deep learning algorithm is used to perform depth based on the various road conditions and scene characteristics. Study and get the scene simulation layer including:

Extracting various scene element information features in the driving scene data, and training the various scene element information features to obtain a completed scene element simulation layer;

Extracting various scene signal features in the driving scene data, and training the various scene signal features to obtain a trained scene signal simulation layer; and

Extracting multiple audio and video signal features in the driving scene data, training on the multiple audio and video signal features, and obtaining a completed driving scene simulation layer.
The method according to claim 1, wherein the dangerous road condition layer includes a dangerous road condition learning layer, and the extraction of multiple dangerous road condition features from historical dangerous road condition scene data includes:

Extracting multiple dangerous road condition features in the historical dangerous road condition scene data through the dangerous road condition learning layer;

Identify the risk factors and risk levels of each dangerous road condition feature; and

According to the risk factors and the risk complexity, the multiple dangerous road condition features are used to generate a dangerous scene factor set.
The method according to claim 5, wherein the dangerous road condition layer comprises a dangerous road condition generation layer, and training an initial confrontation network according to the characteristics of the multiple dangerous road conditions to obtain the dangerous road condition layer comprises:

Generating a random domain of risk factors according to the probability values of the risk factors in the set of dangerous scene elements; and

A generative confrontation network is used to train a variety of dangerous road conditions in the set of dangerous scene factors according to the risk factor random domain to obtain a trained dangerous road condition generation layer; the generated dangerous road condition layer is used to randomly generate dangerous road conditions information.
The method according to any one of claims 1 to 6, wherein the scene simulator is continuously trained using the scene simulation layer and the dangerous road condition layer, until a preset condition is met, and a scene where the training is completed is obtained Simulators, including:

Constructing a scene simulator according to the scene simulation layer and the dangerous road condition layer;

Use a generative countermeasure network algorithm to continuously train the scene simulation layer and the dangerous road condition layer; and

Until the driving scene generated by the scene simulator meets the condition threshold and the generated dangerous road condition meets the probability threshold, the trained scene simulator is obtained.
The method according to claim 1, wherein the method further comprises:

Acquiring a simulation test instruction, and invoking a scene simulator according to the simulation driving instruction;

Generating driving scene information by using the scene simulator, and randomly generating dangerous road condition information in the driving scene information; enabling the vehicle to perform simulated driving in the generated simulated driving scene;

Acquiring vehicle driving data of the vehicle in a simulated driving process; and

Generate vehicle simulation test information according to the driving scene information and the vehicle driving data.
A scene simulator construction device based on deep learning, including:

Data acquisition module, used to acquire driving scene data and historical dangerous road condition data;

The scene simulation training module is used to extract a variety of road condition and scene features in the driving scene data, and use a deep learning algorithm to perform deep learning according to the multiple road condition and scene features to obtain a scene simulation layer;

The dangerous road condition training module is used to extract multiple dangerous road condition features from historical dangerous road condition scene data, and train the initial confrontation network according to the multiple dangerous road condition features to obtain the dangerous road condition layer; and

The scene simulator building module is used to use the scene simulation layer and the dangerous road condition layer to continuously train the scene simulator until the preset conditions are met, and then the trained scene simulator is obtained; the scene simulator is used to perform the vehicle A simulated driving scene is generated during the simulation test.
The device according to claim 9, wherein the dangerous road condition layer comprises a dangerous road condition learning layer, and the dangerous road condition training module is further configured to extract data from the historical dangerous road condition scene data through the dangerous road condition learning layer. Multiple dangerous road condition features; identifying the risk factors and risk levels of each dangerous road condition feature; and using the multiple dangerous road condition features to generate a set of dangerous scene factors according to the risk factors and the risk complexity.
The device according to claim 9, wherein the dangerous road condition layer includes a dangerous road condition generation layer, and the dangerous road condition training module is further configured to generate a random risk factor according to the probability value of the risk factor in the dangerous scene element set. Domain; and using a generative confrontation network according to the risk factor random domain to perform multiple dangerous road condition feature training on the dangerous scene factor set to obtain the dangerous road condition generation layer after the training; the generating dangerous road condition layer is used for random Generate information about dangerous road conditions.
The device according to claim 11, wherein the scene simulator construction module is further configured to construct a scene simulator according to the scene simulation layer and the dangerous road condition layer; and use a generative confrontation network algorithm to continuously check the The scene simulation layer and the dangerous road condition layer perform combined training; and until the driving scene generated by the scene simulator meets the condition threshold and the generated dangerous road condition meets the probability threshold, a trained scene simulator is obtained.
The device according to claim 9, wherein the device further comprises a simulation test module, configured to obtain a simulation test instruction, and call a scene simulator according to the simulated driving instruction; and use the scene simulator to generate driving scene information , And randomly generate dangerous road condition information in the driving scene information; make the vehicle perform simulated driving in the generated simulated driving scene; obtain the vehicle driving data of the vehicle in the simulated driving process; and according to the driving scene information and The vehicle driving data generates vehicle simulation test information.
A computer device includes a memory and one or more processors. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the one or more processors, the one or more Each processor performs the following steps:

Obtain driving scene data and historical dangerous road condition data;

Extracting multiple road condition and scene features in the driving scene data, and using a deep learning algorithm to perform deep learning according to the multiple road condition and scene features to obtain a scene simulation layer;

Extracting multiple dangerous road condition features from historical dangerous road condition scene data, and training the initial confrontation network according to the multiple dangerous road condition features to obtain a dangerous road condition layer; and

The scene simulation layer and the dangerous road condition layer are used to continuously train the scene simulator until the preset conditions are met, and the trained scene simulator is obtained; the scene simulator is used to generate a simulated driving scene when the vehicle is simulated and tested.
The computer device according to claim 14, wherein the processor further executes the following steps when executing the computer-readable instruction:

Performing multi-level feature extraction on the driving scene data, extracting scene element information features, scene signal features, and audio and video signal features in the driving scene data; and

A deep learning algorithm is used to learn and train the initial network model according to the features of the scene element information, the features of the scene signal, and the features of the audio and video signals, to obtain a trained scene simulation layer.
The computer device according to claim 15, wherein the scene simulation layer includes a scene element simulation layer, a scene signal simulation layer, and a driving scene simulation layer, and the processor also executes the following when executing the computer-readable instructions step:

Extracting various scene element information features in the driving scene data, and training the various scene element information features to obtain a completed scene element simulation layer;

Extracting multiple scene signal features in the driving scene data, and training the multiple scene signal features to obtain a trained scene signal simulation layer; and

Extracting multiple audio and video signal features in the driving scene data, training on the multiple audio and video signal features, and obtaining a completed driving scene simulation layer.
The computer device according to claim 14, wherein the dangerous road condition layer comprises a dangerous road condition learning layer, and the processor further executes the following steps when executing the computer-readable instruction:

Extracting multiple dangerous road condition features in the historical dangerous road condition scene data through the dangerous road condition learning layer;

Identify the risk factors and risk levels of each dangerous road condition feature; and

According to the risk factors and the risk complexity, the multiple dangerous road condition features are used to generate a dangerous scene factor set.
The computer device according to claim 17, wherein the dangerous road condition layer comprises a dangerous road condition generation layer, and the processor further executes the following steps when executing the computer readable instruction:

Generating a random domain of risk factors according to the probability values of the risk factors in the set of dangerous scene elements;

And using a generative confrontation network to train multiple dangerous road condition features in the set of dangerous scene factors according to the risk factor random domain, to obtain a trained dangerous road condition generation layer; the generated dangerous road condition layer is used to randomly generate risks Traffic information.
The computer device according to claim 14, wherein the processor further executes the following steps when executing the computer-readable instruction:

Acquiring a simulation test instruction, and invoking a scene simulator according to the simulation driving instruction;

Generating driving scene information by using the scene simulator, and randomly generating dangerous road condition information in the driving scene information; enabling the vehicle to perform simulated driving in the generated simulated driving scene;

Acquiring vehicle driving data of the vehicle in a simulated driving process; and

Generate vehicle simulation test information according to the driving scene information and the vehicle driving data.
One or more non-volatile computer-readable storage media storing computer-readable instructions, which when executed by one or more processors, cause the one or more processors to execute claim 1 To any of the steps described in 8.