KR20220066711A - Autonomous Driving Distributed Simulation Synchronous Control Method - Google Patents

Abstract

The present invention relates to a synchronous control method for autonomous driving distributed simulation, by which virtual sensors, artificial intelligence simulators, and global simulators with characteristics such as various simulation cycles can be synchronized with each other while being processed in parallel. In the autonomous driving simulator for vehicle simulation, which includes loading of control force, dynamics simulation, and a 3D image rendering system, each sensor simulator for generating virtual sensor training data and an AI simulator are provided distinct from a global simulator, in consideration of the processing performance of sensors and artificial intelligence algorithms. The global simulator includes distributed simulation to handle autonomous driving scenarios and vehicle dynamics by using global simulation timestamps. To this end, a parallel simulation method for each simulator using shared buffers and a simulation synchronous control and interpolation method using a fixed time interval manner are presented. The synchronous control method for autonomous driving distributed simulation comprises the steps in which: vehicle control simulation for an AI simulator processes autonomous driving ground truth data collected from a plurality of, sensors through machine learning, and generates vehicle control force; and the AI simulator processes events in parallel with a global simulator by using shared buffers.

Description

Autonomous Driving Distributed Simulation Synchronous Control Method

The present invention relates to a method for controlling autonomous driving distributed simulation synchronous control, and more particularly, to autonomous driving, in which virtual sensors and artificial intelligence simulators having characteristics such as various simulation cycles and global simulators are processed in parallel and synchronized with each other. It relates to a distributed simulation synchronous control method.

According to Application No. 10-2019-0156179 (November 28, 2019), "It relates to a device and method for autonomous driving control based on artificial intelligence VILS, and in particular, realistic driving through a Generative Adversarial Network (GAN) model. BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technology for verifying road operation of an autonomous vehicle through operation on a real road by creating an environment.The present invention provides a state indicating a modeled road environment that requires verification in order to autonomously drive the autonomous vehicle in a virtual environment A driving environment generating unit that generates the virtual environment through driving information including information and information on surrounding vehicles of the autonomous vehicle; A result generating unit that dataizes using layer classification through a map), and an autonomous driving algorithm calculated from the data generated by the result generating unit to control the autonomous vehicle and actually move the autonomous vehicle An artificial intelligence VILS-based autonomous driving control device including an autonomous driving verification unit that tracks and records a trajectory and inputs it into the virtual environment is provided. By using the model for VILS verification, it will be possible to increase the reliability of autonomous driving evaluation.”

According to Application No. 10-2013-0011463 (2013.01.31), "a hybrid system such as a virtual-physical system (CPS) having both physical and computational elements is configured. In performing distributed simulation using a plurality of distributed simulators for subsystem models modeling subsystems to Disclosed is a system and method for performing distributed simulations that allow distributed simulators to perform simulations on subsystem models assigned to them by adjusting local simulation times according to synchronized global simulation times."

1. Application No. 10-2019-0156179 (2019.11.28); AI VILS-based autonomous driving control device and method 2. Application No. 10-2013-0011463 (2013.01.31); Distributed simulation system and method

Recently, a lot of research has been done in relation to autonomous driving such as artificial intelligence, sensor fusion, and communication. In particular, the vast amount of data of various driving environments and scenarios required for AI learning can be easily reproduced through a virtual autonomous driving simulator, and related research is being actively conducted.

Various sensors, such as lidar, radar, camera, GPS, IMU, and vehicle ECU, must be installed in an autonomous vehicle, and each sensor must reflect physical characteristics such as sampling period, resolution, and angle of view. However, the virtual sensor of the autonomous driving simulator has a problem in that the overall integrated simulation performance is degraded because the time synchronization cost increases as the types and number of sensors increase.

In addition, in the autonomous driving simulator, virtual sensor, artificial intelligence, dynamics, and driving scenario processing must be simulated together. The computational load of each processing module is large, and the waiting time from the occurrence of a sensor event to obtaining the virtual world information of the next simulated cycle Therefore, parallel processing with the integrated simulation pipeline is difficult. Since each sensor simulator receives the same simulation result from the global simulator every cycle of the global clock and requires synchronization with each simulator clock cycle, there is a difficulty in parallel execution.

Recently, many studies on distributed simulation methods have been conducted, but the reality is that studies on integrated simulation synchronization methods of multiple simulators including various sensors are lacking. In particular, there is an urgent need to generate realistic sensor images and develop integrated simulation technology through synchronization of simulation and distributed simulation according to the specifications of the actual sensor.

The present invention solves the above problems and satisfies the necessity, and the virtual sensor and artificial intelligence simulators having characteristics such as various simulation cycles and the global simulator are processed in parallel and synchronized with each other, autonomous driving distributed An object of the present invention is to provide a simulation synchronous control method.

Let's look at the features of the present invention for achieving the above object.

Traditional vehicle simulation consists of control force loading, dynamics simulation, and 3D image rendering system. In the autonomous driving simulator of the present invention, in consideration of the processing performance of sensors and artificial intelligence algorithms, each sensor simulator and AI simulator responsible for generating virtual sensor learning data are configured separately from the global simulator, and the global simulation timestamp is configured in the global simulator. Distributed simulations are constructed to handle autonomous driving scenarios and vehicle dynamics.

To this end, we present a parallel simulation method of each simulator using a shared buffer and a simulation synchronous control and interpolation method using a fixed time interval method.

According to a preferred effect of the present invention, 1) establishing a data and time synchronization basis of simulators for sensor fusion, 2) proposing a method for configuring a distributed simulation system for autonomous driving of three types of global, AI, and sensors, 3) sensor scanning It can help to construct a fixed time interval-based scene interpolation algorithm for time interval simulation, 4) has the effect of improving virtual sensor simulation and artificial intelligence data synchronization and parallel distributed processing performance, 5) highly compatible autonomous driving simulation It has the advantage of extending it to the system and improving the model validation and simulation environment.

1 is a flowchart illustrating parallel processing and data synchronization using a shared buffer in the autonomous driving distributed simulation synchronization control method according to the present invention.
FIG. 2 is a flowchart for explaining the acquisition of a global simulation timestamp based on a fixed time interval in the autonomous driving distributed simulation synchronization control method according to the present invention; FIG.
3 is a flowchart illustrating interpolation rendering of a previous scene according to a time elapse rate in the autonomous driving distributed simulation synchronization control method according to the present invention.
4 is a flowchart for explaining a sensor scan time simulation reflecting a dynamic period in the autonomous driving distributed simulation synchronization control method according to the present invention;
5 is a flowchart illustrating a real-time autonomous driving simulation control flow in the autonomous driving distributed simulation synchronous control method according to the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Traditional vehicle simulation consists of control force loading, dynamics simulation, and 3D image rendering system. In the autonomous driving simulator of the present invention, in consideration of the processing performance of sensors and artificial intelligence algorithms, each sensor simulator and AI simulator responsible for generating virtual sensor learning data are configured separately from the global simulator, and the global simulation timestamp is configured in the global simulator. Distributed simulations are constructed to handle autonomous driving scenarios and vehicle dynamics. To this end, we present a parallel simulation method of each simulator using a shared buffer and a simulation synchronous control and interpolation method using a fixed time interval method.

1. AI Simulator: Control Force Loading

The AI simulator performs the function of loading control forces.

The AI simulator's vehicle control simulation processes autonomous driving ground truth data collected from multiple sensors through machine learning and generates a vehicle control force. The correct answer data collected according to the sensor simulation cycle is transmitted in the form of an event to the global simulator in synchronization with the global simulation cycle, and all events including the timestamp of each sensor simulation cycle are added to the simulation message queue.

1 is a flowchart for explaining parallel processing and data synchronization using a shared buffer. According to the previous (t _n-1 ) synchronization signal (①) generated in the AI simulation cycle, the next (t _n+1 ) event is read and They are sorted according to the timestamp of the sensor and delivered to the AI simulator (②). At this time, the AI simulator transmits the control force, which is the result of the previous (t _n-1 ) processing, as a synchronization signal to the global simulator (①), and then performs a new operation from the current (t _n ) event without waiting for the next (t _n+1 ) sensor event. Execute (③) to enable parallel processing with the global simulator.

The AI simulator allows for a quick response by repeating the simulation as often as possible whenever there is a message in the shared request buffer. The request buffer repeats the process of reading the previous operation result (①), writing the next data (②), and operating the current data (③) after the initial buffer is all filled.

2. Global Simulator: Dynamics Simulation

The global simulator performs the dynamics simulation function.

2 is a flowchart for explaining the acquisition of global simulation timestamps based on fixed time intervals. The global simulator simulates various predefined driving scenarios in real time, and loads the control force transmitted from AI to periodically perform vehicle dynamics simulation. .

In general, the global simulation period is higher than the AI period, allowing the simulation of vehicle dynamics with continuous control force even if the control force does not change. However, since the dynamics engine requires a higher cycle to calculate the physical properties, the dynamics simulation cycle must be managed separately. That is, the execution cycle is included so that the dynamics simulation can be performed more than once during the global simulation for updating the 3D scene.

As shown in Equation 1, the global simulator performs rendering as often as possible, and at this time, whenever the measured current system clock exceeds the defined fixed time, the simulation is repeatedly performed (steps), and the time interval (t _delta ) We get the simulation timestamp (t _fixed ) incremented by

The ego vehicle information obtained from the dynamics simulation is transmitted to the AI simulator according to the global simulation cycle of a fixed time so that it can be used in the AI control logic. In addition, the global simulator transmits the 3D scene information obtained by the scenario and dynamics corresponding to the period along with the calculated simulation timestamp to each sensor simulator.

3. Sensor Simulator: Sensor Rendering

The sensor simulator performs the sensor rendering function.

The scene information transmitted from the global simulator includes real-time virtual scenarios such as dynamic objects and environments and information about autonomous vehicles. Since the global simulation cycle and the sensor simulation cycle are different, there is a problem in that the sensor obtains the same result until the state of the 3D virtual world is updated.

3 is a flowchart for explaining interpolation rendering of a previous scene according to a time lapse rate. The above problem can be solved by simulating a time interval before the global simulation time for the sensor rendering time with reference to the timestamp calculation of the global simulation.

As shown in Equation 2, the scene information of the current sensor time (f(20)) and the current simulation time scene information (f(40)) that have already been calculated are linear interpolation of the time ratio f(50)) can be obtained.

In addition, in the case of an actual LiDAR sensor, a 360-degree point cloud cannot be obtained at the same time when scanning a 3D space, so it rotates at a specific angle and scans, so the position and time of the sensor that scanned a scene may be different. As such, even when the time interval that a specific sensor can scan simultaneously is smaller than the time interval of the synchronized global simulation, the matrix of the previous scene and the current scene can be interpolated multiple times, enabling more detailed sensor simulation.

As mentioned earlier, the simulation timestamp is determined by the scene update cycle of the scenario, and the dynamics simulation can generate more detailed dynamic vehicle information than the scenario.

4 is a flowchart for explaining a sensor scan time simulation reflecting the dynamics cycle. The global simulator separately configures and transmits a vehicle information buffer equal to the dynamics cycle rate, and each simulator that receives this interpolates individual vehicle information with the sensor scan cycle. This enables more detailed simulations. That is, since the scene information of the scenario can be obtained by interpolating the global simulation time interval (20), and the autonomous vehicle information can be obtained by interpolating the higher dynamics simulation time interval (2), the sampling period of each sensor (20 Hz) and A more precise sensor simulation result can be obtained by reflecting the physical characteristics of the scan time (75°, 10).

4. Integrated simulator: real-time distributed simulation

The integrated simulator performs a real-time distributed simulation function.

5 is a flowchart for explaining the real-time autonomous driving simulation control flow. The distributed autonomous driving simulator proposed in this patent consists of individual simulators dedicated to processing sensors and AI algorithms to distribute the load, and driving information between the simulators. are synchronized through the synchronization method of fixed time intervals.

In this way, sensor simulation facilitates the realization of photorealism and facilitates extension to various high-performance game rendering engines.

In addition, it has the advantage of laying a scalable foundation for simulation that can process complex deep learning operations of autonomous driving AI algorithms in real time.

When multiple physically separated simulators are connected, start time synchronization is performed from the global clock received from the global simulator, and the rendering time of each simulator is corrected by this time difference to synchronize with the global simulation time.

Capture events generated by each sensor simulator are added to the global message queue as a global simulation cycle, and are delivered to AI by sorting them at each AI simulator cycle. In addition, the global simulator periodically updates the vehicle dynamics information of the next simulation time and transmits it to each sensor simulator at the same time to synchronize the time of the entire simulator.

In addition to autonomous vehicle control, interactions with various surrounding objects and driving environment conditions were reproduced according to the time in a predefined scenario, and 3D scene information was synchronized and updated with the global simulation cycle of the scenario. And AI and sensors selectively use simulation cycles with variable and fixed time intervals according to the deep learning processing performance and the physical characteristics of each sensor.

In addition, the message requested by the simulators is configured to facilitate parallel processing by managing one or more scenes and sensor buffers to perform the next simulation without waiting for data request.

①; Read previous operation result
②; Write next data
③; Current data operation

Claims (5)

Each sensor simulator and AI simulator responsible for generating virtual sensor learning data by considering the processing performance of sensors and artificial intelligence algorithms in the autonomous driving simulator for traditional vehicle simulation including control force loading, dynamics simulation, and 3D image rendering system is configured separately from the global simulator, and distributed simulation is configured to process the autonomous driving scenario and vehicle dynamics of the global simulation timestamp in the global simulator. In an autonomous driving distributed simulation synchronous control method that presents a simulation synchronous control and interpolation method of
Vehicle control simulation of AI simulator loading control force processes autonomous driving ground truth data collected from multiple sensors through machine learning and generates vehicle control force;
Correct answer data collected according to the sensor simulation cycle is transmitted to the global simulator in the form of an event in synchronization with the global simulation cycle;
All events with timestamps of each sensor simulation cycle are added to the simulation message queue;
Parallel processing and data synchronization using a shared buffer reads the next (t _n+1 ) event according to the previous (t _n-1 ) synchronization signal (①) generated in the AI simulation cycle and sorts it according to the sensor’s timestamp. It is transmitted to the AI simulator (②). At this time, the AI simulator transmits the control force, which is the result of the previous (t _n-1 ) processing, as a synchronization signal to the global simulator (①), and then performs a new operation from the current (t _n ) event without waiting for the next (t _n+1 ) sensor event. execute (③) to enable parallel processing with the global simulator;
The AI simulator repeats the simulation as often as possible whenever there is a message in the shared request buffer, enabling a fast response;
Autonomous driving distributed simulation synchronous control, characterized in that the request buffer repeatedly performs the processes of reading the previous operation result (①), writing the next data (②), and calculating the current data (③) after the initial buffer is all filled Way.
The method according to claim 1,
The global simulator performing dynamics simulation simulates various predefined driving scenarios in real time, and loads the control force transmitted from AI to periodically perform vehicle dynamics simulation;
The global simulation period is higher than the AI period, so that vehicle dynamics can be simulated with continuous control force even if the control force does not change, but the dynamics engine requires a higher period to calculate the physical properties, so to manage the dynamics simulation period separately, Configure including the execution cycle so that the dynamics simulation can be performed more than once during the global simulation for updating the 3D scene;
As shown in Equation 1, the global simulator performs rendering as often as possible, and at this time, whenever the measured current system clock exceeds the defined fixed time, the simulation is repeatedly performed (steps) and increased by the time interval (t _delta ). get the simulation timestamp (t _fixed );
(Equation 1)

The ego vehicle information obtained by dynamic simulation is transmitted to the AI simulator according to the global simulation cycle of a fixed time so that it can be used in the AI control logic.
The global simulator is a distributed simulation synchronous control method for autonomous driving, further comprising transmitting 3D scene information obtained through a scenario and dynamics corresponding to a cycle to each sensor simulator together with a calculated simulation timestamp.
The method according to claim 1,
In the sensor simulator for sensor rendering,
The scene information transmitted from the global simulator includes real-time virtual scenarios such as dynamic objects and environments and information of autonomous vehicles;
Because the global simulation cycle and the sensor simulation cycle are different, the sensor gets the same result until the state of the 3D virtual world is updated. Solve by simulation;
As shown in Equation 2, the scene information (f() 50)) can be obtained;
(Equation 2)

In the case of an actual LiDAR sensor, when scanning a 3D space, it is not possible to obtain a 360-degree point cloud at the same time, so the scan is rotated at a specific angle. Even when the possible time interval is smaller than the time interval of the synchronized global simulation, the matrix of the previous scene and the current scene can be interpolated multiple times, enabling a more detailed sensor simulation;
As described above, the simulation timestamp is determined by the scene update period of the scenario, and the dynamics simulation further includes being able to generate more detailed dynamic vehicle information than the scenario.
The method according to claim 1,
In the sensor scan time simulation that reflects the dynamics cycle, the global simulator separately configures and transmits a vehicle information buffer equal to the dynamics cycle ratio, and each simulator that receives this interpolates individual vehicle information with the sensor scan cycle to perform a more detailed simulation. , since the scene information of the scenario can be obtained by interpolating the global simulation time interval (20), and the autonomous vehicle information can be obtained by interpolating the higher dynamics simulation time interval (2), the sampling period (20 Hz) of each sensor and the scan The autonomous driving distributed simulation synchronous control method further comprising allowing more precise sensor simulation results to be obtained by reflecting the physical characteristics of time (75°, 10).
The method according to claim 1,
Distributed autonomous driving simulator, an integrated simulator that performs real-time distributed simulation, distributes the load by configuring a simulator dedicated to processing sensors and AI algorithms into individual simulators, and synchronizes driving information between simulators through a synchronization method at fixed time intervals;
Sensor simulation facilitates the realization of photorealism and facilitates extension to a variety of high-performance game rendering engines;
Laying a scalable foundation for simulation capable of real-time processing of complex deep learning operations of autonomous driving AI algorithms;
When a plurality of physically separated simulators are connected, start time synchronization is performed from the global clock received from the global simulator, and the rendering time of each simulator is corrected by this time difference to synchronize with the global simulation time;
Capture events generated by each sensor simulator are added to the global message queue at the global simulation cycle, sorted and delivered to the AI at each AI simulator cycle;
The global simulator periodically updates the vehicle dynamics information of the next simulation time and simultaneously transmits it to each sensor simulator to synchronize the time of the entire simulator;
Not only autonomous vehicle control, but also interactions with various surrounding objects and driving environment conditions are reproduced according to time in a predefined scenario, and 3D scene information is synchronized and updated with the global simulation cycle of the scenario;
AI and sensors selectively use simulation cycles with variable and fixed time intervals according to the deep learning processing performance and physical characteristics of each sensor;
The method further comprising configuring one or more scenes and sensor buffers to manage one or more scenes and sensor buffers to facilitate parallel processing by performing the next simulation without waiting for data request.

Images