KR102282970B1 - the method for configuring and controling real-time distributed autonomous driving simulation framework - Google Patents

Abstract

The present invention provides a method for configuring and controlling a real-time distributed autonomous driving simulation framework consisting of a server that facilitates verification of autonomous driving algorithms in various scenarios, and a simulator that uses sensor simulation and camera rendering devices as clients, and synchronizes and controls them. It relates to a real-time distributed autonomous driving simulation framework consisting of a server that facilitates verification of autonomous driving algorithms in various scenarios, a simulator that uses sensor simulation and camera rendering devices as clients, and controls them in synchronization with each other. as a method; the autonomous driving simulation framework is composed of a model system including a common simulation model and engine, an image system that reproduces sensors and 3D images, and a control system that controls each system in conjunction with each other based on communication; Each system enables interworking through IPC and Ethernet communication layers in a multi-process, distributed computing environment, and is controlled by the interface layer of an abstract model to facilitate replacement testing while maintaining the simulation relationship with the traditional M&S model; dividing the driving DB for the driving environment, objects, scenarios, etc. of the concept common to the autonomous driving domain and the image DB for 3D environment rendering, and distributing the simulation load by performing sensor emulation by dividing it in multiple clients; Physically, the simulator 20 is configured to be connected between the server 10 and the client 30 .

Description

{the method for configuring and controlling real-time distributed autonomous driving simulation framework}

The present invention provides a method for configuring and controlling a real-time distributed autonomous driving simulation framework consisting of a server that facilitates verification of autonomous driving algorithms in various scenarios, and a simulator that uses sensor simulation and camera rendering devices as clients, and synchronizes and controls them. is about

According to Registration No. 10-1916838 (Simulation Device for Autonomous Driving Vehicle), "The simulation device for autonomous driving vehicle is a plate-shaped base unit installed on the ground and is installed at intervals on the upper part of the base unit, but the vehicle is installed on the upper part of the base unit. A movable unit providing a mounting space to be raised; A rolling unit installed between the base unit and the movable unit to induce inclination in the vehicle width direction, and a balance unit to apply an elastic support force in up and down directions; An inertia unit having a track belt installed and cyclically moved by rolling contact of the wheels of the vehicle and a gear module connected to the track belt and rotated to generate a moment of inertia; One end is fixed to the movable unit and the other end is a track It consists of a steering interlocking unit which is provided to fixedly support the lower body of the vehicle on the front or rear wheels without positional interference with the belt and is provided to rotate according to the steering direction.

The simulation device for an autonomous driving vehicle according to the present invention configured as described above can actively implement and interpret the movement of the vehicle through the realization of a physical driving environment similar to the actual road driving environment, so that autonomous driving big data collection based on this can be achieved. Not only is it possible, but it can also be applied to entertainment or other difficult content, so a useful effect is expected to increase satisfaction with the product.

In addition, users who want to drive in various environments can experience the vivid driving feeling that can be felt during actual driving, so it can be used as content with various game elements and can satisfy differentiated fun and curiosity. there is an effect

In addition, the present invention not only enables localization through the development of a reliable autonomous driving simulator that provides an environment similar to actual driving for the global automobile market, but also creates large financial resources when exporting by opening a market for the global automobile market. It is expected, and a very useful effect is expected in the industry where it is possible to increase the employment of manpower suitable for the autonomous driving market and to enhance the technological competitiveness of domestic and foreign industries and national technology in preparation for the 4th industry."

According to Registration No. 10-1984762 (Autonomous Driving Vehicle Simulator with Network Platform), “It is an autonomous driving vehicle simulator system that operates a simulator based on data in a database. It is divided into datasets, and each dataset is configured to include at least one or more datapackets, so that one data packet in each dataset is selected so that a plurality of users can use their algorithms It relates to an autonomous vehicle simulator system that is provided to enable simulations in more diverse environments through sharing or compatibility with other algorithms."

According to Publication No. 10-2014-0144921 (Autonomous Driving Simulation System for Unmanned Cars Using Virtual Reality), "a virtual machine that performs autonomous driving simulation to verify the autonomous driving algorithm of a virtual driverless car within 3D virtual reality. It relates to an autonomous driving simulation system of an unmanned vehicle using reality.

The autonomous driving simulation system of an unmanned vehicle using virtual reality according to the present invention includes a simulator unit for driving an autonomous driving simulation by providing 3D image information in which a virtual reality environment is modeled, and a 3D image provided by the simulator unit. A simulation server that autonomously drives a virtual car in the , and receives the state information of the autonomously driven virtual car and driving information obtained from the virtual car, and the robot server by receiving the state information and driving information received from the simulation server It is characterized in that it includes a simulation component that converts and outputs data of type data, and a simulation monitor unit that receives OPRoS type data output from the simulation component and displays it as a GUI (Graphical User Interface)."

According to Registration No. 10-1850038 (car simulator apparatus and method), "disclosed is a car simulator apparatus and a method therefor. The car simulator apparatus according to the present invention is configured to include at least one editing module, and the at least one An input unit for inputting road information, object information, traffic information and operation information through the editing module of the virtual reality modeled in the 3D image of the virtual reality environment according to the input road information, object information, traffic information and operation information A simulator unit that autonomously drives a vehicle and acquires state information and driving information about the autonomous vehicle, and a simulator monitoring unit that receives the state information and driving information obtained from the simulator unit and outputs it as a virtual three-dimensional image According to the present invention, a road, an object, a city or a

It has the effect of facilitating the creation and editing of Rio and enabling real-time verification of the modeled 3D image."

1. Registration number 10-1916838 (simulation device for autonomous vehicles) 2. Registration No. 10-1984762 (Autonomous Driving Vehicle Simulator with Network Platform) 3. Publication No. 10-2014-0144921 (Autonomous driving simulation system of driverless car using virtual reality) 4. Registration number 10-1850038 (car simulator device and its method)

In recent years, in relation to autonomous driving technology, a lot of research has been done in relation to autonomous driving such as artificial intelligence, sensor fusion, and communication. In particular, the vast amount of learning data of various driving environments and scenarios required for artificial intelligence can be easily reproduced through a virtual autonomous driving simulator, so research related to this is actively underway.

Autonomous driving simulators are being developed based on game engines due to the advantage of being able to easily implement high-quality driving environments, but they are gradually showing their limitations as they require complex, large-scale and diverse driving environments.

First, for the photorealism of multiple precise sensor point cloud simulation and camera rendering, there is a performance limit in a single system configuration. As for the sensors used in actual autonomous driving, several (usually 10 levels) of lidar, radar, camera, and GPS are used, and when sensors such as ECU, acceleration, and gyro of the vehicle are included, numerous sensor signals are flooded at once. can come

In the case of LiDAR, which is a key sensor in sensor simulation, it is necessary to generate a large amount of point cloud data (approximately one million per second), and each sensor has different sampling period, resolution, effective distance, and noise characteristics. This is mainly developed based on CPU ray casting or GPU depth buffer of the game engine, but for precise simulation of multiple sensors, it is difficult to implement and has limitations in performance.

Camera rendering requires photorealistic rendering using methods such as photogrammetry and physically based rendering, but it is difficult to implement and uses many resources like sensors.

Second, there is a problem that the simulation model is dependent on the rendering model because the game engine generates the correct answer data (ground truth) and scenarios essential for verification and learning of the autonomous driving cognitive algorithm. The simulation model needs to be simplified into conceptual and abstract models such as 3D data such as the location, size, and type of autonomous vehicles and lanes and images extracted from sensors.

The road network creation plug-in supported by the game engine requires a lot of manual work for complex intersections, etc., and does not support lane information and road network topology extraction. There is a need to simplify a common simulation models to support the creation a network of roads manufactured using specialized 3 ^rd party editing tools to edit the lane and 3-D mesh and HD maps.

Third, there is a lack of effective simulator control methods (SIL/HIL, Open/Closed loop, Whole/Partial stack, etc.) for developing, testing, and verifying autonomous driving algorithms quickly and safely. Systematized simulation convergence with external systems to enable various tests such as a single test of some autonomous driving algorithm, replacement test with another autonomous driving system, interworking with actual camera or ECU, rendering engine replacement test, and dynamic scenario change based on the correct answer data Basic technology is urgently needed.

The present invention relates to a method for configuring and controlling a real-time distributed autonomous driving simulation framework consisting of a server that facilitates verification of autonomous driving algorithms in various scenarios, and a simulator that uses sensor simulation and camera rendering devices as clients and controls them in synchronization with each other. will be.

The present invention provides a method for configuring and controlling a real-time distributed autonomous driving simulation framework consisting of a server that facilitates verification of autonomous driving algorithms in various scenarios, and a simulator that uses sensor simulation and camera rendering devices as clients, and synchronizes and controls them. is implemented as

The features of the present invention are as follows.

the autonomous driving simulation framework is composed of a model system including a common simulation model and engine, an image system that reproduces sensors and 3D images, and a control system that controls each system in conjunction with each other based on communication; Each system enables interworking through IPC and Ethernet communication layers in a multi-process, distributed computing environment, and is controlled by the interface layer of an abstract model to facilitate replacement testing while maintaining the simulation relationship with the traditional M&S model; dividing the driving DB for the driving environment, objects, scenarios, etc. of the concept common to the autonomous driving domain and the image DB for 3D environment rendering, and distributing the simulation load by performing sensor emulation by dividing it in multiple clients; Physically, the simulator 20 is configured to be connected between the server 10 and the client 30 .

In an embodiment, the architecture for the autonomous driving simulation is based on MVC, and consists of a model unit 40 , a control unit 50 , and an image unit 60 , and the model unit 40 is an autonomous driving driving algorithm Each driving environment and object state, and actions (behaviors) and interactions for verifying , are expressed as a common conceptual component model, and real or virtual measured surrounding environments and objects with time-stamps are included. It encapsulates and manages information (data) and physical and mathematical control logic, and maintains the relationship between data and control logic of existing subsystems such as various autonomous driving algorithms, traffic conditions, driving scenarios, and vehicle dynamics, while maintaining the common It is abstracted into components including high-level states and actions, and configured so that subsystems can be processed directly in the model unit according to the instructions of the control unit 50 to facilitate load distribution and various scenario tests, and the model unit 40 The communication layer of the communication layer communicates the detected change of the driving state with the image unit 60 and the control unit 50 so that it can be easily extended to a distributed environment, and solves the coupling and synchronization problem by applying the observer mechanism of the traditional MVC, This includes the fact that a separate database for real-time processing of large-scale distributed data can be easily added.

In an embodiment, in the image unit 60, the actual autonomous driving is configured to recognize the surrounding environment such as vehicle, pedestrian, and road information based on data measured by radar, lidar and camera, GPS sensor mounted on the vehicle, etc. In order to simulate various sensors mounted on the vehicle in the virtual driving environment in the simulation, the image DB, sensor and camera emulation system, and image rendering system constitute the image generating device of the autonomous driving simulation, and the image object model is the driving environment. And the simulation state of the scenario is obtained from the model and reproduced as an image, the autonomous driving source data, which is the emulation result of the virtual sensor, is periodically added to the database, and the capture event is transmitted to the controller, and the image unit 60 is a high-performance 3D rendering engine It consists of one or more systems equipped with a virtual camera to distribute the load of signal generation of various virtual cameras and sensors and increase the accuracy of the simulation. To this end, individual image files are used to reproduce the simulation model as a model of each 3D rendering engine. After loading , it includes synchronizing with the driving object information of the model unit 40 through the image object model.

In an embodiment, the control unit 50 performs a role of supervising and controlling the overall simulation such as setting various environments necessary for simulation, HILS/SILS verification, and managing scenarios, event setting, and various data collection and monitoring in test conditions and It enables analysis, the simulator provides an interface between simulation operation/control and subsystems, and a diagnostic function, and the control unit 50 is in charge of processing internal/external events (renderer events, state changes, etc.), and the life cycle and control the logic of the data-synchronized model, in which the control unit processes events that can be performed first, and delegates the processing of impossible events through the communication layer of the model unit 40 to users and other simulations, other application systems, etc. It performs an interface function to connect models, directly/indirectly controls the logic of simulation cycle such as artificial intelligence and vehicle dynamics, and unlike the image unit 60, only loads the system settings at load time, and controls the model unit 40 acts as a proxy for the, and controls the model unit 40 to process the events collected by the image unit 50 at runtime, and to reproduce a signal triggered by a change in the driving state of the model unit 40 as an image. This includes instructing the imaging unit 60 .

In an embodiment, the simulator models the detailed model information of the M&S engine implicitly as a conceptual component of autonomous driving, thereby allowing complex M&S engines to lay a scalable foundation for the development of a simulation environment without loss of integrity, The simulator is configured to interwork with the simulation engine in the form of a plug-in that can be extended to the game engine, and sensors and artificial intelligence algorithms for verification are also managed as plug-ins. There is a communication problem such as format, and it is to solve this problem through the lifecycle management of the simulation engine and the relay role of the communication layer according to the context of the control flow.

According to preferred effects of the present invention, 1) it is possible to configure a distributed simulation framework that does not depend on a 3D image engine, 2) improves the reusability of a component-based autonomous driving model and testability of an AI algorithm, 3) multi- It has the effect of improving the sensor emulation data synchronization and parallel distributed processing performance, 4) can lay the foundation for autonomous driving cloud service through the abstract model of autonomous driving simulation and communication layer, and 5) other simulation systems through the interface layer It has the advantage of improving the extension, validation of the model, and the simulation environment.

1 is a block diagram illustrating a distributed autonomous driving simulation framework according to the present invention.
2 is a block diagram illustrating an MVC-based autonomous driving simulation architecture according to the present invention.
3 is a flowchart illustrating a real-time data processing process for logic control of a sensor input signal according to the present invention.
4 is a flowchart illustrating a real-time data processing process for an updated image/sensor output according to the present invention.
5 is a block diagram showing the configuration of an autonomous driving simulator according to the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1. Framework

Autonomous driving simulation consists of a fusion of various modeling-simulation (M&S) technologies, and requires autonomous driving algorithms, vehicle dynamics simulations, scenarios, virtual sensors, and 3D image rendering engines.

The autonomous driving simulation framework consists of a model system including a common simulation model and engine, an image system that reproduces sensors and 3D images, and a control system that controls each system in conjunction with each other based on communication.

Referring to the distributed autonomous driving simulation framework with reference to FIG. 1 , the upper configuration includes the server 10 , the middle configuration includes the simulator 20 , and the bottom configuration includes the client 30 .

Each system enables interworking through IPC and Ethernet communication layers in a multi-process, distributed computing environment, and is controlled by an interface layer of an abstract model to facilitate replacement testing while maintaining the traditional M&S model and simulation relationship.

It divides the driving DB for the driving environment, objects, and scenarios, which are common concepts in the autonomous driving domain, and the image DB for 3D environment rendering, and distributes the simulation load by performing sensor emulation by dividing it among multiple clients.

2. Model Department

The model unit 40 shown in FIG. 2 expresses each driving environment and object state, and action (action) and interaction for verifying the driving algorithm of autonomous driving as a common conceptual component model.

Here, the actual or virtually measured surrounding environment and entity information (data) including time-stamps and physical and mathematical control logic are encapsulated and managed.

While maintaining the relationship between data and control logic of existing subsystems such as various autonomous driving algorithms, traffic conditions, driving scenarios, and vehicle dynamics, it is abstracted into components including high-level states and behaviors common to each subsystem.

Thereafter, according to the instruction of the control unit 50, the subsystem is configured to be directly processed by the model unit to facilitate load distribution and various scenario tests.

The communication layer of the model unit communicates the detected change of the driving state with the image unit and the control unit to facilitate expansion into a distributed environment. At this time, by applying the observer mechanism of the traditional MVC, the coupling and synchronization problems are solved, and a separate database for real-time processing of large-scale distributed data can be easily added.

3. Imaging department

Actual autonomous driving is configured to recognize the surrounding environment, such as vehicle, pedestrian and road information, based on data measured by radar, lidar and camera, and GPS sensor mounted on the vehicle.

In order to simulate various sensors mounted on a vehicle in a virtual driving environment in the simulation, an image DB, a sensor and camera emulation system, and an image rendering system constitute an image generating device for autonomous driving simulation.

The image object model obtains the simulation state of the driving environment and scenario from the model and reproduces it as an image, periodically adds the autonomous driving source data, which is the emulation result of the virtual sensor, to the database, and delivers the capture event to the controller.

The imaging unit 60 is composed of one or more systems equipped with a high-performance 3D rendering engine to distribute the load of signal generation of various virtual cameras and sensors and to increase the accuracy of simulation.

To this end, after loading individual image files to reproduce the simulation model as a model of each 3D rendering engine, it is synchronized with the driving object information of the model unit [1] through the image object model.

That is, when a change in the state of the driving object model is detected, it is notified to the image unit, and the image unit inquires the change state from the model and reflects it in the image object model. In addition, although the screen rendering is changed as often as possible, each sensor performs sampling at a cycle suitable for characteristics and performs data synchronization at a fixed time step in which the result is transmitted to the control unit 50 as an event.

4. Controls

The control unit 50 performs the role of supervising and controlling the overall simulation, such as setting various environments necessary for the simulation, and verifying HILS/SILS. Scenario management, event setting, and various data collection, monitoring and analysis are possible under test conditions. In addition, the simulator provides an interface between simulation operation/control and subsystems, and diagnostic functions.

The controller 50 is in charge of processing internal/external events (renderer events, state changes, etc.), and controls the logic of the life cycle and data-synchronized model. In this case, the control unit processes the event that can be performed first, and delegates the processing of the impossible event through the communication layer of the model unit [1].

In addition, the control unit 50 performs an interface function to connect the model to the user, other simulations, other application systems, and the like, and directly/indirectly controls the logic of the simulation cycle such as artificial intelligence and vehicle dynamics.

Unlike the image unit 60 , only the system settings are loaded at load time, and the model unit 40 serves as a proxy for control. For example, simulation control such as start, stop, and pause is completed by repeatedly calling a corresponding action for each component of the model unit 40 .

At runtime, the model unit 40 is controlled to process the event collected by the image unit 50, and the image unit 60 is instructed to reproduce a signal triggered by a change in the driving state of the model unit 40 as an image. do.

5. Simulator

The core architecture proposed in the present invention is a fusion between the autonomous driving simulation concept and the classic MVC paradigm, and the autonomous driving simulator configuration has the configuration shown in FIG. 5 .

Referring to FIG. 5 , detailed model information of the M&S engine is implied and modeled as conceptual components of autonomous driving (particularly, road network and correct answer data).

In this way, complex M&S engines have the advantage of providing a scalable foundation for the development of simulation environments without loss of integrity.

The simulator presented in the present invention is configured to interwork with the simulation engine in the form of a plug-in expandable to the game engine, and the sensor and artificial intelligence algorithm for verification are also managed as plug-ins.

Distributed environment simulation control between the model server and the image client has communication problems such as timing, synchronization, parallelism, and data format, and solves these problems through the lifecycle management of the simulation engine that fits the context of the control flow and the relay role of the communication layer do.

In general, the purpose of the simulator is to facilitate user interaction and experimentation of the simulation model. The proposed framework was used to develop a terrain editor for editing driving environments such as terrain, weather, and road networks, a scenario editing tool such as driving scenarios, dynamic objects, and traffic generation, and simulators for data analysis, monitoring, and 3D simulation.

10; server 20; simulator
30; client 40; model department
50; control unit 60; video department

Claims (5)

A method of configuring and controlling a real-time distributed autonomous driving simulation framework comprising a server that facilitates verification of autonomous driving algorithms in various scenarios, a simulator that uses sensor simulation and camera rendering devices as clients, and controls them in synchronization with each other,
The autonomous driving simulation framework consists of a model system including a common simulation model and engine, an image system that reproduces sensors and 3D images, and a control system that controls each system in conjunction with each other based on communication,
Each system enables interworking through the IPC and Ethernet communication layers in a multi-process, distributed computing environment, and maintains the model and simulation relationship of the traditional modeling & simulation (hereinafter referred to as M&S) while maintaining the simulation relationship of the abstract model to facilitate replacement testing. Controlled by the interface layer,
It divides the driving DB with driving environment, objects and scenarios in the autonomous driving domain and the image DB for 3D environment rendering, and distributes the simulation load by performing sensor emulation by dividing it in multiple clients,
Physically, the simulator 20 is configured to be connected between the server 10 and the client 30,
The architecture for the autonomous driving simulation is configured based on the Michigan Vehicle Code (hereinafter referred to as MVC),
In the autonomous driving simulation framework, it is configured to include a model unit 40, a control unit 50, and an image unit 60,
The model unit 40 expresses each driving environment and object state, action (action), and interaction for verifying the driving algorithm of autonomous driving as a common conceptual component model, and a time-stamp is It encapsulates and manages the included real or virtual measured surrounding environment and object information (data) and physical and mathematical control logic together,
While maintaining the relationship between data and control logic of existing subsystems such as various autonomous driving algorithms, traffic conditions, driving scenarios, and vehicle dynamics,
In accordance with the instructions of the control unit 50, the subsystem is configured to be processed directly in the model unit to facilitate load distribution and various scenario tests,
The communication layer of the model unit 40 is a real-time distributed autonomous driving simulation framework including communicating the detected change of the driving state with the image unit 60 and the control unit 50 so as to be easily extended to a distributed environment. How to configure and control.
The method according to claim 1,
In the imaging unit 60,
Actual autonomous driving is configured to recognize the surrounding environment such as vehicle, pedestrian and road information based on data measured by radar, lidar and camera, and GPS sensor mounted on the vehicle.
In order to simulate various sensors mounted on the vehicle in the virtual driving environment in the simulation, the image DB, sensor and camera emulation system, and image rendering system constitute the image generating device of the autonomous driving simulation.
The image object model obtains the simulation state of the driving environment and scenario from the model and reproduces it as an image, periodically adds the autonomous driving source data, which is the emulation result of the virtual sensor, to the database, and delivers the capture event to the controller.
The imaging unit 60 is composed of one or more systems equipped with a high-performance 3D rendering engine to distribute the load of signal generation of various virtual cameras and sensors and to increase the accuracy of the simulation. A method of configuring and controlling a real-time distributed autonomous driving simulation framework comprising loading an individual image file to reproduce it as a model of a rendering engine and synchronizing it with driving object information of the model unit 40 through an image object model.
The method according to claim 1,
The control unit 50 performs the role of supervising and controlling the overall simulation such as setting various environments required for simulation and HILS/SILS verification, and enabling scenario management, event setting, and various data collection, monitoring and analysis under test conditions. The simulator provides interfaces between simulation operation/control and subsystems, and diagnostic functions.
The control unit 50 is in charge of processing internal/external events (renderer event, state change, etc.), and controls the logic of the life cycle and data-synchronized model. Delegates the processing through the communication layer of the model unit 40, performs an interface function that connects models with users, other simulations, other application systems, etc., and directly/executes the logic of the simulation cycle such as artificial intelligence and automobile dynamics. indirectly controlled,
Unlike the image unit 60 , only the system settings are loaded at load time, the model unit 40 acts as a proxy for control, and the model unit 40 is controlled to process the events collected by the image unit 50 at runtime. and instructing the image unit 60 to reproduce a signal triggered by a change in the driving state of the model unit 40 as an image.
The method according to claim 1,
The simulator models the detailed model information of the M&S engine as a conceptual component of autonomous driving, and through this, complex M&S engines lay a scalable foundation for the development of a simulation environment without loss of integrity,
The method of configuring and controlling a real-time distributed autonomous driving simulation framework, comprising configuring the simulator in the form of a plug-in, and allowing the verification sensor and artificial intelligence algorithm to be managed as a plug-in. delete

Images