KR20230095581A - Method and apparatus for generating training data for object recognition - Google Patents

Abstract

A method and apparatus for generating training data for object recognition are disclosed. The method for generating training data for object recognition according to one embodiment of the present disclosure comprises: a step of controlling a driving simulation environment based on simulation environment data and controlling a vehicle simulator to drive own vehicle in the driving simulation environment; a data extraction step of collecting simulation information data based on the field of view (FOV) of the own vehicle while driving; a two-dimensional bounding box generation step of generating a two-dimensional bounding box of at least one object present within the field of view while the own vehicle is driving based on the simulation information data; and a training data generation step of generating a training data image for object recognition of objects based on the bounding box. According to the present invention, time and costs required to generate data can be reduced.

Description

Method and apparatus for generating training data for object recognition {METHOD AND APPARATUS FOR GENERATING TRAINING DATA FOR OBJECT RECOGNITION}

The present disclosure relates to a method and apparatus for generating training data for object recognition automatically based on a vehicle simulator.

For autonomous driving, developing a camera-based perception system requires an enormous amount of training images. Typically, these training images are collected and labeled by human labor. This is costly and error-prone.

In addition, it is very difficult to collect a large amount of real-life driving images with sufficient variety because it is impossible to artificially create unusual corner cases or unusual weather and lighting conditions. Thus, the vast majority of data sets are being conducted by a handful of companies and institutions with significant budgets and resources.

Therefore, in order to generalize the difficult task of generating data sets for recognition systems, there is a need for a technique capable of automatically collecting training data and training a learning model for object recognition based on the automatically collected training data.

The foregoing background art is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and cannot necessarily be said to be known art disclosed to the general public prior to filing the present invention.

Prior Art 1: G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez, The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes, in Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 32343243. Prior Art 2: A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, CARLA: An open urban driving simulator, in Proceedings of the 1st Annual Conference on Robot Learning, 2017, pp. 116.

An object of an embodiment of the present disclosure is to generate a large amount of training data for object recognition for various situations based on a vehicle simulator.

An object of an embodiment of the present disclosure is to conveniently generate training data for object recognition using a driving simulator and convert a 3D bounding box obtained from the driving simulator into a 2D bounding box that matches the object as much as possible.

An object of an embodiment of the present disclosure is to easily collect training data for object recognition by automatically synthesizing driving scenes for situations that rarely occur in actual driving scenarios, such as corner cases, so that objects can be detected in 2D camera images. It is intended to further improve the accuracy of 2D object recognition learning to find the class and location of an object.

The purpose of the embodiments of the present disclosure is not limited to the above-mentioned tasks, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the embodiments of the present invention. will be. It will also be seen that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

A method for generating training data for object recognition according to an embodiment of the present disclosure includes the steps of controlling a driving simulation environment based on simulation environment data and controlling a vehicle simulator to drive a vehicle in the driving simulation environment; Data extraction step of collecting simulation information data based on the field of view (FOV) of the vehicle while driving, and 2D bounding of at least one object existing within the field of view while driving the host vehicle based on the simulation information data A 2D bounding box generating step of generating a box and a training data generating step of generating a training data image for object recognition of an object based on the bounding box may be included.

In addition to this, another method for implementing the present disclosure, another system, and a computer readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to an embodiment of the present disclosure, it is possible to reduce the time and cost required for data generation by generating a considerable amount of realistic driving images together with training data for object recognition within a reasonable time through driving simulation for various situations. there is.

In addition, by simply generating training data for object recognition using a driving simulator and converting the 3D bounding box obtained from the driving simulator into a 2D bounding box that matches the object as much as possible, the object using the generated training data for object recognition The accuracy of recognition learning can be improved.

In addition, object recognition accuracy can be greatly improved in the real world, especially in harsh environments, by using synthetic training images artificially created with abnormal weather and lighting conditions that are difficult to obtain in real driving scenarios.

Effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is an exemplary image of objects recognized in different driving scenarios under various weather conditions in a driving simulator according to an embodiment of the present disclosure.
2 is a diagram schematically illustrating a system environment for generating training data for object recognition according to an embodiment of the present disclosure.
3 is a schematic block diagram of an apparatus for generating training data for object recognition according to an embodiment of the present disclosure.
4 is a diagram for explaining a process of generating training data for object recognition according to an embodiment of the present disclosure.
5 is an exemplary diagram for explaining a post-processing process for generating a final bounding box according to an embodiment of the present disclosure.
6 is a heat map of an object generated from 2D image coordinates according to an embodiment of the present disclosure.
7 is a scalability evaluation graph of an apparatus for generating training data for object recognition according to an embodiment of the present disclosure.
8 is an accuracy evaluation diagram of an apparatus for generating training data for object recognition according to an embodiment of the present disclosure.
9 is a training performance evaluation graph of an apparatus for generating training data for object recognition according to an embodiment of the present disclosure.
10 is an exemplary diagram schematically illustrating a vehicle to which an apparatus for generating training data for object recognition according to an embodiment of the present disclosure is applied.
11 is a flowchart illustrating a method of generating training data for object recognition according to an embodiment of the present disclosure.

Advantages and features of the present disclosure, and methods of achieving them, will become clear with reference to the detailed description of embodiments in conjunction with the accompanying drawings.

However, it should be understood that the present disclosure is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all conversions, equivalents, and substitutes included in the spirit and technical scope of the present disclosure. . The embodiments presented below are provided to complete the present disclosure and to fully inform those skilled in the art of the scope of the disclosure. In describing the present disclosure, if it is determined that a detailed description of related known technologies may obscure the gist of the present disclosure, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present disclosure. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded. Terms such as first and second may be used to describe various components, but components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof are omitted. I'm going to do it.

1 is a diagram schematically illustrating a result of generating training data for object recognition according to an embodiment of the present disclosure.

Figure 1 shows example images composited with four different driving scenarios (row-wise) under various weather conditions (row-wise). A bounding box of each image and the image itself may be automatically generated according to an embodiment of the present disclosure.

An apparatus for generating training data for object recognition according to an embodiment of the present disclosure may collect information according to simulation in an autonomous driving simulation server. An autonomous driving simulation server simulates a predefined virtual world (so-called city) in which data extraction clients interact with the server using APIs (application programming interfaces). The device for generating training data for object recognition can control the town's weather, own vehicles, surrounding vehicles, pedestrians, etc. through the API provided by the autonomous driving simulation server. That is, sensor data (eg, a front camera) of the own vehicle and physical attributes (eg, position, pose, speed) of surrounding vehicles and pedestrians (hereinafter referred to as objects) may be extracted through the information generating software.

2 is a diagram schematically illustrating a system environment for generating training data for object recognition according to an embodiment of the present disclosure.

Referring to FIG. 2 , an object recognition training data generation system 1 according to an embodiment may include an object recognition training data generation apparatus 100 , a server 200 and a network 300 .

In an embodiment, the object recognition training data generating apparatus 100 is for automatically synthesizing training data for 2D object recognition for autonomous driving, and is a target object (eg, a vehicle or a pedestrian) in a 2D camera image. It aims at the two-dimensional object recognition task of finding the class and location of

Object recognition plays an important role in, for example, advanced driver assistance systems (ADAS) and autonomous driving. Accordingly, in an embodiment, a training data set usable in an actual driving scenario may be generated through simulation and used for object recognition model learning.

Furthermore, in an embodiment, it is possible to perform object recognition model learning with better performance by securing corner cases that rarely occur in actual driving scenarios. Therefore, in one embodiment, by generating a training data set synthesized under various weather and lighting conditions, the accuracy of real object recognition can be greatly improved.

To this end, the apparatus 100 for generating training data for object recognition may be largely composed of a part that performs a simulation, a part that collects simulation-related information, and a data generation part that extracts a 2D bounding box from the collected information. A detailed description of this configuration will be described later.

Meanwhile, in one embodiment, a part or the whole of the apparatus 100 for generating training data for object recognition may be implemented as the server 200 . When a part thereof is implemented as the server 200, the apparatus 100 for generating training data for object recognition may generate data by receiving simulation results and simulation-related information performed by the server 200.

In addition, in one embodiment, the training data generation apparatus 100 for object recognition is configured as a configuration including a simulator that performs simulation and a module that performs data extraction after the simulator and data generation based on the extracted data, or in another embodiment. In an example, it may be implemented as a post-processing module that generates data based on data extracted from the simulator.

In one embodiment, the training data generating apparatus 100 for object recognition may be implemented in the apparatus 100 for generating training data for object recognition and/or the server 200, where the server 200 provides training data for object recognition. It may be a server for operating the object recognition training data generating system 1 including the generating device 100 or a server implementing all or part of the object recognition training data generating device 100 .

In one embodiment, the server 200 extracts various information in the simulation step and accurately transforms the 3D bounding boxes of objects in the image into 2D bounding boxes in the overall process of generating a training data set for object recognition learning. It may be a server that controls the operation of the apparatus 100 for generating training data for object recognition. A training data set may refer to data labeled as ground truth.

Also, the server 200 may be a database server that provides data for operating the apparatus 100 for generating training data for object recognition. In addition, the server 200 may include a web server, an application server, or a deep learning network providing server.

In addition, the server 200 may include a big data server and an AI server required to apply various artificial intelligence algorithms, and a calculation server that performs calculations of various algorithms.

Also, in this embodiment, the server 200 may include the aforementioned servers or network with these servers. That is, in this embodiment, the server 200 may include the above web server and AI server or network with these servers.

In the object recognition training data generation system 1 , the object recognition training data generation apparatus 100 and the server 200 may be connected by a network 300 . Such a network 300 may be, for example, a wired network such as local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), and integrated service digital networks (ISDNs), wireless LANs, CDMA, Bluetooth, and satellite communication. However, the scope of the present invention is not limited thereto. Also, the network 300 may transmit and receive information using short-range communication and/or long-distance communication.

In addition, the network 300 may include connections of network elements such as hubs, bridges, routers, switches, and gateways. Network 300 may include one or more connected networks, such as a multiple network environment, including a public network such as the Internet and a private network such as a secure corporate private network. Access to network 300 may be provided via one or more wired or wireless access networks. Furthermore, the network 300 may support an Internet of Things (IoT) network and/or 5G communication in which information is exchanged and processed between distributed components such as things.

3 is a block diagram schematically illustrating an apparatus for generating training data for object recognition according to an embodiment of the present disclosure, and FIG. 4 is a diagram for explaining a process for generating training data for object recognition according to an embodiment of the present disclosure. .

Referring to FIG. 3 , the apparatus 100 for generating training data for object recognition may include a communication unit 110 , a user interface 120 , a memory 130 and a processor 140 .

The communication unit 110 may provide a communication interface necessary to provide a transmission/reception signal between external devices in the form of packet data in conjunction with the network 300 . In addition, the communication unit 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals to and from other network devices through wired or wireless connections.

That is, the processor 140 may receive various data or information from an external device connected through the communication unit 110 and may transmit various data or information to the external device.

In one embodiment, the user interface 120 is an input into which user requests and commands for controlling the operation of the apparatus 100 for generating training data for object recognition (eg, changing simulation conditions, changing parameters of a data generating algorithm, etc.) are input. May contain interfaces. That is, the user interface 120 may mean a client API of a data extraction unit (142 in FIG. 4) to be described later.

In one embodiment, the user interface 120 may include an output interface outputting a result of generating training data for object recognition. That is, the user interface 120 may output results according to user requests and commands. An input interface and an output interface of the user interface 120 may be implemented in the same interface.

The memory 130 can store various information necessary for controlling (operation) the operation of the apparatus 100 for generating training data for object recognition and store control software, and may include a volatile or non-volatile recording medium.

The memory 130 is connected to one or more processors 140 through an electrical or internal communication interface, and when executed by the processor 140, causes the processor 140 to control the apparatus 100 for generating training data for object recognition. You can store code that causes.

Here, the memory 130 may include non-temporary storage media such as magnetic storage media or flash storage media, or temporary storage media such as RAM, but the scope of the present invention is not limited thereto. The memory 130 may include built-in memory and/or external memory, and may include volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, Non-volatile memory such as NAND flash memory, or NOR flash memory, SSD. It may include a compact flash (CF) card, a flash drive such as an SD card, a Micro-SD card, a Mini-SD card, an Xd card, or a memory stick, or a storage device such as an HDD. Also, information related to an algorithm for performing learning according to the present disclosure may be stored in the memory 130 . In addition, various information necessary within the scope of achieving the object of the present disclosure may be stored in the memory 130, and the information stored in the memory 130 may be updated as received from a server or an external device or input by a user. may be

The processor 140 may control overall operations of the apparatus 100 for generating training data for object recognition. Specifically, the processor 140 is connected to the configuration of the apparatus 100 for generating training data for object recognition including the memory 130, and generates training data for object recognition by executing at least one command stored in the memory 130. Overall operation of the device 100 may be controlled.

Processor 140 can be implemented in a variety of ways. For example, the processor 140 may include an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor Processor, DSP) may be implemented as at least one.

The processor 140, as a kind of central processing unit, may control the operation of the apparatus 100 for generating training data for object recognition by driving control software loaded in the memory 130. The processor 140 may include any type of device capable of processing data. Here, a 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or instructions included in a program, for example.

A process for generating training data for object recognition by the processor 140 will be described with reference to FIG. 4 .

As shown in FIG. 4 , the apparatus 100 for generating training data for object recognition may include a simulation unit 141 , a data extraction unit 142 and a data generation unit 143 .

The simulation unit 141 may control a driving simulation environment based on the simulation environment data and control a vehicle simulator to drive the own vehicle in the driving simulation environment. That is, the simulation unit 141 may perform driving simulation and provide simulation information data related to the driving simulation to the data extraction unit 142 . In one embodiment, the simulation unit 141 may refer to a vehicle simulator or a setting unit that sets a driving environment in the vehicle simulator.

The simulation unit 141 simulates a virtual world (a so-called village) and controls a driving simulation environment according to a command to change simulation environment data from the data extraction unit 142 . For example, the data extraction unit 142 may change simulation environment data from a client using an API for communication.

In one embodiment, in the simulation in the simulation unit 141, there are three important concepts of world, actor, and blueprint.

The simulation unit 141 may generate a virtual world using a predefined map. In the virtual world created in this way, the actor can be any object that plays a specific role in the simulation, and generally moves around the virtual world's town. Actors may include, for example, vehicles, pedestrians, and sensors.

And actors can be placed in blueprints, which are 3D models with animation effects. The simulation unit 141 of one embodiment may provide a set of blueprints that can be basically used. A set of blueprints can include, for example, different vehicle models, different types of pedestrians (eg male, female, child, etc.) and different sensors (eg cameras, lidars, radars, etc.).

That is, the simulation unit 141 can perform simulation after creating a virtual world with necessary actors, and controls the weather and lighting conditions according to the simulation environment data change command from the data extraction unit 142, It can move surrounding vehicles and pedestrians.

Meanwhile, in an embodiment, the processor 140 may control a driving environment of the simulation unit 141 to obtain various training data. That is, the processor 140 may control the vehicle simulator by dynamically changing the driving speed of the host vehicle or changing the weather or driving time of the driving simulation environment.

Further, the processor 140 may change data collection according to the driving speed of the own vehicle, and if the driving speed of the own vehicle is fast, the collection period may be increased. That is, the processor 140 may change the period of collecting simulation information data based on the driving speed of the host vehicle.

The data extraction unit 142 may automatically collect simulation information data from the simulation unit 141 . At this time, the data extraction unit 142 may input simulation environment data for data extraction through a client and collect simulation information data according to the input simulation environment data. For example, simulation environment data for data extraction may include vehicle control data, weather and lighting control data, surrounding object control data, and the like.

In particular, in one embodiment, when it is necessary to obtain simulation information data for a corner case, the data extraction unit 142 may request a simulation environment data change command from the simulation unit 141 to change driving simulation conditions for the corresponding corner case. there is. Therefore, the simulation unit 141 may perform a simulation according to the simulation environment data changed by the data extraction unit 142, and the data extraction unit 142 may collect simulation information data according to the simulation of the simulation unit 141. can In addition, the simulation unit 141 analyzes training data collected and accumulated after a certain period of time to control a simulation environment to suit the situation when training data in a specific situation is insufficient. For example, when training data for corner cases is insufficient, a driving route, driving period, or number of driving may be set so that the host vehicle is driven by repeating a route with many corner cases.

At this time, while the simulation is being performed, the data extraction unit 142 may extract simulation information data according to a preset period or according to an event that occurs based on a user's setting. However, it is not limited thereto, and the data extraction unit 142 may extract simulation information data in real time as time progresses.

The data extraction unit 142 may collect simulation information data based on a field of view (FOV) of the own vehicle driving in a driving simulation environment. That is, the data extraction unit 142 may obtain a front image captured by a front camera virtually installed in the vehicle. In addition, the data extraction unit 142 may obtain a 3D bounding box for at least one object present in the front image, and may also obtain a semantic segmentation image of the front image.

Further, the data extraction unit 142 may obtain information on a world coordinate system in which the center of the virtual world is the origin, a vehicle coordinate system in which the center of the own vehicle is the origin, and an image coordinate system in which the top left corner of the front image is the origin. That is, in one embodiment, a world coordinate system (3D) in which the coordinate origin is the center of the village, a vehicle coordinate system (3D) in which the coordinate origin is the center of the own vehicle, and an image coordinate system (2D) in which the coordinate origin is the upper left corner of the image. Simulation and simulation-related data extraction can be performed based on three different coordinate systems.

In other words, the data extraction unit 142 may extract data by controlling the simulated virtual world in terms of weather and lighting conditions and surrounding vehicles and pedestrians. In addition, the data extraction unit 142 may extract simulation information data necessary for generating a 2D bounding box, such as a 3D bounding box, a class, a camera image, and a semantic segmentation image necessary for generating a 2D bounding box.

와 같이 8개의 꼭지점으로 나타낼 수 있다. 그리고 각 점 C는 3차원 좌표에서 (x, y, z)로 나타낼 수 있다. 이때 데이터추출부(142)는

로 나타낼 수 있는 각 행위자의 클래스(차량 또는 보행자)도 함께 추출할 수 있다.A 3D bounding box represents the position and posture of all objects in 3D coordinates. For example, assuming that n actors are active in a village, the data extraction unit 142 may extract n 3D bounding boxes. A 3D bounding box is a set of 3D bounding boxes.

It can be represented by 8 vertices as And each point C can be expressed as (x, y, z) in three-dimensional coordinates. At this time, the data extraction unit 142

The class (vehicle or pedestrian) of each actor that can be represented by can also be extracted.

Also, the camera image may refer to 2D RGB data of a virtual front camera installed in the vehicle. In one embodiment, for example, a front camera capable of obtaining an RGB image with a resolution of 960 X 540 may be installed in the vehicle. Therefore, a camera image obtained from a front camera can be represented by a 960 X 540 matrix I. Here, each element I[x][y] may represent the color code of a pixel at the position (x, y) of the 2D image coordinates.

The semantic segmentation image may refer to an image classified into different color codes (for example, blue for vehicles and red for pedestrians) in units of pixels for each camera image. For example, for each RGB image, a corresponding semantic segmentation image may be extracted as a 960 X 540 matrix S. Here, each element S[x][y] can represent an object class whose pixels are based on a predefined color code.

While the vehicle is being driven, the data generator 143 processes raw data such as a 3D bounding box, camera image, and semantic segmentation image from the data extraction unit 142 to obtain accurate 2-dimensional images for all objects. By finding a dimensional bounding box, it is possible to generate training data for object recognition. In other words, the data generator 143 may generate a 2D bounding box of at least one object existing within the field of view while the vehicle is driving based on the simulation information data.

Meanwhile, in one embodiment, generation of training data for object recognition may be performed as a separate process that operates offline, different from the time point of data extraction. That is, the process performed by the data generator 143 may be referred to as a post-processing process. However, it is not limited thereto, and simulation information data may be collected in real time and training data for object recognition may be generated at the same time.

5 is an exemplary diagram for explaining a post-processing process for generating a final bounding box according to an embodiment of the present disclosure.

Referring to FIG. 5 , the data generator 143 may project vertex coordinates of the 3D bounding box onto an image coordinate system and convert the 3D bounding box projected onto the 2D image coordinate system into a 2D bounding box. And the data generator 143 filters the 2D bounding box of the obscured object based on the semantic segmentation image, and the shape of the 2D bounding box is suitable for the shape of the object corresponding to the 2D bounding box based on the semantic segmentation image. can be changed to create the final bounding box.

In one embodiment, the final bounding box may be saved in an appropriate file format and generated as training data for object recognition. For example, a plain text file format with a two-dimensional bounding box denoted by a center point (x, y) along with an object class number and height (h) and width (w) can be used. Also, a JSON (JavaScript Object Notation) based file format may be used. However, in one embodiment, since the file format is not limited and conversion between different formats is simple, it can be applied to various object recognition algorithms.

Meanwhile, in an embodiment, when there are several objects in the camera image, each object may be processed one by one. Therefore, hereinafter, extracting a bounding box of a single object will be described as an example.

As described above, in the simulation environment of one embodiment, three different coordinate systems may coexist. The simulation unit 141 may provide a 3D bounding box of all objects in the village in vehicle coordinates. Also, the location of the vehicle and the camera may be provided in the world coordinate system.

Therefore, the data generator 143 needs to convert eight points of the 3D bounding box at vehicle coordinates into projected points at 2D image coordinates of the front camera.

To this end, the data generator 143 may apply a transformation matrix to each vertex coordinate of the 3D bounding box based on the center point of the own vehicle in the world coordinate system. In one embodiment, since the center point of the own vehicle is known in the world coordinate system, each point (x, y, z) of the vehicle coordinate system can be transformed into the world coordinate system using a transformation matrix.

Further, the data generator 143 may generate temporary 3D camera coordinates such that the z-axis is parallel to the FOV direction of the front camera based on the position of the front camera in the world coordinate system. That is, the data generator 143 may also integrate the position of the front camera for perspective conversion to the front camera viewpoint, and the temporary 3D camera may generate coordinates along with the z-axis parallel to the viewing direction of the front camera.

Next, the data generator 143 may perform perspective transformation on each vertex of the temporary 3D camera coordinates based on the FOV and image resolution of the front camera.

As shown in FIG. 5(a), the data generator 143 may determine a 2D bounding box having a minimum size including all vertices of the 3D bounding box.

The eight vertices of the cuboid in the leftmost image of FIG. 5 (a) represent a 3D bounding box projected onto 2D image coordinates. And the rectangle in the rightmost image of FIG. 5(a) represents the minimum area bounding box for each rectangular parallelepiped.

As an example, an algorithm for implementing a process of transforming a 3D bounding box projected on a 2D image coordinate system into a 2D bounding box may be schematically shown in Table 1 below.

이

로 표시되는 2차원 바운딩 박스로 변환되는 과정을 나타낸다.Table 1 shows the projected 3D bounding box.

this

It shows the process of converting to a 2-dimensional bounding box represented by .

Such a two-dimensional bounding box may have a problem in that an invisible object exists due to occlusion (occluded by another object). For example, in FIG. 5(a), the object for the 3D bounding box in the upper left corner is an object that needs to be removed because it is behind a building.

Also, the generated 2D bounding box may have a problem in that the gap between the bounding box and the object boundary is wide. If the bounding box does not match the boundary of the object as much as possible, serious object recognition errors may occur in the model trained with the corresponding training data.

Accordingly, the data generator 143 may determine whether the object corresponding to the 2D bounding box is occluded based on the 2D bounding box, the class information of the object corresponding to the 2D bounding box, and the semantic segmentation image. Class information of the object may be extracted by the data extraction unit 142 .

The data generator 143 may obtain coordinates of the center point of the 2D bounding box on the image coordinate system. And, if it is determined that the class information corresponding to the center point coordinates is the same as the class of the object corresponding to the 2D bounding box on the semantic segmentation image, the data generator 143 converts the object corresponding to the 2D bounding box to an unoccluded object. can be determined by

A semantic segmentation image is one in which each pixel is classified with a color code representing the class of an object. For example, as shown in FIG. 5( b ), vehicles and other objects may be displayed in different colors.

The bounding box displayed in the upper left corner of the leftmost image of FIG. 5(b) indicates that the object is not actually visible because it is covered by a building. Accordingly, in one embodiment, the corresponding object may be removed as shown in the second image of FIG. 5(b).

As an embodiment, an algorithm for implementing a process of filtering a 2D bounding box of an occluded object based on a semantic segmentation image may be schematically shown in Table 2 below.

Table 2 shows a process that can confirm the existence of a valid and visible object within a given 2D bounding box. According to Table 2, the data generator 143 may use a 2D bounding box, a corresponding object class, and a semantic segmentation image. Further, the data generator 1430 checks the semantic segmentation image and determines that the central pixel of the bounding box is visible if it is identified as belonging to a given object class.

Meanwhile, in an embodiment, sampling is performed on pixels other than the central pixel so that a partially occluded pixel may be generated as training data.

Referring to FIG. 5(c), the data generator 143 performs the 2D bounding box on the semantic segmentation image until each boundary of the 2D bounding box reaches a pixel within the class segmentation area of the object of the 2D bounding box. By moving each boundary of in a direction closer to the center point, the final bounding box can be created.

The leftmost image of FIG. 5(c) shows that there is a significant gap between the bounding box and the boundary of the actual vehicle. Accordingly, the data generator 143 needs to fit the bounding box to the exact area occupied by the object in order to more accurately generate the bounding box. To this end, the data generator 143 may use a semantic segmentation image.

The middle image of FIG. 5(c) shows a process in which the distance between the 2D bounding box and the object is minimized by moving each boundary of the 2D bounding box in a direction closer to the center point. That is, the data generator 143 may fix the bounding box to the object as shown in the rightmost image of FIG. 5(c) when each boundary of the 2D bounding box reaches a pixel within the class-direction area of the object of the 2D bounding box. there is.

As an embodiment, an algorithm for implementing a process of generating a final bounding box by changing the shape of a 2D bounding box to suit the shape of an object corresponding to the 2D bounding box based on a semantic segmentation image is shown in Table 3 below. can be shown schematically.

가 입력으로 제공되고 최종 바운딩 박스

가 출력되는 최종 바운딩 박스 생성 과정을 나타낸다. Table 3 is the original bounding box

is given as input and the final bounding box

Indicates the final bounding box generation process in which is output.

6 is a heat map for an object generated from 2D image coordinates according to an embodiment of the present disclosure, and FIG. 7 is a scalability evaluation graph of an apparatus for generating training data for object recognition according to an embodiment of the present disclosure. 8 is an accuracy evaluation diagram of an apparatus for generating training data for object recognition according to an embodiment of the present disclosure, and FIG. 9 is a training performance evaluation graph of another apparatus for generating training data for object recognition according to an embodiment of the present disclosure.

6 to 9, experiments were conducted to verify the performance of the apparatus 100 for generating training data for object recognition according to an embodiment.

The apparatus 100 for generating training data for object recognition according to an embodiment may be implemented based on a Carla simulation server. Hereinafter, the training data generating apparatus 100 for object recognition may be described as CarFree.

The apparatus 100 for generating training data for object recognition according to an embodiment may be implemented as a script. In one embodiment, the data extraction unit includes a first script for arranging vehicles and pedestrians necessary for the village, a second script for controlling weather and lighting conditions, and a third script for controlling vehicles and extracting data from a vehicle simulator. It can be implemented with three scripts in the script.

The apparatus 100 for generating training data for object recognition can support two operation modes: periodic data extraction in which data is continuously retrieved for a period specified in a command line option, and event-based data extraction triggered by a user's key press. there is. Event-based data extraction can be useful when a user accidentally finds a corner case scenario while monitoring a simulation.

In one embodiment, the extracted data may be recorded in image files (camera images and semantic segmentation images) and text files (3D bounding boxes) in a designated directory.

In an embodiment, the apparatus 100 for generating training data for object recognition may be verified in terms of scalability, accuracy, and training performance. For verification, 5000 training images from 4 different villages and ground truth data (training data) collected in real driving environments were integrated. During each simulation, 50 random vehicles and 30 random pedestrians were placed on the vehicle simulator's blue screen. And vehicles and pedestrians were allowed to move autonomously around the village by the autopilot module.

The vehicle was deployed with a 90 degree FOV front camera acquiring images at a resolution of 960 X 540. The altitude and azimuth of the sun change rapidly, simulating different lighting conditions. Weather conditions are controlled by changing weather parameters (e.g. cloudiness, precipitation and wind).

6 shows a heat map according to 2D image coordinates to determine the location of an object in the synthesized image. A heatmap may be created by calculating the intensity of an object in units of pixels. To this end, a bounding box covering each pixel for vehicles and pedestrians can be extracted from all synthesized images.

6(a) shows that the vehicle appears more strongly along the horizontal line around the vanishing point. Figure 6(b) shows that pedestrians are more likely to appear on the left and right sides of the image because they mainly walk along the sidewalk. However, some pedestrians arbitrarily cross the road marked with bright dots in the central area. The number of objects may vary randomly in each image. For example, 27% of images may contain only one object, 36% of images may contain two objects, 20% of images may contain three objects, and 18% of images may contain four or more objects. there may be

Referring to FIG. 7 , it is possible to compare the average throughput (ie, images per second) of the apparatus 100 for generating training data for object recognition and when a bounding box is generated by an inexperienced operator and an experienced operator. The y-axis is on a logarithmic scale. 7 shows that the number of objects in each image has a great effect on the labeling performance of human labor, but the effect is not significant in the process of the apparatus 100 for generating training data for object recognition. In addition, the apparatus 100 for generating training data for object recognition processes an average of 5 images per second in the case of 6 objects, which is about 75 times faster than the result of a skilled worker.

As shown in Fig. 8, the bounding box generated by the apparatus 100 for generating training data for object recognition has pixel-level accuracy, whereas manual labeling cannot generate such an accurate bounding box. In addition, a very small object far away is easily ignored by the human eye, but the training data generation apparatus 100 for object recognition does not miss such a small object.

Referring to FIG. 9 , the performance of the data set generated by the apparatus 100 for generating training data for object recognition was evaluated with 200 image test sets randomly selected from other data sets. YOLO v3 DNN based on the Darknet framework was used as an object recognition model for verifying the performance of the data set generated by the apparatus 100 for generating training data for object recognition. In one embodiment, actual driving images from the KITTI data set were used along with synthesized ones for training. Models trained with image configurations such as KITTI only, CarFree only, KITTI+CarFree (random) and KITTI+CarFree (more than 4 objects) were evaluated, respectively.

Here, KITTI only uses only KITTI database images, and CarFree only uses only images generated by the apparatus 100 for generating training data for object recognition. KITTI+CarFree (random) uses the KITTI database image as a reference, but images generated by the apparatus 100 for generating training data for object recognition are randomly and gradually added. KITTI+CarFree (more than 4 objects) selects a random image from a synthetic data set of images generated by the KITTI image and the training data generating apparatus 100 for object recognition, but only images with at least 4 objects are considered .

Fig. 9(a) shows how the mean average precision (mAP) of the trained model is improved by adding more training images. For comparison, we initially trained a DNN using 500 random KITTI images. Then images or composite images from the KITTI dataset were progressively added to the training set according to the rules. In Fig. 9(a), the x-axis represents the total number of training images including the basic KITTI image and additional images.

As shown in Fig. 9(a), the performance generally improved as more images were added. And KITTI + CarFree (random) appears to outperform KITTI only, which may be because the synthesized images are more diverse than the KITTI data set. This trend is more evident in KITTI + CarFree (more than 4 objects), which includes more complex driving scenes.

9(b) compares the performance of different evaluation data sets composed of specially selected images under abnormal weather and lighting conditions such as rainy day, night driving, and sunset driving. In such a harsh driving environment, KITTI exhibits much worse performance than the general driving environment shown in FIG. 9(a). On the other hand, adding the synthesized image significantly improved the performance.

Also, referring to FIG. 9(b), CarFree also outperforms KITTI because most of the KITTI data set in the real world consists of driving images on sunny days without diversity. That is, since the data (CarFree) generated by the apparatus 100 for generating training data for object recognition according to an embodiment uses synthetic images having artificial diversity together with real images, DNN performance can be greatly improved.

As evaluated through the above experiments, the apparatus 100 for generating training data for object recognition can generate a lot of training data (images and bounding boxes) with pixel-level accuracy without human labor. In addition, since the weather and lighting conditions can be freely controlled in the virtual world and complex driving scenes can be artificially composed, a wide variety of synthetic images can be created. Therefore, training data generated by the apparatus 100 for generating training data for object recognition may be more effective for machine learning (e.g., DNN) training than ordinary and boring actual driving images found in most public self-driving data sets. .

Meanwhile, FIG. 10 is an exemplary diagram schematically illustrating a vehicle to which an apparatus for generating training data for object recognition according to an embodiment of the present disclosure is applied.

The vehicle 10 may include a camera that captures a surrounding image, a processor, and a memory electrically connected to the processor and storing at least one code executed by the processor.

In one embodiment, the vehicle 10 receives an image captured by a camera, inputs the image to a machine learning-based learning model for object recognition, outputs information of the recognized object, and based on the information of the recognized object can be controlled In this case, the learning model may be a learning model trained by using training data obtained from a vehicle simulator and images actually captured by a camera of a driving vehicle as training data.

According to the evaluation results according to the above experiment, it can be confirmed that the object recognition performance of the state-of-the-art object recognition model is greatly improved when the synthesized image is used. Performance improvements are even more important when models are evaluated with test images representing unusual weather conditions, which can have significant advantages for the safety of autonomous systems operating in harsh environments.

That is, when the vehicle 10 is controlled based on the training data generated by the object recognition training data generation apparatus 100 according to an embodiment, vehicle driving may be possible according to more accurate object recognition.

11 is a flowchart illustrating a method of generating training data for object recognition according to an embodiment of the present disclosure.

Referring to FIG. 11 , in step S100, the processor 140 controls a driving simulation environment based on the simulation environment data and controls the vehicle simulator to drive the own vehicle in the driving simulation environment.

The simulation unit 141 simulates a virtual world (a so-called village) and controls a driving simulation environment according to a command to change simulation environment data from the data extraction unit 142 . That is, the simulation unit 141 may control the weather and lighting conditions and move the own vehicle, surrounding vehicles, and pedestrians according to the simulation environment data change command from the data extraction unit 142 .

In step S200, the processor 140 collects simulation information data based on a field of view (FOV) of the own vehicle that is driving.

That is, the data extraction unit 142 may obtain a front image captured by a front camera virtually installed in the vehicle. In addition, the data extraction unit 142 may obtain a 3D bounding box for at least one object present in the front image, and may also obtain a semantic segmentation image of the front image.

Further, the data extraction unit 142 may obtain information on a world coordinate system in which the center of the virtual world is the origin, a vehicle coordinate system in which the center of the own vehicle is the origin, and an image coordinate system in which the top left corner of the front image is the origin.

In step S300, the processor 140 generates a 2D bounding box of at least one object existing within the field of view while the vehicle is driving based on the simulation information data.

While the vehicle is being driven, the data generator 143 processes raw data such as a 3D bounding box, camera image, and semantic segmentation image from the data extraction unit 142 to obtain accurate 2-dimensional images for all objects. It is possible to generate training data for object recognition by finding a dimensional bounding box. In other words, the data generator 143 may generate a 2D bounding box of at least one object existing within the field of view while the vehicle is driving based on the simulation information data.

The data generator 143 may project vertex coordinates of the 3D bounding box onto an image coordinate system and convert the 3D bounding box projected onto the 2D image coordinate system into a 2D bounding box. And the data generator 143 filters the 2D bounding box of the obscured object based on the semantic segmentation image, and the shape of the 2D bounding box is suitable for the shape of the object corresponding to the 2D bounding box based on the semantic segmentation image. By changing , the final bounding box can be created.

In step S400, the processor 140 generates a training data image for object recognition of an object based on the bounding box.

That is, the processor 140 may generate accurate bounding boxes for objects in images for various simulation situations in the simulation unit 141 to generate training data images for object recognition. In particular, for object recognition in autonomous driving, training data images for various situations may be generated.

Embodiments according to the present disclosure described above may be implemented in the form of a computer program that can be executed on a computer through various components, and such a computer program may be recorded on a computer-readable medium. At this time, the medium is a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM hardware devices specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the purpose of the present disclosure, or may be known and available to those skilled in the art in the field of computer software. An example of a computer program may include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification of the present disclosure (particularly in the claims), the use of the term "above" and similar indicating terms may correspond to both singular and plural. In addition, when a range is described in the present disclosure, as including the invention to which individual values belonging to the range are applied (unless otherwise stated), each individual value constituting the range is described in the detailed description of the invention Same as

Unless an order is explicitly stated or stated to the contrary for steps comprising a method according to the present disclosure, the steps may be performed in any suitable order. The present disclosure is not necessarily limited to the order of description of the steps. The use of all examples or exemplary terms (eg, etc.) in this disclosure is simply to explain the present disclosure in detail, and the scope of the present disclosure is limited due to the examples or exemplary terms unless limited by the claims. it is not going to be In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Therefore, the spirit of the present disclosure should not be limited to the above-described embodiments, and not only the claims to be described later, but also all ranges equivalent to or equivalent to these claims are within the scope of the spirit of the present disclosure. will be said to belong to

1: Training data generation system for object recognition
100: training data generation device for object recognition
110: Communication Department
120: user interface
130: memory
140: processor
200: server
300: network

Claims (20)

A method for automatically generating training data for object recognition based on a vehicle simulator, wherein at least part of each step is performed by a processor,
a simulation step of controlling a driving simulation environment based on the simulation environment data and controlling a vehicle simulator to drive the host vehicle in the driving simulation environment;
a data extraction step of collecting simulation information data based on a field of view (FOV) of the own vehicle while driving;
a 2D bounding box generating step of generating a 2D bounding box of at least one object existing within a visual field while the vehicle is driving based on the simulation information data; and
A training data generation step of generating a training data image for object recognition of the object based on the bounding box,
A method for generating training data for object recognition.
According to claim 1,
The data extraction step,
obtaining a front image captured by a front camera virtually installed in the vehicle;
obtaining a 3D bounding box for at least one object present in the front image; and
Acquiring a semantic segmentation image of the front image,
A method for generating training data for object recognition.
According to claim 2,
The data extraction step,
Acquiring information on a world coordinate system whose origin is the center of the virtual world of the simulation environment, a vehicle coordinate system whose origin is the center of the own vehicle, and an image coordinate system whose origin is the upper left corner of the front image,
A method for generating training data for object recognition.
According to claim 1,
The data extraction step,
Extracting the simulation information data according to a predetermined period or according to an event that occurs based on a user's setting,
A method for generating training data for object recognition.
According to claim 3,
In the step of generating the two-dimensional bounding box,
projecting the coordinates of vertices of the 3D bounding box onto the image coordinate system;
converting the 3D bounding box projected on the image coordinate system into a 2D bounding box;
filtering a 2D bounding box of an occluded object based on the semantic segmentation image; and
Generating a final bounding box by changing the shape of the 2D bounding box to suit the shape of the object corresponding to the 2D bounding box based on the semantic segmentation image,
A method for generating training data for object recognition.
According to claim 5,
The step of projecting onto the image coordinate system,
applying a transformation matrix to each vertex coordinate of the 3D bounding box based on the center point of the own vehicle in the world coordinate system;
generating temporary 3D camera coordinates such that a z-axis is parallel to the FOV direction of the front camera based on the position of the front camera in the world coordinate system; and
Based on the FOV and image resolution of the front camera, performing perspective transformation on each vertex of the temporary 3D camera coordinates,
A method for generating training data for object recognition.
According to claim 6,
The step of transforming into a two-dimensional bounding box,
Determining a 2-dimensional bounding box of a minimum size including all vertices of the 3-dimensional bounding box,
A method for generating training data for object recognition.
According to claim 7,
The data extraction step further comprises obtaining class information of the object,
In the step of filtering the bounding box,
Determining whether the object corresponding to the 2-dimensional bounding box is occluded based on the 2-dimensional bounding box, the class information of the object corresponding to the 2-dimensional bounding box, and the semantic segmentation image,
A method for generating training data for object recognition.
According to claim 8,
The step of filtering the two-dimensional bounding box,
obtaining coordinates of a center point of the 2D bounding box on the image coordinate system; and
In the semantic segmentation image, if it is determined that the class information corresponding to the center point coordinates is the same as the class of the object corresponding to the 2D bounding box, determining the object corresponding to the 2D bounding box as an unoccluded object. including,
A method for generating training data for object recognition.
According to claim 8,
The step of generating the final bounding box,
On the semantic segmentation image, each boundary of the 2D bounding box is moved in a direction approaching the center point until each boundary of the 2D bounding box reaches a pixel in the class division area of the object of the 2D bounding box. Including the steps of
A method for generating training data for object recognition.
According to claim 1,
Controlling the vehicle simulator,
Dynamically changing the driving speed of the own vehicle or changing the weather or driving time of the driving simulation environment,
A method for generating training data for object recognition.
According to claim 11,
Controlling the vehicle simulator,
Changing a cycle for collecting the simulation information data based on the traveling speed of the own vehicle,
A method for generating training data for object recognition.
An apparatus for automatically generating training data for object recognition based on a vehicle simulator,
Memory; and
a processor coupled with the memory and configured to execute computer readable instructions contained in the memory;
The at least one processor,
A simulation operation of controlling a driving simulation environment based on simulation environment data and controlling a vehicle simulator to drive a vehicle in the driving simulation environment;
A data extraction operation of collecting simulation information data based on a field of view (FOV) of the vehicle in motion;
A 2D bounding box generating operation of generating a 2D bounding box of at least one object existing within a field of view while the vehicle is driving based on the simulation information data; and
Set to perform a training data generation operation of generating a training data image for object recognition of the object based on the bounding box,
A device for generating training data for object recognition.
According to claim 13,
The data extraction operation,
Obtaining a front image captured by a front camera virtually installed in the vehicle;
Obtaining a 3D bounding box for at least one object present in the front image;
Obtaining a semantic segmentation image of the front image, and
Obtaining information about a world coordinate system whose origin is the center of the virtual world of the simulation environment, a vehicle coordinate system whose origin is the center of the own vehicle, and an image coordinate system whose origin is the upper left corner of the front image,
A device for generating training data for object recognition.
15. The method of claim 14,
The operation of generating the two-dimensional bounding box,
Projecting the coordinates of vertices of the 3-dimensional bounding box onto the image coordinate system;
converting the 3D bounding box projected on the image coordinate system into a 2D bounding box;
Filtering a 2D bounding box of an occluded object based on the semantic segmentation image; and
Based on the semantic segmentation image, generating a final bounding box by changing the shape of the 2D bounding box to suit the shape of the object corresponding to the 2D bounding box.
A device for generating training data for object recognition.
According to claim 15,
The operation of projecting onto the image coordinate system,
Applying a transformation matrix to each vertex coordinate of the 3-dimensional bounding box based on the center point of the own vehicle in the world coordinate system;
Based on the position of the front camera in the world coordinate system, generating temporary 3D camera coordinates such that the z-axis is parallel to the FOV direction of the front camera, and
Based on the FOV and image resolution of the front camera, performing perspective transformation on each vertex of the temporary 3D camera coordinates,
A device for generating training data for object recognition.
17. The method of claim 16,
The operation of converting to the two-dimensional bounding box,
Including an operation of determining a 2-dimensional bounding box of a minimum size including all vertices of the 3-dimensional bounding box,
A device for generating training data for object recognition.
18. The method of claim 17,
The data extraction operation further includes obtaining class information of the object,
The operation of filtering the bounding box,
Determining whether an object corresponding to the 2-dimensional bounding box is occluded based on the 2-dimensional bounding box, the class information of the object corresponding to the 2-dimensional bounding box, and the semantic segmentation image,
A device for generating training data for object recognition.
According to claim 18,
The operation of generating the final bounding box,
On the semantic segmentation image, each boundary of the 2D bounding box is moved in a direction approaching the center point until each boundary of the 2D bounding box reaches a pixel in the class division area of the object of the 2D bounding box. Including the action to
A device for generating training data for object recognition.
A camera that captures surrounding images;
processor; and
A memory electrically connected to the processor and storing at least one code executed by the processor;
When the memory is executed through the processor, the processor receives an image captured by the camera, inputs the image to a machine learning-based learning model for object recognition, outputs information of the recognized object, and recognizes the object Stores a code that causes the vehicle to be controlled based on the information of
The learning model is a learning model trained by using training data obtained from a vehicle simulator and images actually captured by a camera of a driving vehicle as training data,
vehicle.

Images