WO2024025063A1

WO2024025063A1 - System and method for labeling scenarios using sensor data measurements of autonomous vehicle

Info

Publication number: WO2024025063A1
Application number: PCT/KR2023/002913
Authority: WO
Inventors: 김태형; 허준호; 김봉섭; 윤경수
Original assignee: 재단법인 지능형자동차부품진흥원
Priority date: 2022-07-27
Filing date: 2023-03-03
Publication date: 2024-02-01
Also published as: KR20240015792A

Abstract

An embodiment of the present invention provides a system for labeling scenarios using sensor data measurements of an autonomous vehicle, wherein the system collects weather and illumination states around the autonomous vehicle in real time, and automatically labels overall measurement environment data, which combines calculated dynamic environment measurement data and static environment measurement data including road information, by inferring the overall environment as the collected information.

Description

Scenario labeling system and method using sensor data measurement of autonomous vehicles

The present invention relates to a scenario labeling system and method using sensor data measurement of an autonomous vehicle.

Recently, research on intelligent vehicle technologies such as autonomous driving and driver assistance systems (DAS) has been actively conducted. Among these, research is being actively conducted recently on technologies that can utilize sensor data from autonomous vehicles as artificial intelligence learning data.

In order to construct/use such autonomous driving sensor data as artificial intelligence learning data, labeling of raw sensor data is necessary.

In order to label sensor data, object-level sensor data labeling is important, but in terms of big data management, it is necessary to infer the environment in which the data was measured and manage it by labeling it as a scenario.

Most conventional measurement/labeling systems focus on objects (vehicles/pedestrians, etc.), and in the case of open data sets, data labeled on an object basis is mostly distributed.

Existing technology related to this includes an offline labeling method in which the user manually labels each data frame after data acquisition for labeling objects or environmental conditions (weather, lighting), etc., but separate time is required to label the measured data. There is a problem with additional demands.

Additionally, in the prior art, there is a method of labeling data in the form of a trigger input by the user while acquiring data, but since it requires manipulation for trigger input while driving, there are safety issues and the need for additional manpower, and there are problems with sudden environmental changes. There was a problem that was difficult to respond to.

The present invention was created to solve the above-described conventional problems. The purpose of the present invention is to infer the data measurement environment based on precision road maps, precipitation/illuminance sensors, and GPS, and automatically label it when measuring data. To provide a scenario labeling system and method using sensor data measurement of autonomous vehicles.

The present invention may include the following examples to achieve the above object.

An embodiment of the present invention collects weather and illumination conditions around an autonomous vehicle in real time, and combines dynamic environmental measurement data calculated by inferring the entire environment from the collected information and static environmental measurement data including road information. We provide a scenario labeling system using sensor data measurement from autonomous vehicles that automatically labels the entire measurement environment data.

In addition, the present invention, as another embodiment, includes an autonomous driving measurement unit that outputs sensor measurement data in real time as any one of a camera, lidar, radar, and IMU, and at least one of weather, time, and lighting conditions around the autonomous vehicle. A dynamic environment measurement unit that measures and analyzes the environment, a static environment measurement unit that analyzes road and terrain information according to the location of the autonomous vehicle based on GPS location information and a precise road map, and an environment that fuses static and dynamic environment measurement data. Final measurement including a fusion unit, an inference unit that infers the measurement environment through the entire fused measurement data, a scenario labeling unit that labels the inferred measurement environment data, and data labeled for each frame of real-time video data from the autonomous driving measurement unit. It includes a scenario labeling system using sensor data measurement of an autonomous vehicle, including a final measurement data derivation unit that calculates and outputs data.

In addition, another embodiment of the present invention is a scenario labeling method using sensor data measurement of an autonomous vehicle, a) detecting precipitation and illumination in the autonomous vehicle, GPS information including location and time, and precision road Analyzing the map layer, b) Outputting raw data measured in real time from at least one of a camera, lidar sensor, and radar sensor installed in the autonomous vehicle, and c) Autonomous driving using the precipitation measurement data a step of analyzing the lighting state using weather conditions around the vehicle and illuminance measurement data, d) a step of fusing the analysis results of the weather condition and the analysis result of the lighting condition, and e) the analysis result of the weather condition and the lighting. f) inferring the entire measurement data around the autonomous vehicle by fusing the state-fused dynamic environment data and the road information of the road map analysis module; f) automatically labeling the entire inferred measurement data and labeling each video frame of the raw data; It includes a scenario labeling method using sensor data measurement of an autonomous vehicle, which includes combining labeled data to calculate final data.

Therefore, the present invention increases the amount of information included in scenario labeling by utilizing precision road maps for measurement environment inference, and can respond to environments that change in real time by reflecting real-time sensing results such as precipitation/illuminance sensors in scenario labeling. .

In addition, the present invention infers the data measurement environment based on precision road maps, precipitation/illuminance sensors, and GPS, and automatically labels this when measuring data, thereby reducing costs such as time/manpower for data labeling and user convenience. Convenience can be increased.

Figure 1 is a block diagram for explaining the outline of the present invention.

Figure 2 is a block diagram showing a scenario labeling system using sensor data measurement of an autonomous vehicle according to the present invention.

Figure 3 is a block diagram showing a static environment measurement unit.

Figure 4 is a block diagram showing a dynamic environment measurement unit.

Figures 5 and 6 are diagrams showing an example of a day and night environmental data measurement process based on a precision road map.

Figure 7 is a diagram illustrating an example of a convergence environment determination process.

Figure 8 is a diagram showing an example of a data measurement environment inference process.

Figure 9 is a diagram showing an example of final measurement data.

Figure 10 is a flowchart showing a sirio labeling method using sensor data measurement of an autonomous vehicle according to the present invention.

Although the present invention may be subject to various changes and may have various embodiments, specific embodiments will be described in detail by illustrating them in the drawings. This is not intended to limit the present invention to specific embodiments, and is not intended to limit the present invention to any of the changes, equivalents, or substitutes included in the spirit and scope of the present invention for connecting and/or fixing structures extending in different directions. It must be understood as applicable.

The terms used herein are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

Additionally, when describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, embodiments of the present invention will be described with reference to the attached drawings.

Figure 1 is a block diagram for explaining the outline of the present invention, Figure 2 is a block diagram showing a scenario labeling system using sensor data measurement of an autonomous vehicle according to the present invention, and Figure 3 is a block diagram showing a static environment measurement unit. , Figure 4 is a block diagram showing the dynamic environment measurement unit.

Referring to Figures 1 and 2, the present invention measures the dynamic environment and correction environment around an autonomous vehicle, infers the measurement environment through the measured static and dynamic data, and automatically labels the entire inferred measurement data. It is characterized by combining and outputting raw data detected in real time.

Here, the static environment means that it is maintained regardless of the environment, such as GPS measurement information and precise road maps, and the dynamic environment means weather, time, lighting, illuminance, and surrounding information of the autonomous vehicle (e.g., surrounding vehicle information, traffic information, etc.). refers to a changing environment such as information, etc.).

Specifically, the present invention is a system that automatically labels by combining measurement data from dynamic and static environments and raw data sensed in real time.

To this end, the present invention includes a static environment measuring unit 100 that measures the static environment, a dynamic environment measuring unit 200 that measures the dynamic environment, and an autonomous driving measuring unit 300 that measures the road environment before or while driving the autonomous vehicle. ), an environment fusion unit 400 that fuses the measurement data of the static environment and the dynamic environment, an inference unit 500 that infers the entire measurement data through the fused measurement information, and a device that labels the inferred measurement data in the scenario. It includes a scenario labeling unit 600 and a final measurement data deriving unit 700 that calculates final measurement data.

Referring to FIG. 3, the static environment measurement unit 100 includes a GPS measurement module 110 that measures GPS data and a road map analysis module 120 that analyzes layers of a precision road map.

The GPS measurement module 110 is a sensor that measures the location and time of the vehicle. It receives signals transmitted from two or more GPS satellites, calculates the distance, calculates the location, and provides time information using the atomic clock inside the satellite. do.

The road map analysis module 120 extracts information about roads and surrounding facilities from a precise road map based on the current location of the autonomous vehicle on the road map, the designated route, and the current location.

Precision road maps are infrastructure for determining the location of autonomous vehicles, setting/changing routes, and recognizing road/traffic regulations, including network information (nodes, links), road section information (tunnels, bridges, etc.), and sign information (traffic safety). It may refer to a precise electronic map expressing at least one of (signs, lanes, crosswalks, etc.) and facility information (traffic lights, vehicle protection and safety facilities, etc.).

In addition, the precise road map includes driving route nodes/links, roadways/accessory sections, parking surfaces, safety signs, road surface/road line markings, traffic lights, kilo posts, vehicle protection and safety facilities, speed bumps, height obstacles, supports, etc. and each can be configured in the form of a hierarchy (layer).

The road map analysis module 120 can add layers to the layer structure of an existing precision road map. For example, the road map analysis module 120 is designed to label road surface materials (e.g., asphalt, concrete, unpaved), etc. on a precision road map based on the location of the autonomous vehicle measured by the GPS measurement module 110. This data adds a layer to the precision road map. Here, additional layers can be selected arbitrarily by the user.

Referring to FIG. 4, the dynamic environment measurement unit 200 includes a precipitation detection module 210 for detecting the amount of precipitation, an illuminance detection module 220 for detecting the illuminance, and a weather analysis module 230 for detecting weather conditions. , includes a lighting state analysis module 240 that analyzes the lighting state.

The precipitation detection module 210, for example, is a precipitation sensor that measures the amount of precipitation and can detect the wavelength between the infrared rays being emitted and the infrared rays returning to the sensor. Additionally, the precipitation detection module 210 can estimate the amount of precipitation using the intensity of reflected light that varies depending on the amount of rain. This precipitation detection module 210 is mounted on the exterior of an autonomous vehicle and detects the amount of precipitation.

Alternatively, the precipitation detection module 210 may further include a device that detects the presence or absence of snow and the amount of snow.

The illuminance detection module 220 is a sensor that measures the surrounding brightness. When it receives optical energy (light), moving electrons are generated inside it, and the conductivity changes, thereby changing the intensity of the output voltage. The illumination detection module 220 is installed on the outside of the vehicle to detect the intensity of light from the outside, and detects the characteristics of the light source (for example, the direction of incidence) by installing sensors in four or more directions. It is desirable to reduce measurement errors.

The weather analysis module 230 receives data measured by the precipitation detection module 210, analyzes weather conditions, and outputs a result. For example, the weather analysis module 230 can calculate rainfall and snowfall amounts. Alternatively, the weather analysis module 230 may calculate and output the analysis results of weather conditions as simple information such as little rainfall, heavy rainfall, little snowfall, and heavy snowfall.

The lighting condition analysis module 240 uses the illuminance measurement data from the illuminance detection module 220, the time measurement data from the GPS measurement module 110, and the road information from the road map analysis module 120 to determine the current time and the autonomous vehicle. Analyze the surrounding conditions (e.g., whether it is day, night, or in a tunnel, etc.). This is explained with reference to FIGS. 5 and 6.

FIG. 5 is a diagram illustrating an example of a daytime environmental data measurement process, and FIG. 6 is a diagram illustrating an example of a nighttime environment data measurement process.

Referring to FIGS. 5 and 6, the lighting condition analysis module 240 provides current location information (e.g., whether on a road or in a tunnel) on a precise road map through measurement data of the road map analysis module, and the illuminance detection module 220. )'s illuminance detection data and GPS time information are fused to analyze the current time (night or day) and lighting conditions and output the results (for example, while passing through a tunnel during the day or driving on the road at night).

The autonomous driving measurement unit 300 includes at least one of a lidar sensor, radar sensor, camera, GPS, and IMU installed in the autonomous vehicle and measures the surroundings of the autonomous vehicle. The data measured here consists of two-dimensional and/or three-dimensional data such as camera images and lidar point clouds.

The environment fusion unit 400 analyzes the surrounding environment by fusing weather, time, and lighting state analysis results. The environment fusion unit 400 will be described with reference to FIG. 7 .

Referring to FIG. 7, the environment fusion unit 400 fuses the weather condition information and time and lighting information of the weather analysis module 230 and the lighting condition analysis module 240 to determine the current time (e.g., daytime) and driving information. Result data including roads (e.g., general roads) and weather conditions (e.g., cloudy) can be calculated.

The inference unit 500 infers the overall data measurement environment including data in the static environment and data in the dynamic environment. This is explained with reference to FIG. 8.

Referring to FIG. 8 , for example, weather, time, and lighting data of the environment fusion unit 400 are dynamic environment data. Therefore, the inference unit 500 adds static environment data to the dynamic environment data calculated by the environment fusion unit 400 to infer the entire data measurement environment.

Here, static environmental data includes network information (nodes, links) of the road map analysis module, road section information (tunnels, bridges, etc.), sign information (traffic safety signs, lanes, crosswalks, etc.), and facility information (traffic lights, vehicle protection and safety). (Soil, etc.), number of lanes, lane number, road type, road grade, lane type, roadway section, and road line marking type.

Additionally, the inference unit 500 may add measurement data (eg, camera image) around the autonomous vehicle measured by the autonomous driving measurement unit 300.

The scenario labeling unit 600 defines the inferred measurement environment data that is a fusion of static environment data and dynamic environment data as a scenario, configures it as labeling data, and fuses it with the raw data to generate learning data.

Here, the raw data is data from the autonomous driving measurement unit 300 (e.g., camera image, video image, and lidar point cloud data), and the labeling data is data that constitutes the scenario defined in the scenario lavalem department.

The final measurement data deriving unit 700 derives the final measurement data as learning data that is a fusion of raw data and labeling data. Such final measurement data will be described with reference to FIG. 8.

Figure 8 is a diagram showing an example of final measurement data.

Referring to FIG. 8, the final measurement data deriving unit 700 fuses the raw data from the autonomous driving measurement unit and the labeling data from the scenario labeling unit 600 and outputs them. At this time, labeling data may be output for each frame of raw data, as shown in FIG. 10.

That is, the final data is output including labeling data for each frame of the raw data.

Additionally, the present invention includes a scenario labeling method using sensor data measurement of an autonomous vehicle using the above configuration. This is explained with reference to FIG. 10.

Figure 10 is a flowchart showing a scenario labeling method using sensor data measurement of an autonomous vehicle according to the present invention.

Referring to FIG. 10, the present invention includes step S110 of calculating measurement data, step S120 of measuring autonomous driving sensor data, step S130 of analyzing static environmental data, step S140 of determining the fusion environment, and data measurement. It includes a step S150 for inferring the environment, a scenario labeling step for calculating learning data, and a step S170 for deriving final measurement data.

Step S110 is a step of outputting sensor measurement data. Here, step S110 includes step S111 of measuring precipitation data, step S112 of measuring illuminance data, step S113 of measuring GPS data, and step S114 of analyzing the precision road map layer.

Step S111 is a step in which the precipitation detection module 210 detects the presence or absence of precipitation and/or the amount of precipitation. The precipitation detection module 210 detects the presence or absence of precipitation and/or the amount of precipitation around the autonomous vehicle. At this time, the precipitation detection module 210 can output the measured amount of precipitation by classifying it as high or low.

Step S112 is a step in which the illuminance detection module 220 detects the illuminance around the autonomous vehicle. The illuminance detection module 220 measures the illuminance around the autonomous vehicle and outputs illuminance measurement data.

Step S113 is a step in which the GPS measurement module 110 calculates location information and time information.

Step S114 is a step in which the road map analysis module 120 analyzes a layer at the current location of the autonomous vehicle on a precise road map using GPS measurement data. Here, the road map analysis module 120 can add a new layer, such as the color or material of the road surface, to the current location of the autonomous vehicle on the precision road map according to set conditions or input commands.

In addition, the road map analysis module 120 provides current location information (e.g., number of lanes, lane number, road type, presence or absence of tunnel, road surface material, road grade, lane type, roadway section type, road surface) through a precise road map. Outputs precision analysis data such as line display type.

Step S120 is a step in which raw data is output from the autonomous driving measurement unit 300. Raw data is information measured by cameras, lidar sensors, and radar sensors installed in autonomous vehicles.

Step S130 includes step S131 for analyzing weather conditions and step S132 for analyzing time/lighting conditions.

Among them, in step S131, the weather analysis module 230 analyzes the data measured by the precipitation detection module 210 to determine the current location of the autonomous vehicle and the presence or absence of rain and/or snowfall in the surrounding area, and the amount of rainfall and/or snowfall. Analyze.

In step S131, the current time and autonomous vehicle are measured through the illuminance measurement data around the autonomous vehicle from the lighting condition analysis module 240, the time measurement data from the GPS measurement module 110, and the surrounding terrain data from the road map analysis module 120. This is the step to analyze the surrounding lighting conditions of the driving vehicle.

Step S140 is a step in which the environmental fusion unit 400 fuses the weather state analysis results and the time/lighting state analysis results. The environment fusion unit 400 fuses weather, time, and lighting state analysis results to produce dynamic environmental data. Here, the dynamic environmental data may include the current time (eg, daytime), driving road and/or terrain information (eg, general road, tunnel), and weather conditions (eg, cloudy).

Step S150 is a step in which the inference unit 500 infers the data measurement environment. The inference unit 500 infers the data measurement environment by fusing the static environment data of the road map analysis module with the dynamic environment data calculated in step S140 and outputs the result.

Step S160 is a step of labeling the entire measurement data inferred by the scenario labeling unit 600 and generating learning data combined with the raw data. The scenario labeling unit 600 labels the entire inferred measurement data and defines it as a scenario.

Step S170 is a step of deriving final measurement data from the final measurement data derivation unit 700. The final measurement data deriving unit 700 fuses the scenarios defined in the scenario labeling unit 600 for each frame of raw data, that is, labeled data, with the raw data to produce final measurement data.

In this way, the scenario labeling method using sensor data measurement of autonomous vehicles according to the present invention reflects real-time sensor measurement information of autonomous vehicles in scenario labeling and increases the amount of information by using precise road maps for measurement environment inference. As all of these processes are automated, the time and/or manpower required for scenario labeling can be significantly reduced compared to the past.

Claims

The weather and illumination conditions around the autonomous vehicle are collected in real time, and the entire environment is automatically measured by combining dynamic environment measurement data calculated by inferring the entire environment from the collected information and static environment measurement data including road information. labeling as; A scenario labeling system using sensor data measurement from autonomous vehicles.
An autonomous driving measurement unit 300 that outputs sensor measurement data in real time as one of a camera, lidar, radar, and IMU;

a dynamic environment measurement unit 200 that measures and analyzes at least one of weather, time, and lighting conditions around the autonomous vehicle;

A static environment measurement unit 100 that analyzes road and terrain information according to the location of the autonomous vehicle according to GPS location information and a precise road map;

An environmental fusion unit 400 that fuses static environmental measurement data and dynamic environmental measurement data;

an inference unit 500 that infers the measurement environment through the entire fused measurement data;

A scenario labeling unit 600 that labels the inferred measurement environment data; and

a final measurement data deriving unit 700 that calculates and outputs final measurement data including data labeled for each frame of the real-time image data of the autonomous driving measurement unit 300; Scenario labeling system using sensor data measurement of autonomous vehicles including.
The method in claim 1, wherein the dynamic environment measurement unit 200

A precipitation detection module 210 that detects the amount of precipitation;

An illuminance detection module 220 that detects illuminance;

a weather analysis module 230 that detects weather conditions through measurement data from the precipitation detection module 210; and

a lighting condition analysis module 240 that detects lighting conditions around the autonomous vehicle through measurement data from the illuminance detection module 220; Scenario labeling system using sensor data measurement of autonomous vehicles including.
The method in claim 1, wherein the static environment measurement unit 100

A GPS measurement module 110 that receives signals transmitted from GPS satellites, calculates the location, and provides time information; and

A road map analysis module 120 that extracts road information based on the current location of the autonomous vehicle from a precise road map; Scenario labeling system using sensor data measurement of autonomous vehicles including.
The method of claim 4, wherein the precision road map is

Infrastructure for determining the location of autonomous vehicles, setting/changing routes, and recognizing at least one of road/traffic regulations, including at least one of network information, road section information, sign information, and facility information; A scenario labeling system using sensor data measurement from autonomous vehicles.
The method of claim 5, wherein the road map analysis module 120

At least one of the driving path nodes/links, roadways/accessory sections, parking surfaces, safety signs, road surface/road line markings, traffic lights, kilo posts, vehicle protection safety facilities, speed bumps, height obstacles, and supports, each of which is configured in a hierarchical form. A scenario labeling system using sensor data measurement of an autonomous vehicle further comprising:
In a scenario labeling method using sensor data measurement of an autonomous vehicle,

a) detecting precipitation and illumination in an autonomous vehicle, analyzing GPS information including location and time, and a precise road map layer;

b) outputting raw data measured in real time from at least one of a camera, lidar sensor, and radar sensor installed in the autonomous vehicle;

c) analyzing weather conditions around the autonomous vehicle using precipitation measurement data and lighting conditions using illuminance measurement data;

d) fusing the analysis results of the weather condition and the analysis result of the lighting condition;

e) inferring the entire measurement data around the self-driving vehicle by fusing dynamic environmental data combining weather condition analysis results and lighting conditions with road information from the road map analysis module; and

f) automatically labeling the entire inferred measurement data and combining the labeled data for each video frame of the raw data to calculate final data; Scenario labeling method using sensor data measurement of autonomous vehicles including.