KR102325367B1 - Method, apparatus and computer program for conducting automatic driving data labeling - Google Patents

Abstract

A method, apparatus and computer program for data labeling of autonomous driving data are provided. The data labeling method of autonomous driving data according to various embodiments of the present invention, in a method performed by a computing device, is labeled on 2D data generated by converting 3D point cloud data (Labeling) Obtaining a user input pointing to a target, selecting a sensor data coordinate value corresponding to a 3D point cloud data coordinate value of the labeling target from sensor data previously matched with the 2D form data, and using the sensor data coordinate value and performing labeling on the sensor data.

Description

Method, device and computer program for data labeling of autonomous driving data

Various embodiments of the present invention relate to a data labeling method, apparatus, and computer program of autonomous driving data for labeling autonomous driving data for supervised learning.

For the convenience of users driving a vehicle, various sensors and electronic devices (eg, an advanced driver assistance system (ADAS)) are being provided, and in particular, an autonomous driving system of the vehicle. Technological development is being actively carried out.

Here, the autonomous driving system refers to a vehicle that recognizes the surrounding environment without driver intervention and automatically drives to a given destination by itself according to the recognized surrounding environment.

In general, in the case of an autonomous driving system, a model is trained by using autonomous driving data (eg, data on road conditions or driving routes) as learning data, and the vehicle can drive to a destination using the learned model. provide a path

In order to supervise a model using autonomous driving data as training data, a large number of sensor data (e.g., laser scan data obtained from a laser scan sensor and image (video or image) data detected through a camera sensor) and a vehicle based on sensor data We need a ground truth that points to the location of . Therefore, in order to provide a correct answer on each sensor data, labeling of autonomous driving data is essential.

On the other hand, in the related art, in order to label autonomous driving data, since a person has to directly label each sensor data, there is a problem that it takes a lot of time for the labeling operation. In addition, since more manpower needs to be input to shorten the time, there is a problem that costs such as labor cost are required.

Korean Patent Publication No. 10-2017-0118501 (2017.10.25)

The problem to be solved by the present invention is to provide a user interface (UI) capable of performing a labeling operation, and by inputting a simple input (eg, selecting a labeling target) on 2D data output through the UI. An object of the present invention is to provide a data labeling method, apparatus, and computer program for autonomous driving data capable of performing labeling on a plurality of sensor data.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

Data labeling method according to an embodiment of the present invention for solving the above problems, in a method performed by a computing device, a 2D form generated by converting 3D point cloud data (Point Cloud Data) on an XY plane obtaining a user input pointing to a labeling target on the data of , selecting a sensor data coordinate value corresponding to a 3D point cloud data coordinate value of the labeling target from sensor data previously matched with the 2D form data; And The method may include performing labeling on the sensor data by using the coordinate values of the sensor data.

A data labeling apparatus according to another embodiment of the present invention for solving the above problems may include a memory for storing one or more instructions and a processor for executing the one or more instructions stored in the memory, The processor executes the one or more instructions, thereby converting 3D point cloud data to obtain a user input pointing to a labeling target on the 2D form data generated by converting 3D point cloud data, and matching with the 2D form data A data labeling method comprising: selecting a sensor data coordinate value corresponding to a coordinate value of the 3D point cloud data of the labeling target from the sensor data; and performing labeling on the sensor data using the sensor data coordinate value can be performed.

A data labeling computer program according to another embodiment of the present invention for solving the above-described problems is combined with a computer (or computing device) that is hardware, and a 2D form generated by converting 3D point cloud data obtaining a user input pointing to a labeling target on the data of , selecting a sensor data coordinate value corresponding to a 3D point cloud data coordinate value of the labeling target from sensor data previously matched with the 2D form data; And It may be stored in a computer (or computing device) readable recording medium to perform a data labeling method including performing labeling on the sensor data using the sensor data coordinate values.

Other specific details of the invention are included in the detailed description and drawings.

According to various embodiments of the present disclosure, a UI capable of performing a labeling operation is provided, and a simple operation (eg, an operation of selecting a labeling target by touch, click, and drag input) is provided on 2D data output through the UI. ) to perform labeling on a plurality of sensor data, there is an advantage in that the time and cost required for individually labeling the plurality of sensor data can be reduced.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram showing the configuration of a data labeling system according to an embodiment of the present invention.
2 is a hardware configuration diagram of a data labeling apparatus according to various embodiments of the present invention.
3 is a flowchart illustrating a data labeling method according to various embodiments of the present invention.
4 is a diagram exemplarily illustrating a UI provided by a data labeling apparatus to a user terminal according to various embodiments.
5 is a diagram illustrating data in a 2D form output through a UI provided by a data labeling apparatus according to various embodiments of the present disclosure;
6 is a diagram illustrating a form in which a user input indicating a labeling target is input on 2D data output through a UI provided by a data labeling apparatus according to various embodiments of the present disclosure;
7 is a diagram illustrating a bounding box in the form of a 3D figure output on a UI according to a user input when sensor data is 3D data, according to various embodiments.
8 is a diagram illustrating a bounding box in the form of a 2D figure output on a UI according to a user input when sensor data is 2D data, according to various embodiments.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

As used herein, the term “unit” or “module” refers to a hardware component such as software, FPGA, or ASIC, and “unit” or “module” performs certain roles. However, “part” or “module” is not meant to be limited to software or hardware. A “unit” or “module” may be configured to reside on an addressable storage medium or may be configured to reproduce one or more processors. Thus, by way of example, “part” or “module” refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, Includes procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Components and functionality provided within “parts” or “modules” may be combined into a smaller number of components and “parts” or “modules” or as additional components and “parts” or “modules”. can be further separated.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe the correlation between a component and other components. A spatially relative term should be understood as a term that includes different directions of components during use or operation in addition to the directions shown in the drawings. For example, when a component shown in the drawing is turned over, a component described as “beneath” or “beneath” of another component may be placed “above” of the other component. can Accordingly, the exemplary term “below” may include both directions below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

In this specification, a computer means all types of hardware devices including at least one processor, and may be understood as encompassing software configurations operating in the corresponding hardware device according to embodiments. For example, a computer may be understood to include, but is not limited to, smart phones, tablet PCs, desktops, notebooks, and user clients and applications running on each device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Each step described in this specification is described as being performed by a computer, but the subject of each step is not limited thereto, and at least a portion of each step may be performed in different devices according to embodiments.

1 is a diagram showing the configuration of a data labeling system according to an embodiment of the present invention.

Referring to FIG. 1 , a data labeling system according to an embodiment of the present invention may include a data labeling apparatus 100 , a user terminal 200 , and an external server 300 .

Here, the data labeling system shown in FIG. 1 is according to an embodiment, and its components are not limited to the embodiment shown in FIG. 1 , and may be added, changed, or deleted as necessary.

In an embodiment, the data labeling apparatus 100 may obtain 3D point cloud data and sensor data (eg, image data captured from a camera) from the external server 300 , and coordinate values and sensor data of the 3D point cloud data 3D point cloud data and sensor data can be calibrated by matching the coordinate values of .

In an embodiment, the data labeling apparatus 100 may generate 2D data by converting 3D point cloud data matched with sensor data (eg, image data captured from a camera) into 2D. For example, the data labeling apparatus 100 converts the 3D point cloud data matched with the sensor data based on the X-axis and the Y-axis into 2D to generate 2D data (Brid-eye-view of 3D point cloud data) view, BEV) to be output on the XY plane to generate 2D data).

In one embodiment, the data labeling apparatus 100 provides a UI including 2D data and sensor data (eg, image data captured from a camera) to the user terminal 200, thereby labeling from the user terminal 200 . You can get user input pointing to an object.

In various embodiments, the data labeling apparatus 100 may include a separate communication module (not shown), and the communication module is connected to the user terminal 200 through the network 400 to provide a UI to the user terminal 200 . (eg, FIG. 4 ) may be provided, or a user input input may be obtained through the provided UI. In addition, the communication module may be connected to the external server 300 through the network 400 to receive point cloud data and sensor data (eg, image data captured by a camera) from the external server 300 .

In an embodiment, the data labeling apparatus 100 may extract the 3D point cloud data coordinate value of the labeling object by using a user input indicating the labeling object obtained from the user terminal 200, and the extracted 3D point of the labeling object A sensor data coordinate value corresponding to a cloud data coordinate value (eg, a coordinate value on image data captured from a camera) may be selected, and labeling may be performed on the sensor data using the selected sensor data coordinate value.

In an embodiment, the user terminal 200 may include a display in at least a portion of the user terminal 200 , and may output a UI provided from the data labeling apparatus 100 through the display. For example, the user terminal 200 may include at least one of a smartphone, a tablet PC, a notebook computer, and a desktop, but is not limited thereto.

In various embodiments, the user terminal 200 may be connected to the data labeling apparatus 100 by wire or wireless, may receive a UI from the data labeling apparatus 100, and may receive a user input pointing to a labeling target input through the UI as data. It may be provided to the labeling apparatus 100 . For example, the user may be wirelessly connected to the data labeling apparatus 100 using a communication module included in his/her smartphone, and may output a UI provided from the data labeling apparatus 100 on the smartphone screen, By touch-inputting a labeling target on 2D data provided through a UI output to the phone, a labeling target can be selected. Thereafter, information on the labeling target (eg, the label name of the labeling target, coordinate values, etc.) may be provided to the data labeling apparatus 100 using a communication module included in the smartphone. However, the present invention is not limited thereto.

In an embodiment, the external server 300 includes sensor data detected from a plurality of sensors (eg, laser scan data obtained from a laser scan sensor and image (video or image) data detected through a camera sensor) and a 3D point cloud data can be saved. For example, the external server 300 receives sensor data (eg, camera image (video and image) data and laser scan data) detected while the information providing vehicle having a plurality of sensors moves in a predetermined area. can be saved However, the present invention is not limited thereto.

In various embodiments, the external server 300 may receive a transmission request of sensor data for a predetermined area from the data labeling apparatus 100 through the network 400, Only sensor data may be extracted and provided to the data labeling apparatus 100 .

In various embodiments, the external server 300 may group sensor data sensed at the same time point according to a time point at which the sensor data is sensed, and provide the grouped sensor data to the data labeling apparatus 100 . For example, a plurality of sensor data is grouped into sensor data at a first time point, sensor data at a second time point, and sensor data at a third time point, and the grouped sensor data is provided to the data labeling apparatus 100 . can

Here, the data labeling apparatus 100 receives various data from the external server 300, processes and analyzes the provided data to perform labeling on sensor data, but is not limited thereto, and the data labeling apparatus ( 100) is a control signal for performing a labeling operation of sensor data to the external server 300 (eg, a control instructing matching sensor data extraction signal of a simultaneous point for a predetermined area, 3D point cloud data and sensor data) Signals and 2D-type data generation signals, etc.) can be performed only, and the external server 300 uses the control signal received from the data labeling apparatus 100 to perform a labeling operation according to the control signal. can

2 is a hardware configuration diagram of a data labeling apparatus according to various embodiments of the present invention.

Referring to FIG. 2 , a data labeling apparatus 100 (hereinafter, "computing apparatus 100") according to another embodiment of the present invention may include a processor 110 and a memory 120 . In various embodiments, the computing device 100 may further include a network interface (or communication interface) (not shown), storage (not shown), and a bus (not shown).

In an embodiment, the processor 110 may control the overall operation of each component of the computing device 100 . The processor 110 may include a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), or any type of processor well known in the art.

In various embodiments, the processor 110 may perform an operation on at least one application or program for executing the method according to the embodiments of the present invention. In various embodiments, the processor 110 includes one or more cores (not shown) and a graphic processing unit (not shown) and/or a connection path (eg, a bus, etc.) for transmitting and receiving signals to and from other components. can do.

In various embodiments, the processor 110 temporarily and/or permanently stores a signal (or data) processed inside the processor 110 , a random access memory (RAM) and a read access memory (ROM). -Only Memory, not shown) may be further included. In addition, the processor 110 may be implemented in the form of a system on chip (SoC) including at least one of a graphic processing unit, a RAM, and a ROM.

In an embodiment, the processor 110 may perform a method (eg, a data labeling method) to be described with reference to FIGS. 3 to 8 by executing one or more instructions stored in the memory 120 . For example, the processor 110 executes one or more instructions stored in the memory 120, thereby converting 3D point cloud data to obtain a user input pointing to a labeling target on the 2D form data generated, 2D form data The operation of selecting the sensor data coordinate value corresponding to the 3D point cloud data coordinate value of the labeling target from the previously matched sensor data (eg, image data taken from the camera) and labeling on the sensor data using the sensor data coordinate value A data labeling method including an operation of performing

In one embodiment, memory 120 may store various data, commands, and/or information. The memory 120 may store programs (one or more instructions) for processing and controlling the processor 110 . Programs stored in the memory 120 may be divided into a plurality of modules according to functions.

In various embodiments, steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may contain random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains. Hereinafter, a data labeling method performed by the computing device 100 will be described with reference to FIGS. 3 to 8 .

3 is a flowchart illustrating a data labeling method according to various embodiments of the present invention.

In one embodiment, the computing device 100 obtains a user input pointing to a labeling target on 2D data in the data labeling method according to various embodiments of the present disclosure, and includes sensor data previously matched with the 2D data (eg: A sensor data coordinate value corresponding to a 3D point cloud data coordinate value of a labeling target is selected from image data taken from a camera), and labeling may be performed on the sensor data using the selected sensor data coordinate value. To this end, the computing device 100 may generate 2D data matched with sensor data before performing the data labeling method according to another embodiment of the present invention.

Here, the data in the 2D form may refer to data converted into a form that can output 3D point cloud data to a 2D screen (eg, the display of the user terminal 200 ) as shown in FIG. 5 . For example, the computing device 100 may select only sensor data at the same point in time among a plurality of sensor data obtained from the external server 300 and match the coordinate values of the selected sensor data with the coordinate values of the 3D point cloud data. have. Thereafter, the computing device 100 projects the 3D point cloud data on the XY plane in the form of a bird-eye-view (BEV), so that when it is output through the UI, it is output in a 2D form in a 2D form. data can be created.

In various embodiments, the computing device 100 may generate 2D data by quantizing 3D point cloud data on a 2D grid map. For example, the computing device 100 quantizes 3D point cloud data on a 2D grid map in the form of a bird's eye view, and uses at least one of an average value, a maximum value, and a density value as a representative value. By expressing it, it is possible to generate data in 2D form. However, the present invention is not limited thereto.

In various embodiments, the computing device 100 may generate 2D data for each viewpoint of sensor data (eg, image data captured from a camera). For example, the computing device 100 may generate 2D data for sensor data of a first viewpoint and 2D data for a second viewpoint different from the first viewpoint. However, the present invention is not limited thereto.

Referring to FIG. 3 , in step S110 , the computing device 100 outputs 2D data and sensor data (eg, image data captured from a camera) to the user terminal 200 , and a labeling target on the 2D data A UI may be provided for obtaining a user input pointing to , and outputting a first bounding box and a second bounding box corresponding to the user input. Hereinafter, a UI provided by the computing device 100 will be described with reference to FIG. 4 .

4 is a diagram exemplarily illustrating a UI provided by a data labeling apparatus to a user terminal according to various embodiments.

Referring to FIG. 4 , in various embodiments, the UI 40 may include a first UI 41 , a second UI 42 , and a third UI 43 .

In an embodiment, the computing device 100 may output the data 10 in a 2D form through the first UI 41 , and obtain a user input indicating a labeling target from the user. Thereafter, by analyzing the user input, information on the labeling object such as the type (or label name) of the labeling object and the 3D point cloud data coordinate value of the labeling object may be obtained.

In various embodiments, the computing device 100 may provide a function of enlarging or reducing the data 10 in the 2D form through the first UI 41 . In various embodiments, the computing device 100 may output the data 10 in the 2D form for a predetermined area on one screen through the first UI 41 .

In an embodiment, the computing device 100 may output the sensor data 20 (eg, a plurality of image data captured by a plurality of cameras) through the second UI 42 . For example, the computing device 100 may output the sensor data 20 previously matched with the 2D data output through the first UI 41 to the second UI 42 .

In various embodiments, the computing device 100 may provide the output UI 40 only to the first UI 41 and the third UI 43 , and a user input pointing to a labeling target through the first UI 41 . When obtaining , the second UI 42 that outputs the sensor data 20 (eg, image data) having sensor data coordinates corresponding to the 3D point cloud data coordinate values of the labeling target is displayed in a pop-up window form. can be output as

In an embodiment, the computing device 100 may output various tools for manipulating the first UI 41 and the second UI 42 through the third UI 43 . For example, the computing device 100, through the third UI 43, a tool for selecting the size and shape of the bounding box (eg, 11 and 12 in FIG. 6 ) according to the labeling target, the pre-generated bounding box A tool for changing properties and a tool for changing settings of the first UI 41 , the second UI 42 , and the third UI 43 may be provided. However, the present invention is not limited thereto, and the computing device 100 provides a UI 40 that outputs only the first UI 41 and the second UI 42, and only when a preset condition is satisfied, the third UI ( 43) can be output (eg, when the user makes a touch input for a specific location on the UI 40 , the third UI 43 is output).

Referring back to FIG. 3 , in step S120 , the computing device 100 may receive a user input indicating a labeling target through the UI provided to the user terminal 200 in step S110 . For example, the computing device 100 may receive a user input generated by touch-inputting a labeling target on 2D data output to the UI or a user input generated by clicking and inputting a labeling target with a mouse pointer using a mouse. can

In various embodiments, the computing device 100 generates a first bounding box in the form of a 2D figure for a labeling target based on the type and size of the labeling target, and uses the generated first bounding box to receive a user input on the UI. It can be output to a location (eg, a location where a user input is input on 2D data). Hereinafter, it will be described with reference to FIG.

6 is a diagram illustrating a form in which a user input indicating a labeling target is input on 2D data output through a UI provided by a data labeling apparatus according to various embodiments of the present disclosure;

Referring to FIG. 6 , in an embodiment, when the type of a labeling object on which the user wants to perform labeling is a car, the computing device 100 configures the first bounding boxes 11 and 12 in a rectangular shape based on the size of the car. ) can be created. The computing device 100 may output the first bounding boxes 11 and 12 in the form of a rectangle at a position corresponding to a user input (eg, a click input and a keyboard input through the mouse pointer 13) on the data in the 2D form. have.

In various embodiments, the computing device 100 provides a bounding box addition mode for generating a first bounding box through a UI according to a user input (eg, a keyboard input, etc.) Any one of a box correction mode and a Region of Uninterest (ROU) mode for excluding a specific region from data for learning may be provided.

In various embodiments, the computing device 100 may preset the size and shape of the first bounding boxes 11 and 12 generated according to a user input based on the type of labeling target, and the UI is a bounding box addition mode. In this case, the first bounding boxes 11 and 12 having a preset size and shape may be output at a position corresponding to the user input on the 2D data. For example, when the type of labeling object is a car, a 2D figure shape having a predetermined size according to the size and shape of the car (eg, the size and shape of a light car, motorcycle, small car, medium car, truck, and bus, etc.) For example, a template may be preset for the first bounding boxes 11 and 12 of a rectangular shape.

Then, when the computing device 100 obtains a user input including information on the position of the labeling target and the type of the labeling target through the UI, the rectangular first bounding boxes 11 and 12 set in advance at positions corresponding to the user input ) template can be printed. For example, if the type of the label object input by the user is a 40-seater bus, the first bounding box 11 of a size and shape corresponding to the 40-seater bus among templates of the first bounding boxes 11 and 12 created in advance , 12) may be output to the position of the labeling target corresponding to the user input. However, the present invention is not limited thereto, and the computing device 100 applies the method as described above even when the type of labeling target is a person, a fixed obstacle such as a construction site, a guard rail, and a sign to the first bounding box 11, 12) may be generated, and the templates of the first bounding boxes 11 and 12 previously generated according to the type of the label target corresponding to the user input may be output.

In various embodiments, when the UI is in the bounding box addition mode, the computing device 100 may include the first bounding boxes 11 and 12 corresponding to a user input (an input indicating adding a bounding box and an input indicating a type of a labeling target). can be selected, and the shape of the mouse pointer can be set in the same shape as the template of the selected first bounding boxes 11 and 12 (eg, a mouse pointer having a rectangular shape). In response to the user's mouse movement, the computing device 100 may move a mouse pointer having the same shape as the template of the first bounding boxes 11 and 12 on the screen, and the location of the user's click input on the 2D data. The first bounding boxes 11 and 12 having the same shape as the mouse pointer may be added to the .

In various embodiments, when an additional user input (eg, a keyboard input) is received in addition to the input indicating generation of the first bounding boxes 11 and 12 , the computing device 100 receives the first bounding box ( 11 , 12 ) according to the additional user input. 11, 12), the shape of the mouse pointer having the same shape as the template (eg, the length, width, and angle of the mouse pointer) can be changed. For example, when receiving a request for generating a first bounding box for the compact car from the user, the computing device 100 may set the mouse pointer in the same form as the template of the first bounding boxes 11 and 12 corresponding to the compact car. have. In this case, when the computing device 100 receives a first keyboard input from the user indicating that the labeling target is changed from a small car to a motorcycle, a mouse set in the same shape as the template of the first bounding boxes 11 and 12 corresponding to the small car By changing the length, width, and angle of the pointer, the mouse pointer may be changed in the same form as the template of the first bounding box 11 and 12 corresponding to the motorcycle. In various embodiments, the computing device 100 may be configured to A template of a plurality of first bounding boxes 11 and 12 having different shapes (eg, size, shape, angle, etc.) may be provided, and a plurality of first bounding boxes 11 and 12 may be provided according to a user input. At least one template among the templates of . Thereafter, the computing device 100 may automatically determine the type of the labeling target indicated by the first bounding box according to the shape (size, shape) of the generated template.

In various embodiments, the computing device 100 may receive a direct input of the first bounding boxes 11 and 12 (eg, a drag input of a mouse pointer) through the UI, and the directly inputted first bounding boxes 11 and 12 . ) by using the size and shape of the labeling target can be determined. For example, the computing device 100 uses the directly input size and shape of the first bounding box 11 and 12 and a pre-stored template of the first bounding box 11 and 12 for each labeling target to the first bounding box ( 11, 12) can determine the type of the inputted labeling target. Through this, the computing device 100 has the advantage of being able to identify the type of the labeling target without the user separately inputting the type of the labeling target through the UI.

In various embodiments, the computing device 100 may generate in advance a template of the first bounding boxes 11 and 12 in the form of a 2D figure having a predetermined size regardless of the type and size of the labeling object, and A template of the first bounding boxes 11 and 12 in the form of a 2D figure having a predetermined size generated in advance can be output at a corresponding position, and a 2D having a predetermined size according to the modification input from the user through the UI The size, shape, and position of the first bounding boxes 11 and 12 in the shape of a figure may be modified. For example, the computing device 100 uses a size adjustment request (eg, a mouse pointer 13 ) from a user with respect to the first bounding boxes 11 and 12 output through the UI, and the first bounding boxes 11 , 11 , 12), when dragging is input while clicking the corner or vertex), requesting a shape change (eg, selecting (or activating) the first bounding box 11, 12, and selecting the shape of the shape to be changed. When selecting) and a request for position adjustment (for example, when dragging input to a desired position while selecting (or activating) the first bounding boxes 11 and 12), correction contents including input can be received .

Thereafter, when the UI is in the bounding box modification mode, the computing device 100 sets the first boundary in the form of a 2D figure having a predetermined size according to the modification including at least one of a size adjustment request, a shape change request, and a position adjustment request. The shape of the boxes 11 and 12 can be changed.

In various embodiments, the computing device 100 may output the first bounding box 11 to which the correction content is not inputted from the user and the first bounding box 12 to which the correction content is inputted or inputted from each other and output. can For example, the computing device 100 may display the first bounding box 11 in which correction information is not input in red, and the first bounding box 12 in which correction information is input or being input, in blue. have. Through this, the user using the UI can determine the labeling target for which the labeling operation has been completed (eg, the first bounding box 11 in which the correction information is not input) and the labeling target for which the labeling operation is being performed (eg, the correction information is input or is being inputted). The first bounding box 12) may be intuitively distinguished.

In various embodiments, the computing device 100 is not activated because the first bounding box 12 is selected and activated by the user and is not selected from the user with respect to the plurality of first bounding boxes 11 and 12 included in the UI. The color of the first bounding box 11 that is not there may be different from each other and output. For example, by displaying the first bounding box 11 that is not selected and activated by the user in red, and the first bounding box 12 that is selected and activated by the user in blue, the user is currently It may be possible to intuitively check whether a labeling operation is being performed on a labeling target.

In various embodiments, the computing device 100 inputs modifications to the first bounding boxes 11 and 12 from the user (eg, length, width, angle, height, increase and decrease of the ground, etc. through keyboard input). When receiving, the first bounding box 12 selected and activated by the user among the plurality of first bounding boxes 11 and 12 may be modified according to the correction inputted by the user.

In various embodiments, when the UI is in the ROU mode, the computing device 100 may receive a region setting of any size input from the user, and may exclude an input region of any size from the data for learning. have. For example, the computing device 100 designates an area of any size through a drag operation in a state where the UI is in the ROU mode, and maintains the clicked state of the mouse, and clicks the mouse when the area designation is completed. An area of an arbitrary size can be input through the tearing operation, and the received area of an arbitrary size can be excluded from the data for learning.

Referring back to FIG. 3 , in step S130 , the computing device 100 may select sensor data coordinates (eg, coordinates on image data) for a labeling target. For example, when the computing device 100 receives a user input for a labeling target on 2D data, 3D point cloud data coordinates for a labeling target corresponding to the user input may be extracted.

Thereafter, the computing device 100 may select the sensor data coordinates corresponding to the 3D point cloud data coordinate values of the labeling target from the sensor data previously matched with the 2D form data. For example, the computing device 100 uses the coordinates (three-dimensional coordinates of x, y, and z) of the 3D point cloud data matched with the data in the 2D form projected onto the XY plane in the form of a bird's eye view, the 3D point cloud Coordinate values of data and projected sensor data (eg image data) can be selected.

In various embodiments, the computing device 100 may generate a second bounding box to be output on the sensor data by using the first bounding box generated on the 2D data. Thereafter, the computing device 100 may select the sensor data coordinates using a matrix capable of mapping the 3D coordinate values (X, Y, Z) of the second bounding box and the sensor data coordinate values. .

In various embodiments, the computing device 100 performs a 2D shape or a 3D shape on sensor data (eg, image data) based on the first bounding box (eg, 11 and 12 of FIG. 6 ) generated in step S120 . It is possible to create a second bounding box of . Hereinafter, it will be described with reference to FIGS. 7 and 8 .

7 is a diagram illustrating a bounding box in the form of a 3D figure output on a UI according to a user input when sensor data is 3D data, according to various embodiments of the present disclosure;

Referring to FIG. 7 , when the sensor data is 3D data, the computing device 100 provides a second bounding box 31a in the form of a 3D figure based on the first bounding box 11a , 11b , 11c generated in step S120 , 31b, 31c) can be created.

In various embodiments, when the sensor data is 3D data (eg, 3D laser scan data), the computing device 100 uses height information for at least one point inside the first bounding boxes 11a, 11b, and 11c. to calculate the bottom surface and height of the labeling object, and use the calculated bottom surface and height of the labeling object to place a second bounding box 31a in the form of a 3D figure at a position corresponding to the sensor data coordinate value on the 3D sensor data 30 , 31b, 31c) can be generated.

For example, when the type of the labeling object is a car and the first bounding boxes 11a, 11b, and 11c have a rectangular shape, the computing device 100 is located inside the rectangular first bounding boxes 11a, 11b, and 11c. The average height, minimum height, and maximum height of the included points may be calculated, and the floor surface and height of the vehicle to be labeled may be calculated using the calculated average height, minimum height, and maximum height.

Thereafter, the computing device 100 is a center point (x, y, z), angle (yaw), size (length (length), width (width), height (height)) using the floor surface and height of the vehicle to be labeled may be calculated, and the second bounding boxes 31a, 31b, and 31c in the form of a rectangular parallelepiped having the calculated center point, angle, and size may be generated. Also, the computing device 100 may output the generated second bounding boxes 31a, 31b, and 31c in the form of a cuboid at positions corresponding to the sensor data coordinate values of the labeling target on the 3D sensor data 30 . However, the present invention is not limited thereto.

In various embodiments, the computing device 100 cross-sections (eg, XZ plane and YZ plane) the periphery of the second bounding box 31a , 31b , 31c generated using the calculated labeling target car floor surface and height. It can be cut and output on the UI in the form of a pop-up window. Through this, even if the floor surface and height of the vehicle calculated by the computing device 100 are not accurate, the user checks the cross sections of the second bounding boxes 31a, 31b, and 31c output in the form of a pop-up window and corrects them in more detail and can make it change.

In various embodiments, the computing device 100 receives, from a user, a user input for adjusting the center point, angle, and size of the second bounding box (eg, directly inputting values of the center point, angle, and size through a keyboard) and calculates the calculated value. The center point, angle and size values can be modified.

In various embodiments, the computing device 100 may directly receive the size and shape of the first bounding box 11a, 11b, 11c through the UI, and even if the user does not directly input the type of the labeling target, the UI The type of the labeling target indicated by each of the first bounding boxes 11a, 11b, and 11c may be determined by using the sizes and shapes of the first bounding boxes 11a, 11b, and 11c directly input through the .

For example, the type of the labeling target may include not only the type of vehicle traveling on the road (eg, light vehicle, motorcycle, small vehicle, medium-sized vehicle, truck, and bus, etc.), but also a person, It may include, but is not limited to, obstacles, guard rails, and signs. The type of the labeling target may be obtained based on a user's input as described above, or may be determined based on at least a part of the shape and size of the first bounding box and data included in the first bounding box. When information on the type of the labeling object is previously stored, the computing device 100 may acquire information on the three-dimensional shape and size of the corresponding labeling object, and according to an embodiment, Even in this case, the three-dimensional shape and size of the corresponding labeling target may be estimated at a predetermined ratio based on the size of the first bounding box.

Thereafter, the computing device 100 selects the templates of the second bounding boxes 31a, 31b, and 31c generated in advance based on the type of labeling target indicated by each of the first bounding boxes 11a, 11b, and 11c. Selects the template of the second bounding box 31a, 31b, 31c corresponding to the type of , and sets the template of the selected second bounding box 31a, 31b, 31c to the sensor data coordinates of the labeling target on the 3D sensor data 30 It can be output to the position corresponding to the value.

In various embodiments, the computing device 100 may receive an input of correction for the second bounding boxes 31a, 31b, and 31c from the user, and set the second bounding boxes 31a, 31b, and 31c according to the correction. Simultaneously with the correction, the first bounding boxes 11a, 11b, and 11c corresponding to the second bounding boxes 31a, 31b, and 31c may also be modified.

8 is a diagram illustrating a bounding box in the form of a 2D figure output on a UI according to a user input when sensor data is 2D data, according to various embodiments of the present disclosure;

Referring to FIG. 8 , when sensor data is 2D data, second bounding boxes 21a, 21b, and 21c in the form of a 2D figure are generated based on the first bounding boxes 11a, 11b, and 11c generated in step S120. can do.

In various embodiments, when the sensor data is 2D data (eg, image data captured from a camera), the computing device 100 generates a second bounding box in the above method (eg, when the sensor data is 3D data) method), the coordinate values of the second bounding box (eg, 31a, 31b, 31c in FIG. 7 ) in the form of a 3D figure are converted into an image coordinate system through the camera parameters in the form of a matrix, and the 2D sensor data 20 The second bounding boxes 21a, 21b, and 21c in the form of a 2D figure converted into the image coordinate system may be generated at positions corresponding to the data coordinate values.

For example, when the type of the labeling object is a car and the first bounding boxes 11a, 11b, and 11c have a rectangular shape, the computing device 100 is located inside the rectangular first bounding boxes 11a, 11b, and 11c. The average height, minimum height, and maximum height of the included points may be calculated, and the floor surface and height of the vehicle to be labeled may be calculated using the calculated average height, minimum height, and maximum height.

Thereafter, the computing device 100 calculates a center point, an angle, and a size using the floor surface and height of the vehicle as a labeling target, and a second bounding box in the form of a cuboid having the calculated center point, angle, and size (eg, in FIG. 31a, 31b, 31c) of 7 can be created.

Thereafter, the computing device 100 converts the vertex coordinate image coordinate system of the second bounding box (eg, 31a, 31b, 31c of FIG. 7 ) of a rectangular parallelepiped shape to the second bounding box 21a, 21b, 21c in a rectangular shape. may be generated, and the generated rectangular second bounding boxes 21a, 21b, and 21c may be output at positions corresponding to the sensor data coordinate values of the labeling target in the 2D sensor data 20 . However, the present invention is not limited thereto.

According to various embodiments, the computing device 100 is modified when receiving modifications of the previously created first bounding box (eg, a size adjustment request, a shape change request, a location change request, etc. for the first bounding box). The first bounding box may be changed according to the content, and the second bounding box may be changed according to the changed content of the changed first bounding box. For example, when a correction requesting that the shape of the first bounding box be changed from a rectangle to a square is input through the UI, the computing device 100 changes the shape of the first bounding box from a rectangle to a square according to the changed content , and the shape of the second bounding box may be changed from a cuboid to a cube based on the change in the shape of the first bounding box.

In various embodiments, when a modification to the previously generated first bounding box is input through the UI, the computing device 100 may simultaneously change the shapes of the first bounding box and the second bounding box according to the modified content. . For example, when the first modification input through the UI is a size change request, the computing device 100 may sequentially change the sizes of the first bounding box and the second bounding box according to the size change request. Also, when a second modification requesting a position change is input through the UI, the positions of the first bounding box and the second bounding box may be sequentially changed according to the second correction. Through this, the user can see how the second bounding box changes according to the content of the modification of the shape of the first bounding box through the first UI (eg, 41 in FIG. 4 ) through the second UI (eg, 42 in FIG. 4 ). It has the advantage of being able to check it immediately.

In various embodiments, the computing device 100 applies all modifications at the point in time when all the modifications are completed when the previously generated first bounding box is inputted through the UI to form the second bounding box. can be changed. For example, in a state in which correction content is being input to the first bounding box (eg, 12 in FIG. 6 ), when the first correction content and the second correction content are sequentially input, the computing device 100 performs the first The shape of the first bounding box may be changed by sequentially reflecting the correction content and the second correction content, and when the correction content input to the first bounding box is completed (eg, 11 in FIG. 6 ), the first correction content and the shape of the second bounding box may be changed by collectively reflecting the second modification. When the user repeatedly inputs corrections with respect to the first bounding box, the amount of computation increases if the second bounding box is also repeatedly changed according to the repeatedly inputted corrections. In this case, the computing device 100 aggregates the corrections for the first bounding box input repeatedly as described above and collectively reflects the corrections in the second bounding box, thereby increasing the amount of computation by repeatedly modifying the second bounding box. This has the advantage of being able to prevent it.

In various embodiments, the computing device 100 receives a first boundary based on the size and shape of the first boundary box modified according to the modification when a modification to the previously generated first boundary box is input through the UI. The type of labeling target corresponding to the box can be changed. Also, the computing device 100 may change the size and shape of the second bounding box as the type of the labeling target is changed.

In various embodiments, the computing device 100 may receive an input of correction for the second bounding boxes 21a, 21b, and 21c from the user, and set the second bounding boxes 21a, 21b, and 21c according to the correction. Simultaneously with the correction, the first bounding boxes 11a, 11b, and 11c corresponding to the second bounding boxes 21a, 21b, and 21c may also be modified.

Referring back to FIG. 3 , in step S140 , the computing device 100 may perform labeling on the sensor data using the sensor data coordinate value selected in step S130 . For example, the computing device 100 may include sensor data, data indicating the type (or label name) (eg, car, person, etc.) of the labeling object (eg, the type of the labeling object directly input from the user or the first bounding box) data indicating the type of labeling target determined according to the size and shape of ), a 3D point cloud data coordinate value of the labeling target, and a data set including the sensor data coordinate value of the labeling target may be configured. However, the present invention is not limited thereto.

According to various embodiments, when one 2D data and a plurality of sensor data are matched, a data set may be configured for each of the plurality of sensor data.

In various embodiments, when the labeling operation of the sensor data at the first time point among the sensor data is completed, the computing device 100 performs 2D data matching with the sensor data at the second time point after the first time point. can be loaded automatically. For example, when the labeling operation on the first 2D form data previously matched with the sensor data at the first time point is completed (eg, when a user input indicating labeling completion is received through the UI), the first time point Data in a second 2D form previously matched with sensor data at a subsequent second time point may be automatically loaded. Through this, the user can maintain the continuity of the operation in that the user performing the labeling on the sensor data can perform the labeling operation on the sensor data after the completed sensor data without taking a separate action. have.

In various embodiments, the computing device 100 may receive a load request for at least one 2D data from among a plurality of 2D data for which a labeling operation has been completed. Computing device 100 provides at least one 2D data requested from a user and information on labeling performed on the at least one 2D data (eg, a data set generated by a labeling operation and a detection algorithm result) By providing a UI including data (such as 2D data and data indicating a result of automatically performing a labeling operation on sensor data through a detection algorithm) to the user terminal 200, at least one 2D data A new bounding box creation may be input on the upper bounding box, or modifications to a pre-created bounding box may be input.

The data labeling method described above has been described with reference to the flowchart shown in the drawings. For the sake of simplicity, the data labeling method has been described by showing a series of blocks, but the present invention is not limited to the order of the blocks, and some blocks may be performed in an order different from that illustrated and described herein, or may be performed simultaneously. can In addition, new blocks not described in this specification and drawings may be added, or some blocks may be deleted or changed.

The components of the present invention may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. The components of the present invention may be implemented as software programming or software components, and similarly, embodiments may include various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs, including C, C++ , Java, assembler, etc. may be implemented in a programming or scripting language. Functional aspects may be implemented in an algorithm running on one or more processors.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: data labeling device (or computing device)
200: user terminal
300 : external server

Claims (11)

A data labeling method performed by a computing device, comprising:
obtaining a user input pointing to a labeling target on 2D data generated by converting 3D point cloud data for a predetermined region;
3D of the labeling target using sensor data previously matched with the 3D point cloud data, wherein the sensor data includes at least one of laser scan data obtained from a laser scan sensor and image data captured through a camera sensor selecting a coordinate value on the sensor data corresponding to the point cloud data coordinate value; and
performing labeling on the sensor data by using the coordinate values on the sensor data,
The step of obtaining the user input includes:
Providing a user interface (UI) for outputting the 2D data and the sensor data, and obtaining a user input indicating the labeling target on the 2D data output through the provided UI , data labeling methods. According to claim 1,
The step of obtaining the user input includes:
generating a first bounding box in the form of a 2D figure for the labeling object based on the size of the labeling object,
The step of selecting a coordinate value on the sensor data comprises:
and generating a second bounding box in the form of a 2D or 3D figure on the sensor data by using the first bounding box. 3. The method of claim 2,
The generating of the second bounding box includes:
When the sensor data is 3D data, the bottom surface and height of the labeling object are calculated using height information on at least one point inside the first bounding box, and the calculated bottom surface and height of the labeling object are calculated generating a second bounding box in the form of a 3D figure at a position corresponding to a coordinate value on the sensor data on the sensor data by using; and
When the sensor data is 2D data, the coordinate value of the second bounding box of the 3D figure form is converted into an image coordinate system, and the first converted into the image coordinate system at a position corresponding to the coordinate value on the sensor data on the sensor data 2 A method for labeling data, comprising generating a bounding box. 3. The method of claim 2,
The step of obtaining the user input includes:
and outputting the first bounding box and the second bounding box corresponding to the user input through the UI. 5. The method of claim 4,
The step of obtaining the user input includes:
Further comprising the step of receiving a correction input for the pre-generated first bounding box through the UI,
The step of selecting a coordinate value on the sensor data comprises:
The method further comprising the step of changing the shape of the second bounding box according to the modified content of the pre-generated first bounding box. 5. The method of claim 4,
The step of obtaining the user input includes:
The method further comprising the step of outputting any one template among a plurality of templates of a plurality of first bounding boxes having different shapes previously generated according to the user input on the UI,
The step of performing the labeling comprises:
Further comprising the step of determining the type of the labeling target according to the form of any one of the output template, the data labeling method. According to claim 1,
The data labeling method comprises:
matching the coordinate values of the sensor data at the same time point among the sensor data and the coordinate values of the 3D point cloud data; and
At least one of a method of projecting the 3D point cloud data in the form of a bird-eye-view (BEV) based on the X-axis and the Y-axis and a method of quantizing the 3D point cloud data on a 2D grid map The method further comprising the step of generating the data in the 2D form using the method, data labeling method. 8. The method of claim 7,
The data labeling method comprises:
When the labeling operation of the sensor data at the first time point among the sensor data is completed, 2D form data generated by converting the 3D point cloud data matched with the sensor data at the second time point after the first time point Further comprising the step of automatically loading the data labeling method. delete a memory storing one or more instructions; and
a processor executing the one or more instructions stored in the memory;
The processor by executing the one or more instructions,
An apparatus for performing the method of claim 1 . A computer program stored in a computer-readable recording medium in combination with a computer, which is hardware, to perform the method of claim 1.

Images