KR102378890B1 - A method of reducing data load of images for annotation, and computer program recorded on record-medium for executing method thereof - Google Patents

Abstract

The present invention provides a method capable of preprocessing three-dimensional point group data in connection with annotation work of data for machine-learning artificial intelligence (AI). The method comprises: a step in which a learning data generation device receives three-dimensional point group data acquired through lidar fixed and installed in a vehicle for machine learning artificial intelligence (AI) usable in autonomous driving of the vehicle; a step in which the learning data generation device reduces the volume of the three-dimensional point group data based on distance values of points included in the three-dimensional point group data; and a step in which the learning data generation device distributes the three-dimensional point group data with a reduced volume to a plurality of annotation devices.

Description

{A method of reducing data load of images for annotation, and computer program recorded on record-medium for executing method thereof}

The present invention relates to the processing of data for artificial intelligence (AI) learning. More specifically, in relation to the data annotation operation for machine learning of artificial intelligence (AI), a method for reducing the data load of 3D point cloud for annotation and a computer program recorded on a recording medium for executing the same is about

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning ability, reasoning ability, and perception ability using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the type of data for learning.

In general, the design of data for artificial intelligence (AI) learning proceeds in the stages of data structure design, data collection, data purification, data processing, data expansion, and data verification.

To describe each step in more detail, the design of the data structure is made through the definition of an ontology, a definition of a classification system, and the like. Data collection is performed by collecting data through direct shooting, web crawling, or association/professional organizations. Data purification is performed by removing duplicate data from the collected data and de-identifying personal information. Data processing is performed by performing annotations and inputting metadata. Data expansion is performed by performing ontology mapping, and supplementing or extending the ontology as necessary. And, the verification of the data is performed by verifying the validity according to the set target quality using various verification tools.

Meanwhile, automatic driving of a vehicle refers to a system capable of driving by determining the vehicle itself. Such autonomous driving may be divided into gradual stages from non-automation to full automation according to the degree to which the system is involved in driving and the degree to which the driver controls the vehicle. In general, the stage of autonomous driving is divided into six levels classified by the Society of Automotive Engineers (SAE) International. According to the six levels classified by the International Association of Automobile Engineers, Level 0 is non-automated, Level 1 is driver assistance, Level 2 is partial automation, Level 3 is conditional automation, Level 4 is highly automated, and Level 5 is The steps are fully automated steps.

Autonomous driving of a vehicle is performed through mechanisms of perception, localization, path planning, and control. Currently, several companies are developing to implement cognitive and path planning among autonomous driving mechanisms using artificial intelligence (AI). And, the data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving consists of a large number ranging from a few thousand to a maximum of several million.

Numerous workers participate in the process of individually annotating a large number of data to be used for machine learning with artificial intelligence (AI) to be used for autonomous driving. However, the computing power of each local computing device used by numerous workers to individually perform annotation work is not uniform, and the bandwidth and response of the network line that transmits and receives data to and from each local computing device The response time is also not uniform.

Also, the non-uniform computing power of the local computing device and the bandwidth and response speed of the network line affect the efficiency of the annotation operation.

Korean Patent Application Laid-Open No. 10-2020-0042629, ‘Method for generating touch-based annotations and images in mobile devices for artificial intelligence learning, and device therefor’, (published on April 24, 2020)

It is an object of the present invention to provide a method for reducing the data load of 3D point cloud for annotation in relation to an annotation operation of data for machine learning of artificial intelligence (AI).

Another object of the present invention is a computer program recorded on a recording medium to execute a method of reducing the data load of 3D point cloud for annotation in relation to an annotation operation of data for machine learning of artificial intelligence (AI). is to provide

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical task as described above, the present invention proposes a method capable of preprocessing 3D point cloud data in relation to annotating data for machine learning of artificial intelligence (AI). The method is a 3D point cloud (3D) obtained through a lidar fixed to a vehicle in order for the learning data generating device to machine learning artificial intelligence (AI) that can be used for autonomous driving of the vehicle. points group) receiving data; reducing, by the learning data generating apparatus, a capacity of each 3D point cloud data based on a distance value of points included in the 3D point cloud data; and distributing, by the training data generating device, the 3D point cloud data with the reduced capacity to a plurality of annotation devices.

Specifically, the step of reducing the capacity may include: identifying object candidates by extracting points forming a crowd within a preset threshold range from among points included in each 3D point cloud data; and changing or removing some of the points included in the 3D point cloud data based on a distance value of points constituting the identified object candidates or a size value of the identified object candidates.

The receiving of the 3D point cloud data further includes receiving a 2D image photographed through a camera fixedly installed in the vehicle, and the identifying of the object candidates includes RGB of a region corresponding to the object candidate in the received 2D image. Verification may be performed on the identified object candidates based on the (Red, Green, Blue) values.

The step of changing or removing some of the points estimates a class for each of the identified object candidates, and an identifiable distance for each of the object candidates based on pre-determined identifiable distance values for each type of class. A value may be calculated, and the points to be changed or removed may be identified by comparing the distance value from the lidar to each object candidate and the calculated identifiable distance value.

In this case, the step of changing or removing some of the points includes a width on an X-axis, a height on a Y-axis, a depth on a Z-axis, and a preset dimension for each class of the object candidate. ), it is possible to estimate a class for each of the object candidates by comparing the range.

According to an embodiment, changing or removing some of the points may remove points constituting a distant object candidate from the 3D point cloud data. In this case, the distant object candidate may be an object candidate having a minimum distance value equal to or greater than a preset threshold distance among distance values of points constituting the object candidate.

According to another embodiment, in the step of changing or removing some of the points, points constituting an object candidate having a distance value greater than or equal to the identifiable distance value from the lidar are replaced with a single preset garbage value. can do.

According to another embodiment, in the changing or removing of some of the points, points constituting an object candidate having a distance value greater than or equal to the identifiable distance value from the lidar may be removed from the 3D point cloud data.

According to another embodiment, the step of changing or removing some of the points includes changing or removing points constituting an object candidate estimated as a type of a class set in advance as irrelevant to machine learning of the artificial intelligence (AI). You can also identify the points to be removed.

In order to achieve the technical task as described above, the present invention relates to an annotation operation of data for machine learning of artificial intelligence (AI) recorded on a recording medium to execute a method capable of pre-processing 3D point cloud data. Suggest a computer program. The computer program includes a memory; transceiver; and a processor for processing instructions resident in the memory. The computer program may include: receiving, by the processor, 3D point cloud data acquired through a lidar fixed in a vehicle through the transceiver to machine-learning artificial intelligence (AI) that can be used for autonomous driving of the vehicle; reducing, by the processor, a capacity of each 3D point cloud data based on a distance value of points included in the 3D point cloud data; and distributing, by the processor, the 3D point cloud data with the reduced capacity to a plurality of annotation devices through the transceiver, it may be a computer program recorded on a recording medium.

Specific details of other embodiments are included in the detailed description and drawings.

According to an embodiment of the present invention, by pre-processing point clouds of low quality to the extent that objects cannot be identified in 3D point cloud data and reducing their capacity, the impact on the annotation operation result is minimized, and at the same time, the local computing device and data It is possible to reduce the influence of the bandwidth and response speed of the network line for sending and receiving.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.
2 is an exemplary diagram for explaining an apparatus for collecting learning data according to an embodiment of the present invention.
3 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.
4 is a logical configuration diagram of a data processing unit of an apparatus for generating learning data according to an embodiment of the present invention.
5 is a hardware configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.
6 and 7 are exemplary views for explaining a 2D image partitioned into a plurality of regions according to some embodiments of the present invention.
8 and 9 are exemplary diagrams for explaining a 2D image with reduced capacity according to some embodiments of the present invention.
10 is an exemplary diagram for describing identified object candidates according to an embodiment of the present invention.
11 is an exemplary diagram for explaining a process of estimating a class of an object candidate according to an embodiment of the present invention.
12 is an exemplary diagram for explaining 3D point cloud data with reduced capacity according to an embodiment of the present invention.
13 is an exemplary diagram for explaining a process of selecting a reference object candidate according to an embodiment of the present invention.
14 is an exemplary diagram for explaining a process of collectively applying a work result to a reference object candidate according to an embodiment of the present invention.
15 is an exemplary diagram for explaining a process of evaluating the capability of a local device according to an embodiment of the present invention.
16 is an exemplary diagram for explaining a process of rewriting an annotation work result according to an embodiment of the present invention.
17 is a flowchart illustrating a method of generating learning data according to an embodiment of the present invention.
18 is a flowchart illustrating a data processing method according to an embodiment of the present invention.

It should be noted that the technical terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in this specification should be interpreted as meanings generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise specifically defined in this specification, and excessively inclusive It should not be construed in the meaning of a human being or in an excessively reduced meaning. In addition, when the technical terms used in the present specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be understood by being replaced with technical terms that can be correctly understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted according to the definition in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or various steps described in the specification, and some of the components or some steps are included. It should be construed that it may not, or may further include additional components or steps.

Also, terms including ordinal numbers such as first, second, etc. used herein may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but another component may exist in between. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be interpreted as extending to all changes, equivalents or substitutes other than the accompanying drawings.

As described above, the computing power of each local computing device used by numerous workers to individually perform annotation work is not uniform, and the bandwidth and response speed of the network line for transmitting and receiving data to and from each local computing device are also not uniform. . Also, the non-uniform computing power of the local computing device and the bandwidth and response speed of the network line affect the efficiency of the annotation operation.

In order to solve this difficulty, the present invention intends to propose various means that can minimize the influence received from the computing power of the local computing device or the quality of the network line.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.

As shown in FIG. 1 , an artificial intelligence learning system according to an embodiment of the present invention includes a learning data collection device 100 , a learning data generating device 200 , a plurality of annotation devices 300 , and an artificial intelligence learning device ( 400) may be included.

As such, since the components of the artificial intelligence learning system according to an embodiment are merely functionally distinct elements, two or more components are integrated with each other in the actual physical environment, or one component is the actual physical environment. may be implemented separately from each other.

Description of each component, the learning data collection device 100 is a device that can be used to collect data for machine learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

To this end, the learning data collection apparatus 100 may be configured to include one or more of a lidar, a camera, a radar, and an ultrasonic sensor, but is not limited thereto, and one It may be a computing device that collects data measured by the above lidar, camera, radar or ultrasonic sensor.

An example in which the learning data collection device 100 collects data for artificial intelligence (AI) machine learning will be described later with reference to FIG. 2 .

With the following configuration, the learning data generating device 200 is a device that can be used to design and generate data for machine learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

As such, the learning data generating apparatus 200 is basically a device distinct from the learning data verifying apparatus 400, but in an actual physical environment, the learning data generating apparatus 200 and the learning data verifying apparatus 400 are one device. It may be integrated and implemented.

Characteristically, the learning data generating apparatus 200 according to embodiments of the present invention processes the 2D images and 3D point cloud data received from the learning data collection apparatus 100 using a plurality of annotation apparatuses 300 . Therefore, it is possible to minimize the influence received from the computing power of the annotation device 300 corresponding to the local computing device or the quality of the network line.

The training data generating device 200 having such a characteristic transmits and receives data to and from the training data collection device 100 , the annotation device 300 , and the artificial intelligence learning device 400 , and performs an operation based on the transmitted/received data Any device that can do this is acceptable. For example, the learning data generating apparatus 200 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

With the following configuration, the annotation device 300 is a local computing device that can be used to perform an annotation operation on 2D images or 3D point cloud data distributed by the training data generating device 200 . As such, all or a part of the annotation device 300 may be a device in which an annotator performs an annotation operation through a clouding service.

Specifically, the annotation apparatus 300 may output one 2D image or 3D point cloud data to be annotated among the 2D images or 3D point cloud data received from the training data generating apparatus 200 to the display.

The annotation device 300 may select a tool according to a signal input from the user through the input/output device. Here, the tool is a tool for setting a bounding box specifying one or more objects included in a 2D image or 3D point cloud data.

The annotation device 300 may receive coordinates according to the selected tool through the input/output device. In addition, the annotation apparatus 300 may specify an object included in the 2D image or 3D point cloud data by setting a bounding box based on the input coordinates. Here, the bounding box is an area for specifying an object to be subjected to artificial intelligence (AI) learning among objects included in the image. As such, the bounding box may have a shape of a rectangle or a cube, but is not limited thereto.

For example, the annotation device 300 receives two coordinates through an input/output device, and uses the input two coordinates as the coordinates of the upper-left vertex and the coordinates of the lower-right vertex in the 2D image. Bounding based on a rectangle By setting the box, you can specify the object included in the 2D image. In this case, the two coordinates may be set by the user inputting one type of input signal twice (eg, clicking the mouse), or the user may inputting two types of input signals once (eg, dragging the mouse). However, the present invention is not limited thereto.

The annotation apparatus 300 may generate 2D image or 3D point cloud data to be annotated, or metadata for a set object according to a signal input from a user through the input/output device. Here, the metadata is 3D point cloud data or 2D image and data for describing an object specified from 3D point cloud data or 2D image. Such metadata includes a category of an object specified from 3D point cloud data or a 2D image, the ratio of the object cut by the angle of view, the ratio of the object to another object or object, the tracking ID of the object, the time the image was taken, the image may include, but are not limited to, the weather conditions on the day that ' was taken, file size, image size, copyright holder, resolution, bit value, aperture transmission amount, exposure time, ISO sensitivity, focal length, aperture opening number, angle of view, white Balance, RGB depth, class name, tag, shooting location, type of road, road surface information or traffic jam information may be further included.

The annotation apparatus 300 may generate an annotation work result based on an object set from the 2D image or 3D point cloud data and the generated metadata. In this case, the annotation operation result may have a JSON (Java Script Object Notation) file format, but is not limited thereto. The annotation apparatus 300 may transmit the generated annotation work result to the training data generating apparatus 200 . In addition to the generated annotation work result, the annotation apparatus 300 may transmit a 2D image or 3D point cloud data in which an object is set to the learning data generating apparatus 200 for verification.

Characteristically, the annotation apparatus 300 according to an embodiment of the present invention configures the annotation apparatus 300 without transmitting the 2D image or 3D point cloud data in which the result of the annotation work and the object are set to the learning data generating apparatus 200 . It is also possible to transmit the control data of the input/output device in operation to the learning data generating device 200 .

Here, the control data of the input/output device is data stored in time-series of one or more signals input by the user to control the input/output device while the annotation device 300 performs an annotation operation on the 2D image or 3D point cloud data. can be Here, the user may be referred to as an operator, performer, labeler, or data labeler, but is not limited thereto.

For example, when the annotation device 300 is driven by an operating system according to an event-driven architecture, one or more signals included in the control data of the input/output device are the annotation device 200 . It may be an event message generated by the operating system in response to the control of the input/output device. Also, the annotation device 300 may generate control data of the input/output device by duplicating a system queue in which event messages generated by the operating system are stored in a first-in first-out structure.

As a more specific example, when the operating system of the annotation device 300 corresponds to Windows, the control data of the input/output device includes WM_LBUTONDOWN generated in response to a left button click of the mouse, WM_KEYDOWN generated in response to a keyboard input, etc. Event messages may be included.

The annotation device 300 having the above-described characteristics may be any device as long as it is a device capable of transmitting and receiving data to and from the training data generating device 200 and performing an operation based on the transmitted/received data. For example, the annotation device 300 may be a desktop, a stationary computing device such as a workstation or a server, or a smart phone, a laptop, a tablet, a phablet, or a portable multimedia player. It may be any one of a portable computing device such as a portable multimedia player (PMP), a personal digital assistant (PDA), or an e-book reader (E-book reader).

With the following configuration, the artificial intelligence learning device 400 is a device that can be used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Specifically, the artificial intelligence learning apparatus 400 may transmit a requirement for achieving the purpose of artificial intelligence (AI) that can be used for autonomous driving of a vehicle to the learning data generating apparatus 200 . The artificial intelligence learning apparatus 400 may receive data for artificial intelligence (AI) learning from the learning data generating apparatus 200 . In addition, the artificial intelligence learning apparatus 400 may machine-learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle by using the received artificial intelligence (AI) learning data.

As such, the artificial intelligence learning apparatus 400 may be any device as long as it is capable of transmitting and receiving data to and from the learning data generating apparatus 200 and performing an operation using the transmitted/received data. For example, the artificial intelligence learning device 400 may be any one of a desktop, a workstation, or a stationary computing device such as a server, but is not limited thereto.

As described above, the learning data collection device 100 , the learning data generating device 200 , the plurality of annotation devices 300 , and the artificial intelligence learning device 400 have a security line directly connecting the devices, a common Data may be transmitted/received using a network in which one or more of a wired communication network or a mobile communication network is combined.

For example, public wired networks include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). However, it is not limited thereto. In addition, the mobile communication network includes Code Division Multiple Access (CDMA), Wideband CDMA, WCDMA, High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile communication may be included, but are not limited thereto.

Hereinafter, an example in which the learning data collection apparatus 100 as described above collects data will be described in detail.

2 is an exemplary diagram illustrating an apparatus for collecting learning data according to an embodiment of the present invention.

As shown in FIG. 2 , the learning data collection apparatuses 100a, 100b, 100c, 100d, 100e, 100f, and 100g 100 according to an embodiment of the present invention are fixedly installed in the vehicle 10 and are used for autonomous driving. It can collect data to machine learning artificial intelligence (AI) that can be used.

For example, the learning data collection apparatus 100 may be configured to include a lidar 100a and a plurality of cameras 100b, 100c, 100d, 100e, 100f, and 100g. However, the present invention is not limited thereto, and the learning data collection apparatus 100 may include one or more radar or ultrasonic sensors. Furthermore, the learning data generating apparatus 100 may be a computing device that collects data measured by one or more lidar, camera, radar, or ultrasonic sensor.

The lidar 100a included in the learning data collection device 100 is fixedly installed on the vehicle 10 , emits a laser pulse around the vehicle 10 and is reflected by objects located around the vehicle 10 . By detecting the returned light, 3D point cloud data corresponding to a 3D image of the surroundings of the vehicle 10 may be generated. Accordingly, the 3D point cloud data obtained by the lidar 100a may include a set of points reflecting the laser pulse emitted by the lidar 100a into a three-dimensional space.

A plurality of cameras 100b, 100c, 100d, 100e, 100f, and 100g included in the learning data collection device 100 are radially fixed with the vehicle 10 as the center, and are installed around the vehicle 10. Each of the dimensional images can be taken. 2, the learning data collection apparatus 100 is illustrated as including six cameras 100b, 100c, 100d, 100e, 100f and 100g, but the present invention is applied to fewer than six or more than six cameras. It will be apparent to those of ordinary skill in the art that the present invention may be implemented based on the 2D image taken by the present invention.

Hereinafter, the configuration of the learning data generating apparatus 200 as described above will be described in more detail.

3 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.

As shown in FIG. 3 , the learning data generating device 200 includes a communication unit 205 , an input/output unit 210 , a data structure design unit 215 , a data collection and refinement unit 220 , a data processing unit 225 and It may be configured to include a learning data generator 230 .

As such, the components of the learning data generating apparatus 200 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is implemented with each other in the actual physical environment. It may be implemented separately.

When each component is described, the communication unit 205 may transmit/receive data to and from one or more of the learning data collection apparatus 100 , the annotation apparatus 300 , and the artificial intelligence learning apparatus 400 .

Specifically, the communication unit 205 may receive 2D images and 3D point cloud data from the learning data collection apparatus 100 . Here, the 2D images may be images captured by a camera fixedly installed in the vehicle in order to machine-learning artificial intelligence (AI) that can be used for autonomous driving of the vehicle. In addition, the 3D point cloud data may be point cloud data acquired through a lidar fixedly installed in a vehicle in order to machine-learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

The communication unit 205 may distribute and transmit a plurality of 2D images or 3D point cloud data to be annotated to a plurality of annotation apparatuses 200 . The communication unit 205 may receive an annotation work result from each of the plurality of annotation apparatuses 300 . Also, the communication unit 205 may receive control data of the input/output device from each of the plurality of annotation devices 300 . Here, the control data of the input/output device includes the annotation device 300 that has received one or more of the 2D images or 3D point cloud data. In the process of performing an annotation operation on the received 2D image or 3D point cloud data, the user may use the annotation device It may be data stored in time series of one or more signals input to control the input/output device constituting the 300 .

In addition, the communication unit 205 may transmit data for artificial intelligence (AI) learning to the artificial intelligence learning apparatus 300 .

With the following configuration, the input/output unit 210 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 210 may receive a control signal for designing a data structure for artificial intelligence (AI) learning from a user. The input/output unit 210 may receive input from a user, such as a quota for distributing annotation tasks to the plurality of annotation apparatuses 300 .

With the following configuration, the data structure design unit 215 may design a data structure for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Specifically, the data structure design unit 215 may design a data structure for machine learning of artificial intelligence (AI) based on the user's control input through the input/output unit 110 . For example, the data structure design unit 215 may define an ontology for artificial intelligence (AI) learning and a classification system for data for artificial intelligence (AI) learning based on the input user's control. .

With the following configuration, the data collection and refinement unit 220 collects and performs data for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle based on the data structure designed by the data structure design unit 215 . can be purified.

Specifically, the data collection and refinement unit 220 may receive 2D images and 3D point cloud data from the learning data collection apparatus 100 through the communication unit 205 . However, the present invention is not limited thereto, and the data collection and refinement unit 220 performs web crawling through the communication unit 205 to collect a 2D image, or an external institution that has 2D image and 3D point cloud data. You can also download data from your device.

The data collection and refinement unit 220 may remove duplicate or extremely similar images from among the received 2D images and 3D point cloud data. In addition, the data collection and refinement unit 220 may de-identify personal information included in the collected 2D images and 3D point cloud data.

With the following configuration, the data processing unit 225 may distribute and process the data collected and refined by the data collection and refinement unit 220 to the annotation device 300 .

Characteristically, in the data processing (ie, annotation operation) process, the data processing unit 225 according to an embodiment of the present invention receives from the computing power of the annotation device 300 corresponding to the local computing device or the quality of the network line. impact can be minimized.

A detailed configuration of the data processing unit 225 having such a characteristic will be described in more detail later with reference to FIG. 4 .

With the following configuration, the learning data generating unit 230 generates data that can be used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle based on the data processed by the data processing unit 225 . can do.

Specifically, the learning data generator 230 may generate data for artificial intelligence (AI) learning by packaging the processed data. In addition, the learning data generating unit 230 may transmit the generated artificial intelligence (AI) learning data to the artificial intelligence learning apparatus 400 through the communication unit 205 .

Hereinafter, as described above, the data processing unit 225 will be described in more detail.

4 is a logical configuration diagram of a data processing unit of an apparatus for generating learning data according to an embodiment of the present invention.

As shown in FIG. 4 , the data processing unit 225 of the learning data generating apparatus 200 includes a 2D image preprocessing module 225-1, a 3D point cloud preprocessing module 225-2, and a local capability correspondence module 225- 3), the distribution and collection module 225-4, the result rewriting module 225-5, and the same object post-processing module 225-6 may be included.

As such, the components of the data processing unit 225 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is separated from each other in the actual physical environment. and can be implemented.

Description of each component, the 2D image preprocessing module 225-1 recompresses the 2D images prior to data processing (ie, annotation operation) using the annotation device 300 to increase its capacity. A pretreatment to reduce it can be performed.

First, the 2D image pre-processing module 225 - 1 may divide each 2D image into a plurality of sectors with respect to the 2D images received through the communication unit 205 .

According to an embodiment of the present invention, the 2D image pre-processing module 225-1 may partition a region based on an edge of the 2D image. Specifically, the 2D image preprocessing module 225 - 1 may extract an edge for each 2D image using either a Laplacian of Gaussian (LoG) algorithm or a Difference of Gaussian (DoG) algorithm.

When using the LoG algorithm, the 2D image preprocessing module 225 - 1 may use a Gaussian filter to remove noise existing in the 2D image. The 2D image preprocessing module 225 - 1 may apply a Laplacian filter to the 2D image from which noise has been removed. In addition, the 2D image preprocessing module 225-1 may extract an edge by detecting zerocrossing in the image to which the Laplacian filter is applied.

When the DoG algorithm is used, the 2D image preprocessing module 225-1 generates two Gaussian masks having different variances from the 2D image. The 2D image preprocessing module 225 - 1 subtracts another mask from one generated mask. In addition, the 2D image preprocessing module 225 - 1 may extract an edge by applying the subtracted mask to the 2D image.

The 2D image preprocessing module 225 - 1 may binarize the 2D images from which edges are extracted. In addition, the 2D image preprocessing module 225 - 1 may partition a plurality of regions in the 2D image based on enclosures closed by edges in the binarized 2D image.

Alternatively, according to another embodiment of the present invention, the 2D image pre-processing module 225-1 simply grids each 2D image based on a preset grid size, thereby providing a plurality of regions within the 2D image. can also be partitioned. In order to reduce the amount of computation on 2D images to be pre-processed, the training data generating apparatus 200 may simply partition regions based on the grid size.

Next, the 2D image preprocessing module 225 - 1 may calculate a probability that an object exists in each of the plurality of partitioned regions. Specifically, the 2D image pre-processing module 225-1 sets the minimum value among the distance values of points included in the region corresponding to the plurality of regions partitioned from the 2D image in the 3D point cloud data received together with the 2D images. A probability that an object exists in the area may be calculated based on a difference between the identifiable distance value and the preset identifiable distance value.

In this case, the 3D point cloud data will be data acquired by the lidar fixedly installed in the same vehicle at the same time point as when the 2D image is captured by the camera fixedly installed in the vehicle.

Next, the 2D image preprocessing module 225 - 1 may recompress each of the 2D images centering on regions in which the calculated probability is equal to or greater than a preset threshold value.

According to an embodiment of the present invention, the 2D image pre-processing module 225-1 sets the data value of a region in which the calculated probability is equal to or less than a threshold value among the partitioned regions in the 2D image to a preset single garbage value. can be replaced with

According to another embodiment of the present invention, the 2D image pre-processing module 225-1 calculates the image compression rate of a region in which the calculated probability is less than or equal to a threshold value among the partitioned regions in the 2D image, Recompression can be performed by setting it higher than the image compression rate.

According to another embodiment of the present invention, the 2D image pre-processing module 225 - 1 sets the resolution of a region in which the calculated probability is equal to or less than the threshold value among the partitioned regions in the 2D image, and the calculated probability is equal to or greater than the threshold value. Recompression can be performed by setting it lower than the resolution of the region.

According to another embodiment of the present invention, the 2D image preprocessing module 225-1 may exclude the 2D image from the annotation operation target when the calculated probabilities for all the partitioned regions in the 2D image are less than or equal to a threshold value.

With the following configuration, the 3D point cloud preprocessing module 225 - 2 is based on the distance values of points included in the 3D point cloud data prior to data processing (ie, annotation operation) using the annotation device 300 . , it is possible to perform preprocessing to reduce the capacity of each 3D point cloud data.

According to an embodiment of the present invention, the 3D point cloud preprocessing module 225 - 2 may perform preprocessing to reduce the capacity of each 3D point cloud data according to characteristics of points included in the 3D point cloud data.

Specifically, the 3D point cloud preprocessing module 225-2 extracts points forming a crowd within a preset threshold range from among points included in each 3D point cloud data, and uses them as object candidates. can be identified. In this case, the 3D point cloud pre-processing module 225 - 2 is configured to apply to the identified object candidates based on the RGB (Red, Green, Blue) values of the regions corresponding to the object candidates in the 2D image received together with the 3D point cloud data. verification can be performed.

The 3D point cloud preprocessing module 225 - 2 may change or remove some of the points included in the 3D point cloud data based on the distance values of points constituting the identified object candidates or the size values of the identified object candidates.

In more detail, the 3D point cloud preprocessing module 225 - 2 may remove points constituting the distant object candidate from the 3D point cloud data. Here, the distant object candidate may be an object candidate having a minimum distance value equal to or greater than a preset threshold distance among distance values of points constituting the object candidate.

The 3D point cloud preprocessing module 225 - 2 may estimate a class for each of the identified object candidates. For example, the 3D point cloud preprocessing module 225 - 2 determines the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis that the object candidate occupies in the 3D point cloud data in advance. A class for each of the object candidates may be estimated by preparing a set dimension range for each class.

The 3D point cloud preprocessing module 225-2 may calculate an identifiable distance value for each of the object candidates based on the class estimated for each object candidate and the identifiable distance values preset for each class type. . The 3D point cloud preprocessing module 225-2 compares the distance value from the lidar fixed installed in the vehicle to each object candidate with the identifiable distance value calculated for each object candidate, and changes or removes it from the 3D point cloud data points to be identified.

For example, the 3D point cloud preprocessing module 225 - 2 may replace points constituting an object candidate whose distance from the lidar is equal to or greater than the calculated identifiable distance value with a single preset garbage value.

As another example, the 3D point cloud preprocessing module 225 - 2 may simply remove points constituting an object candidate whose distance from the lidar is equal to or greater than the calculated identifiable distance value from the 3D point cloud data.

In addition, the 3D point cloud preprocessing module 225-2 configures the estimated object candidate as the type of the class set in advance as irrelevant to artificial intelligence (AI) machine learning, based on the class estimated for each object candidate. Points may be identified as points to be changed or removed from the 3D point cloud data.

According to another embodiment of the present invention, the 3D point cloud preprocessing module 225-2 may perform preprocessing to reduce the capacity of each 3D point cloud data by collectively processing the same objects included in the 3D point cloud data. there is. In this case, the 3D point cloud data may be 3D point cloud data continuously acquired in time series through the lidar fixedly installed in the vehicle.

Specifically, the 3D point cloud preprocessing module 225 - 2 may estimate the class of object candidates included in the 3D point cloud data. To this end, the 3D point cloud preprocessing module 225 - 2 may extract points forming a cluster within a preset threshold range among points included in each 3D point cloud data and identify them as object candidates. In this case, the 3D point cloud preprocessing module 225 - 2 may verify the identified object candidates based on the RGB values of the regions corresponding to the object candidates in the 2D image received together with the 3D point cloud data. .

The 3D point cloud preprocessing module 225-2 compares the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis that the object candidate occupies in the 3D point cloud data, and a preset size range for each class to identify the identified object candidates. You can infer a class for each.

The 3D point cloud preprocessing module 225-2 compares the qualities in 3D point cloud data for object candidates having the same estimated class type with each other, and selects one reference object candidate from among object candidates having the same class type. .

For example, the 3D point cloud preprocessing module 225-2 may select, as a reference object candidate, an object candidate including a point having a minimum distance from a lidar fixedly installed in a vehicle from among object candidates having the same class type. there is.

As another example, the 3D point cloud preprocessing module 225 - 2 may select an object candidate having the largest number of points constituting the object candidate from among object candidates having the same class type as the reference object candidate.

Prior to selecting the reference object candidate, the 3D point cloud preprocessing module 225 - 2 may filter object candidates having heterogeneous class types by comparing dimensions of object candidates having the same class type with each other.

Also, the 3D point cloud preprocessing module 225 - 2 may remove points that do not constitute a reference object candidate among points that respectively constitute object candidates having the same class type from the 3D point cloud data.

With the following configuration, the local capability correspondence module 225 - 3 corresponds to the capability of the annotation device 300 corresponding to the local computing device, and corresponds to the 2D images and 3D point cloud data to be distributed to each annotation device 300 . can be pre-processed.

Specifically, the local capability correspondence module 225 - 3 may evaluate the capabilities of the plurality of annotation devices 300 . Here, the annotation device 300 is a local computing device to perform an annotation operation on 2D images and 3D point cloud data for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle. In addition, the local capability correspondence module 225 - 3 may reduce each capacity by recompressing 2D images or 3D point cloud data in response to the evaluated capability of the annotation apparatus 300 .

According to an embodiment of the present invention, the local capability correspondence module 225 - 3 may collect hardware information for each of the plurality of annotation devices 300 . The local capability correspondence module 225 - 3 may calculate the arithmetic capability of each of the plurality of annotation devices 300 based on the collected hardware information. In addition, the local capability correspondence module 225 - 3 may determine the recompression quality of 2D images or 3D point cloud data to be transmitted to each annotation apparatus 300 in response to the calculated computational capability.

The local capability correspondence module 225 - 3 may determine not to transmit 3D point cloud data to the annotation device for which a computational capability lower than a preset recommended computational capability is calculated, but to transmit only 2D images.

According to another embodiment of the present invention, the local capability correspondence module 225-3 calculates the time taken until a response after transmitting an Internet Control Message Protocol (ICMP) signal to each of the plurality of annotation devices 300 . can The local capability correspondence module 225 - 3 may determine the recompression quality of 2D images or 3D point cloud data to be transmitted to each annotation apparatus 300 in response to a time taken until a response.

The local capability correspondence module 225-3 transmits an ICMP PING signal to the annotation device 300, and for the annotation device 300, the number of identified hops is equal to or greater than a preset threshold number, 2D images And it may be determined to exclude from the transmission target of the 3D point cloud data.

According to another embodiment of the present invention, the local capability correspondence module 225 - 3 may collect bandwidth information of each Internet Service Provider (ISP) to which the plurality of annotation devices 300 are connected. The local capability correspondence module 225 - 3 may determine the recompression quality of 2D images or 3D point cloud data to be transmitted to each annotation device 300 in response to the collected bandwidth information.

With the following configuration, the distribution and collection module 225-4 performs 2D images preprocessed by the 2D image preprocessing module 225-1, the 3D point cloud preprocessing module 225-2, or the local capability correspondence module 225-3. And 3D point cloud data may be distributed and transmitted to the plurality of annotation devices 300 .

Thereafter, the distribution and collection module 225 - 4 may receive an annotation work result from each of the plurality of annotation devices 300 .

Characteristically, the distribution and collection module 225 - 4 may receive control data of input/output devices constituting the annotation device 300 without receiving the results of the annotation work from the plurality of annotation devices 300 . Here, the control data of the input/output device is data stored in time series of one or more signals input by the user to control the input/output device while the annotation device 300 performs an annotation operation on the 2D image or 3D point cloud data. can be

With the following configuration, when the result rewriting module 225-5 does not receive the annotation work result from the annotation device 300 and receives control data of the input/output device, a 2D image based on the received control data of the input/output device It is possible to perform annotation work on field and 3D point cloud data.

Specifically, the result rewriting module 225-5 may generate a virtual machine in response to hardware information of the annotation device 300 that has generated control data of the input/output device. The result rewriting module 225 - 5 may perform an annotation operation by processing one or more signals included in the control data received on the generated virtual machine according to the generation order.

For example, when the annotation device 300 is driven by the operating system according to the event-driven architecture, one or more signals included in the control data of the input/output device are transmitted to the operating system in response to the control of the input/output device of the annotation device 200 . It can be an event message generated by Also, the annotation device 300 may generate control data of the input/output device by duplicating a system queue in which event messages generated by the operating system are stored in a first-in, first-out structure.

In this case, the result rewriting module 225-5 may perform the annotation work by sequentially dequeueing event messages from the received control data of the input/output device one by one and processing them through a message processing loop.

With the following configuration, the same object post-processing module 225 - 6 may collectively reflect the results of the annotation work processed on the reference object candidates to the results of the annotation work on the 3D point cloud data including the same object candidates. .

Specifically, among the results of the annotation work on 3D point cloud data including object candidates having the same class type, only the annotation device 300 receiving the 3D point cloud data including the reference object candidate is the same object candidate. Annotation work would have been performed. Accordingly, the same object post-processing module 225-6 may collectively include the results of the annotation operation on the reference object candidate for the results of the annotation operation on the 3D point cloud data including object candidates having the same class type. can

In addition, the same object post-processing module 225 - 6 may assign the same tracking identifier to the objects to which the results of the annotation operation on the reference object candidates are collectively reflected. Here, the tracking ID may be a unique identifier of an object assigned to track the same object from 3D point cloud data.

Hereinafter, hardware for implementing the logical components of the apparatus 200 for generating learning data as described above will be described in more detail.

5 is a hardware configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.

As shown in FIG. 5 , the learning data generating device 200 includes a processor 250 , a memory 255 , a transceiver 260 , an input/output device 265 , and a data bus. (Bus, 270) and storage (Storage, 275) may be configured to include.

The processor 250 may implement the operations and functions of the learning data generating apparatus 200 based on an instruction according to the software 280a in which the method according to the embodiments of the present invention is implemented residing in the memory 255 . . In the memory 255, the software 280a in which the method according to the embodiments of the present invention is implemented may be loaded. The transceiver 260 may transmit/receive data to and from the learning data collection apparatus 100 , the annotation apparatus 300 , and the artificial intelligence learning apparatus 400 . The input/output device 265 may receive data necessary for the operation of the learning data generating device 200 , and output collected and preprocessed 2D images, 3D point cloud data, and annotation work results. The data bus 270 is connected to the processor 250 , the memory 255 , the transceiver 260 , the input/output device 265 , and the storage 275 . can play a role.

The storage 275 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 280a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 275 may store the software 280b in which the method according to the embodiments of the present invention is implemented. Also, the storage 275 may store information necessary for performing the method according to the embodiments of the present invention.

According to an embodiment of the present invention, the software 280a and 280b for implementing the annotation distribution processing method, resident in the memory 255 or stored in the storage 275, is used by the processor 250 for autonomous driving of the vehicle. Evaluating the capability of a plurality of annotation devices to annotate 2D images and 3D point cloud data for machine learning capable artificial intelligence (AI); After distributing the 2D images and 3D point cloud data to a plurality of annotation devices, receiving control data of the input/output device from each of the pre-processed plurality of annotation devices, and a processor 250 of the received input/output device. Based on the control data, it may be a computer program recorded in a recording medium to execute the step of performing an annotation operation on the 2D images and the 3D point cloud data.

According to another embodiment of the present invention, the software 280a, 280b for implementing the 2D image pre-processing method, resident in the memory 255 or stored in the storage 275, is the processor 250 to be used for autonomous driving of the vehicle. Receiving 2D images photographed through a camera fixed in a vehicle to machine-learning capable artificial intelligence (AI) through the transceiver, the processor 250 recompressing the 2D images to reduce the capacity; and the processor 250 distributing the reduced capacity 2D images to a plurality of annotation devices through the transceiver 260 may be a computer program recorded on the recording medium.

According to another embodiment of the present invention, software 280a, 280b for implementing a method of reducing the data load of a 3D point cloud for annotation, resident in memory 255 or stored in storage 275, is processor 250 ) receiving 3D point cloud data obtained through the lidar fixed in the vehicle through the transceiver to machine-learning artificial intelligence (AI) that can be used for autonomous driving of the vehicle, the processor 250 receiving the 3D point cloud reducing the capacity of each 3D point cloud data based on the distance value of points included in the data, and the processor 250 transmits the reduced capacity 3D point cloud data through the transceiver 260 . It may be a computer program recorded on a recording medium in order to execute the step of distributing it to the annotation device.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the 3D point cloud data batch processing method, resident in the memory 255 or stored in the storage 275, the processor 250 is autonomous of the vehicle. Receiving, through the transceiver 260, 3D point cloud data acquired through a lidar fixedly installed in a vehicle to machine-learning an artificial intelligence (AI) that can be used for driving, by a processor 250, the 3D point cloud data estimating the class of object candidates included in , the processor 250 compares the quality in the 3D point cloud data for the object candidates having the same class type, among the object candidates having the same class type Selecting one reference object candidate, and after the processor 250 removes some of the points included in the 3D point cloud data based on the selected reference object candidate, distribution to a plurality of annotation devices through the transceiver It may be a computer program recorded on a recording medium to execute the steps.

More specifically, the processor 250 may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 255 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 260 may include a baseband circuit for processing wired and wireless signals. The input/output device 265 includes an input device such as a keyboard, a mouse, and/or a joystick, and a liquid crystal display (LCD), an organic light emitting diode (OLED) and/or an input device such as a joystick. Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiment included in this specification is implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. Modules reside in memory 255 and may be executed by processor 250 . The memory 255 may be internal or external to the processor 250 , and may be coupled to the processor 250 by various well-known means.

Each component shown in FIG. 5 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present invention provides one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs ( Field Programmable Gate Arrays), a processor, a controller, a microcontroller, a microprocessor, etc. may be implemented.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored in a recording medium readable through various computer means. can be recorded. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), and a floppy disk. magneto-optical media, such as a disk, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler. Such hardware devices may be configured to operate as one or more software to perform the operations of the present invention, and vice versa.

Hereinafter, the features of the artificial intelligence learning system according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

6 and 7 are exemplary views for explaining a 2D image partitioned into a plurality of regions according to some embodiments of the present invention. 8 and 9 are exemplary views for explaining a 2D image with reduced capacity according to some embodiments of the present invention.

The learning data generating apparatus 200 according to an embodiment of the present invention may perform preprocessing for reducing the capacity of 2D images prior to data processing (ie, annotation operation) using the annotation apparatus 300 .

Specifically, the learning data generating apparatus 200 may divide each 2D image into a plurality of regions with respect to the 2D images.

As shown in FIG. 6 , the learning data generating apparatus 200 according to an embodiment of the present invention extracts an edge with respect to a 2D image, and after binarizing the 2D image from which the edge is extracted, within the 2D image A plurality of regions may be partitioned in the 2D image based on regions closed by edges.

As shown in FIG. 7 , the apparatus 200 for generating learning data according to another embodiment of the present invention simply lattices each 2D image based on a preset grid size, so that a plurality of It is also possible to partition the domains of dogs.

Next, the learning data generating apparatus 200 may calculate a probability that an object exists in each of the plurality of partitioned regions. For example, the training data generating apparatus 200 may include a minimum value and a preset value among distance values of points included in an area corresponding to a plurality of areas partitioned from the 2D image in the 3D point cloud data received together with the 2D images. Based on the difference in the identifiable distance values, the probability that the object exists in the area may be calculated. In this case, the 3D point cloud data will be data acquired by the lidar fixedly installed in the same vehicle at the same time point as when the 2D image is captured by the camera fixedly installed in the vehicle.

Next, the learning data generating apparatus 200 may recompress each of the 2D images centering on regions in which the calculated probability is equal to or greater than a preset threshold value.

As shown in FIG. 8 , the apparatus 200 for generating learning data according to an embodiment of the present invention sets the data value of a region in which the calculated probability is equal to or less than a threshold value among the partitioned regions in the 2D image to a preset single value. It can be replaced with a garbage value.

As shown in FIG. 9 , the apparatus 200 for generating learning data according to another embodiment of the present invention calculates the image compression rate of the region in which the calculated probability is less than or equal to the threshold value among the partitioned regions in the 2D image, the calculated probability is Recompression may be performed by setting the image compression rate higher than the image compression rate of the region greater than the threshold value.

In addition, the apparatus 200 for generating learning data according to another embodiment of the present invention determines the resolution of an area in which the calculated probability is equal to or less than the threshold value among the partitioned areas in the 2D image, and the calculated probability is equal to or greater than the threshold value. Recompression may be performed by setting it lower than the resolution of the region.

Accordingly, the learning data generating apparatus 200 according to the embodiments of the present invention preprocesses regions with a relatively low probability of an object in the 2D images to reduce the capacity, thereby minimizing the influence on the annotation work result, , it is possible to reduce the impact of the bandwidth and response speed of the network line that transmits and receives data to and from the local computing device at the same time.

10 is an exemplary diagram for describing identified object candidates according to an embodiment of the present invention. 11 is an exemplary diagram for explaining a process of estimating a class of an object candidate according to an embodiment of the present invention. And, FIG. 12 is an exemplary diagram for explaining 3D point cloud data with reduced capacity according to an embodiment of the present invention.

The training data generating apparatus 200 according to an embodiment of the present invention may perform preprocessing for reducing the capacity of 3D point cloud data prior to data processing (ie, annotation operation) using the annotation apparatus 300 .

Specifically, as shown in FIG. 10 , the learning data generating apparatus 200 extracts points forming a cluster within a preset threshold range from among the points included in each 3D point cloud data to be object candidates. (object candidates) can be identified. In this case, the training data generating apparatus 200 may verify the identified object candidates based on the RGB values of the regions corresponding to the object candidates in the 2D image received together with the 3D point cloud data.

As shown in FIG. 11 , the learning data generating apparatus 200 occupies the 3D point cloud data in the X-axis width w, the Y-axis height h, and the Z-axis depth for each of the identified object candidates. In comparison with (d) and a preset dimension range for each class, a class for each of the object candidates may be estimated.

The learning data generating apparatus 200 may calculate an identifiable distance value for each of the object candidates based on the class estimated for each object candidate and the identifiable distance values preset for each class type.

As shown in FIG. 12 , the learning data generating apparatus 200 determines in advance the points constituting the object candidates with respect to the object candidates whose distance value from the lidar fixedly installed in the vehicle 10 is equal to or greater than the identifiable distance value. It can be replaced with a set garbage value or removed from the 3D point cloud data.

Accordingly, the learning data generating apparatus 200 according to the embodiments of the present invention preprocesses point clouds of low quality to the extent that an object cannot be identified in the 3D point cloud data and reduces the capacity, thereby reducing the impact on the annotation operation result. It is possible to minimize the impact on the bandwidth and response speed of the network line that transmits and receives data to and from the local computing device at the same time.

13 is an exemplary diagram for explaining a process of selecting a reference object candidate according to an embodiment of the present invention. And, FIG. 14 is an exemplary diagram for explaining a process of collectively applying a work result to a reference object candidate according to an embodiment of the present invention.

The training data generating apparatus 200 according to an embodiment of the present invention may perform preprocessing for reducing the capacity of each 3D point cloud data by collectively processing the same objects included in the 3D point cloud data. In this case, the 3D point cloud data may be 3D point cloud data continuously acquired in time series through the lidar fixedly installed in the vehicle.

Specifically, as shown in FIG. 13 , the training data generating apparatus 200 may estimate the classes of object candidates included in the 3D point cloud data. To this end, the learning data generating apparatus 200 may extract points forming a cluster within a preset threshold range among points included in each 3D point cloud data and identify them as object candidates. The learning data generating apparatus 200 compares the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis that the object candidate occupies in the 3D point cloud data, and a preset size range for each class, to each of the identified object candidates. class can be estimated.

The learning data generating apparatus 200 may select one reference object candidate from among object candidates having the same class type by comparing the qualities in the 3D point cloud data of object candidates having the same estimated class type. Prior to selecting a reference object candidate, the learning data generating apparatus 200 may filter object candidates having different class types by comparing dimensions of object candidates having the same class type with each other.

For example, the learning data generating apparatus 200 may select an object candidate including a point having a minimum distance from a lidar fixedly installed in a vehicle from among object candidates having the same class type as the reference object candidate.

As another example, the learning data generating apparatus 200 may select an object candidate having the largest number of points constituting the object candidate from among object candidates having the same class type as the reference object candidate.

Also, the learning data generating apparatus 200 may remove points that do not constitute a reference object candidate among points that respectively constitute object candidates having the same class type from the 3D point cloud data.

As shown in FIG. 14 , after receiving the annotation work results from the plurality of annotation apparatuses 300 , the training data generating apparatus 200 annotates 3D point cloud data including object candidates having the same class type. With respect to the work result, it is possible to collectively include the result of the annotation work on the reference object candidate.

In addition, the learning data generating apparatus 200 may give the same tracking ID to the objects to which the results of the annotation operation for the reference object candidates are collectively reflected.

Accordingly, the learning data generating apparatus 200 according to embodiments of the present invention selects only one object having a relatively highest quality from among point clouds for the same object in 3D point cloud data and performs data processing on the remaining 3D point cloud data. By collectively applying the point cloud data, it is possible to reduce the capacity of 3D point cloud data to be annotated and to improve the quality of the result of the annotation work.

15 is an exemplary diagram for explaining a process of evaluating the capability of a local device according to an embodiment of the present invention. And, FIG. 16 is an exemplary diagram for explaining a process of rewriting an annotation work result according to an embodiment of the present invention.

The training data generating apparatus 200 according to an embodiment of the present invention corresponds to the capability of the annotation device 300 corresponding to the local computing device, and 2D images and 3D point cloud data to be distributed to each annotation device 300 . can be pre-processed.

15 , the learning data generating apparatus 200 may evaluate the capabilities of the plurality of annotation apparatuses 300 . Specifically, the learning data generating apparatus 200 may collect hardware information for each of the plurality of annotation apparatuses 300 . The learning data generating apparatus 200 may calculate the arithmetic capability of each of the plurality of annotation apparatuses 300 based on the collected hardware information.

The training data generating apparatus 200 may calculate a time taken until a response after transmitting the ICMP signal to each of the plurality of annotation apparatuses 300 . The learning data generating apparatus 200 may collect bandwidth information of each ISP to which the plurality of annotation apparatuses 300 are connected. The training data generating apparatus 200 may calculate the networking capability of each of the plurality of annotation apparatuses 300 based on the time taken from transmission of the calculated ICMP signal to the response and bandwidth information of the ISP.

In addition, the training data generating apparatus 200 may reduce the capacity of each of the 2D images or 3D point cloud data by recompressing the 2D images or 3D point cloud data in response to the evaluated computational capability and networking capability of the annotation apparatus 300 .

Meanwhile, as shown in FIG. 16 , the learning data generating apparatus 200 may receive control data of the input/output device without receiving an annotation work result from the annotation apparatus 300 .

In this case, the learning data generating apparatus 200 may generate a virtual machine in response to hardware information of the annotation apparatus 300 that has generated control data of the input/output device. In addition, the training data generating apparatus 200 may perform an annotation operation by processing one or more signals included in the control data received on the generated virtual machine according to the generation order.

For example, when the annotation device 300 is driven by the operating system according to the event-driven architecture, one or more signals included in the control data of the input/output device are transmitted to the operating system in response to the control of the input/output device of the annotation device 200 . It can be an event message generated by Also, the annotation device 300 may generate control data of the input/output device by duplicating a system queue in which event messages generated by the operating system are stored in a first-in, first-out structure.

In addition, the learning data generating apparatus 200 may perform an annotation operation by sequentially taking out event messages one by one from the received control data of the input/output device and processing them through a message processing loop.

Hereinafter, the operation of the learning data generating apparatus 200 according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

17 is a flowchart illustrating a method of generating learning data according to an embodiment of the present invention.

Referring to FIG. 17 , the learning data generating apparatus 200 according to an embodiment of the present invention may design a data structure for machine learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle (S100). . For example, the learning data generating apparatus 200 may define an ontology for artificial intelligence (AI) learning and a classification system for data for artificial intelligence (AI) learning.

The learning data generating apparatus 200 may receive 2D images and 3D point cloud data from the learning data collecting apparatus 100 based on the designed data structure ( S200 ). However, the present invention is not limited thereto, and the learning data generating apparatus 200 may perform web crawling to collect 2D images, or may download data from an external device having 2D image and 3D point cloud data.

The training data generating apparatus 200 removes duplicate or extremely similar images from among the received 2D images and 3D point cloud data, and de-identifies personal information included in the 2D images and 3D point cloud data to perform data purification. can be performed (S300).

The learning data generating apparatus 200 may process the data by distributing the collected and refined data to the annotation apparatus 300 ( S400 ). Such a data processing step will be described later with reference to FIG. 18 .

The learning data generating apparatus 200 may package the processed data to generate data that can be used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle (S500).

And, the learning data generating apparatus 200 may transmit the generated artificial intelligence (AI) learning data to the artificial intelligence learning apparatus 400 (S600).

18 is a flowchart illustrating a data processing method according to an embodiment of the present invention.

Referring to FIG. 18 , the learning data generating apparatus 200 according to an embodiment of the present invention may perform preprocessing to reduce the capacity of 2D images and 3D point cloud data ( S410 ).

Specifically, the learning data generating apparatus 200 divides each 2D image into a plurality of regions, calculates a probability that an object exists in the region for each of the plurality of partitioned regions, and the calculated probability is a preset threshold value Each of the 2D images may be recompressed based on the abnormal regions.

The learning data generating apparatus 200 extracts points forming a cluster within a preset threshold range among points included in each 3D point cloud data, identifies them as object candidates, and the distances between points constituting the identified object candidates. Based on the value or the size value of the identified object candidates, some of the points included in the 3D point cloud data may be changed or removed.

In addition, the learning data generating apparatus 200 estimates the class of object candidates included in each 3D point cloud data, and compares the quality in the 3D point cloud data of the object candidates having the same estimated class type with each other to classify the class. One reference object candidate may be selected from among object candidates having the same type of , and points that do not constitute the reference object candidate among points constituting object candidates having the same class type may be removed from the 3D point cloud data.

Next, the learning data generating apparatus 200 may evaluate the capabilities of the plurality of annotation apparatuses 300 corresponding to the local computing apparatus ( S420 ).

In detail, the learning data generating apparatus 200 may collect hardware information on each of the plurality of annotation apparatuses 300 , and may calculate the arithmetic capability of each of the plurality of annotation apparatuses 300 based on this.

The training data generating apparatus 200 calculates a time taken until a response after transmitting the ICMP signal to each of the plurality of annotation apparatuses 300 , and transmits bandwidth information of each ISP to which the plurality of annotation apparatuses 300 are connected. collected, and based on this, a networking capability for each of the plurality of annotation devices 300 may be calculated.

In addition, the training data generating apparatus 200 may reduce the capacity of each of the 2D images or 3D point cloud data by recompressing the 2D images or 3D point cloud data in response to the evaluated computational capability and networking capability of the annotation apparatus 300 .

Next, the training data generating apparatus 200 may distribute and transmit the preprocessed 2D images and 3D point cloud data to the plurality of annotation apparatuses 300 ( S430 ).

Thereafter, the training data generating apparatus 200 may receive an annotation work result from each of the plurality of annotation apparatuses 300 . Characteristically, the training data generating apparatus 200 may receive control data of the input/output devices constituting the annotation apparatus 300 without receiving the results of the annotation work from the plurality of annotation apparatuses 300 ( S440 ).

Finally, the training data generating apparatus 200 may perform post-processing on the received data or rewrite the annotation work result (S450).

Specifically, when the control data of the input/output device is received without receiving the annotation work result from the annotation device 300 , the learning data generating device 200 generates one or more signals included in the control data on the virtual machine according to the generation order. It can be processed to perform annotation operations.

Also, the learning data generating apparatus 200 may collectively include the results of the annotation operation on the reference object candidate with respect to the results of the annotation operation on the 3D point cloud data including object candidates having the same class type.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention pertains that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein. It is obvious to those with ordinary knowledge. In addition, although specific terms have been used in the present specification and drawings, these are only used in a general sense to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Training data collection device: 100 Training data generation device: 200
Annotation device: 300 AI learning device: 400
Communication unit: 205 Input/output unit: 210
Data structure design department: 215 Data collection and refinement department: 220
Data processing unit: 225 Learning data generation unit: 230
2D image preprocessing module: 225-1 3D point cloud preprocessing module: 225-2
Local Capability Response Module: 225-3 Distribution and Collection Module: 225-4
Output Rewrite Module: 225-5 Same Object Post-Processing Module: 225-6

Claims (10)

Receiving, by the learning data generating apparatus, 3D point group data obtained through lidar in order to machine learning artificial intelligence (AI);
reducing, by the learning data generating apparatus, a capacity of each 3D point cloud data based on a distance value of points included in the 3D point cloud data; and
distributing, by the learning data generating device, the 3D point cloud data with reduced capacity to a plurality of annotation devices;
The step of reducing the capacity is
Including; identifying object candidates by extracting points forming a crowd within a preset threshold range among points included in each 3D point cloud data;
Changing or removing, by the learning data generating apparatus, some of the points included in the 3D point cloud data based on the distance values of points constituting the identified object candidates or the size values of the identified object candidates and
Receiving the 3D point cloud data includes:
Receive more 2D images taken through a camera fixedly installed in the vehicle,
The step of identifying the object candidates comprises:
Based on the RGB (Red, Green, Blue) values of the regions corresponding to the object candidates in the received 2D image, it is characterized in that verification of the identified object candidates is performed,
The method further includes, by the learning data generating device, performing pre-processing on 3D point cloud data to be distributed to each annotation device in response to the capabilities of the plurality of annotation devices;
Performing the pre-processing step,
Evaluating, by the learning data generating apparatus, the capability of the plurality of annotation apparatuses to perform an annotation operation on the 3D point cloud data;
Performing the pre-processing step,
In response to the evaluated capability, characterized in that the 3D point cloud data is recompressed to reduce the capacity of each,
Assessing the ability
Collecting hardware information for each of the plurality of annotation devices, calculating a computational capability for each of the plurality of annotation devices based on the collected hardware information, and transmitting the calculated computational capability to each annotation device It is characterized in that the recompression quality of the 3D point cloud data is determined,
The distributing step is
Reducing the data load of 3D point cloud for annotation, characterized in that it is determined to transmit only the 2D image without transmitting the 3D point cloud data to the annotation device for which the calculation capability lower than the preset recommended computing power is calculated method.
The method of claim 1, wherein changing or removing some of the points comprises:
Remove the points constituting the distant object candidate from the 3D point cloud data,
The distant object candidate is
A method of reducing the data load of a 3D point cloud for annotation, characterized in that the minimum distance value among the distance values of points constituting the object candidate is an object candidate having a preset threshold distance or more.
The method of claim 1, wherein changing or removing some of the points comprises:
Estimate a class for each of the identified object candidates, calculate an identifiable distance value for each of the object candidates based on identifiable distance values determined in advance for each type of class, and calculate each object from the lidar A method of reducing a data load of a 3D point cloud for annotation, characterized in that the points to be changed or removed are identified by comparing the distance value to the candidate and the calculated identifiable distance value.
4. The method of claim 3, wherein changing or removing some of the points comprises:
By comparing the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis of the object candidate, and the preset size range for each class, the class for each of the object candidates A method of reducing the data load of a 3D point cloud for annotation, comprising estimating.
4. The method of claim 3, wherein changing or removing some of the points comprises:
Reducing the data load of 3D point cloud for annotation, characterized in that the points constituting the object candidate whose distance value from the lidar is greater than or equal to the identifiable distance value are replaced with a single preset garbage value method.
4. The method of claim 3, wherein changing or removing some of the points comprises:
A method of reducing a data load of a 3D point cloud for annotation, characterized in that points constituting an object candidate whose distance value from the lidar is equal to or greater than the identifiable distance value is removed from the 3D point cloud data.
4. The method of claim 3, wherein changing or removing some of the points comprises:
3D point cloud data for annotation, characterized in that points constituting an object candidate estimated as a type of a class set in advance as irrelevant to machine learning of the artificial intelligence (AI) are identified as the points to be changed or removed How to reduce the load.
memory;
transceiver; and
Combined with a computing device configured to include a processor (processor) for processing instructions resident in the memory,
receiving, by the processor, 3D point cloud data obtained through lidar through the transceiver to machine-learning artificial intelligence (AI);
reducing, by the processor, a capacity of each 3D point cloud data based on a distance value of points included in the 3D point cloud data; and
The processor is for executing the step of distributing the reduced capacity 3D point cloud data to a plurality of annotation devices through the transceiver,
The step of reducing the capacity is
Identifying object candidates by extracting points forming a crowd within a preset threshold range from among points included in each 3D point cloud data;
changing or removing, by the processor, some of the points included in the 3D point cloud data based on a distance value of points constituting the identified object candidates or a size value of the identified object candidates;
Receiving the 3D point cloud data includes:
Receive more 2D images taken through a camera fixedly installed in the vehicle,
The step of identifying the object candidates comprises:
Based on the RGB (Red, Green, Blue) values of the regions corresponding to the object candidates in the received 2D image, it is characterized in that verification of the identified object candidates is performed,
Further comprising; performing, by the processor, pre-processing on 3D point cloud data to be distributed to each annotation device in response to the capabilities of the plurality of annotation devices;
Performing the pre-processing step,
evaluating, by the processor, capabilities of the plurality of annotation devices to perform an annotation operation on the 3D point cloud data;
Performing the pre-processing step,
In response to the evaluated capability, characterized in that the 3D point cloud data is recompressed to reduce the capacity of each,
Assessing the ability
Collecting hardware information for each of the plurality of annotation devices, calculating a computational capability for each of the plurality of annotation devices based on the collected hardware information, and transmitting the calculated computational capability to each annotation device It is characterized in that the recompression quality of the 3D point cloud data is determined,
The distributing step is
A computer program recorded on a recording medium, characterized in that it is determined to transmit only the 2D image without transmitting the 3D point cloud data with respect to the annotation device for which the calculation power lower than the preset recommended computing power is calculated.
The method of claim 8, wherein changing or removing some of the points comprises:
Estimate a class for each of the identified object candidates, calculate an identifiable distance value for each of the object candidates based on identifiable distance values determined in advance for each type of class, and calculate each object from the lidar A computer program recorded on a recording medium, characterized in that the points to be changed or removed are identified by comparing the distance value to the candidate and the calculated identifiable distance value.
10. The method of claim 9, wherein changing or removing some of the points comprises:
By comparing the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis of the object candidate, and the preset size range for each class, the class for each of the object candidates A computer program recorded on a recording medium, characterized in that for estimating.

Images