KR102310604B1 - Method for processing data collected by multiple sensors, and computer program recorded on record-medium for executing method therefor - Google Patents

Abstract

The present invention proposes a method capable of processing data collected by multiple sensors for collecting data for machine learning of artificial intelligence (AI). The method includes: receiving, by a learning data collection device, sensing data obtained by a radar fixedly installed in a vehicle and a plurality of pieces of 3D point group data obtained by a lidar fixedly installed in the vehicle; identifying, by the learning data collection device, an object from the received sensing data; and filtering, by the learning data collection device, the pieces of 3D point cloud data obtained by the lidar in response to a distance between the vehicle and the object. According to the present invention, a plurality of pieces of data collected by the multiple sensors of the vehicle are filtered to perform machine learning on the artificial intelligence (AI) used for autonomous driving, so that data having relatively low importance among the data for the machine learning of the artificial intelligence (AI) is reduced.

Description

Method for processing data collected by multiple sensors, and computer program recorded on record-medium for executing method therefor

The present invention relates to the processing of data for artificial intelligence (AI) learning. More specifically, it relates to a method capable of processing data collected by multiple sensors for collecting data for machine learning of artificial intelligence (AI), and a computer program recorded on a recording medium for executing the method.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning ability, reasoning ability, and perception ability using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the form of data for learning.

In general, the design of data for artificial intelligence (AI) learning proceeds in the stages of data structure design, data collection, data purification, data processing, data expansion, and data verification.

To describe each step in more detail, the design of the data structure is made through the definition of an ontology, a definition of a classification system, and the like. The collection of data is made by collecting data through direct shooting, web crawling, or association/professional organizations. Data purification is performed by removing duplicate data from the collected data and de-identifying personal information. Data processing is performed by performing annotations and inputting metadata. Data expansion is performed by performing ontology mapping and supplementing or extending the ontology as needed. And, the verification of the data is performed by verifying the validity according to the set target quality using various verification tools.

Meanwhile, automatic driving of a vehicle refers to a system capable of driving by determining the vehicle itself. Such autonomous driving may be divided into gradual stages from non-automation to full automation according to the degree to which the system is involved in driving and the degree to which the driver controls the vehicle. In general, the stages of autonomous driving are divided into six levels classified by the Society of Automotive Engineers (SAE) International. According to the six stages classified by the International Association of Automobile Engineers, level 0 is non-automated, level 1 is driver assistance, level 2 is partial automation, level 3 is conditional automation, level 4 is highly automated, and level 5 is The steps are fully automated steps.

Autonomous driving of a vehicle is performed through mechanisms of perception, localization, path planning, and control. Currently, several companies are developing to implement cognitive and path planning among autonomous driving mechanisms using artificial intelligence (AI).

Data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of these vehicles is collected by various types of sensors installed in the vehicle. For example, data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle is a lidar, a camera, a radar, and an ultrasonic sensor fixed in the vehicle. ) may be data acquired, photographed or sensed by, but is not limited thereto.

However, an object to be learned by artificial intelligence (AI) is not included in all data continuously acquired, photographed, or measured by various sensors installed in the vehicle. That is, among all the data continuously acquired, photographed, or measured by various sensors, a lot of data meaningless for machine learning of artificial intelligence (AI) that can be used for autonomous driving are included.

In addition, various sensors for acquiring, photographing, or measuring data used for machine learning of artificial intelligence (AI) cannot be physically installed at the same point, but are inevitably installed separately from multiple points. Accordingly, data acquired, photographed, or measured by various sensors inevitably have differences depending on the location in which each sensor is installed.

Korean Patent Application Laid-Open No. 10-2020-0042629, ‘Method for generating touch-based annotations and images of mobile devices for artificial intelligence learning, and device therefor’, (published on April 24, 2020)

One object of the present invention is to provide a method capable of processing data collected by multiple sensors for collecting data for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Another object of the present invention is to provide a computer program recorded on a recording medium for executing a method capable of processing data collected by multiple sensors for collecting data for machine learning of artificial intelligence (AI).

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical task as described above, the present invention proposes a method capable of processing data collected by multiple sensors for collecting data for machine learning of artificial intelligence (AI). In the method, the learning data collection device receives sensing data obtained by a radar fixedly installed in the vehicle and a plurality of 3D point group data obtained by a lidar fixedly installed in the vehicle. to do; identifying, by the learning data collection device, an object from the received sensed data; and filtering, by the learning data collection apparatus, a plurality of 3D point cloud data acquired by the lidar in response to the distance between the vehicle and the object. In this case, the sensed data is information on points at which the electromagnetic wave signal emitted by the radar toward the driving direction of the vehicle is reflected, and the 3D point cloud data is radiated around the vehicle by the lidar. It can be 3D information about the points that reflected the laser pulse.

Specifically, the step of identifying the object may identify the object by extracting points forming a crowd within a preset threshold range among points included in the sensed data.

In addition, in the step of identifying the object, a region to which the laser pulse of the LIDAR can reach is selected from among the coordinates of points included in the 3D point cloud data, with respect to the region where the identified object is located. It can be divided into a region of interest and a region of uninterested.

The filtering of the 3D point cloud data includes identifying a first sampling rate corresponding to the distance between the vehicle and the object from a preset sampling table, and according to the identified first sampling rate, the plurality of By sampling the 3D point cloud data, the number of 3D point cloud data per unit time may be reduced.

Furthermore, the filtering of the 3D point cloud data may include identifying a second sampling rate corresponding to the distance between the vehicle and the object from the sampling table, and selecting the plurality of 3D point cloud data according to the identified second sampling rate. By sampling, the number of points constituting each 3D point cloud data can be reduced.

Meanwhile, the receiving of the data may further receive 2D images photographed by a plurality of cameras fixedly installed in the vehicle.

The filtering of the 3D point cloud data may include selecting 2D images photographed by a camera photographing the non-interest region among the received 2D images with a lower resolution than 2D images photographed by a camera photographing the region of interest ( resolution) can be recompressed.

In the filtering of the 3D point cloud data, the number of 2D images per unit time may be reduced by sampling the 2D images captured by the camera capturing the non-interested region.

In addition, the filtering of the 3D point cloud data may extend the size of chroma subsampling for compressing color information of 2D images captured by the camera capturing the uninterested region.

Meanwhile, the receiving of the data may further receive distance information measured by a plurality of ultrasonic sensors fixedly installed in the vehicle.

Also, in the filtering of the 3D point cloud data, when the distance between the vehicle and the object is greater than a preset specific possible range, distance information detected by the ultrasonic sensor may be discarded.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute a method capable of processing data collected by multiple sensors. The computer program includes a memory; transceiver; and a processor for processing instructions resident in the memory. And, the computer program is the processor is a plurality of three-dimensional point group (3D points group) data acquired by the detection data acquired by the radar (radar) fixedly installed in the vehicle and the lidar (lidar) fixedly installed in the vehicle receiving through the transceiver, the processor identifying an object from the received sensed data; and a computer program recorded on a recording medium to execute, by the processor, filtering the plurality of 3D point cloud data obtained by the lidar in response to the distance between the vehicle and the object. have.

Specific details of other embodiments are included in the detailed description and drawings.

According to an embodiment of the present invention, by filtering data collected by multiple sensors of a vehicle to machine-learning artificial intelligence (AI) that can be used for autonomous driving, relative among data for machine learning of artificial intelligence (AI) This can reduce low-importance data. In this way, by reducing data of relatively low importance, it is possible to lower the burden of data processing for machine learning of artificial intelligence (AI).

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.
2 is an exemplary diagram for explaining multiple sensors according to an embodiment of the present invention.
3 is a logical configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.
4 is a hardware configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.
5 to 7 are exemplary diagrams for explaining a process of controlling multiple sensors according to an embodiment of the present invention.
8 and 9 are exemplary views for explaining a process of post-processing collected data according to an embodiment of the present invention.
10 to 12 are exemplary views for explaining a process of correcting errors of multiple sensors according to an embodiment of the present invention.
13 is a flowchart illustrating a data collection method according to an embodiment of the present invention.

It should be noted that technical terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in this specification should be interpreted in the meaning generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise defined in this specification, and excessively inclusive. It should not be construed in the meaning of a human being or in an excessively reduced meaning. In addition, when the technical terms used in the present specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be understood by being replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “consisting of” or “having” should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are included. It should be construed that it may not, or may further include additional components or steps.

Also, terms including ordinal numbers such as first, second, etc. used herein may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but another component may exist in between. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents, or substitutes other than the accompanying drawings.

As described above, an object to be learned by artificial intelligence (AI) is not included in all data continuously acquired, photographed, or measured by various sensors installed in the vehicle. That is, among all the data continuously acquired, photographed, or measured by various sensors, a lot of data meaningless for machine learning of artificial intelligence (AI) that can be used for autonomous driving are included.

In addition, various sensors for acquiring, photographing, or measuring data used for machine learning of artificial intelligence (AI) cannot be physically installed at the same point, but are inevitably installed separately from multiple points. Accordingly, data acquired, photographed, or measured by various sensors inevitably have differences depending on the location in which each sensor is installed.

In order to solve this difficulty, the present invention can effectively control multiple sensors that collect data for machine learning of artificial intelligence (AI), reduce meaningless data, and minimize the error between data collected by multiple sensors. We would like to propose a variety of means.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.

As shown in FIG. 1 , the artificial intelligence learning system according to an embodiment of the present invention includes a learning data collection device 100 , a learning data generating device 200 , a plurality of annotation devices 300 , and an artificial intelligence learning device ( 400) may be included.

As such, since the components of the AI learning system according to an embodiment are merely functionally distinct elements, two or more components are integrated with each other in the actual physical environment, or one component is the actual physical environment. may be implemented separately from each other.

When describing each component, the learning data collection device 100 is a device that can be used to collect data for machine learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

That is, the learning data collection apparatus 100 may acquire, photograph, or detect data by controlling multiple sensors. In addition, the learning data collection apparatus 100 may transmit the acquired, photographed, or sensed data to the learning data generating apparatus 200 so that the acquired, photographed, or sensed data may be processed.

In this case, the multi-sensor to be controlled by the learning data collection apparatus 100 may include one or more of a lidar, a camera, a radar, and an ultrasonic sensor. It is not limited.

Such multiple sensors as a control target of the learning data collection apparatus 100 will be described in more detail later with reference to FIG. 2 .

Characteristically, the learning data collection apparatus 100 according to embodiments of the present invention effectively controls multiple sensors for collecting data, reduces meaningless data, and reduces errors between data collected by multiple sensors. It has features that can be minimized.

Such features of the learning data collection apparatus 100 will be described in more detail later with reference to FIGS. 3 and 4 .

With the following configuration, the learning data generating device 200 is a device that can be used to design and process data for machine learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Specifically, the learning data generating apparatus 200 may receive the properties of the project related to artificial intelligence (AI) learning from the artificial intelligence learning apparatus 400 . The learning data generating device 200 is based on the user's control and the properties of the received project, the design of the data structure for artificial intelligence (AI) learning, the purification of the collected data, the processing of the data, the expansion of the data and the verification can be performed.

First, the learning data generating apparatus 200 may design a data structure for artificial intelligence (AI) learning. For example, the learning data generating apparatus 200 is based on the user's control and the properties of the received project, an ontology for artificial intelligence (AI) learning, a classification system of data for artificial intelligence (AI) learning can be defined.

The learning data generating apparatus 200 may collect data for artificial intelligence (AI) learning based on the designed data structure. To this end, the learning data generating apparatus 200 may receive sensing data, 3D point cloud data, 2D images, and sensing data from the learning data collecting apparatus 100 . However, the present invention is not limited thereto, and the learning data generating apparatus 200 may perform web crawling or may download data from an external device.

The training data generating apparatus 200 may remove duplicate or extremely similar data from among the collected sensing data, 3D point cloud data, 2D images, and sensing data. The learning data generating apparatus 200 may de-identify personal information included in the collected sensed data, 3D point cloud data, and 2D images.

The training data generating apparatus 200 may distribute and transmit the collected and refined sensing data, 3D point cloud data, 2D images, and sensing data to the plurality of annotation apparatuses 300 . In this case, the learning data generating apparatus 200 generates sensed data, 3D point cloud data, 2D images, and sensed data in response to a pre-allocated amount for an operator (ie, a labeler) of the annotation apparatus 300 . can be distributed

The training data generating apparatus 200 may receive an annotation work result from each annotation apparatus 300 . The training data generating apparatus 200 may generate data for artificial intelligence (AI) learning by packaging the received annotation work result. In addition, the learning data generating apparatus 200 may transmit the generated artificial intelligence (AI) learning data to the artificial intelligence learning apparatus 400 .

The learning data generating apparatus 200 having such a characteristic transmits and receives data to and from the learning data collection apparatus 100 , the annotation apparatus 300 , and the artificial intelligence learning apparatus 400 , and performs an operation based on the transmitted and received data Any device that can do it is allowed. For example, the learning data generating apparatus 200 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

With the following configuration, the annotation device 300 is a device that can be used to annotate the sensed data, 3D point cloud data, 2D images, and sensed data distributed by the training data generating device 200 . As such, all or a part of the annotation device 300 may be a device in which an annotator performs an annotation operation through a clouding service.

Specifically, the annotation apparatus 300 includes one sensed data to be annotated from among the sensed data, 3D point cloud data, 2D images, and sensed data received from the learning data generating device 200, 3D point cloud data, It can be output to a 2D image or sensed data display.

The annotation device 300 may select a tool according to a signal input from a user through the input/output device. Here, the tool is a tool for setting a bounding box specifying one or more objects included in the sensed data, 3D point cloud data, 2D image, or sensed data.

The annotation device 300 may receive coordinates according to the selected tool through the input/output device. In addition, the annotation apparatus 300 may set a bounding box based on the input coordinates to specify the object included in the sensed data, 3D point cloud data, 2D image, or sensed data.

Here, the bounding box is an area for specifying an object to be subjected to artificial intelligence (AI) learning among objects included in sensed data, 3D point cloud data, 2D image, or sensed data. As such, the bounding box may have a shape of a rectangle or a cube, but is not limited thereto.

For example, the annotation device 300 receives two coordinates through the input/output device, and based on the input two coordinates, the coordinates of the upper left vertex in the sensed data, 3D point cloud data, 2D image, or sensing data and An object can be specified by setting a bounding box based on a rectangle having the coordinates of the lower right vertex. In this case, the two coordinates may be set by the user inputting one type of input signal twice (eg, clicking the mouse), or setting the user by inputting two types of input signals once (eg, dragging the mouse). However, the present invention is not limited thereto.

The annotation device 300 generates, according to a signal input from the user through the input/output device, sensing data, 3D point cloud data, 2D image, sensing data, or metadata for a set object to be annotated. can Here, the metadata is sensing data, 3D point cloud data, 2D image, sensing data, or data for describing an object. As such, the metadata includes the category of a specified object, the rate at which the object is cut by the angle of view, the rate at which the object is obscured by other objects or objects, the tracking ID of the object, the time the image was taken, the weather conditions on the day the image was captured. These may include, but are not limited to, file size, image size, copyright holder, resolution, bit value, aperture transmission, exposure time, ISO sensitivity, focal length, aperture opening value, angle of view, white balance, RGB depth, class name , tags, shooting locations, types of roads, road surface information, or traffic jam information may be further included.

The annotation apparatus 300 may generate an annotation work result based on the specified object and the generated metadata. In this case, the annotation operation result may have a JSON (Java Script Object Notation) file format, but is not limited thereto. The annotation apparatus 300 may transmit the generated annotation work result to the training data generating apparatus 200 .

Any device that can transmit and receive data to and from the training data generating device 200 and perform an operation based on the transmitted/received data is acceptable for the annotation device 300 having such a characteristic. For example, the annotation device 100 may be a fixed computing device such as a desktop, a workstation, or a server, or a smart phone, a laptop, a tablet, or a tablet computer. It may be any one of a mobile computing device such as a phablet, a portable multimedia player (PMP), a personal digital assistant (PDA), or an e-book reader (E-book reader).

In the following configuration, the artificial intelligence learning device 400 is a device that can be used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Specifically, the artificial intelligence learning apparatus 400 may transmit a requirement for achieving the purpose of artificial intelligence (AI) that can be used for autonomous driving of a vehicle to the learning data generating apparatus 200 . The artificial intelligence learning apparatus 400 may receive data for artificial intelligence (AI) learning from the learning data generating apparatus 200 . And, the artificial intelligence learning apparatus 400 may machine-learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle by using the received artificial intelligence (AI) learning data.

As such, the artificial intelligence learning device 400 may be any device as long as it is capable of transmitting and receiving data to and from the learning data generating device 200 and performing an operation using the transmitted/received data. For example, the artificial intelligence learning apparatus 400 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

As described above, the learning data collection device 100 , the learning data generating device 200 , the plurality of annotation devices 300 , and the artificial intelligence learning device 400 have a security line directly connecting the devices, a common Data may be transmitted/received using a network in which one or more of a wired communication network or a mobile communication network is combined.

For example, public wired networks include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). However, it is not limited thereto. In addition, the mobile communication network includes Code Division Multiple Access (CDMA), Wideband CDMA, WCDMA, High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile communication may be included, but are not limited thereto.

Hereinafter, a multi-sensor that is a control target of the learning data collection apparatus 100 as described above will be described in more detail.

2 is an exemplary diagram for explaining multiple sensors according to an embodiment of the present invention.

As shown in FIG. 2 , the learning data collection apparatus 100 according to an embodiment of the present invention includes a radar 20, a lidar 30, a camera 40, and an ultrasonic sensor ( 50) to acquire, photograph, or detect data for machine learning of artificial intelligence (AI).

Here, the vehicle 10 is a vehicle in which a radar 20, a lidar 30, a camera 40, and an ultrasonic sensor 50 are installed for collecting basic data for machine learning artificial intelligence (AI). It can be distinguished from a vehicle that performs autonomous driving by intelligence (AI).

The radar 20 is fixedly installed in the vehicle 10 , and emits an electromagnetic wave toward the driving direction of the vehicle 10 , and is reflected back by an object positioned in front of the vehicle 10 . , the vehicle 10 may generate sensing data corresponding to an image of the front.

In other words, the sensed data is information on points at which electromagnetic waves emitted toward the driving direction of the vehicle by the radar 20 fixedly installed in the vehicle 10 are reflected. Accordingly, the coordinates of the points included in the sensed data may have values corresponding to the position and shape of the object positioned in front of the vehicle 10 . The sensed data may be two-dimensional information, but is not limited thereto and may be three-dimensional information.

The lidar 30 is fixedly installed in the vehicle 10, emits a laser pulse around the vehicle 10, and detects the light reflected by an object located around the vehicle 10 and returned. , 3D point cloud data corresponding to a 3D image of the surroundings of the vehicle 10 may be generated.

In other words, the 3D point cloud data is three-dimensional information about points that reflect a laser pulse radiated around the vehicle by the lidar 30 fixedly installed in the vehicle 10 . Accordingly, coordinates of points included in the 3D point cloud data may have values corresponding to the position and formation of an object located around the vehicle 10 .

The camera 40 may be fixedly installed on the vehicle 10 to capture a two-dimensional image of the surroundings of the vehicle 10 . As such, a plurality of cameras 40 may be configured according to the angle of view. For example, FIG. 2 shows an example in which six cameras 40 are installed in the vehicle 10 , but the present invention shows that the number of cameras 40 that can be installed in the vehicle 10 may be various. It will be apparent to those of ordinary skill in the art to which this belongs.

In other words, the 2D image is an image captured by the camera 40 fixedly installed in the vehicle 10 . Accordingly, the 2D image may include color information of an object positioned in the direction the camera 40 faces.

The ultrasonic sensor 50 is fixedly installed in the vehicle 50 , emits ultrasonic waves around the vehicle 10 , and detects a sound wave reflected by an object positioned adjacent to the vehicle 10 , and the vehicle Distance information corresponding to the distance between the ultrasonic sensor 50 installed in 10 and the object may be generated. In general, the ultrasonic sensor 50 is composed of a plurality and may be fixedly installed in front, rear, front and rear sides of the vehicle 10 that is easy to come into contact with an object.

In other words, the distance information is information about a distance from an object sensed by the ultrasonic sensor 50 fixedly installed in the vehicle 10 .

Hereinafter, the configuration of the learning data collection apparatus 100 as described above will be described in more detail.

3 is a logical configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.

As shown in FIG. 3 , the learning data collection apparatus 100 includes a communication unit 105 , an input/output unit 110 , a multi-sensor control unit 115 , a data post-processing unit 120 , an error correction unit 125 , and a data control unit. It may be configured to include a study 130 .

As such, the components of the learning data collection device 100 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is implemented with each other in the actual physical environment. It may be implemented separately.

When each component is described, the communication unit 105 may transmit/receive data to/from the multiple sensors and the learning data generating apparatus 200 .

Specifically, the communication unit 105 transmits detection data, 3D point cloud data, 2D image, and distance information from the radar 20 , the lidar 30 , the camera 40 and the ultrasonic sensor 50 fixedly installed in the vehicle 10 . can receive

To this end, the learning data collection apparatus 100 and the multiple sensors may be directly connected to each other by a data transmission/reception cable, but is not limited thereto.

In addition, the communication unit 105 may transmit, to the learning data generating apparatus 200 , the sensed data, 3D point cloud data, 2D image, and distance information on which post-processing and error correction have been performed, under the control of the data providing unit 130 . .

With the following configuration, the input/output unit 110 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 110 may receive a threshold range input from the user. Here, the critical range is a size range for determining a crowd of points that can be identified as an object among points included in the sensed data or 3D point cloud data, A value determined according to the type may be input.

The input/output unit 110 may receive an input of a lidar recognition range from a user. Here, the lidar recognition range is a range in which the laser pulse emitted from the lidar 30 arrives to recognize an object, and a value determined according to the output or type of the lidar 30 may be input.

The input/output unit 110 may receive a specific possible range input from the user. Here, the specific possible range is a range in which an object can be specified from the 2D image taken by the camera 40, the type of object to be subjected to machine learning of artificial intelligence (AI), and the minimum number of pixels A value determined according to, etc. may be input.

The input/output unit 110 may receive a touchable range input from the user. Here, the contactable range is a range in which the vehicle 10 may have a possibility of coming into contact with another object when it behaves within a common-sense range, and a value determined according to the goal of machine learning of artificial intelligence (AI) may be input. have.

The input/output unit 110 may receive a sampling table input from a user. Here, the sampling table is a table in which reference information for sampling only some data from the 3D point cloud data obtained by the lidar and the distance between the vehicle 10 and the object are mapped to each other. The first sampling rate included in the sampling data is information about the number of 3D point cloud data to be sampled per unit time, and the second sampling rate is information about the number of points constituting one 3D point cloud data.

With the following configuration, the multi-sensor controller 115 may control the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 fixedly installed in the vehicle 10 .

Specifically, the multi-sensor control unit 115 may receive one or more of sensing data, 3D point cloud data, 2D image, and distance information through the communication unit 105 . For example, the multi-sensor control unit 115 may receive sensing data through the communication unit 105 , and may additionally receive one or more of 3D point cloud data, 2D image, and distance information according to circumstances.

Here, the sensed data is information on points at which electromagnetic waves emitted toward the driving direction of the vehicle by the radar 20 fixedly installed in the vehicle 10 are reflected. The 3D point cloud data is three-dimensional information about points that reflect a laser pulse radiated around the vehicle by the lidar 30 fixedly installed in the vehicle 10 . The 2D image is an image taken by the camera 40 fixedly installed in the vehicle 10 . And, the distance information is information about a distance from an object sensed by the ultrasonic sensor 50 fixedly installed in the vehicle 10 .

The multi-sensor controller 115 may identify an object from the received sensing data. For example, the multi-sensor control unit 125 extracts points forming a cluster within a threshold range preset by the input/output unit 110 from among the points included in the sensed data, thereby identifying an object from the sensed data. have.

The multi-sensor control unit 115 corresponds to the distance between the vehicle 10 and the object identified from the sensed data, and is fixedly installed in the vehicle 10 and includes a lidar 30 to emit laser pulses, a camera to take a 2D image ( 40) and one or more of the ultrasonic sensor 50 to emit ultrasonic waves.

In more detail, when no object is identified from the sensed data, the multi-sensor controller 115 may control the lidar 30 not to interfere with the laser pulse.

When the distance between the vehicle 10 and the object is greater than the previously set LiDAR recognition range through the input/output unit 110, the multi-sensor controller 115 determines that the LiDAR 30 does not emit a laser pulse, or It can be controlled to emit a laser pulse only with the intensity set in .

When the distance between the vehicle 10 and the object is greater than the lidar recognition range, the multi-sensor controller 115 does not capture the 2D image by the plurality of cameras 40 fixed to the vehicle 10, and the ultrasonic sensor 50 ) can be controlled not to emit ultrasonic waves.

When the distance between the vehicle 10 and the object is within the lidar recognition range, the multi-sensor controller 115 lengthens or shortens the laser pulse emission period of the lidar 30 in proportion to the distance from the object. can be controlled

When the distance between the vehicle 10 and the object is within a specific possible range preset through the input/output unit 110 , the multi-sensor control unit 115 is configured to display a plurality of cameras 40 fixedly installed in the vehicle 10 as a 2D image. can be controlled to shoot. In this case, the specific possible range may correspond to a narrower range than the lidar recognition range.

The multi-sensor controller 115 may set different resolutions of the 2D images to be captured by the plurality of cameras 40 in response to the distance between the vehicle 10 and the object. For example, the multi-sensor controller 115 may set the resolution of the 2D image to be captured by the plurality of cameras 40 to increase as the distance between the vehicle 10 and the object increases.

Also, the multi-sensor controller 115 may estimate the movement path of the object based on time-series change in the position of the object including the sensed data. The multi-sensor controller 115 may identify a camera that captures a direction corresponding to the estimated movement path of the object from among the plurality of cameras 40 . In addition, the multi-sensor controller 115 may set a photographing cycle of the camera for photographing a direction corresponding to the movement path of the object to be shorter than that of the camera for photographing another direction.

In addition, when the distance between the vehicle 10 and the object is within the contactable range set in advance through the input/output unit 110 , the multi-sensor control unit 115 includes a plurality of ultrasonic sensors 50 fixedly installed in the vehicle 10 . They can be controlled to emit ultrasonic waves. In this case, the contactable range may correspond to a narrower range than the specific possible range.

With the following configuration, the data post-processing unit 120 targets the detection data, 3D point cloud data, 2D image, and distance information collected from the radar 20 , the lidar 30 , the camera 40 and the ultrasonic sensor 50 . Post-processing can be performed.

Specifically, the data post-processing unit 120 may receive one or more of sensed data, 3D point cloud data, 2D images, and distance information through the communication unit 105 . For example, the data post-processing unit 120 may receive sensed data and 3D point cloud data through the communication unit 105 , and may additionally receive one or more of 2D images and distance information according to circumstances.

The data post-processing unit 120 may identify an object from the received sensed data. For example, the data post-processing unit 125 may identify an object from the sensed data by extracting points forming a cluster within a threshold range preset by the input/output unit 110 from among the points included in the sensed data. have.

In addition, the data post-processing unit 120 selects a region to which the laser pulse of the LiDAR 30 can reach with the center of the region where the identified object is located among the coordinates included in the 3D point cloud data as a region of interest. It can be divided into a region of interest and a region of uninterested.

The data post-processing unit 120 corresponds to the distance between the vehicle 10 and the object identified from the sensed data, a plurality of 3D point cloud data acquired by the lidar 30 , and a plurality of images captured by the camera 40 . At least one of the 2D images of , and distance information sensed by the ultrasonic sensor 50 may be filtered.

In more detail, the data post-processing unit 120 may identify a first sampling rate corresponding to the distance between the vehicle 10 and the object from the sampling table preset by the input/output unit 110 . . The data post-processing unit 120 may reduce the number of 3D point cloud data per unit time by sampling a plurality of 3D point cloud data according to the identified first sampling rate.

Also, the data post-processing unit 120 may identify the second sampling rate corresponding to the distance between the vehicle 10 and the object from the sampling table. The data post-processing unit 120 may reduce the number of points constituting each 3D point cloud data by sampling a plurality of 3D point cloud data according to the identified second sampling rate.

On the other hand, the data post-processing unit 120 selects 2D images photographed by the camera 40 for photographing the non-interest region from among the plurality of 2D images, rather than 2D images photographed by the camera 40 for photographing the region of interest. It can be recompressed at a lower resolution.

The data post-processing unit 120 may reduce the number of 2D images per unit time by sampling the 2D images captured by the camera 40 which has captured the non-interested region among the plurality of 2D images.

The data post-processing unit 120 expands the size of chroma subsampling for compressing color information of the 2D images photographed by the camera 40 photographing the non-interested region among the plurality of 2D images, and then re- can be compressed.

In addition, when the distance between the vehicle 10 and the object is greater than a specific possible range preset through the input/output unit 110 , the data post-processing unit 120 discards the distance information detected by the ultrasonic sensor 50 . may be

With the following configuration, the error correcting unit 125 may correct an error according to the installation positions of the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 fixedly installed in the vehicle 10 . .

Specifically, the error correcting unit 125 may receive one or more of sensed data, 3D point cloud data, 2D images, and distance information through the communication unit 105 . For example, the data post-processing unit 120 may receive sensed data and 3D point cloud data through the communication unit 105 , and may additionally receive one or more of 2D images and distance information according to circumstances.

The error correcting unit 125 may identify a first point at which the radar 20 fixedly installed in the vehicle 10 emits an electromagnetic wave signal. That is, the first point means a point at which the radar 20 is fixedly installed in the vehicle 10 . If a plurality of radars 20 are installed in the vehicle 10 , the number of first points may also be a plurality.

The error correcting unit 125 may identify a second point at which the lidar 30 fixedly installed in the vehicle 10 emits a laser pulse. That is, the second point means a point where the lidar 30 is fixedly installed in the vehicle 10 . If a plurality of lidars 30 are installed in the vehicle 10 , the number of second points may also be a plurality.

As such, the first point and the second point may be preset in advance according to the vehicle model, size, and learning goal of artificial intelligence (AI) of the vehicle 10 in which the radar 20 and the lidar 30 are fixedly installed. , but not limited thereto.

The error correcting unit 125 may correct the coordinates of the points included in the sensed data and the coordinates of the points included in the 3D point cloud data so that the identified first point and the second point are recognized as the same location.

In more detail, the error correcting unit 125 may identify the object from the sensed data. For example, the error correcting unit 125 may identify an object from the sensed data by extracting points forming a cluster within a threshold range preset by the input/output unit 110 from among the points included in the sensed data. have.

The error correction unit 125 corrects the coordinates of the points included in the sensed data and the coordinates of the points included in the 3D point cloud data only when the object is identified from the sensed data, and detects when no object is identified from the sensed data. The coordinates of the points included in the data and the coordinates of the points included in the 3D point cloud data can be maintained as they are.

Meanwhile, the error correcting unit 125 sets a virtual standard point having three-dimensional coordinates, and then coordinates the points included in the sensed data so that the first point and the second point are recognized as being located at the reference point. and coordinates of points included in the 3D point cloud data can be corrected.

For example, the error correcting unit 125 sets a center point of the vehicle 10 as a reference point, and then a point included in the sensed data so that the first point and the second point are recognized as being located at the reference point. The coordinates of the points and the coordinates of the points included in the 3D point cloud data can be corrected.

On the other hand, the sensing data acquired by the radar 20 and the 3D point cloud data acquired by the lidar 30 are composed of coordinates of points reflecting electromagnetic waves or laser pulses, , the first point and the second point) can be matched. However, since the 2D image captured by the camera 40 is composed of color information, the viewpoint cannot be matched. Accordingly, the error correction unit 125 provides a formula capable of correcting the first point of the radar 20 and the second point of the lidar 30 as a point at which the camera 40 captures the 2D image. can do.

In more detail, the error correcting unit 125 may calculate 3D vector values between the respective photographing points where the plurality of cameras 40 are installed and the reference point, respectively. The error correction unit 125 may correct the sensed data or 3D point cloud data so that the first point or the second point may be recognized as being located at each imaging point based on the calculated 3D vector values. can be derived.

In addition, the error correcting unit 125 is 3D irrespective of whether the lidar 30 fixedly installed in the vehicle 10 corresponds to a static lidar or a rotational lidar. It can be calibrated to handle point cloud data.

In more detail, the error correcting unit 125 includes a plurality of lidar devices when the lidar 30 fixedly installed in the vehicle 10 is configured with a plurality of fixed lidar devices installed to be spaced apart from each other at different points in the vehicle 10 . By combining the coordinates of the points included in the 3D point cloud data respectively acquired by the device, one 3D point cloud data having a spherical shape obtained by a rotating lidar that rotates around a reference point and emits a laser pulse is generated. can do.

On the other hand, the error correcting unit 125 is configured to use a rotating lidar device in which the lidar 30 fixedly installed in the vehicle 10 rotates around the second point and emits a laser pulse. The coordinates of points included in one 3D point cloud data obtained by can create

Meanwhile, the error correcting unit 125 may correct distance values included in the distance information so that each of the detection points where the plurality of ultrasonic sensors 50 are installed is recognized as being located at a reference point.

With the following configuration, the data providing unit 130 may provide basic data that can be used for machine learning of artificial intelligence (AI) to the learning data generating apparatus 200 .

Specifically, the data providing unit 130 provides detection data, 3D point cloud data, and data through the radar 20 , the lidar 30 , the camera 40 and the ultrasonic sensor 50 controlled by the multi-sensor controller 115 , After the 2D image and distance information are collected, and the collected sensing data, 3D point cloud data, 2D image and distance information are post-processed by the data post-processing unit 120, post-processed sensing data, 3D point cloud data, 2D image and When the distance information is corrected by the error correcting unit 125 , the corrected sensing data, 3D point cloud data, 2D image, and distance information may be transmitted to the learning data generating apparatus 200 through the communication unit 105 .

Hereinafter, hardware for implementing the logical components of the learning data collection apparatus 100 as described above will be described in more detail.

4 is a hardware configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.

As shown in FIG. 4 , the learning data collection device 100 includes a processor 150 , a memory 155 , a transceiver 160 , an input/output device 165 , and a data bus. (Bus, 170) and may be configured to include a storage (Storage, 175).

The processor 150 may implement the operations and functions of the learning data collection apparatus 100 based on an instruction according to the software 180a in which the method according to the embodiments of the present invention is implemented residing in the memory 155 . . The memory 155 may be loaded with software 180a in which methods according to embodiments of the present invention are implemented. The transceiver 160 may transmit/receive data to and from the radar 20 , the lidar 30 , the camera 40 , the ultrasonic sensor 50 , and the learning data generating apparatus 200 . The input/output device 165 may receive data necessary for the operation of the learning data collection device 100 , and may output the collected sensed data, 3D point cloud data, 2D image, and distance information. The data bus 170 is connected to the processor 150 , the memory 155 , the transceiver 160 , the input/output device 165 and the storage 175 . can play a role.

The storage 175 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 180a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 175 may store the software 180b in which the method according to the embodiments of the present invention is implemented. In addition, the storage 175 may store information necessary for performing the method according to the embodiments of the present invention.

According to an embodiment of the present invention, the software 180a and 180b for implementing the control method of multiple sensors resident in the memory 155 or stored in the storage 175 is the processor 150 fixed to the vehicle 10 . Receiving detection data from the installed radar 20 through the transceiver 160, the processor 150 identifying the object from the received detection data, and the processor 150 the distance from the identified object Correspondingly, it may be a computer program recorded on a recording medium to execute the step of controlling the lidar 30 to be fixedly installed in the vehicle to emit laser pulses.

According to another embodiment of the present invention, the software (180a, 180b) for implementing the data processing method resident in the memory 155 or stored in the storage 175 is a radar processor 150 is fixedly installed in the vehicle 10. Receiving the sensed data acquired by (20) and a plurality of three-dimensional point cloud data acquired by the lidar (30) fixedly installed in the vehicle (10) through the transceiver (160), the processor (150) identifying an object from the received sensed data, and a plurality of 3D point cloud data obtained by the lidar 30 by the processor 150 corresponding to the distance between the vehicle 10 and the object It may be a computer program recorded on a recording medium in order to execute the filtering step.

According to another embodiment of the present invention, the software 180a and 180b for implementing the error correction method resident in the memory 155 or stored in the storage 175 is the processor 150 is fixedly installed in the vehicle 10 . Receiving the sensed data acquired by the radar 20 and the 3D point cloud data acquired by the lidar 30 fixedly installed in the vehicle 10 through the transceiver 160, the processor 150 The radar 20 identifies a first point at which the electromagnetic wave signal is emitted, the lidar 30 identifies a second point at which the laser pulse is emitted, and the processor 150 determines the first point and the It may be a computer program recorded on a recording medium to execute the step of correcting the coordinates of the points included in the sensed data and the coordinates of the points included in the 3D point cloud data so that the second point is recognized as the same position. have.

More specifically, the processor 150 may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 155 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 160 may include a baseband circuit for processing wired and wireless signals. The input/output device 165 includes an input device such as a keyboard, a mouse, and/or a joystick, and a liquid crystal display (LCD), an organic light emitting diode (OLED) and/or an input device such as a joystick. Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiment included in this specification is implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. The module resides in the memory 155 and may be executed by the processor 150 . The memory 155 may be internal or external to the processor 150 , and may be connected to the processor 150 by various well-known means.

Each component shown in FIG. 4 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present invention is one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs ( Field Programmable Gate Arrays), a processor, a controller, a microcontroller, a microprocessor, and the like.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored in a recording medium readable through various computer means. can be recorded. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), and a floppy disk. magneto-optical media, such as a disk, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software to perform the operations of the present invention, and vice versa.

Hereinafter, the features of the artificial intelligence learning system according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

5 to 7 are exemplary views for explaining a process of controlling multiple sensors according to an embodiment of the present invention.

5 to 7 , the learning data collection apparatus 100 according to an embodiment of the present invention includes a radar 20, a lidar 30, a camera 40 and The ultrasonic sensor 50 may be controlled.

Specifically, the learning data collection apparatus 100 receives at least one of detection data, 3D point cloud data, 2D image, and distance information from the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 . can do.

Here, the sensed data is information on points at which electromagnetic waves emitted toward the driving direction of the vehicle by the radar 20 fixedly installed in the vehicle 10 are reflected. The 3D point cloud data is three-dimensional information about points that reflect a laser pulse radiated around the vehicle by the lidar 30 fixedly installed in the vehicle 10 . The 2D image is an image taken by the camera 40 fixedly installed in the vehicle 10 . And, the distance information is information about a distance from an object sensed by the ultrasonic sensor 50 fixedly installed in the vehicle 10 .

The learning data collection apparatus 100 may identify the object 60 from the received sensed data. For example, the learning data collection apparatus 100 may identify the object 60 from the sensed data by extracting points forming a cluster within a preset threshold range among points included in the sensed data.

The learning data collection device 100 corresponds to the distance d between the vehicle 10 and the object 60, is fixedly installed in the vehicle 10, and the lidar 30 to emit a laser pulse, to take a 2D image One or more of the camera 40 and the ultrasonic sensor 50 to emit ultrasonic waves may be controlled.

In more detail, when the distance d between the vehicle 10 and the object 60 is greater than the lidar recognition range 31, the learning data collection apparatus 100 determines that the lidar 30 does not emit a laser pulse. Or, it can be controlled to emit a laser pulse only with a preset intensity. In addition, when the distance d between the vehicle 10 and the object 60 is greater than the lidar recognition range 31 , the learning data collection device 100 does not capture the 2D image by the plurality of cameras 40 and , it is possible to control the ultrasonic sensor 50 not to emit ultrasonic waves.

When the distance d between the vehicle 10 and the object 60 is within the lidar recognition range 31 , the learning data collection device 100 is a lidar in proportion to the distance d from the object 60 . The laser pulse emission period of (30) can be lengthened or controlled to be short.

When the distance d between the vehicle 10 and the object 60 is within a specific possible range 41, the learning data collection apparatus 100 includes a plurality of cameras 40 fixedly installed in the vehicle 10 as a 2D image. can be controlled to shoot. In this case, the specific possible range 41 may correspond to a narrower range than the lidar recognition range 31 .

Also, the learning data collection apparatus 100 may set different resolutions of the 2D images to be captured by the plurality of cameras 40 in response to the distance d between the vehicle 10 and the object 60 . The learning data collection apparatus 100 estimates a movement path of the object 60 based on a time-series position change of the object 60 including the sensed data, and takes a direction corresponding to the estimated movement path of the object 60 . A camera may be identified, and a shooting cycle of a camera that takes a picture in a direction corresponding to the movement path of the object 60 may be set shorter than that of a camera that takes pictures of another direction.

When the distance d between the vehicle 10 and the object 60 is within the contactable range 51 , the learning data collection apparatus 100 includes a plurality of ultrasonic sensors 50 fixedly installed in the vehicle 10 . can be controlled to fire. In this case, the contactable range 51 may correspond to a narrower range than the specific possible range 41 .

Therefore, the learning data collection apparatus 100 according to an embodiment of the present invention controls the data collection period of multiple sensors or the quality of data to be collected, and thus has relatively low importance among the data for machine learning of artificial intelligence (AI). data can be reduced.

8 and 9 are exemplary views for explaining a process of post-processing collected data according to an embodiment of the present invention.

As shown in FIGS. 8 and 9 , the learning data collection apparatus 100 according to an embodiment of the present invention includes a radar 20 , a lidar 30 , a camera 40 and Post-processing may be performed on the sensing data, 3D point cloud data, 2D image, and distance information collected from the ultrasonic sensor 50 .

In detail, the learning data collection apparatus 100 includes the detection data, 3D point cloud data, and 2D images from the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 . and distance information.

The learning data collection apparatus 100 may identify the object 60 from the received sensed data. For example, the learning data collection apparatus 100 may identify the object 60 from the sensed data by extracting points forming a cluster within a preset threshold range among points included in the sensed data.

The learning data collection apparatus 100 selects an area to which the laser pulse of the LiDAR 30 can reach with the center of the area where the identified object 60 is located among the coordinates included in the 3D point cloud data as the area of interest ( ROI) and non-interest regions.

The learning data collection device 100 corresponds to the distance between the vehicle 10 and the object 60 , a plurality of 3D point cloud data acquired by the lidar 30 , and a plurality of images photographed by the camera 40 . At least one of 2D images and distance information sensed by the ultrasonic sensor 50 may be filtered.

In more detail, the learning data collection apparatus 100 identifies a first sampling rate corresponding to the distance between the vehicle 10 and the object 60 from the sampling table, and according to the identified first sampling rate, a plurality of 3D By sampling the point cloud data, the number of 3D point cloud data per unit time may be reduced.

In addition, the learning data collection apparatus 100 identifies a second sampling rate corresponding to the distance between the vehicle 10 and the object 60 from the sampling table, and according to the identified second sampling rate, a plurality of 3D point cloud data By sampling the points, the number of points constituting each 3D point cloud data can be reduced.

On the other hand, the learning data collection apparatus 100 selects a region of interest (ROI) from the 2D images photographed by the camera 40 photographing the non-interested region among the plurality of 2D images photographed by the plurality of cameras 40 . It can be recompressed to a lower resolution than the 2D images captured by the captured camera 40 .

The learning data collection apparatus 100 may reduce the number of 2D images per unit time by sampling the 2D images captured by the camera 40 which has captured the non-interested region among the plurality of 2D images.

The learning data collection apparatus 100 may expand the size of the chroma subsampling for compressing the color information of the 2D images photographed by the camera 40 photographing the non-interested region among the plurality of 2D images and then recompress them. have.

Also, when the distance between the vehicle 10 and the object 60 is greater than a specific possible range, the learning data collection apparatus 100 may discard distance information detected by the ultrasonic sensor 50 .

Therefore, the learning data collection apparatus 100 according to an embodiment of the present invention filters data collected by multiple sensors, thereby reducing data of relatively low importance among data for machine learning of artificial intelligence (AI). can As a result, by reducing the amount of data of relatively low importance, it is possible to lower the burden of data processing for artificial intelligence (AI) machine learning.

10 to 12 are exemplary views for explaining a process of correcting errors of multiple sensors according to an embodiment of the present invention.

10 to 12 , the learning data collection apparatus 100 according to an embodiment of the present invention includes a radar 20 , a lidar 30 , a camera 40 and An error according to an installation position of the ultrasonic sensor 50 may be corrected.

In detail, the learning data collection apparatus 100 includes the detection data, 3D point cloud data, and 2D images from the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 . and distance information.

The learning data collection apparatus 100 may identify the first point 25 at which the radar 20 fixedly installed in the vehicle 10 emits an electromagnetic wave signal. That is, the first point 25 means a point at which the radar 20 is fixedly installed in the vehicle 10 . If a plurality of radars 20 are installed in the vehicle 10 , the number of the first points 25 may also be a plurality.

The learning data collection apparatus 100 may identify the second point 35 at which the lidar 30 fixedly installed in the vehicle 10 emits a laser pulse. That is, the second point 35 means a point at which the lidar 30 is fixedly installed in the vehicle 10 . If a plurality of lidars 30 are installed in the vehicle 10 , the number of second points 35 may also be a plurality.

The learning data collection apparatus 100 is configured to recognize the coordinates of the points included in the sensed data and the points included in the 3D point cloud data so that the identified first point 25 and the second point 35 can be recognized as the same location. Coordinates can be corrected. In more detail, the learning data collection apparatus 100 sets a virtual standard point having three-dimensional coordinates, and then recognizes that the first point 25 and the second point 35 are located at the reference point. As much as possible, the coordinates of the points included in the sensed data and the coordinates of the points included in the 3D point cloud data may be corrected.

On the other hand, the learning data collection apparatus 100 calculates three-dimensional vector values between each of the photographing points 45 and the reference point where the plurality of cameras 40 are installed, and based on the calculated three-dimensional vector values, the second A formula for correcting the sensed data or 3D point cloud data may be derived so that the first point 25 or the second point 35 can be recognized as being located at each of the photographing points 45 .

On the other hand, in the case where the learning data collection device 100 includes a plurality of fixed lidar devices installed at different points in the vehicle 10 , the lidar 30 fixed to the vehicle 10 is configured to be spaced apart from each other, the plurality of lidar devices By combining the coordinates of the points included in the 3D point cloud data respectively obtained by can

On the contrary, the learning data collection apparatus 100 is a rotation type lidar device when the lidar 30 fixedly installed in the vehicle 10 is configured as a rotation type lidar device that rotates around a second point and emits laser pulses. The coordinates of the points included in one 3D point cloud data obtained by can also create

Therefore, the learning data collection apparatus 100 according to an embodiment of the present invention can directly apply the position of an object recognized by a specific sensor to data acquired from another sensor by correcting an error according to the installation position of multiple sensors. there will be As a result, it is possible to integrate and manage the results of the annotation work on the data respectively acquired by the multiple sensors.

Hereinafter, the operation of the learning data collection apparatus 100 according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

13 is a flowchart illustrating a data collection method according to an embodiment of the present invention.

Referring to FIG. 13 , the learning data collection apparatus 100 according to an embodiment of the present invention includes sensing data and 3D point cloud data from a radar 20 , a lidar 30 , a camera 40 , and an ultrasonic sensor 50 . , one or more of the 2D image and distance information may be received (S100).

The learning data collection device 100 identifies an object from the received sensing data, and is fixedly installed in the vehicle 10 in response to a distance between the identified object and the vehicle 10 to emit a laser pulse 30 . ), one or more of the camera 40 for capturing a 2D image and the ultrasonic sensor 50 for emitting ultrasonic waves can be controlled (S200).

A detailed description of a process in which the learning data collection apparatus 100 controls the lidar 30 , the camera 40 and the ultrasonic sensor 50 is the same as that described with reference to FIGS. 5 to 7 , so it will not be repeated. .

The learning data collection apparatus 100 performs post-processing on the detection data, 3D point cloud data, 2D image, and distance information received from the radar 20 , the lidar 30 , the camera 40 and the ultrasonic sensor 50 . can be performed (S300).

A detailed description of a process in which the learning data collection apparatus 100 performs post-processing of the sensed data, the 3D point cloud data, the 2D image, and the distance information is the same as that described with reference to FIGS. 8 and 9 , and thus will not be repeated.

The learning data collection apparatus 100 may correct an error according to the installation positions of the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 ( S400 ).

The learning data collection device 100 detects errors in detection data, 3D point cloud data, 2D image, and distance information according to the installation positions of the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 . A detailed description of the correction process is the same as that described with reference to FIGS. 10 to 12 , and thus will not be repeated.

The learning data collecting apparatus 100 may transmit the corrected sensing data, 3D point cloud data, 2D image, and distance information to the learning data generating apparatus 200 ( S500 ).

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention pertains that other modifications based on the technical idea of the present invention can be implemented in addition to the embodiments disclosed herein. It is obvious to those with ordinary knowledge. In addition, although specific terms have been used in the present specification and drawings, these are only used in a general sense to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Vehicle: 10 Radar: 20
Lidar: 30 Camera: 40
Ultrasonic sensor: 50
Training data collection device: 100 Training data generation device: 200
Annotation device: 300 AI learning device: 400
Communication unit: 105 Input/output unit: 110
Multi-sensor controller: 115 Data post-processor: 120
Error correction unit: 125 Data providing unit: 130

Claims (10)

A method of collecting data for machine learning of artificial intelligence (AI) that can be used for autonomous driving, the method comprising:
Receiving, by the learning data collection device, sensing data acquired by a radar fixedly installed in the vehicle and a plurality of 3D point group data acquired by a lidar fixedly installed in the vehicle ;
identifying, by the learning data collection device, an object from the received sensed data; and
Comprising the step of filtering, by the learning data collection device, a plurality of 3D point cloud data acquired by the lidar in response to the distance between the vehicle and the object,
The sensed data is information on points at which the electromagnetic wave signal emitted by the radar toward the driving direction of the vehicle is reflected,
The 3D point cloud data is three-dimensional information about points reflecting a laser pulse emitted by the lidar around the vehicle,
The step of identifying the object
Identify the object by extracting points forming a cluster within a preset threshold range from among the points included in the sensed data,
The step of filtering the 3D point cloud data is
Identifies a first sampling rate corresponding to the distance between the vehicle and the object from a preset sampling table, and samples the plurality of 3D point cloud data according to the identified first sampling rate in 3D per unit time Reduce the number of point cloud data,
The step of filtering the 3D point cloud data is
Identifying a second sampling rate corresponding to the distance between the vehicle and the object from the sampling table, and sampling the plurality of 3D point cloud data according to the identified second sampling rate to configure each 3D point cloud data A method of processing collected data, characterized in that the number of points is reduced. According to claim 1, wherein the step of identifying the object
Among the coordinates of the points included in the 3D point cloud data, a region to which the laser pulse of the LIDAR can reach is defined as a region of interest and a region of non-interest based on the region where the identified object is located. of uninterested), characterized in that, the processing method of the collected data. The method of claim 1, wherein receiving the data comprises:
The method of processing collected data, characterized in that it further receives distance information measured by a plurality of ultrasonic sensors fixedly installed in the vehicle. The method of claim 3, wherein the filtering of the 3D point cloud data comprises:
When the distance between the vehicle and the object is greater than a preset specific possible range, distance information sensed by the ultrasonic sensor is discarded. memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
The processor receives, through the transceiver, sensing data obtained by a radar fixedly installed in the vehicle and a plurality of 3D point group data obtained by a lidar fixedly installed in the vehicle. to do;
identifying, by the processor, an object from the received sensed data; and
Execute, by the processor, filtering a plurality of 3D point cloud data obtained by the lidar in response to the distance between the vehicle and the object,
The sensed data is information on points at which the electromagnetic wave signal emitted by the radar toward the driving direction of the vehicle is reflected,
The 3D point cloud data is three-dimensional information about points reflecting a laser pulse emitted by the lidar around the vehicle,
The step of identifying the object
Identify the object by extracting points forming a cluster within a preset threshold range from among the points included in the sensed data,
The step of filtering the 3D point cloud data is
Identifies a first sampling rate corresponding to the distance between the vehicle and the object from a preset sampling table, and samples the plurality of 3D point cloud data according to the identified first sampling rate in 3D per unit time Reduce the number of point cloud data,
The step of filtering the 3D point cloud data is
Identifies a second sampling rate corresponding to the distance between the vehicle and the object from the sampling table, and samples the plurality of 3D point cloud data according to the identified second sampling rate to construct each 3D point cloud data A computer program recorded on a recording medium, characterized in that the number of dots is reduced. The method of claim 5, wherein identifying the object comprises:
Among the coordinates of the points included in the 3D point cloud data, a region to which the laser pulse of the LIDAR can reach is defined as a region of interest and a region of non-interest based on the region where the identified object is located. of uninterested), a computer program recorded on a recording medium, characterized in that. 6. The method of claim 5, wherein receiving the data comprises:
A computer program recorded on a recording medium, characterized in that it further receives distance information measured by a plurality of ultrasonic sensors fixedly installed in the vehicle. The method of claim 7, wherein the filtering of the 3D point cloud data comprises:
When the distance between the vehicle and the object is greater than a preset specific possible range, the distance information sensed by the ultrasonic sensor is discarded, the computer program recorded on the recording medium.
delete delete

Images