KR102356910B1 - Method for preemptively detecting object, and computer program recorded on record-medium for executing method therefor - Google Patents

Abstract

The present invention proposes a method for preemptively detecting an object from data for machine learning of artificial intelligence (AI) that can be used for the autonomous driving of a vehicle. The method comprises: a step in which a learning data collection device collects 3D point group data and 2D images simultaneously acquired and photographed by a lidar and a camera installed in a vehicle; a step in which the learning data collection device sets an object region in which an object is predicted to be present in the 2D image based on the three-dimensional coordinates of points included in the 3D point group data; and a step in which the learning data collection device transmits the 2D image and information about the set object region to a learning data generation device. In this case, the object region may include a preset number of vertices and edges connecting the vertices by corresponding to the computing power of the learning data collection device. Accordingly, it is possible to reduce the amount of work of annotation.

Description

Method for preemptively detecting object, and computer program recorded on record-medium for executing method therefor

The present invention relates to the collection of data for artificial intelligence (AI) machine learning. More particularly, it relates to a method for preemptively detecting an object from data for machine learning of artificial intelligence (AI), and a computer program recorded on a recording medium for executing the method.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning ability, reasoning ability, and perception ability using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the type of data for learning.

In general, the design of data for artificial intelligence (AI) learning proceeds in the stages of data structure design, data collection, data purification, data processing, data expansion, and data verification.

To describe each step in more detail, the design of the data structure is made through the definition of an ontology, a definition of a classification system, and the like. Data collection is performed by collecting data through direct shooting, web crawling, or association/professional organizations. Data purification is performed by removing duplicate data from the collected data and de-identifying personal information. Data processing is performed by performing annotations and inputting metadata. Data expansion is performed by performing ontology mapping, and supplementing or extending the ontology as necessary. And, the verification of the data is performed by verifying the validity according to the set target quality using various verification tools.

Meanwhile, automatic driving of a vehicle refers to a system capable of driving by determining the vehicle itself. Such autonomous driving may be divided into gradual stages from non-automation to full automation according to the degree to which the system is involved in driving and the degree to which the driver controls the vehicle. In general, the stage of autonomous driving is divided into six levels classified by the Society of Automotive Engineers (SAE) International. According to the 6 levels classified by the Society of Automotive Engineers (SAE), Level 0 is no automation, Level 1 is driver assistance, Level 2 is partial automation, and Level 3 is The stage is a conditional automation stage, the level 4 stage is a high automation stage, and the level 5 stage is a full automation stage.

Autonomous driving of a vehicle is performed through mechanisms of perception, localization, path planning, and control. Currently, several companies are developing to implement cognitive and path planning among autonomous driving mechanisms using artificial intelligence (AI).

Data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of these vehicles is collected by various types of sensors installed in the vehicle. For example, data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle is a lidar, a camera, a radar, and an ultrasonic sensor that are fixedly installed in the vehicle. ) and data acquired, photographed, or detected by a Global Positioning System (GPS), etc., but is not limited thereto.

However, when collecting data in real time using various types of sensors installed in vehicles for use in machine learning of artificial intelligence (AI), the capacity of the collected data increases exponentially, but the bandwidth and shadow of mobile communication There is a difficulty in that it is impossible to transmit data collected by a region or the like to the outside in real time.

In addition, data collected by various sensors may include useless data or personal information. Furthermore, when the physical conditions of the vehicle in which various sensors for collecting data are installed and the vehicle that will actually perform autonomous driving are different, the performance of autonomous driving by machine-learned artificial intelligence (AI) using the collected data is different. There are limitations that cannot be guaranteed.

Korean Patent Application Laid-Open No. 10-2020-0042629, ‘Method for generating touch-based annotations and images in mobile devices for artificial intelligence learning, and device therefor’, (published on April 24, 2020)

One object of the present invention is to provide a method for preemptively detecting an object from data for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Another object of the present invention is to provide a computer program recorded on a recording medium for executing a method capable of preemptively detecting an object from data for machine learning.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical task as described above, the present invention proposes a method for preemptively detecting an object from data for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle. The method includes: collecting, by a learning data collection device, 3D point group data and 2D images simultaneously acquired and photographed by a lidar and a camera installed in a vehicle; setting an object region in which an object is predicted to exist in the 2D image based on the three-dimensional coordinates of points included in the 3D point cloud data by the learning data collection device; and transmitting, by the learning data collection device, the 2D image and information about the set object region to the training data generating device. In this case, the object region may include a preset number of vertices corresponding to the computing power of the learning data collection apparatus and edges connecting the vertices.

Specifically, the setting of the object region may increase the number of vertices constituting the object region in proportion to the size of a period during which the camera captures the 2D image. Alternatively, the setting of the object region may increase the number of vertices constituting the object region in proportion to the moving speed of the vehicle.

Meanwhile, in the setting of the object region, points forming a crowd within a preset threshold range among points included in the 3D point cloud data are identified, and the identified cluster width on the X-axis, Y The type of object may be identified based on the height on the axis and the depth on the Z axis. In this case, the step of setting the object area includes the width on the X-axis, the height on the Y-axis of the identified cluster, and the It is also possible to identify the type of object corresponding to the depth on the Z-axis.

In the step of setting the object area, a 3D model provided in advance according to the type of the identified object is three-dimensionally rotated according to an optical axis direction of the camera, and then the rotated 3D model is rotated. A two-dimensional shape viewed from the optical axis direction may be identified as the object region.

In the step of setting the object region, an edge is extracted for the identified object region, and a type of the identified object and an edge pattern corresponding to the extracted edge pattern from a database provided in advance. The identified object region may be verified based on whether an object detection rule exists. In this case, the object detection rule is distributed by the learning data generating device, and may be a rule enumerating patterns of edges classified by type of object so as to verify the object identified in the 2D image.

In the step of setting the object area, if the object detection rule does not exist in the database, an update request message for the object detection rule including the 2D image, the type of the identified object, and the pattern of the extracted edge After generating, the generated update request message may be transmitted to the learning data generating device.

Also, in the setting of the object region, when no object is identified in the 2D image, 3D point cloud data acquired at the same time as the 2D image in which the object is not identified may be removed.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium for executing a method for preemptively detecting an object. The computer program includes a memory; transceiver; and a processor for processing instructions resident in the memory. The computer program may include: collecting, by the processor, 3D point cloud data and 2D images simultaneously acquired and photographed by the lidar and the camera installed in the vehicle; identifying, by the processor, an object region in which an object is expected to exist in the 2D image, based on three-dimensional coordinates of points included in the 3D point cloud data; and transmitting, by the processor, the 2D image and the information on the predicted area to the learning data generating apparatus, it may be a computer program recorded on a recording medium.

Specific details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, by preemptively detecting an object included in data collected in real time to machine-learning artificial intelligence (AI) that can be used for autonomous driving, it is possible to reduce the amount of annotation work. there will be

Furthermore, according to an embodiment of the present invention, an object is detected without a bottleneck from large-capacity data collected in real time from various sensors installed in a vehicle by detecting an object in response to the computational capability of a device for collecting machine learning data. can be detected.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.
2 is an exemplary diagram for explaining sensors according to an embodiment of the present invention.
3 is an exemplary diagram for explaining different object recognition rates of cameras installed at different heights.
4 is a logical configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.
5 is a hardware configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.
6 is an exemplary diagram for explaining a process of identifying a type of an object included in 3D point cloud data according to an embodiment of the present invention.
7 is an exemplary diagram for explaining a process of identifying a region of an object in a 2D image according to an embodiment of the present invention.
8 and 9 are exemplary views for explaining a result of detecting an object according to some embodiments of the present invention.
10 is an exemplary diagram for explaining a result of de-identification processing of personal information according to an embodiment of the present invention.
11 is an exemplary diagram for explaining a process of transmitting data according to an embodiment of the present invention.
12 is an exemplary diagram for explaining a process of transmitting data in the worst communication environment according to an embodiment of the present invention.
13 is a flowchart illustrating a data collection method according to an embodiment of the present invention.

It should be noted that the technical terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in this specification should be interpreted as meanings generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise specifically defined in this specification, and excessively inclusive It should not be construed in the meaning of a human being or in an excessively reduced meaning. In addition, when the technical terms used in the present specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be understood by being replaced with technical terms that can be correctly understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted according to the definition in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or various steps described in the specification, and some of the components or some steps are included. It should be construed that it may not, or may further include additional components or steps.

Also, terms including ordinal numbers such as first, second, etc. used herein may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but another component may exist in between. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be interpreted as extending to all changes, equivalents or substitutes other than the accompanying drawings.

As described above, when data is collected in real time using various types of sensors installed in a vehicle for use in machine learning of artificial intelligence (AI), the capacity of the collected data increases exponentially, but There was a difficulty in that it was impossible to transmit the collected data to the outside in real time due to bandwidth, shaded area, etc. In addition, data collected by various sensors may include useless data or personal information. Furthermore, when the physical conditions of the vehicle in which various sensors for collecting data are installed and the vehicle that will actually perform autonomous driving are different, the performance of autonomous driving by machine-learned artificial intelligence (AI) using the collected data is different. There was a limit that could not be guaranteed.

In order to overcome this limitation, the present invention provides an apparatus for collecting data for machine learning of artificial intelligence (AI) to efficiently control various sensors installed in a vehicle, and to perform pre-processing of object detection and personal information de-identification from data collected in real time. We intend to propose means for efficiently transmitting data according to communication quality.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.

1, the artificial intelligence learning system according to an embodiment of the present invention includes a plurality of learning data collection devices 100a, 100b, ..., 100n; 100), a learning data generating device 200, a plurality of It may be configured to include the annotation devices 300a, 300b, ..., 300n; 300 and the artificial intelligence learning device 400 .

As such, since the components of the artificial intelligence learning system according to an embodiment are merely functionally distinct elements, two or more components are integrated with each other in the actual physical environment, or one component is the actual physical environment. may be implemented separately from each other.

When explaining each component, the learning data collection device 100 collects data for machine learning artificial intelligence (AI) that can be used for autonomous driving, a lidar installed in a vehicle, a camera ( A device that collects data in real time from one or more of a camera), a radar, and an ultrasonic sensor.

Characteristically, the learning data collection apparatus 100 according to various embodiments of the present disclosure can efficiently control a plurality of cameras installed at different heights in a vehicle and simultaneously photographing the same direction in real time. The learning data collection apparatus 100 may automatically detect an object included in the collected data before transmitting the collected data to the learning data generating apparatus 200 and automatically de-identify personal information. In addition, the learning data collecting apparatus 100 may efficiently transmit the collected data according to the communication quality with the learning data generating apparatus 200 .

The types of sensors that are control objects of the learning data collection device 100 and installed in a vehicle to acquire, photograph, or detect machine learning data include a lidar, a camera, a radar, and an ultrasonic sensor. (ultrasonic sensor) may be included, but is not limited thereto. In addition, the control target of the learning data collection device 100 and the sensor installed in the vehicle to acquire, photograph, or detect data for machine learning is not limited to being provided one for each type, and a plurality of sensors even of the same type are provided. can be

The types of sensors that are control objects of the learning data collection device 100 and installed in a vehicle to acquire, photograph, or detect data for machine learning will be described later in more detail with reference to FIG. 2 . In addition, the characteristics of the learning data collection apparatus 100 will be described in more detail later with reference to FIGS. 3 and 4 .

In the following configuration, the learning data generating apparatus 200 receives data collected in real time by each of the learning data collecting apparatuses 100 using mobile communication from each of the plurality of learning data collecting apparatuses 100 . and distributes the received data to a plurality of annotation devices 300 , and generates data for machine learning of artificial intelligence (AI) based on the results of the annotation work received from each annotation device 300 .

Specifically, the learning data generating apparatus 200 may receive the properties of the project related to artificial intelligence (AI) learning from the artificial intelligence learning apparatus 400 . The learning data generating apparatus 200 is based on the user's control and the properties of the received project, the design of the data structure for artificial intelligence (AI) learning, the purification of the collected data, the processing of the data, the expansion of the data, and the verification can be performed.

First, the learning data generating apparatus 200 may design a data structure for artificial intelligence (AI) learning. For example, the learning data generating device 200 is based on the user's control and the properties of the received project, an ontology for artificial intelligence (AI) learning, a classification system of data for artificial intelligence (AI) learning can be defined.

The learning data generating apparatus 200 may collect data for artificial intelligence (AI) learning based on the designed data structure. To this end, the learning data generating apparatus 200 may receive sensing data, 3D point cloud data, 2D images, and distance information from the learning data collecting apparatus 100 . However, the present invention is not limited thereto, and the learning data generating apparatus 200 may perform web crawling or may download data from an external device.

The training data generating apparatus 200 may remove duplicate or extremely similar data from among the collected sensing data, 3D point cloud data, 2D images, and distance information. The learning data generating apparatus 200 may verify whether the personal information automatically de-identified by the learning data collecting apparatus 100 for the 2D image is correct.

Meanwhile, the learning data generating apparatus 200 detects a machine-learned object based on a result of an existing annotation operation on a plurality of 2D images in order to support the automation of object detection performed by the learning data collection apparatus 100 . It can be equipped with artificial intelligence (AI). In addition, the learning data generating apparatus 200 may generate an object detection rule using object detection artificial intelligence (AI) and distribute the generated object detection rule to a plurality of learning data collection apparatuses 100 .

In this case, the object detection rule may be a rule enumerating patterns of edges classified by type of object to verify the object identified in the 2D image.

The learning data generating apparatus 200 may receive an update request for an object detection rule from the learning data collecting apparatus 100 . In this case, the update request may include, but is not limited to, a 2D image that has not been correctly verified by an existing object detection rule, a type of an object identified from the 2D image, and an edge pattern extracted with respect to the identified object. The training data generating apparatus 200 may perform reinforcement learning of object detection artificial intelligence (AI) based on the received update request. Also, when a new object detection rule is generated as a result of reinforcement learning, the learning data generating apparatus 200 may redistribute the new object detection rule to the plurality of learning data collecting apparatuses 100 .

In addition, in order to support the automation of the de-identification processing performed by the learning data collection apparatus 100, the learning data generating apparatus 200 is based on images in which personal information has been previously de-identified for a plurality of 2D images. It may be equipped with machine-learned de-identification processing artificial intelligence (AI). Then, the learning data generating device 200 generates a non-identification processing rule using non-identification processing artificial intelligence (AI), and distributes the generated non-identification processing rule to a plurality of learning data collection devices 100 . can

In this case, the de-identification processing rule may be a rule enumerating patterns of edges with respect to the non-identification area existing in the object so as to determine the area to be de-identified in the 2D image. In addition, de-identification processing is image processing so that a person cannot identify personal information included in a 2D image. A specific example of the non-identification processing method may include, but is not limited to, blurring processing of a part of a 2D image.

The learning data generating apparatus 200 may receive an update request for the non-identification processing rule from the learning data collecting apparatus 100 . In this case, the update request may include, but is not limited to, a 2D image that has not been processed by the existing de-identification processing rule, a type of an object identified from the 2D image, and an edge pattern extracted with respect to the identified object. The learning data generating apparatus 200 may perform reinforcement learning of non-identification processing artificial intelligence (AI) based on the received update request. In addition, when a new de-identification processing rule is generated as a result of reinforcement learning, the learning data generating apparatus 200 may redistribute the new de-identification processing rule to the plurality of learning data collection apparatuses 100 .

Meanwhile, the learning data generating apparatus 200 may distribute and transmit the collected and refined sensing data, 3D point cloud data, 2D images, and distance information to the plurality of annotation apparatuses 300 . In this case, the learning data generating apparatus 200 generates sensed data, 3D point cloud data, 2D images, and distance information in response to an amount previously allocated to an operator (ie, a labeler) of the annotation apparatus 300 . can be distributed

The training data generating apparatus 200 may receive an annotation work result from each annotation apparatus 300 . The learning data generating apparatus 200 may generate data for artificial intelligence (AI) learning by packaging the received annotation work result. In addition, the learning data generating apparatus 200 may transmit the generated artificial intelligence (AI) learning data to the artificial intelligence learning apparatus 400 .

The training data generating device 200 having such a characteristic transmits and receives data to and from the training data collection device 100 , the annotation device 300 , and the artificial intelligence learning device 400 , and performs an operation based on the transmitted/received data Any device that can do this is acceptable. For example, the learning data generating apparatus 200 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

With the following configuration, the annotation device 300 is a device that can be used to perform an annotation operation on data collected from lidar, camera, radar, or ultrasound. That is, the annotation apparatus 300 is a device that can be used to perform an annotation operation on the sensed data, 3D point cloud data, 2D images, and distance information distributed by the learning data generating apparatus 200 .

As such, all or a part of the annotation device 300 may be a device in which an annotator performs an annotation operation through a clouding service.

Specifically, the annotation apparatus 300 includes one sensed data to be annotated from among the sensed data, 3D point cloud data, 2D images, and distance information received from the learning data generating apparatus 200, 3D point cloud data, A 2D image or distance information can be output to the display.

The annotation device 300 may select a tool according to a signal input from the user through the input/output device. Here, the tool is a tool for setting a bounding box specifying one or more objects included in the sensed data, 3D point cloud data, 2D image, or distance information.

The annotation device 300 may receive coordinates according to the selected tool through the input/output device. In addition, the annotation apparatus 300 may set a bounding box based on the input coordinates to specify an object included in the sensed data, 3D point cloud data, 2D image, or distance information.

Here, the bounding box is an area for specifying an object to be subjected to artificial intelligence (AI) learning among objects included in sense data, 3D point cloud data, 2D image, or distance information. As such, the bounding box may have a shape of a rectangle or a cube, but is not limited thereto.

For example, the annotation device 300 receives two coordinates through the input/output device, and based on the input two coordinates, the coordinates of the upper left vertex in the sensed data, 3D point cloud data, 2D image, or distance information, and An object can be specified by setting a bounding box based on a rectangle having the coordinates of the lower right vertex. In this case, the two coordinates may be set by the user inputting one type of input signal twice (eg, clicking the mouse), or the user may inputting two types of input signals once (eg, dragging the mouse). However, the present invention is not limited thereto.

Alternatively, the annotation apparatus 300 outputs the 2D image received from the learning data generating apparatus 200 and information about the object automatically detected by the learning data generating apparatus 100 with respect to the 2D image on the display. can do. In addition, the annotation apparatus 300 may receive a result of verifying whether the automatically detected information on the object is correct from the user. If there is an error in the automatically detected object information, the annotation device 300 may re-specify the object included in the 2D image according to a signal input from the user through the input/output device.

The annotation device 300 generates, according to a signal input from the user through the input/output device, detection data, 3D point cloud data, 2D image, distance information, or metadata for a set object to be annotated. can Here, the metadata is sensing data, 3D point cloud data, 2D image, distance information, or data for describing an object. Such metadata includes the category of a specified object, the rate at which the object is cut by the angle of view, the rate at which the object is obscured by other objects or objects, the tracking ID of the object, the time the image was taken, the weather conditions on the day the image was taken These may include, but are not limited to, file size, image size, copyright holder, resolution, bit value, aperture transmission, exposure time, ISO sensitivity, focal length, aperture opening value, angle of view, white balance, RGB depth, class name , tag, shooting location, type of road, road surface information, or traffic jam information may be further included.

The annotation apparatus 300 may generate an annotation work result based on the specified object and the generated metadata. In this case, the annotation operation result may have a JSON (Java Script Object Notation) file format, but is not limited thereto. The annotation apparatus 300 may transmit the generated annotation work result to the training data generating apparatus 200 .

The annotation device 300 having such a characteristic may be any device as long as it is capable of transmitting and receiving data to and from the training data generating device 200 and performing an operation based on the transmitted/received data. For example, the annotation device 100 may be a fixed computing device such as a desktop, a workstation, or a server, or a smart phone, a laptop, a tablet, or a tablet computer. It may be any one of a mobile computing device such as a phablet, a portable multimedia player (PMP), a personal digital assistant (PDA), or an e-book reader (E-book reader).

With the following configuration, the artificial intelligence learning device 400 is a device that can be used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Specifically, the artificial intelligence learning apparatus 400 may transmit a requirement for achieving the purpose of artificial intelligence (AI) that can be used for autonomous driving of a vehicle to the learning data generating apparatus 200 . The artificial intelligence learning apparatus 400 may receive data for artificial intelligence (AI) learning from the learning data generating apparatus 200 . In addition, the artificial intelligence learning apparatus 400 may machine-learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle by using the received artificial intelligence (AI) learning data.

As such, the artificial intelligence learning apparatus 400 may be any device as long as it is capable of transmitting and receiving data to and from the learning data generating apparatus 200 and performing an operation using the transmitted/received data. For example, the artificial intelligence learning device 400 may be any one of a desktop, a workstation, or a stationary computing device such as a server, but is not limited thereto.

As described above, the plurality of learning data collecting apparatuses 100 and learning data generating apparatus 200 may transmit/receive data using a mobile communication network. For example, in mobile communication networks, Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed Packet Access (HSPA), Long Term Evolution (Long Term) Evolution, LTE) and 5th generation mobile telecommunication may be included, but are not limited thereto.

In addition, the learning data generating device 200, the plurality of annotation devices 300, and the artificial intelligence learning device 400 are a combination of one or more of a security line directly connecting the devices, a public wired communication network, or a mobile communication network. can be used to transmit and receive data. For example, public wired networks include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). However, it is not limited thereto.

Hereinafter, sensors that are control objects of the learning data collection apparatus 100 as described above and are installed in a vehicle to acquire, photograph, or detect data for machine learning will be described in more detail.

2 is an exemplary diagram for explaining sensors according to an embodiment of the present invention.

As shown in FIG. 2 , the learning data collection apparatus 100 according to an embodiment of the present invention includes a radar 20 , a lidar 30 , a camera 40 , and an ultrasonic sensor fixed to the vehicle 10 ( 50), it is possible to collect basic data for machine learning of artificial intelligence (AI) by controlling one or more of them.

Here, the vehicle 10 is a vehicle in which a radar 20, a lidar 30, a camera 40, and an ultrasonic sensor 50 are installed for collecting basic data for machine learning artificial intelligence (AI). It can be distinguished from a vehicle that performs autonomous driving by intelligence (AI).

The radar 20 is fixedly installed in the vehicle 10 , and emits an electromagnetic wave in the direction of travel of the vehicle 10 , and the electromagnetic wave returned by being reflected by an object positioned in front of the vehicle 10 . , the vehicle 10 may generate sensing data corresponding to an image of the front.

In other words, the sensed data is information about points at which electromagnetic waves emitted toward the driving direction of the vehicle by the radar 20 fixedly installed in the vehicle 10 are reflected. Accordingly, the coordinates of the points included in the sensed data may have values corresponding to the position and shape of the object located in front of the vehicle 10 . The sensed data may be two-dimensional information, but is not limited thereto and may be three-dimensional information.

The lidar 30 is fixedly installed in the vehicle 10, radiates a laser pulse around the vehicle 10, and detects light reflected by an object located around the vehicle 10 and returned. , 3D point cloud data corresponding to a 3D image of the surroundings of the vehicle 10 may be generated.

In other words, the 3D point cloud data is three-dimensional information about points that reflect a laser pulse radiated around the vehicle by the lidar 30 fixedly installed in the vehicle 10 . Accordingly, coordinates of points included in the 3D point cloud data may have values corresponding to the position and formation of an object located around the vehicle 10 .

The camera 40 may be fixedly installed on the vehicle 10 to capture a two-dimensional image of the surroundings of the vehicle 10 . As such, a plurality of cameras 40 may be installed to be horizontally or horizontally spaced apart from the ground surface so as to take pictures in different directions. For example, FIG. 2 shows an example of a vehicle 10 in which six cameras 40 capable of photographing six different directions are fixedly installed, but the camera 40 that can be installed in the vehicle 10 is It will be apparent to those of ordinary skill in the art to which the present invention pertains that it may be configured in various numbers.

In other words, the 2D image is an image captured by the camera 40 fixedly installed in the vehicle 10 . Accordingly, the 2D image may include color information of an object positioned in the direction the camera 40 faces.

The ultrasonic sensor 50 is fixedly installed in the vehicle 50 , emits ultrasonic waves around the vehicle 10 , and detects a sound wave reflected by an object positioned adjacent to the vehicle 10 , and the vehicle Distance information corresponding to the distance between the ultrasonic sensor 50 installed in 10 and the object may be generated. In general, the ultrasonic sensor 50 is composed of a plurality and may be fixedly installed in the front, rear, front, and rear of the vehicle 10 that is easy to come into contact with an object.

In other words, the distance information is information about a distance from an object sensed by the ultrasonic sensor 50 fixedly installed in the vehicle 10 .

3 is an exemplary diagram for explaining different object recognition rates of cameras installed at different heights.

The vehicle height of a vehicle to perform autonomous driving using machine-learned artificial intelligence (AI) may vary, and the installation height of a camera installed in a vehicle to perform autonomous driving may also vary.

Therefore, when artificial intelligence (AI) is machine-learned based on only 2D images taken at a certain height, the cameras are placed at different heights in vehicles that will perform autonomous driving using the machine-learned artificial intelligence (AI). When installed, there is a limit that the detection rate of an object is different depending on the height of the camera installed in each vehicle.

In order to overcome this limitation, as shown in FIG. 3 , a plurality of cameras 40_high and 40_low are vertically spaced apart from the ground surface in a vehicle 10 that collects basic data for machine learning artificial intelligence (AI). A plurality of cameras 40_high and 40_low installed to be spaced apart from the ground surface in a vertical direction may be configured to capture 2D images 45_high and 45_low, respectively.

However, all 2D images (45_high, 45_low) taken by a plurality of cameras (40_high, 40_low) that are photographed in the same direction in overlapping two-dimensional perspectives horizontal to the ground surface are duplicated for image processing and transmitted to the outside. is inefficient

The learning data collection apparatus 100 according to an embodiment of the present invention defines and adjusts priorities among a plurality of cameras 40_high and 40_low installed at different heights, so that the plurality of cameras 40_high and 40_low are photographed It is possible to reduce the computational load and network load for image processing of all the 2D images 45_high and 45_low.

Hereinafter, the configuration of the learning data collection apparatus 100 having the above-described characteristics will be described in more detail.

4 is a logical configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.

As shown in FIG. 4 , the learning data collection apparatus 100 according to an embodiment of the present invention includes a communication unit 105 , an input/output unit 110 , a multi-sensor control unit 115 , an object identification unit 120 , and a It may be configured to include an identification processing unit 125 , a data providing unit 130 , and a storage unit 135 .

As such, the components of the learning data collection apparatus 100 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is implemented with each other in the actual physical environment. It may be implemented separately.

Each of the components will be described. The communication unit 105 may transmit/receive data to/from multiple sensors installed in the vehicle 10 and the learning data generating device 200 .

Specifically, the communication unit 105 transmits detection data, 3D point cloud data, 2D image and distance information from the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 fixedly installed in the vehicle 10 . can receive

The communication unit 105 may measure the communication quality of mobile communication with the learning data generating apparatus 200 every preset period. For example, the communication unit 105 transmits an Internet Control Message Protocol (ICMP) ping packet (echo request) to the learning data generating apparatus 200 at a preset period, and a response packet from the learning data generating apparatus 200 (echo reply) can be received.

The communication unit 105 may transmit the sensed data, 3D point cloud data, 2D image, and distance information to the learning data generating apparatus 200 under the control of the data providing unit 130 . In this case, the 2D image may include information about the area occupied by the object identified from the 2D image by the object identification unit 120 and information about de-identified personal information.

Then, the communication unit 105 transmits the update request message of the object detection rule or the update request message of the de-identification processing rule to the learning data generating apparatus 200 under the control of the object identification unit 120 and the de-identification processing unit 125 . can be transmitted

With the following configuration, the input/output unit 110 may receive a signal from a user through a user interface (UI) or may output an operation result to the outside.

Specifically, the input/output unit 110 may receive an input from the user of a period for measuring the communication quality with the learning data generating apparatus 200 . The input/output unit 110 may receive, from a user, a basic size of a buffer for storing 3D point cloud data, 2D image, sensed data, and distance information and a priority of data to be preferentially stored in the buffer.

The input/output unit 110 may receive, from the user, a threshold range, which is a size range for determining a crowd of points that can be identified as objects among points included in the sensed data or 3D point cloud data. In this case, as the threshold range, a value determined according to the type of object to be subjected to machine learning of artificial intelligence (AI) may be input. The input/output unit 110 may receive, from a user, the number of vertices for setting an object region for an object included in the 2D image.

The input/output unit 110 may receive, from a user, a minimum separation distance between an edge of the object and an area for de-identifying an object included in the 2D image. The input/output unit 110 may output the de-identified 2D image through the display.

In addition, the input/output unit 110 may receive an input from a user of priorities for a plurality of cameras installed at different heights in the vehicle 10 . The input/output unit 110 may receive input from the user the number of times to attempt object detection for each 2D image captured by each camera.

With the following configuration, the multi-sensor control unit 115 controls one or more of the radar 20 , the lidar 30 , the camera 40 and the ultrasonic sensor 50 fixedly installed in the vehicle 10 , and the radar 20 . , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 may collect data in real time.

Specifically, the multi-sensor control unit 115 may collect one or more of sensing data, 3D point cloud data, 2D image, and distance information through the communication unit 105 . Here, the sensed data is information on points at which electromagnetic waves emitted toward the driving direction of the vehicle by the radar 20 fixedly installed in the vehicle 10 are reflected. The 3D point cloud data is three-dimensional information about points that reflect a laser pulse radiated around the vehicle by the lidar 30 fixedly installed in the vehicle 10 . The 2D image is an image captured by the camera 40 fixedly installed in the vehicle 10 . In addition, the distance information is information about a distance from an object sensed by the ultrasonic sensor 50 fixedly installed in the vehicle 10 .

The multi-sensor control unit 115 does not always need to collect all of the sensed data, 3D point cloud data, 2D image, and distance information, and the multi-sensor control unit 115 is the object identification unit 120, de-identification processing unit 125 or In response to a request from the data controller 130 , one or more of sensing data, 3D point cloud data, 2D image, and distance information may be collected.

For example, the multi-sensor controller 115 may collect only 3D point cloud data and 2D images simultaneously acquired and photographed by the lidar 30 and the camera 40 . Alternatively, the multi-sensor controller 115 may collect 2D images simultaneously photographed by a plurality of cameras 40 installed at different heights in the vehicle 10 .

In addition, the multi-sensor control unit 115 stores data collected in real time from one or more of the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 in the buffer of the data providing unit 130 . can

Characteristically, the multi-sensor controller 115 according to an embodiment of the present invention collects 2D images simultaneously photographed by a plurality of cameras installed at different heights in the vehicle 10, The cameras can be controlled according to priority.

In more detail, the multi-sensor control unit 115 is configured to identify an object included in a 2D image captured by a camera having the highest priority based on a priority between a plurality of cameras installed at different heights. , a 2D image captured by a camera having the highest priority may be provided to the object identification unit 120 .

If the identification of the object from the 2D image taken by the camera having the highest priority fails, the multi-sensor control unit 115 tries to identify the object from the 2D image photographed by the camera having the next priority, The 2D image captured by the camera having the next priority may be provided to the object identification unit 120 .

In addition, when the identification of an object from all 2D images simultaneously photographed by a plurality of cameras installed at different heights fails, the multi-sensor control unit 115 performs The 3D point cloud data may be removed from the storage unit 135 .

As described above, the priority among a plurality of cameras installed at different heights for processing 2D images is basically based on the garage of the vehicle that will perform autonomous driving using artificial intelligence (AI). (200) may be temporarily set.

However, when the identification of the object performed on the 2D images taken by the camera having the highest priority fails continuously for a preset number of times, the multi-sensor control unit 115 is configured to perform a function between a plurality of cameras installed at different heights. can adjust the priorities.

In more detail, the multi-sensor control unit 115 may identify an object included in each 2D image through the object identification unit 120 for 2D images captured by a plurality of cameras installed at different heights. have. In addition, the multi-sensor controller 115 may adjust the priority among the plurality of cameras installed at different heights based on the integrity of the object identified from each 2D image.

In this case, the object's completeness means a ratio included in the 2D image without being cut by the camera's angle of view or other objects among the overall shape of the object.

To this end, the multi-sensor control unit 115 evaluates the integrity of the object by comparing the 3D point cloud data with the object identified from the 2D image, or the completeness of the object based on how many essential points are included in the 2D image itself. can be evaluated.

As an example, the multi-sensor controller 115 further collects 3D point cloud data obtained by the lidar 30 installed in the vehicle 10 , and selects three points included in the 3D point cloud data through the object identification unit 120 . Objects included in 3D point cloud data can be identified based on dimensional coordinates. In addition, the object identification unit 120 may evaluate the completeness of the object based on a result of comparing the object identified from the 3D point cloud data and the object identified from each 2D image with each other.

As another example, the multi-sensor control unit 115 may include a plurality of essential points preset according to the type of object identified by the object identification unit 120 among points constituting the object identified from each 2D image. The integrity of the object can be evaluated according to whether or not it contains the number of points corresponding to the points. In this case, the essential points may be the points required to distinguish the type of an object from other types.

With the following configuration, the object identification unit 120 may identify an object from the sensing data, 3D point cloud data, 2D image, and distance information collected by the multi-sensor control unit 115 .

Basically, the object identification unit 120 according to an embodiment of the present invention sets an object area in which an object is predicted to exist in a 2D image based on the three-dimensional coordinates of points included in the 3D point cloud data. can In this case, the object region may be a two-dimensional region composed of vertices and edges connecting the vertices to each other.

The number of vertices constituting the object region may be basically set in advance by the learning data generating apparatus 200 in response to the computing power of the learning data collecting apparatus 100 .

However, when the importance of the 2D image captured by the camera 40 is changed, the object identification unit 120 may adjust the number of vertices constituting the object region.

For example, the object identification unit 120 may increase the number of vertices constituting the object region in proportion to the size of a period during which the camera 40 installed in the vehicle 10 captures a 2D image.

As another example, the object identification unit 120 may increase the number of vertices constituting the object area in proportion to the moving speed of the vehicle 10 in which the camera 40 is installed.

Meanwhile, in order to identify the object in the 2D image from the 3D point cloud data, the object identification unit 120 may identify points forming a cluster within a preset threshold range among points included in the 3D point cloud data.

The object identification unit 120 may identify the type of object based on a width on an X-axis, a height on a Y-axis, and a depth on the Z-axis of the identified cluster. In more detail, the object identification unit 120 determines the width on the X-axis and the height on the Y-axis of the identified cluster based on a rate relation of width, height, and depth provided in advance for each type of object in the database. and the type of object corresponding to the depth on the Z-axis may be identified.

The object identification unit 120 may rotate the 3D model previously provided in the database according to the type of the identified object according to the optical axis direction of the camera 40 photographing the corresponding 2D image. In addition, the object identification unit 120 may identify the two-dimensional shape of the three-dimensionally rotated 3D model viewed from the optical axis direction of the camera 40 as the object region.

In addition, the object identification unit 120 may configure the vertices and edges by reflecting the two-dimensional shape identified as the object region in the 2D image, thereby setting the object region in which the object is predicted to exist in the 2D image.

On the other hand, the object identification unit 120 extracts an edge with respect to the identified object region, and determines whether an object detection rule corresponding to the type of object and the extracted edge pattern from the database exists. Based on the identified object area can be verified. In this case, the object detection rule is a rule that is distributed by the learning data generating device 200 and enumerates edge patterns classified by type of object to verify the object identified in the 2D image.

When the object detection rule does not exist in the database, the object identification unit 120 updates the object detection rule by including the 2D image, the type of object identified from the 3D point cloud data, and the edge extracted from the 2D image. You can create a request message. When the update request message is generated, the object identification unit 120 may transmit the generated update request message to the learning data generating apparatus 200 through the communication unit 105 .

In addition, when no object is identified in the 2D image, the object identification unit 120 may remove the 3D point cloud data obtained at the same time as the 2D image in which the object is not identified from the storage unit 135 . .

With the following configuration, the de-identification processing unit 125 may de-identify a part of the 2D image in response to the type of the object identified by the object identification unit 120 .

Specifically, when the object identification unit 120 sets the object area using the 3D model, the de-identification processing unit 125 performs de-identification processing on the area corresponding to the non-identification processing area previously given to the 3D model. can be done

To this end, the de-identification processing unit 125 extracts the edge of the 2D image, the type of object identified by the object identification unit 120 from the database, and the de-identification rule corresponding to the pattern of the edge extracted from the 2D image. can be inquired. When the de-identification processing rule is queried from the database, the de-identification processing unit 125 may de-identify a part of the 2D image according to the inquired de-identification processing rule. In this case, the de-identification processing rule is distributed by the learning data generating device 200, and a rule enumerating the pattern of edges for the non-identification area existing in the object so as to determine the area to be de-identified within the 2D image. to be.

The de-identification processing unit 125 includes the inclusion of the 2D image, the type of object identified from the 3D point cloud data, and the edge extracted from the 2D image when the de-identification processing rule is not queried from the database, to the de-identification processing rule. You can create an update request message for When the update request message is generated, the non-identification processing unit 125 may transmit the generated update request message to the learning data generating apparatus 200 through the communication unit 105 .

On the other hand, when the de-identification processing unit 125 performs the de-identification process, the boundary line of the area to be de-identified and the boundary line of the closed area by the edge extracted from the 2D image is set in advance. When adjacent within the minimum separation distance, the size of the area to be de-identified may be reduced so that the boundary line of the area to be de-identified and the boundary line of the closed area are spaced apart by more than the minimum separation distance. In this case, the minimum separation distance may be the minimum number of pixels positioned between the two boundary lines.

In this way, the de-identification processing unit 125 allows the boundary line of the area to be de-identified and the edge of the 2D image to not overlap each other and to be spaced apart from each other by more than the minimum separation distance, so that the de-identification processing of personal information affects the annotation work. can be minimized.

Meanwhile, the de-identification processing unit 125 as described above may perform de-identification processing by blurring a part of the 2D image. In this case, since the visibility of the 2D image to be subjected to the de-identification process can be changed according to the moving speed of the vehicle 10, the non-identification processing unit 125 measures the intensity of blurring for the de-identification process to the camera. (40) may be set in inverse proportion to the moving speed of the installed vehicle (10).

In addition, the de-identification processing unit 125 may crop a region to be de-identified before performing the blurring process, and then include the cropped image to generate restored data.

With the following configuration, the data providing unit 130 includes the sensed data collected by the multi-sensor control unit 115 , 3D point cloud data, a 2D image de-identified by the de-identification processing unit 125 , distance information, and an object identification unit Information on the object region set by 120 may be transmitted to the learning data generating apparatus 200 .

Characteristically, the data providing unit 130 according to an embodiment of the present invention measures the communication quality of mobile communication with the learning data generating device 200 at a preset period, and detects data in response to the measured communication quality; Adjust the size of the buffer to store 3D point cloud data, 2D image and distance information, and when the accumulated data in the buffer is full, the sensed data, 3D point cloud data, 2D image and distance information stored in the buffer are used as training data. It can be transmitted to the generating device 200 . That is, the buffer is a temporary storage space for storing data collected from the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 .

Basically, the data providing unit 130 may measure the communication quality of mobile communication with the learning data generating apparatus 200 every preset period, and may adjust the size of the buffer in response to the measured communication quality.

Specifically, the data providing unit 130 transmits an ICMP ping packet to the learning data generating apparatus 200 at a set period through the communication unit 105, and the RTT ( Round Trip Time) and TTL (Time To Live) can determine the size of the buffer. In this case, the data providing unit 130 transmits the ICMP ping packet in inverse proportion to the moving speed of the vehicle 10 in which the radar 20 , the lidar 30 , the camera 40 and the ultrasonic sensor 50 are installed. cycle can be adjusted.

On the other hand, when it is determined that all packets are lost based on the TTL included in the response packet, the data providing unit 130 does not directly perform mobile communication with the learning data generating device 200, but with another learning data collecting device ( It may be controlled to perform mobile communication with the learning data generating apparatus 200 via 100).

If the size of the buffer determined by the data providing unit 130 is smaller than a preset basic transmission size, the data providing unit 130 may include the radar 20 , the lidar 30 , the camera 40 and the ultrasonic sensor. Some of the data collected from one or more of (50) may be discarded. In this case, the basic transmission size is the sum of all data simultaneously acquired, photographed, and detected by the radar 20 , the lidar 30 , the camera 40 and the ultrasonic sensor 50 during the preset basic collection time. can be size.

First, the data providing unit 130 according to a preset priority, the detection data acquired by the radar 20, the 3D point cloud data acquired by the lidar 30, and the 2D photographed by the camera 40 Distance information detected by the image and ultrasonic sensor 50 may be sequentially removed.

In this case, the data providing unit 130 estimates the distance at which the vehicle 10 is separated from the object based on the detected data obtained by the radar 20 , and obtains it by the radar 20 based on the estimated distance. It is possible to adjust the priority among the detected data, the 3D point cloud data obtained by the lidar 30 , the 2D image captured by the camera 40 , and the distance information detected by the ultrasonic sensor 50 .

In addition, when the data providing unit 130 needs to remove the 2D images photographed by the camera 40 according to the priority, the object is included among the plurality of 2D images photographed by the camera 40 during the basic collection time. 2D images that have not been removed can be removed preferentially.

In this case, since it is sufficient to determine whether an object is included in the 2D image without accurately specifying the object from the 2D image, the data providing unit 130 does not identify the correct object through the object identification unit 120 . It is also possible to determine only whether an object is included in the 2D image by simply matching the edge of the 2D image with the boundary of the points forming the cluster among the points included in the 3D point cloud data.

With the following configuration, the storage unit 135 may store data necessary for the operation of the learning data collection apparatus 100 .

Specifically, the storage unit 135 may include a buffer for storing 3D point cloud data, 2D image, sensing data, and distance information. In addition, the storage unit 135 may be configured to include a database for storing rules and basic data required for image processing and data transmission.

The database constituting the storage unit 135 may store object detection rules, non-identification processing rules, and 3D models for each type of object. Here, the object detection rule may be a rule enumerating patterns of edges classified by type of object to verify the object identified in the 2D image. The de-identification processing rule may be a rule enumerating patterns of edges for the de-identification area existing in the object so that the area to be de-identified in the 2D image can be determined. And, the 3D model is three-dimensional shape data of an object classified by type of object.

Hereinafter, hardware for implementing the logical components of the learning data collection apparatus 100 as described above will be described in more detail.

5 is a hardware configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.

5, the learning data collection device 100 includes a processor 150, a memory 155, a transceiver 160, an input/output device 165, and a data bus. (Bus, 170) and may be configured to include a storage (Storage, 175).

The processor 150 may implement the operations and functions of the learning data collection apparatus 100 based on an instruction according to the software 180a in which the method according to the embodiments of the present invention is implemented residing in the memory 155 . . The memory 155 may be loaded with software 180a in which methods according to embodiments of the present invention are implemented. The transceiver 160 may transmit/receive data to and from the radar 20 , the lidar 30 , the camera 40 , the ultrasonic sensor 50 , and the learning data generating apparatus 200 . The input/output device 165 may receive data necessary for the operation of the learning data collection device 100 , and may output the collected sensed data, 3D point cloud data, 2D image, and distance information. The data bus 170 is connected to the processor 150 , the memory 155 , the transceiver 160 , the input/output device 165 and the storage 175 , and is a movement path for transferring data between each component. can play a role.

The storage 175 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 180a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 175 may store the software 180b and the database 185 in which the method according to the embodiments of the present invention is implemented. The database 185 may store object detection rules, non-identification processing rules, and 3D models for each type of object.

According to an embodiment of the present invention, the software 180a and 180b for implementing the control method of the sensors resident in the memory 155 or stored in the storage 175 is configured so that the processor 150 is different from each other in the vehicle 10 . Collecting 2D images taken simultaneously by a plurality of cameras 40 installed at a height. step of the processor 150 based on the priority among the plurality of cameras 40, identifying an object included in the 2D image taken by the camera having the highest priority, and the processor 150 is the It may be a computer program recorded on a recording medium in order to execute the step of transmitting the 2D image captured by the camera 40 with the highest priority and information about the identified object to the learning data generating device 200 . .

According to another embodiment of the present invention, the software 180a and 180b for implementing the method of preemptively detecting an object resident in the memory 155 or stored in the storage 175 is the processor 150 installed in the vehicle 10 . Collecting 3D point cloud data and 2D images simultaneously acquired and photographed by the lidar 20 and the camera 30, the processor 150 based on the 3D coordinates of the points included in the 3D point cloud data, the In order to execute the step of identifying an object region in which an object is predicted to exist in the 2D image, and the processor 150 transmitting the 2D image and information about the predicted region to the learning data generating device 200 . It may be a computer program recorded on a recording medium.

According to another embodiment of the present invention, the software 180a, 180b for implementing the personal information de-identification processing method resident in the memory 155 or stored in the storage 175 is the processor 150 and the vehicle 10 Collecting 3D point cloud data and 2D images simultaneously acquired and photographed by the lidar 20 and the camera 40 installed in the , identifying an object included in the 2D image, and the processor 150 de-identifying a part of the 2D image corresponding to the identified type of object. It may be a computer program recorded in

More specifically, the processor 150 may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 155 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 160 may include a baseband circuit for processing wired/wireless signals. The input/output device 165 includes an input device such as a keyboard, a mouse, and/or a joystick, and a liquid crystal display (LCD), an organic light emitting diode (OLED) and/or an input device such as a joystick. Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiment included in this specification is implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. The module resides in the memory 155 and may be executed by the processor 150 . The memory 155 may be internal or external to the processor 150 , and may be connected to the processor 150 by various well-known means.

Each component shown in FIG. 5 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present invention provides one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs ( Field Programmable Gate Arrays), a processor, a controller, a microcontroller, a microprocessor, etc. may be implemented.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored in a recording medium readable through various computer means. can be recorded. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), and a floppy disk. Magneto-Optical Media, such as a disk, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler. Such hardware devices may be configured to operate as one or more software to perform the operations of the present invention, and vice versa.

Hereinafter, the features of the artificial intelligence learning system according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

6 is an exemplary diagram for explaining a process of identifying a type of an object included in 3D point cloud data according to an embodiment of the present invention. 7 is an exemplary diagram for explaining a process of identifying a region of an object in a 2D image according to an embodiment of the present invention. 8 and 9 are exemplary views for explaining a result of detecting an object according to some embodiments of the present invention.

Referring to FIG. 6 , the learning data collection apparatus 100 according to an embodiment of the present invention identifies an object in a 2D image from 3D point cloud data, a cluster 60 within a critical range among points included in 3D point cloud data. points forming the .

The learning data collection apparatus 100 views the cluster 60 from the identified 3D point cloud data on the X-axis when viewed from the top (60_top), the width on the X-axis, and the cluster 60 from the side The type of object may be identified based on the depth on the Z-axis when viewed (60_side) and the height on the Y-axis when the cluster 60 is viewed from the front (60_front).

Specifically, the learning data collection apparatus 100 determines the width on the X-axis, the height on the Y-axis, and the Z-axis of the cluster 60 based on the ratio relationship between the width, height, and depth of the object provided for each type of object in the database. It is possible to identify the type of object corresponding to the depth of the image.

Referring to FIG. 7 , the learning data collection apparatus 100 uses the 3D model 70 previously provided in the database according to the type of the identified object in the optical axis direction of the camera 40 for photographing the corresponding 2D image. can be rotated in three dimensions.

In this case, the 3D model 70 is 3D shape data of an object classified by type of object. As such, the 3D model 70 may be provided with a non-identification processing area 90 for removing personal information for each type of object.

In addition, the learning data collection apparatus 100 may identify a two-dimensional shape viewed from the optical axis direction of the camera 40 as the object region 80 of the three-dimensionally rotated 3D model.

In this case, the object region 80 is a two-dimensional region composed of the vertices 85 and the edges connecting the vertices 85 to each other, and is a region in which an object is predicted to exist in the 2D image. As such, the vertex 85 may be referred to as a magic pin in the sense that it is a point for automatically specifying an object.

As such, when the number of vertices 85 constituting the object region 80 is large, a large load is placed on the learning data collection apparatus 100 that must process numerous 2D images captured in real time by the camera 40 . may occur.

Accordingly, as shown in FIGS. 8 and 9 , the vertices 85 constituting the object region 80 basically correspond to the computing power of the learning data collecting apparatus 100 , and the learning data generating apparatus 200 . It may be composed of a number set by However, when the importance of the 2D image captured by the camera 40 is changed, the learning data collection apparatus 100 may actively adjust the number of vertices 85 constituting the object region 80 .

For example, the learning data collection device 200 includes the vertices 85 constituting the object region 80 in proportion to the size of the period during which the camera 40 installed in the vehicle 10 captures the 2D image. ) can be increased. Also, the learning data collection apparatus 200 may increase the number of vertices 85 constituting the object area 80 in proportion to the moving speed of the vehicle 10 in which the camera 40 is installed.

As described above, the learning data collection device 200 reduces the amount of annotation work by preemptively detecting objects included in the data collected in real time to machine-learning artificial intelligence (AI) that can be used for autonomous driving. be able to

In addition, the learning data collection apparatus 200 detects the object in response to the computing power of the apparatus, so that the object can be detected without a bottleneck from the large amount of data collected in real time.

10 is an exemplary diagram for explaining a result of de-identification processing of personal information according to an embodiment of the present invention.

As shown in FIG. 10 , the learning data collection apparatus 100 according to an embodiment of the present invention may de-identify a part of the 2D image (ie, an area including personal information).

Specifically, when the object area 80 is set using the 3D model 70 , the learning data collection apparatus 100 compares the area corresponding to the non-identification processing area 90 given to the 3D model 70 . Identification processing may be performed.

Characteristically, when the learning data collection apparatus 100 performs de-identification processing, the boundary line of the area to be de-identified and the boundary line of the closed area by the edge extracted from the 2D image are adjacent within a preset minimum separation distance. In this case, the size of the region to be de-identified may be reduced so that the boundary line of the region to be de-identified is spaced apart from the boundary of the closed region by a minimum separation distance or more. Through this, the learning data collection apparatus 100 may minimize the influence of the de-identification processing of personal information on the annotation work.

In addition, the de-identification process performed by the learning data collection apparatus 100 may be performed by blurring a part of the 2D image. In this case, since the sharpness of the 2D image to be subjected to the de-identification process may be changed according to the moving speed of the vehicle 10 , the learning data collection apparatus 100 calculates the intensity of blurring for the de-identification process by the camera 40 . ) may be set in inverse proportion to the moving speed of the installed vehicle 10 .

On the other hand, the learning data collection apparatus 100 may generate restored data by including the cropped image after cropping the area to be de-identified before performing the blurring process.

As described above, the learning data collection device 200 can not expose personal information to the outside by immediately de-identifying the personal information included in the data collected in real time.

In this case, there is no need for a person to directly de-identify the personal information contained in the data, by simply reviewing the already de-identified area. The efficiency of the data purification process can be improved.

11 is an exemplary diagram for explaining a process of transmitting data according to an embodiment of the present invention. And, FIG. 12 is an exemplary diagram for explaining a process of transmitting data in the worst communication environment according to an embodiment of the present invention.

11 , the learning data collecting apparatus 100 according to an embodiment of the present invention measures the communication quality of mobile communication with the learning data generating apparatus 200 every preset period, and the measured communication quality You can adjust the size of the buffer in response. In this case, the buffer is a temporary storage space for storing data collected from the radar 20 , the lidar 30 , the camera 40 , and the ultrasonic sensor 50 .

Specifically, the training data collection device 100 transmits an ICMP ping packet to the training data generation device 200 at a preset period, and a Round Trip Time (RTT) included in a response packet received from the training data generation device. and TTL (Time To Live) can determine the size of the buffer.

In this case, the learning data collection apparatus 100 transmits the ICMP ping packet in inverse proportion to the moving speed of the vehicle 10 in which the radar 20 , the lidar 30 , the camera 40 and the ultrasonic sensor 50 are installed. The transmission period can be adjusted.

On the other hand, when it is determined that all packets are lost based on the TTL included in the response packet, as shown in FIG. 12 , the learning data collection apparatus 100a directly performs mobile communication with the learning data generating apparatus 200 . Instead, it may be controlled to perform mobile communication with the learning data generating apparatus 200 via another learning data collecting apparatus 100b.

In addition, when the size of the buffer determined by the learning data collection apparatus 100 is smaller than a preset basic transmission size, the learning data collection apparatus 100 includes the radar 20 , the lidar 30 , the camera 40 and Some of the data collected from one or more of the ultrasonic sensors 50 may be removed.

First, the learning data collection apparatus 100 performs the detection data acquired by the radar 20, the 3D point cloud data acquired by the lidar 30, and the camera 40 according to a preset priority. The 2D image and distance information detected by the ultrasonic sensor 50 may be sequentially removed.

In this case, the learning data collection apparatus 100 estimates the distance at which the vehicle 10 is separated from the object based on the detected data acquired by the radar 20, and uses the radar 20 based on the estimated distance. It is possible to adjust the priority between the acquired detection data, the 3D point cloud data acquired by the lidar 30 , the 2D image photographed by the camera 40 , and the distance information detected by the ultrasonic sensor 50 . In addition, when the learning data collection apparatus 100 needs to remove the 2D images photographed by the camera 40 according to the priority, the object is selected from among the plurality of 2D images photographed by the camera 40 during the basic collection time. It is also possible to preferentially remove 2D images that are not included.

As described above, since the learning data collection apparatus 200 efficiently transmits the collected data in real time, it is not necessary to stop the driving of the vehicle for stable data transmission, and it is not necessary to provide a large-capacity storage space.

Hereinafter, the operation of the learning data collection apparatus 100 according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

13 is a flowchart illustrating a data collection method according to an embodiment of the present invention.

Referring to FIG. 13 , the learning data collection apparatus 100 according to an embodiment of the present invention provides sensing data and 3D point cloud data from a radar 20 , a lidar 30 , a camera 40 , and an ultrasonic sensor 50 . , it is possible to collect one or more of the 2D image and distance information (S100). In this case, when the learning data collection apparatus 100 collects 2D images simultaneously photographed by a plurality of cameras installed at different heights in the vehicle 10 , the plurality of cameras installed at different heights are prioritized according to priority. can be controlled

Next, the learning data collection apparatus 100 may identify the object from the collected sensing data, 3D point cloud data, 2D image, and distance information ( S200 ). In this case, the learning data collection apparatus 100 may set the object region in which the object is predicted to exist in the 2D image based on the 3D coordinates of the points included in the 3D point cloud data. In this case, the object region may be a two-dimensional region composed of vertices and edges connecting the vertices to each other. In addition, the vertices constituting the object region may be basically configured in a number corresponding to the computing power of the learning data collection apparatus 100 .

Next, the learning data collection apparatus 100 may de-identify the personal information included in the 2D image corresponding to the identified object type ( S300 ). In this case, the learning data collection device 100 inquires a de-identification rule corresponding to the type of object and the pattern of the edge extracted from the 2D image, and a part of the 2D image (ie, personal information) according to the inquired de-identification processing rule. ) can be de-identified.

Next, the learning data collecting apparatus 100 may transmit the collected sensed data, 3D point cloud data, de-identified 2D image, distance information, and information on the object area to the learning data generating apparatus 200 (S400). ).

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention pertains that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein. It is obvious to those with ordinary knowledge. In addition, although specific terms have been used in the present specification and drawings, these are only used in a general sense to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Vehicle: 10 Radar: 20
Lidar: 30 Camera: 40
Ultrasonic sensor: 50
Training data collection device: 100 Training data generation device: 200
Annotation device: 300 AI learning device: 400
Communication unit: 105 Input/output unit: 110
Multi-sensor control unit: 115 Object identification unit: 120
De-identification processing unit: 125 Data providing unit: 130
Storage: 135

Claims (10)

A method of collecting data for machine learning of artificial intelligence (AI) that can be used for autonomous driving, the method comprising:
Collecting, by the learning data collection device, 3D point group data and 2D images simultaneously acquired and photographed by a lidar and a camera installed in the vehicle;
setting, by the learning data collection device, an object region in which an object is predicted to exist in the 2D image, based on the three-dimensional coordinates of points included in the 3D point cloud data; and
Comprising the step of transmitting, by the learning data collection device, information about the 2D image and the set object region to the learning data generating device,
The object region is characterized in that it is composed of a preset number of vertices and edges connecting the vertices corresponding to the computing power of the learning data collection device,
The step of setting the object area is
It characterized in that the number of vertices constituting the object area is increased in proportion to the size of the period during which the camera takes the 2D image,
The step of setting the object area is
The method of preemptively detecting an object, characterized in that the number of vertices constituting the object area is increased in proportion to the moving speed of the vehicle.
The method of claim 1, wherein the setting of the object area comprises:
Among the points included in the 3D point cloud data, points forming a crowd within a preset critical range are identified, and the identified cluster is based on the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis. A method of preemptively detecting an object, characterized in that the type of the object is identified with a
The method of claim 2, wherein the setting of the object area comprises:
Based on the rate relation of width, height, and depth provided in advance for each type of object, the type of object corresponding to the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis of the identified cluster is determined. A method for preemptive detection of an object, characterized in that for identification.
The method of claim 3, wherein the setting of the object area comprises:
After three-dimensional rotation of a 3D model provided in advance according to the type of the identified object according to an optical axis direction of the camera, a two-dimensional shape viewed from the optical axis direction of the rotated 3D model A method of preemptively detecting an object, characterized in that it is identified as the object region.
The method of claim 3, wherein the setting of the object area comprises:
Edge detection is performed with respect to the identified object area, and whether an object detection rule corresponding to the type of the identified object and the pattern of the extracted edge exists from a database provided in advance. Verifies the identified object area based on,
The object detection rule is
Distributed by the learning data generating device and characterized in that it is a rule enumerating patterns of edges classified by type of object so as to verify the object identified in the 2D image, the method for preemptive detection of an object.
The method of claim 5, wherein the setting of the object area comprises:
When the object detection rule does not exist in the database, after generating an update request message for the object detection rule including the 2D image, the type of the identified object, and the pattern of the extracted edge, the generated update A method for preemptively detecting an object, characterized in that it transmits a request message to the learning data generating device.
The method of claim 1, wherein the setting of the object area comprises:
If no object is identified in the 2D image, 3D point cloud data acquired at the same time as the 2D image in which the object is not identified is taken is removed.
memory;
transceiver; and
Combined with a computing device configured to include a processor (processor) for processing instructions resident in the memory,
collecting, by the processor, 3D point cloud data and 2D images simultaneously acquired and photographed by the lidar and the camera installed in the vehicle;
identifying, by the processor, an object region in which an object is predicted to exist in the 2D image, based on the three-dimensional coordinates of points included in the 3D point cloud data; and
Execute, by the processor, transmitting information about the 2D image and the predicted area to a learning data generating device,
The object region is characterized in that it is composed of a preset number of vertices and edges connecting the vertices corresponding to the computing power of the learning data collection device,
The step of identifying the object area is
It characterized in that the number of vertices constituting the object area is increased in proportion to the size of the period during which the camera takes the 2D image,
The step of identifying the object area is
A computer program recorded on a recording medium, characterized in that the number of vertices constituting the object area is increased in proportion to the moving speed of the vehicle.
The method of claim 8, wherein the identifying the object area comprises:
Among the points included in the 3D point cloud data, points forming a crowd within a preset critical range are identified, and the identified cluster is based on the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis. A computer program recorded on a recording medium, characterized in that for identifying the type of the object with
The method of claim 9, wherein the identifying the object area comprises:
Based on the rate relation of width, height, and depth provided in advance for each type of object, the type of object corresponding to the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis of the identified cluster is determined. A computer program recorded on a recording medium, characterized in that it is identified.

Images