KR102313944B1 - Method for tracking object crossing the boundary of the angle of view, and computer program recorded on record-medium for executing method therefor - Google Patents

Abstract

The present invention proposes a method for tracking an object crossing the boundary of a camera angle of view. The method may include the steps of: allowing an annotation device to specify an object from each of a plurality of 2D images taken by a plurality of cameras fixedly installed in a vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of the vehicle; allowing the annotation device to specify the same object as the object from 3D point group data continuously acquired by a lidar fixedly installed in the same vehicle as the vehicle in which the plurality of cameras are installed; allowing the annotation device to identify the movement path of the object based on a change in the position of the object specified from the 3D point cloud data; allowing the annotation device to assign the same metadata to the specified object from each of the plurality of 2D images, based on the identified movement path.

Description

Method for tracking object crossing the boundary of the angle of view, and computer program recorded on record-medium for executing method therefor

The present invention relates to the processing of data for artificial intelligence (AI) learning. More specifically, in relation to an annotation operation of data for machine learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle, a method for tracking an object crossing the boundary of a camera angle of view and a method for tracking the same It relates to a computer program recorded on a recording medium for execution.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning ability, reasoning ability, and perception ability using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the form of data for learning.

In general, the design of data for artificial intelligence (AI) learning proceeds in the stages of data structure design, data collection, data purification, data processing, data expansion, and data verification.

To describe each step in more detail, the design of the data structure is made through the definition of an ontology, a definition of a classification system, and the like. The collection of data is made by collecting data through direct shooting, web crawling, or association/professional organizations. Data purification is performed by removing duplicate data from the collected data and de-identifying personal information. Data processing is performed by performing annotations and inputting metadata. Data expansion is performed by performing ontology mapping and supplementing or extending the ontology as needed. And, the verification of the data is performed by verifying the validity according to the set target quality using various verification tools.

Meanwhile, automatic driving of a vehicle refers to a system capable of driving by determining the vehicle itself. Such autonomous driving may be divided into gradual stages from non-automation to full automation according to the degree to which the system is involved in driving and the degree to which the driver controls the vehicle. In general, the stages of autonomous driving are divided into six levels classified by the Society of Automotive Engineers (SAE) International. According to the six stages classified by the International Association of Automobile Engineers, level 0 is non-automated, level 1 is driver assistance, level 2 is partial automation, level 3 is conditional automation, level 4 is highly automated, and level 5 is The steps are fully automated steps.

Autonomous driving is performed through the mechanisms of perception, localization, path planning, and control. And, various companies are developing to implement perception and route planning in autonomous driving mechanisms using artificial intelligence (AI).

For example, Tesla is developing artificial intelligence (AI) for autonomous driving that minimizes location information and operates based on information input from cameras and sensors. To this end, Tesla collects its own vehicle driving data and manages it in a data center, and based on this, performs imitation learning to imitate the driving pattern of a real driver with respect to artificial intelligence (AI).

In contrast, Google's Waymo and Hyundai Motor's and Aptiv's Motional provide vehicle location information, terrain and road information obtained from map information, and cameras and sensors. We are developing artificial intelligence (AI) for autonomous driving that operates based on input information. To this end, the companies are setting driving algorithms based on the information obtained through test driving of the information collection vehicle, and performing artificial intelligence (AI) reinforcement learning using simulators based on the driving algorithm.

As described above, the data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving consists of a large number ranging from a few thousand to a maximum of several million. Accordingly, various means for more easily processing a large number of data are required.

Republic of Korea Patent Publication No. 10-2230144, ‘Artificial intelligence deep learning target detection and speed potential field algorithm-based obstacle avoidance and autonomous driving method and device’, (2021.03.15. Announcement)

An object of the present invention is to provide a method for tracking an object crossing the boundary of a camera angle of view in relation to an annotation operation of data for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle. will provide

Another object of the present invention is to provide a computer program recorded on a recording medium for executing a method capable of tracking an object crossing a boundary of a camera angle of view.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical problem as described above, the present invention proposes a method for tracking an object crossing a boundary of a camera angle of view. In the method, in order for the annotation device to learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle, an object is extracted from each of a plurality of 2D images taken by a plurality of cameras fixedly installed in the vehicle. specifying; specifying, by the annotation device, the same object as the object from 3D point group data continuously acquired by a lidar fixedly installed in the same vehicle as the vehicle in which the plurality of cameras are installed; identifying, by the annotation device, a movement path of the object based on a change in the position of the object specified from the 3D point cloud data; and assigning, by the annotation device, identical metadata to the specified object from each of the plurality of 2D images, based on the identified movement path.

Specifically, in the step of identifying the movement path, when two 2D images obtained by photographing spatially continuous angles of view among the plurality of 2D images each include different parts of one object, the movement path of the object can be identified.

To this end, the step of identifying the movement path is the sum of the ratio truncated by the angle of view of the first 2D image and the ratio truncated by the angle of view of the second image among the two 2D images obtained by photographing the spatially continuous angles of view. When it is equal to or greater than the preset threshold ratio, it may be determined that the two 2D images each include different parts of one object.

In the step of identifying the movement path, when the sum of the ratios is equal to or greater than the threshold ratio, the RGB (Red, Green, Blue) values of the object specified from the first 2D image and the RGB of the object specified from the second 2D image Based on the value, it may be verified whether the two 2D images each include different parts of one object.

In addition, in the step of identifying the movement path, when the sum of the ratios is equal to or greater than the threshold ratio, the edge of the object specified from the first 2D image is based on the boundary line between the first 2D image and the second 2D image. And based on whether the edges of the object specified from the second 2D image are continuous, it may be verified whether the two 2D images each include different parts of one object.

In the step of identifying the movement path, when the object specified from the 3D point cloud data crosses the boundary line between two 2D images obtained by photographing spatially continuous angles of view among the plurality of 2D images, the movement of the object path can be identified.

In the step of specifying the same object, the plurality of 2D images are arranged in a circle according to the optical axis angle of the camera so that the vehicle in which the lidar and the camera are installed is a virtual circular center point, and in the 3D point cloud data The same object may be specified by acquiring a top view image composed of only X-axis and Z-axis data, and matching the plurality of 2D images arranged in a circle with the plane viewpoint image.

In the step of providing the metadata, not the relative size occupied by the object in the 2D image, but an absolute size identified from the 3D point cloud data and determined by the unique shape of the object to each of the plurality of 2D images It can be equally included in the metadata of the object specified from

In addition, the assigning of the metadata may assign the same tracking identifier to the object specified from each of the plurality of 2D images. In this case, the tracking ID may be a unique identifier of an object assigned to track the object in the 2D images.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute the verification method as described above. The computer program includes a memory; transceiver; and a processor for processing instructions resident in the memory. And, in the computer program, in order for the processor to learn artificial intelligence (AI) that can be used for autonomous driving of the vehicle, specifying an object from each of a plurality of 2D images taken by a plurality of cameras fixedly installed in the vehicle ; specifying, by the processor, the same object as the object from 3D point cloud data continuously acquired by a lidar fixedly installed in the same vehicle as the vehicle in which the plurality of cameras are installed; identifying, by the processor, a movement path of the object based on a change in the position of the object specified from the 3D point cloud data; and assigning, by the processor, identical metadata to an object specified from each of the plurality of 2D images based on the identified movement path, a computer program recorded on a recording medium.

Specific details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, in order to learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle, each of a plurality of 2D images acquired through a plurality of cameras fixedly installed in a vehicle is not individually annotated. Instead, it is possible to organically annotate a plurality of 2D images based on 3D point cloud data acquired through lidar.

Through this, even if one object crosses the boundary between the two 2D images (ie, the boundary of the camera angle of view), it is possible to ensure that objects individually specified from the two 2D images can be specified as the same object.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 and 2 are block diagrams of an artificial intelligence learning system according to various embodiments of the present invention.
3 is a logical configuration diagram of an annotation device according to an embodiment of the present invention.
4 is a logical configuration diagram of an object specifying unit according to an embodiment of the present invention.
5 is a hardware configuration diagram of an annotation apparatus according to an embodiment of the present invention.
6 is an exemplary diagram illustrating a vehicle that collects data for artificial intelligence (AI) machine learning according to an embodiment of the present invention.
7 is an exemplary diagram for explaining a process of automatically specifying an object from 3D point cloud data according to an embodiment of the present invention.
8 is an exemplary diagram for explaining a process of improving the quality of 3D point cloud data according to an embodiment of the present invention.
9A to 9C are exemplary diagrams for explaining a process of tracking an object from 3D point cloud data according to an embodiment of the present invention.
10A to 10C are exemplary diagrams for explaining a process of tracking an object from 2D images according to an embodiment of the present invention.
11 is an exemplary diagram for explaining a process of performing synchronization between 3D point cloud data and a 2D image according to an embodiment of the present invention.
12 and 13 are exemplary diagrams for explaining a process of tracking an object crossing a boundary of an angle of view according to an embodiment of the present invention.
14 is an exemplary diagram for explaining a process of pre-processing a 2D image using 3D point cloud data according to an embodiment of the present invention.
15 is a flowchart illustrating a method of generating data for artificial intelligence (AI) learning according to an embodiment of the present invention.
16 is a flowchart illustrating an annotation method according to an embodiment of the present invention.

It should be noted that technical terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in this specification should be interpreted in the meaning generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise defined in this specification, and excessively inclusive. It should not be construed in the meaning of a human being or in an excessively reduced meaning. In addition, when the technical terms used in the present specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be understood by being replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “consisting of” or “having” should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are included. It should be construed that it may not, or may further include additional components or steps.

Also, terms including ordinal numbers such as first, second, etc. used herein may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but another component may exist in between. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents, or substitutes other than the accompanying drawings.

According to conventional means, in order to machine-learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle, a plurality of 3D point group data obtained through a lidar fixedly installed in a vehicle And, a lot of time and manpower was required in the process of processing a number of 2D images acquired through the cameras fixedly installed in the vehicle.

In order to overcome this limitation, the present invention intends to propose various means for more easily processing a plurality of 3D point cloud data and 2D images.

1 and 2 are block diagrams of an artificial intelligence learning system according to various embodiments of the present invention.

As shown in Figure 1, the artificial intelligence learning system according to an embodiment of the present invention is a learning data generating device 100, one or more annotation devices (200-1, 200-2, ..., 200-n; 200) , the learning data verification apparatus 300 and the artificial intelligence learning apparatus 400 may be included.

In addition, as shown in FIG. 2, the artificial intelligence learning system according to another embodiment of the present invention includes one or more annotation devices 200-a, 200-b, ..., 200-m; 200 and a learning data verification device ( 300-a, 300-b, …, 300-m; 300) a plurality of groups (Group-a, Group-b …, Group-m), a learning data generating apparatus 100 and an artificial intelligence learning apparatus ( 400) may be included.

As such, since the components of the AI learning system according to various embodiments are merely functionally distinct elements, two or more components are integrated with each other in the actual physical environment, or one component is the actual physical environment. may be implemented separately from each other.

Each component will be described. The learning data generating apparatus 100 is a device that can be used to design and generate data for machine learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

As such, the learning data generating apparatus 100 is basically a device distinct from the learning data verifying apparatus 300, but in an actual physical environment, the learning data generating apparatus 100 and the learning data verifying apparatus 300 are one device. It may be integrated and implemented.

Specifically, the learning data design apparatus 100 may receive the properties of the project related to artificial intelligence (AI) learning from the artificial intelligence learning apparatus 400 . The learning data design device 100 is based on the user's control and the properties of the received project, the design of the data structure for artificial intelligence (AI) learning, the purification of the collected data, the processing of the data, the expansion of the data, and the verification can be performed.

First, the learning data design apparatus 100 may design a data structure for artificial intelligence (AI) learning. For example, the learning data design apparatus 100 is based on the user's control and the properties of the received project, an ontology for artificial intelligence (AI) learning, a classification system of data for artificial intelligence (AI) learning can be defined.

The learning data design apparatus 100 may collect data for artificial intelligence (AI) learning based on the designed data structure. To this end, the learning data design apparatus 100 receives 3D point cloud data and 2D images from the outside, performs web crawling to collect 3D point cloud data and 2D images, or 3D point cloud from an external device. Data and 2D images can be downloaded.

Here, the 3D point cloud data is data acquired by a lidar fixedly installed in a vehicle. The lidar fixedly installed in the vehicle may generate 3D point cloud data corresponding to a 3D image of the vehicle's surroundings by emitting a laser pulse and detecting the light reflected back by objects located around the vehicle. That is, the point cloud constituting the 3D point cloud data refers to a set of points that reflect the laser pulse emitted by the lidar into the 3D space.

And, the 2D image is an image taken by a plurality of cameras fixedly installed in the vehicle. For autonomous driving, a plurality of cameras are fixedly installed in one vehicle, and two-dimensional images of the surroundings of the vehicle can be captured, respectively. For example, six cameras may be installed in one vehicle, but the present invention is not limited thereto.

The learning data design apparatus 100 may remove duplicate or extremely similar data from the collected 3D point cloud data and 2D images. The learning data design apparatus 100 may de-identify personal information included in the collected 3D point cloud data and 2D images.

The learning data design apparatus 100 may distribute and transmit the collected and refined 3D point cloud data and 2D images to the plurality of annotation apparatuses 200 . In this case, the learning data design apparatus 100 may distribute the 3D point cloud data and the 2D images according to an amount previously allocated to the operator (ie, the labeler) of the annotation apparatus 200 .

The learning data design apparatus 100 may receive an annotation operation result directly from the annotation apparatus 200 , or may receive an annotation operation result and an inspection result from the learning data verification apparatus 300 .

The learning data design apparatus 100 may generate data for artificial intelligence (AI) learning by packaging the received annotation work result. In addition, the learning data design apparatus 100 may transmit the generated artificial intelligence (AI) learning data to the artificial intelligence learning apparatus 400 .

The training data generating apparatus 100 having such a characteristic transmits and receives data to and from the annotation apparatus 200, the learning data verification apparatus 300, and the artificial intelligence learning apparatus 400, and performs an operation based on the transmitted and received data Any device that can do it is allowed. For example, the training data generating apparatus 100 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

With the following configuration, the annotation device 200 is a device that can be used to annotate 3D point cloud data and 2D images distributed by the training data generating device 100 .

Characteristically, the annotation apparatus 200 according to an embodiment of the present invention may provide various means for more easily processing 3D point cloud data and 2D images distributed by the learning data generating apparatus 100 .

Any device capable of transmitting and receiving data to and from the training data generating device 100 and the training data verifying device 300, and performing an operation based on the transmitted/received data, may use the annotation device 200 having these characteristics. may be allowed. For example, the annotation device 100 may be a desktop, a stationary computing device such as a workstation or a server, or a smart phone, a laptop, a tablet, a phablet, or a portable multimedia player. It may be any one of a portable computing device such as a Portable Multimedia Player (PMP), Personal Digital Assistants (PDA), or an E-book reader.

In addition, all or a part of the annotation apparatus 200 may be a device in which an annotator performs an annotation through a clouding service.

A detailed configuration and operation of the annotation apparatus 200 will be described later with reference to FIGS. 3 to 5 .

With the following configuration, the learning data verification device 300 is a device that can be used to verify data for artificial intelligence (AI) learning. That is, the learning data verification apparatus 300 verifies whether the annotation operation result generated by the annotation apparatus 200 meets a preset target quality or whether the annotation operation result is valid for artificial intelligence (AI) learning. It is a device that can

Specifically, the training data verification apparatus 300 may receive an annotation work result from the annotation apparatus 200 . Here, the annotation work result may include coordinates of an object specified from 3D point cloud data and 2D images, and metadata about the image or object. Metadata of the result of annotation work includes the specified object category (category), the rate at which the object is truncated by the angle of view of the 2D image (truncation), the rate at which the object is occluded by other objects or objects (occlusion), and the tracking ID (tracking) of the object. identifier), the time at which the image was captured, weather conditions on the day the image was captured, and the like, but is not limited thereto. Such an annotation operation result may have a JSON (Java Script Object Notation) file format, but is not limited thereto.

The training data verification apparatus 300 may inspect the received annotation work result. To this end, the learning data verification apparatus 300 may perform inspection using a script on the annotation work result. Here, the script is a code for verifying whether a preset target quality is met or whether data is valid with respect to the results of the annotation work.

In addition, the training data verification apparatus 300 may transmit the annotation work result and the inspection result received from the annotation apparatuses 200 to the training data generating apparatus 100 .

The training data verification apparatus 300, having the above-described characteristics, transmits/receives data to and from the annotation apparatus 200 and the training data generation apparatus 100, and any device capable of performing an operation based on the transmitted/received data. A device may be acceptable. For example, the training data verification apparatus 300 may be any one of a desktop, a workstation, or a stationary computing device such as a server, but is not limited thereto.

With the following configuration, the artificial intelligence learning device 400 is a device that can be used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

As such, the artificial intelligence learning apparatus 400 may be any device as long as it is capable of transmitting and receiving data to and from the learning data generating apparatus 100 and performing an operation using the transmitted/received data. For example, the artificial intelligence learning apparatus 400 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

As described above, the training data generating device 100 , one or more annotation devices 200 , the training data verification device 300 , and the artificial intelligence learning device 400 are a secure line directly connecting the devices and a public wired communication network. Alternatively, data may be transmitted/received using a combined network of one or more of the mobile communication networks.

For example, public wired networks include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). However, it is not limited thereto. In addition, the mobile communication network includes Code Division Multiple Access (CDMA), Wideband CDMA, WCDMA, High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile communication may be included, but are not limited thereto.

Hereinafter, the configuration of the annotation apparatus 200 as described above will be described in more detail.

3 is a logical configuration diagram of an annotation device according to an embodiment of the present invention.

As shown in FIG. 3 , the annotation apparatus 200 according to an embodiment of the present invention includes a communication unit 205 , an input/output unit 210 , a storage unit 215 , an object specifying unit 220 , and a metadata generation unit. 225 and a result generating unit 230 may be included.

As such, the components of the annotation apparatus 200 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is separated from each other in the actual physical environment. could be implemented.

When each component is described, the communication unit 205 may transmit/receive data to and from the learning data generating apparatus 100 and the learning data verifying apparatus 300 .

Specifically, the communication unit 205 may receive 3D point cloud data and 2D images from the learning data generating apparatus 100 . The communication unit 205 may individually receive the 3D point cloud data and the 2D images from the training data generating apparatus 100 , or may receive them collectively.

Here, the 3D point cloud data is data obtained by a lidar fixedly installed in a vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle. Such 3D point cloud data may be continuously acquired in time series by the lidar installed in the vehicle. In addition, the communication unit 205 may receive 3D point cloud data continuously acquired in time series.

And, the 2D image is an image taken by a plurality of cameras fixedly installed in the vehicle. Such 2D images may be continuously captured in time series by a camera installed in the vehicle. In addition, the communication unit 205 may receive 2D images continuously photographed in time series.

In addition, the communication unit 205 may transmit the annotation work result to the training data verification apparatus 300 . Such an annotation work result may have a JSON file format, but is not limited thereto.

With the following configuration, the input/output unit 210 may receive a signal from a user through a user interface (UI) or may output an operation result to the outside. Here, the user means a person who performs the annotation work. As such, a user may be referred to as an operator, performer, labeler, or data labeler, but is not limited thereto.

Specifically, the input/output unit 210 may output an image to be annotated. The input/output unit 110 may receive a control signal for setting a bounding box from a user. In addition, the input/output unit 110 may output the 3D point cloud data or the 2D image by overlaying the bounding box.

In addition, the input/output unit 110 may receive a control signal for setting metadata of an image from a user.

Specifically, the storage unit 215 may store 3D point cloud data and 2D images received through the communication unit 205 . Also, the storage unit 215 may store the properties of the project, the properties of the image, or the properties of the user received through the communication unit 205 .

With the following configuration, the object specifying unit 220 may specify an object to be subjected to machine learning of artificial intelligence (AI) with respect to each of the 3D point cloud data and the 2D image that is the object of the annotation operation.

Characteristically, the object specifying unit 220 according to an embodiment of the present invention may provide various means for more easily performing annotations on 3D point cloud data and 2D images.

Such a detailed configuration and operation of the object specifying unit 220 will be described later with reference to FIG. 4 .

With the following configuration, it is possible to generate metadata for 3D point cloud data and 2D images to be annotated.

Here, the metadata is 3D point cloud data or 2D image and data for describing an object specified from the 3D point cloud data or 2D image. As such, metadata includes 3D point cloud data or a category of an object specified from a 2D image, the ratio of the object being cut by the angle of view of the 2D image, the ratio of the object being obscured by other objects or objects, the tracking ID of the object, the image taken Time, weather conditions on the day the image was taken, etc. may include, but are not limited to, file size, image size, copyright holder, resolution, bit value, aperture transmission, exposure time, ISO sensitivity, focal length, aperture opening value, Angle of view, white balance, RGB depth, class name, tag, shooting location, type of road, road surface information, or traffic jam information may be further included.

The metadata generator 225 may basically set specific values to be included in the metadata according to a control signal input from the user through the input/output unit 210 .

Characteristically, metadata may be preset for the 2D images preprocessed by the object specifying unit 220 according to an embodiment of the present invention, and the metadata generating unit 225 may simply modify the preset metadata. Alternatively, it may only perform the role of verification.

With the following configuration, the result generating unit 230 may generate an annotation work result.

Specifically, the result generating unit 230 performs annotations based on the coordinates of the object specified from the 3D point cloud data and 2D images by the object specifying unit 220 and the metadata generated by the metadata generating unit 225 . You can create work results. In this case, the annotation work result may have a JSON file format, but is not limited thereto.

In addition, the result generating unit 230 may directly transmit the generated annotation work result to the learning data generating apparatus 100 through the communication unit 205 , or may transmit it to the learning data verifying apparatus 300 .

Hereinafter, as described above, the configuration of the object specifying unit 220 constituting the annotation apparatus 200 according to an embodiment of the present invention will be described in more detail.

4 is a logical configuration diagram of an object specifying unit according to an embodiment of the present invention.

As shown in FIG. 4 , the object specifying unit 220 according to an embodiment of the present invention includes a 3D data object specifying module 220-1, a 2D image object specifying module 220-2, and a quality management module 220 -3), an object automatic tracking module 220-4, a synchronization processing module 220-5, and a pre-processing module 220-6 may be included.

Since the components of the object specifying unit 220 are merely functionally distinct elements, two or more components are integrated with each other in the actual physical environment, or one component is separated from each other in the actual physical environment. and can be implemented.

When each component is described, the 3D data object specifying module 220 - 1 may specify an object from 3D point cloud data.

Specifically, the 3D data object specifying module 220 - 1 may receive 3D point cloud data through the communication unit 205 . Here, the 3D point cloud data is data obtained by a lidar fixedly installed in a vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Basically, the 3D data object specifying module 220 - 1 may receive six coordinates according to a control signal input through the input/output unit 210 . The 3D data object specifying module 220-1 may specify an object included in the 3D point cloud data by setting a 3D bounding box based on six input coordinates.

Here, the 3D bounding box is an area for specifying an object to be the target of the artificial intelligence (AI) learning from among the objects included in the 3D point cloud data. As such, the 3D bounding box may have a shape of a hexahedron, but is not limited thereto.

For example, the 3D data object specifying module 220-1 receives six coordinates from the user through the input/output unit 210, and a hexahedral 3D bounding box having the six coordinates as vertex coordinates. can be set in the 3D point cloud data.

In this case, the six coordinates are set by the user inputting one type of input signal six times (eg, clicking a mouse or inputting a key on the keyboard), or the user inputs two types of input signals once (eg, may be set by dragging the mouse or inputting two keys on the keyboard), but is not limited thereto.

Characteristically, the 3D data object specifying module 220 - 1 may automatically specify an object from the 3D point cloud data.

Specifically, the 3D data object specification module 220-1 configures the X-axis (ie, the width of the 3D point cloud data), the Y-axis (ie, the height of the 3D point cloud data) and the Z-axis (ie, the 3D point cloud data) of the received 3D point cloud data. In the depth of the data), three 2D images with one axis fixed can be identified.

For example, the 3D data object specifying module 220-1 provides a side view image consisting of only data of the Y and Z axes in which the X axis is fixed in the 3D point cloud data, and the Y axis is fixed in the 3D point cloud data. It is possible to identify a top view image composed of only the X-axis and Z-axis data, and a front view image composed only of the data of the X-axis and the Y-axis in which the Z-axis is fixed in the three-dimensional point cloud data.

The 3D data object specifying module 220-1 may identify a set of some points included in the image with respect to each of the three identified 2D images. In addition, the 3D data object specifying module 220-1 may set a 2D temporary bounding box to each of the three 2D images so that the identified set of some points is included.

Here, the 2D temporary bounding box is not an area for directly specifying an object for artificial intelligence (AI) learning. , an area for temporarily specifying a two-dimensional surface of an object from a front view image and a side view image).

For example, the 3D data object specifying module 220-1 may perform the following for each of a planar view image, a front view image, and a side view image.

First, the 3D data object specification module 220-1 sets one point located at one end as a reference point for points included in the planar view image, and a preset adjacent distance on the X-axis based on the reference point Search for adjacent points located within. The 3D data object specific module 220-1 performs a process of recursively searching for a new adjacent point located within the adjacent distance on the X-axis by setting the found adjacent point as a new reference point when the adjacent point is searched. can

The 3D data object specifying module 220-1 searches for an adjacent point located within the adjacent distance on the Z-axis based on the point where the adjacent point is not searched when the adjacent point located within the adjacent distance on the X-axis is not searched. When an adjacent point is found on the Z-axis, the 3D data object specifying module 220-1 may recursively perform a process of searching for the adjacent point on the Z-axis by setting the found adjacent point as a new reference point.

The 3D data object specification module 220-1 is configured to include both the reference point and the adjacent point found on the X-axis and the reference point and the adjacent point found on the Z-axis when the adjacent point located within the adjacent distance on the Z-axis is not searched. You can set a 2D temporary bounding box within the viewpoint image.

In the same manner as in the process of setting the 2D temporary bounding box in the planar view image as described above, the 3D data object specifying module 220-1 targets the points included in the side view image in the Y axis and the Z axis respectively. A 2D temporary bounding box may be set in the side view image to recursively perform the setting and search for adjacent points, and to include both reference points and adjacent points found on the Y-axis and Z-axis, respectively.

In addition, the 3D data object specifying module 220-1 recursively performs a process of setting a reference point and searching for adjacent points on the X-axis and Y-axis, respectively, for points included in the front view image, and the X-axis and Y-axis A 2D temporary bounding box can be set in the front view image so that both the reference point and the adjacent point searched for each axis are included.

As described above, after each setting of the 2D temporary bounding box, the 3D data object specifying module 220 - 1 moves the 2D temporary bounding box in parallel in response to a direction indication input by the user through the input/output unit 210 . (parallel transference) is possible.

For example, the 3D object specifying module 220-1 may move a 2D temporary bounding box set in a planar view image in parallel on the X-axis in response to a direction indication input from a user, or 2D temporary bounding set in a side view image. The box can be translated on the Z-axis, or the 2D temporary bounding box set in the front view image can be translated on the Y-axis.

And, the 3D data object specifying module 220-1 uses the 2D temporary bounding box set for each of the three 2D images (that is, a plane view image, a side view image, and a front view image), which is included in the 3D point cloud data. You can automatically set a 3D bounding box to specify an object.

For example, in the 3D data object specification module 220-1, the 2D temporary bounding box set in the planar view image and the 2D temporary bounding box set in the side view image are superimposed on each other in the Z-axis, and the 2D temporary bounding box set in the planar view image is superimposed on each other. When the box and the 2D temporary bounding box set in the front view image overlap each other on the X axis, and the 2D temporary bounding box set in the side view image and the 2D temporary bounding box set in the front view image overlap each other on the Y axis, the plane view image The 2D temporary bounding box set in the 2D temporary bounding box, the 2D temporary bounding box set in the side view image, and the 2D temporary bounding box set in the front view image are included as a two-dimensional plane constituting a 3D bounding box for specifying an object in the 3D point cloud data. can

In this case, the 3D bounding box has the 2D temporary bounding box set in the plane view image as a plane (top) and bottom (bottom), and the 2D temporary bounding box set in the side view image is left lateral, right lateral ), and has a hexahedron shape with the 2D temporary bounding box set in the front view image as the front and rear.

With the following configuration, the 2D image object specifying module 220 - 2 may specify an object from the 2D image.

Specifically, the 2D image object specifying module 220 - 2 may receive a plurality of 2D images through the communication unit 205 . Here, the 2D image is an image taken by a plurality of cameras fixedly installed in the vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of the vehicle.

Basically, the 2D data object specifying module 220 - 2 may receive a plurality of coordinates according to a control signal input through the input/output unit 210 . The 2D data object specifying module 220 - 2 may specify an object included in the 2D image by setting a 2D bounding box based on a plurality of input coordinates.

Here, the 2D bounding box is an area for specifying an object to be the target of the artificial intelligence (AI) learning from among the objects included in the 2D image. Such a 2D bounding box set in a 2D image may have a three-dimensional hexahedral shape having six vertices, or a two-dimensional rectangular shape having four vertices, depending on the implementation method of the present invention. and is not limited thereto.

Characteristically, the 2D data object specifying module 220-2 receives the 2D bounding box pre-arranged by the object automatic tracking module 220-4 or the pre-processing module 220-6 through the input/output unit 210. It can also be simply modified according to the control signal.

With the following configuration, the quality management module 220 - 3 may improve the quality of continuously acquired 3D point cloud data.

Specifically, the quality management module 220 - 3 specifies an object included in the 3D point cloud data through the 3D data object specifying module 220 - 1 for each of the 3D point cloud data received through the communication unit 205 . can do. Here, the 3D point cloud data is data continuously acquired according to a time series by a lidar that is fixedly installed in a vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

The quality management module 220 - 3 may identify a plurality of 3D point cloud data including the same object from among 3D point cloud data, based on the specified object.

For example, the quality management module 220 - 3 compares the tracking IDs provided by the meta data generator 225 with respect to the objects included in each of the 3D point cloud data with each other to obtain a plurality of objects including the same object. 3D point cloud data can be identified. In this case, the tracking ID may be a unique identifier of an object assigned to track the object in 3D point cloud data and 2D images.

The quality management module 220 - 3 may compare the qualities of the same objects included in a plurality of 3D point cloud data including the same object with each other, and select one object having the best quality.

For example, the quality management module 220-3 calculates a volume occupied by objects specified as each of the 3D point cloud data in the 3D point cloud data, and selects one object having the largest calculated volume. can In this case, the quality control module 220-3 multiplies the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis with respect to the 3D bounding box set to specify the object. volume can be calculated. When there are a plurality of objects with the largest calculated volume, the quality management module 220-3 calculates the number of points constituting each of the plurality of objects, and selects one object having the largest number of points. can

When selecting one object based on the calculated volume, the quality management module 220 - 3 may preferentially compare volumes of objects having a minimum occlusion ratio of the object to another object.

The quality control module 220-3 specifies an object to which the same tracking ID as the object specified from the 3D point cloud data is assigned in the 2D image taken at the same time as the time at which the 3D point cloud data was acquired, and specified within the 2D image. It is also possible to preferentially compare the volumes of the objects whose truncation is the minimum by the angle of view of the 2D image.

In addition, the quality management module 220-3 includes a plurality of 2D images captured at the same time as the time at which the 3D point cloud data was acquired, and the first 2D image and the second 2D image among the plurality of 2D images are different from each other of the same object. When an object having a cropped ratio of 0 can be derived by combining two parts of the object included in the first 2D image and the second 2D image, respectively, the object derived by combining the two parts Based on volume, one object with the highest quality can be selected.

To this end, the quality control module 220-3 compares the hue, brightness, and gamma of two parts included in the first 2D image and the second 2D image, respectively, to obtain the first 2D It may be verified whether two parts included in the image and the second 2D image are different parts of the same object.

In addition, the quality management module 220-3 configures one object selected as having the best quality with respect to points constituting the same object included in a plurality of 3D point cloud data including the same object, respectively. points can be replaced.

With the following configuration, the object automatic tracking module 220 - 4 may track the same object from continuously acquired 3D point cloud data or continuously acquired 2D images.

First, a process in which the automatic object tracking module 220 - 4 tracks the same object from continuously acquired 3D point cloud data will be described.

The object automatic tracking module 220-4 is configured for each of the first 3D point cloud data, the second 3D point cloud data, and the third 3D point cloud data, one included in the 3D point cloud data through the 3D data object specifying module 220-1. More than one object can be specified.

Here, the first 3D point cloud data, the second 3D point cloud data, and the third 3D point cloud data are successively in time series through the lidar installed in the vehicle to learn artificial intelligence (AI) that can be used for autonomous driving of the vehicle. It is 3D point cloud data included in the acquired data. In addition, the first 3D point cloud data may be data acquired chronologically prior to the second 3D point cloud data, and the second 3D point cloud data may be data acquired chronologically preceding the third 3D point cloud data.

The object automatic tracking module 220 - 4 may identify a common object commonly included in the first 3D point cloud data and the second 3D point cloud data.

For example, the object automatic tracking module 220-4 compares the tracking IDs given to the objects included in each of the first 3D point cloud data and the second 3D point cloud data with each other, so that the objects to which the same tracking ID is assigned are shared. It can be identified as an object.

The object automatic tracking module 220 - 4 may calculate the movement speed of the identified common object.

For example, the automatic object tracking module 220 - 4 may determine the separation distance between the common object specified from the first 3D point cloud data and the common object specified from the second 3D point cloud data, and the acquisition time of the first 3D point cloud data. The moving speed of the common object may be calculated by using the separation time between the acquisition time of the second 3D point cloud data and the 3D point cloud data.

The object automatic tracking module 220 - 4 may identify an area in which the common object is estimated to be located in the third 3D point cloud data, based on the calculated moving speed.

The object automatic tracking module 220 - 4 may verify whether a common object actually exists in an area where the common object is estimated to be located in the third 3D point cloud data. More specifically, the object automatic tracking module 220-4 determines the area occupied by the common object in the first 2D image captured at the same time as the time at which the first 3D point cloud data was obtained, and the area at which the third point cloud data was obtained. It is possible to verify whether an object corresponding to the common object actually exists in the area where the common object is estimated to be located by comparing the areas where the common object is estimated to be located in the third 2D image taken at the same time as the time. have. In this case, the first 2D image and the third 2D image may be images taken by a camera fixedly installed in the same vehicle as the vehicle in which the lidar used to obtain the first 3D point cloud data and the third 3D point cloud data is installed. have.

For example, the object automatic tracking module 220-4 predicts that the RGB (Red, Green, Blue) pattern of the area occupied by the common object in the first 2D image and that the common object will be located in the third 2D image Based on a result of comparing the RGB patterns of the estimated area with each other, it may be verified whether an object corresponding to the common object exists in the area where the common object is estimated to be located. In this case, the RGB pattern may be a pattern for the ratio of the RGB values to the pixels in the region and the arrangement order of the RGB values.

For example, the automatic object tracking module 220 - 4 determines the edge pattern of the area occupied by the common object in the first 2D image, and the area where the common object is estimated to be located in the third 2D image. Based on the result of comparing the edge patterns of , it may be verified whether an object corresponding to the common object exists in the region where the common object is estimated to be located. In this case, the edge pattern may be a pattern for the number of pixels included in an enclosure closed by the edge within the area and a relative position where the edge is located within the area.

The object automatic tracking module 220-4 is configured to correspond to an area where the common object is estimated to be located among objects specified from the third 3D point cloud data, based on the location and size of the area where the common object is estimated to be located. The position and size of a preset 3D bounding box with respect to an object (that is, it may be a common object) may be readjusted.

In addition, the object automatic tracking module 220 - 4 provides a common object (that is, may be a common object) corresponding to an area in which the common object is estimated to be located among objects specified from the third 3D point cloud data. Pre-set metadata can be assigned to an object. In this case, the object automatic tracking module 220 - 4 may assign a size to an object (that is, may be a common object) corresponding to an area where the common object is estimated to be located in the third 3D point cloud data. .

If the distance between the observer and the object is changed, the relative size of the object observed by the observer will change. However, the object automatic tracking module 220-4 determines the absolute size determined by the unique shape of the common object regardless of the distance the common object is spaced apart from the vehicle in which the lidar is fixedly installed, the area in which the common object is estimated to be located. It can be given as the size of the object corresponding to .

Next, a process in which the object automatic tracking module 220 - 4 tracks the same object from successively acquired 2D images will be described.

The object automatic tracking module 220-4 specifies one or more objects included in the 2D image through the 2D data object specifying module 220-2 for each of the first 2D image, the second 2D image, and the third 2D image. can do.

Here, the first 2D image, the second 2D image, and the third 2D image are images continuously taken in a time series through a camera fixedly installed in the vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of the vehicle. 2D image included in In addition, the first 2D image may be an image photographed chronologically before the second 2D image, and the second 2D image may be an image photographed chronologically before the third 2D image.

The object automatic tracking module 220 - 4 may identify a common object commonly included in the first 2D image and the second image.

For example, the object automatic tracking module 220-4 compares the tracking IDs given to the objects included in each of the first 2D image and the second 2D image with each other, and sets the object to which the same tracking ID is assigned as a common object. can be identified.

The object automatic tracking module 220 - 4 may calculate the movement speed of the identified common object.

For example, if the 2D bounding box set in the 2D image has a three-dimensional hexahedral shape, the object automatic tracking module 220-4 sets the volume of the bounding box set to specify a common object in the first 2D image and the second A separation distance between the common object specified from the first 2D image and the common object specified from the second 2D image may be estimated by comparing the volume of the bounding box set to specify the common object in the 2D image.

Alternatively, when the 2D bounding box set in the 2D image has a two-dimensional rectangular shape, the object automatic tracking module 220-4 sets the width of the bounding box set to specify a common object in the first 2D image and the second 2D The separation distance between the common object specified from the first 2D image and the common object specified from the second 2D image may be estimated by comparing the width of a bounding box set to specify the common object in the image.

The object automatic tracking module 220 - 4 may calculate a separation time between the photographing time of the first image and the photographing time of the second image, and calculate the moving speed of the common object using the separation distance and the separation time.

The object automatic tracking module 220 - 4 may identify an area where the common object is estimated to be located in the third 2D image, based on the calculated moving speed.

The object automatic tracking module 220 - 4 may verify whether a common object actually exists in an area where the common object is estimated to be located in the third 2D image. More specifically, the object automatic tracking module 220-4 is configured to include an area occupied by a common object in the first 3D point cloud data acquired at the same time as the time at which the first 2D image was captured, and the area at which the third 2D image was captured. By comparing regions where the common object is estimated to be located in the third 3D point cloud data acquired at the same time as the time, it is possible to verify whether an object corresponding to the common object actually exists in the region where the common object is estimated to be located. can In this case, the first 3D point cloud data and the third 3D point cloud data may be point cloud data acquired by a lidar fixedly installed in the same vehicle as the vehicle in which the camera used to photograph the first image and the third image is installed.

For example, the object automatic tracking module 220-4 determines the number of points included in the 3D bounding box set to specify the common object in the first 3D point cloud data, and the location of the common object in the third 3D point cloud data. Based on whether the difference in the number of points included in the area estimated to be performed is included within a preset quality change range, it can be verified whether an object corresponding to the common object exists in the area where the common object is estimated to be located. have.

For another example, the object automatic tracking module 220-4 may include an RGB pattern of an area occupied by a common object in the first 2D image and an RGB pattern of an area where a common object is estimated to be located in the third 2D image. Based on the result of comparing the , it may be verified whether an object corresponding to the common object exists in the region where the common object is estimated to be located.

As another example, the automatic object tracking module 220-4 may include an edge pattern of an area occupied by a common object in the first 2D image, and an edge of an area where the common object is estimated to be located in the third 2D image. Based on a result of comparing the patterns with each other, it may be verified whether an object corresponding to the common object exists in an area where the common object is estimated to be located.

Object automatic tracking module 220-4 is based on the location and size of the area where the common object is estimated to be located, the object corresponding to the area where the common object is estimated to be located among the objects specified from the third image ( That is, it is possible to readjust the position and size of the preset 2D bounding box with respect to (which may be a common object).

In addition, the object automatic tracking module 220-4 is configured to, with respect to an object (ie, may be a common object) corresponding to an area in which the common object is estimated to be located, among objects specified from the third 2D image, the common object. Pre-specified metadata can be given to . In this case, the object automatic tracking module 220-4 determines the unique shape of the common object with respect to the object (that is, may be a common object) corresponding to the region where the common object is estimated to be located in the third image. can be given an absolute size determined by

With the following configuration, the synchronization processing module 220 - 5 may synchronize the object specified from the 3D point cloud data of the lidar fixedly installed in the same vehicle and the object specified from the 2D image of the camera with each other.

Specifically, the synchronization processing module 220 - 5 may specify an object included in the 3D point cloud data through the 3D data object specifying module 220 - 1 with respect to the 3D point cloud data. Here, the 3D point cloud data is data continuously acquired according to a time series by a lidar that is fixedly installed in a vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

The synchronization processing module 220 - 5 may specify the same object as the object specified from the 3D point cloud data from the 2D image. In this case, the 2D image is an image taken at the same time as the time at which the 3D point cloud data was obtained through a camera fixedly installed in the same vehicle as the vehicle in which the lidar obtained the 3D point cloud data was acquired.

The synchronization processing module 220 - 5 may determine whether the object specified from the 3D point cloud data and the object specified from the 2D image correspond to each other.

For example, the synchronization processing module 220-5 rotates the 3D bounding box three-dimensionally so that the observation direction of the 3D bounding box corresponds to the optical axis direction of the camera photographing the 2D image, It may be determined whether the shapes of the 3D rotated 3D bounding box and the 2D bounding box correspond to each other. In this case, both the 3D bounding box and the 2D bounding box may have a hexahedral shape.

The synchronization processing module 220-5 adjusts the position or size of the 3D bounding box set in the 3D point cloud data to specify the same object so that the object specified from the 3D point cloud data and the object specified from the 2D image correspond to each other. , or it is possible to adjust the position or size of the 2D bounding box set in the 2D image to specify the same object.

For example, the synchronization processing module 220-5 determines the position of the border line of the 2D bounding box set in the 2D image to specify the same object based on the depth value included in the 3D point cloud data. can be readjusted. Here, the depth value may be a value indicating a distance from the lidar to the object reflecting the laser pulse emitted from the lidar.

In this case, the synchronization processing module 220-5 identifies an inflection line in which the depth value included in the 3D point cloud data changes beyond a preset threshold range, and specifies the same object in the 2D image. The position of the boundary line of the 2D bounding box may be readjusted so that the boundary line of the set 2D bounding box corresponds to the identified inflection line.

As another example, the synchronization processing module 220 - 5 may readjust the position of the boundary line of the 3D bounding box set to specify the same object based on the RGB values included in the 2D image. In addition, the synchronization processing module 220-5 detects an edge included in the 2D image, and the boundary line of the 3D bounding box set in the 3D point cloud data to specify the same object corresponds to the detected edge from the 2D image. It is also possible to readjust the position of the boundary line of the 3D bounding box to do so.

As another example, the synchronization processing module 220-5 is configured to: When the volume of the 3D bounding box for specifying the same object is smaller than the volume of the 2D bounding box for specifying the same object, the 2D bounding box The volume of the 2D bounding box may be reduced such that the volume of the 2D bounding box becomes the same as the volume of the 3D bounding box.

In this case, the synchronization processing module 220-5 converts each of the line segments constituting the 2D bounding box within the size of the chroma subsampling applied in the process of compressing the 2D image to a file of the 2D bounding box. The volume of the 2D bounding box can be reduced by moving it inward.

In addition, the synchronization processing module 220 - 5 may temporarily store the adjusted position or size value after the position or size of the 3D bounding box or the position or size of the 2D bounding box is adjusted. In this way, the temporarily stored position or size value may be applied to 3D point cloud data or 2D image and subsequent 3D point cloud data or 2D image continuously acquired or photographed in time series.

With the following configuration, the pre-processing module 220 - 6 may pre-process the 2D images by tracking the same object irrespective of the angles of view of cameras that have photographed the 2D images based on the continuously acquired 3D point cloud data.

First, a process in which the pre-processing module 220 - 6 tracks the same object regardless of the angle of view of the cameras that have respectively captured the 2D images will be described.

The pre-processing module 220 - 6 may specify an object included in the 2D image through the 2D data object specifying module 220 - 2 for each of the plurality of 2D images received through the communication unit 205 . Here, the plurality of 2D images are images taken by a plurality of cameras fixedly installed in the vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of the vehicle.

The preprocessing module 220 - 6 may specify the same object as the object specified from the 2D image from the 3D point cloud data. Here, the 3D point cloud data is data continuously acquired in a time series by a lidar fixedly installed in the same vehicle as a vehicle in which a plurality of cameras are installed for capturing a 2D image.

For example, the pre-processing module 220 - 6 may arrange a plurality of 2D images in a circle according to the angle of the optical axis of the camera so that the vehicle in which the lidar and the camera are installed becomes the center point of a virtual circle. The preprocessing module 220 - 6 may acquire a top view image composed of only X-axis and Z-axis data from the 3D point cloud data. The pre-processing module 220 - 6 may specify the same object by matching a plurality of circularly arranged 2D images with the plane view image.

The preprocessing module 220 - 6 may identify the movement path of the same object based on the change in the position of the same object specified from the 3D point cloud data.

In more detail, the pre-processing module 220 - 6 determines the movement path of the same object when two 2D images obtained by photographing spatially continuous angles of view from among a plurality of 2D images each include different parts of one object. can be identified.

To this end, the pre-processing module 220-6 calculates the sum of the ratio truncated by the angle of view of the first 2D image and the ratio truncated by the angle of view of the second image among two 2D images obtained by photographing spatially continuous angles of view. When it is equal to or greater than a preset threshold ratio, it may be determined that the two 2D images each include different parts of one object.

The pre-processing module 220-6 is configured to, when the sum of the ratio of the cropped by the angle of view of the first 2D image and the ratio cropped by the angle of view of the second image is equal to or greater than a threshold ratio, the RGB value of the object specified from the first 2D image and the second Based on the RGB value of the object specified from the 2D image, it may be verified whether the two 2D images each include different parts of one object.

Alternatively, the pre-processing module 220-6 is configured to perform a process between the first 2D image and the second 2D image when the sum of the ratio of the cropped by the angle of view of the first 2D image and the ratio of the cropped by the angle of view of the second image is equal to or greater than a threshold ratio. Based on the boundary line, based on whether the edge of the object specified from the first 2D image and the edge of the object specified from the second 2D image are continuous, the two 2D images each include different parts of one object, It can be verified whether or not

In addition, the preprocessing module 220-6 is configured to move the same object when an object specified from the 3D point cloud data crosses the boundary line between two 2D images obtained by taking spatially continuous angles of view among the plurality of 2D images. A path can also be identified.

In addition, the preprocessing module 220 - 6 may assign the same metadata to the object specified from each of the plurality of 2D images based on the identified movement path of the same object.

For example, the preprocessing module 220 - 6 may assign the same tracking ID to an object specified from each of a plurality of 2D images. In addition, the preprocessing module 220-6 is identified from 3D point cloud data, not the relative size occupied by the object in the 2D image, and the absolute size determined by the unique shape of the object is specified from each of the plurality of 2D images. The same can be included in the object's metadata.

Next, a process of preprocessing the 2D images by the preprocessing module 220-6 will be described.

The pre-processing module 220 - 6 may specify an object included in each of the 3D point cloud data according to a time sequence obtained by the lidar with respect to each of the 3D point cloud data received through the communication unit 205 . . Here, the 3D point cloud data is data continuously acquired according to a time series by a lidar that is fixedly installed in a vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

If no object is included in all of the 3D point cloud data, the preprocessing module 220 - 6 converts the 2D images captured by a plurality of cameras during the time the 3D point cloud data was acquired by the lidar to the specific object. can be excluded from the subject.

The preprocessing module 220 - 6 may identify the movement path of the object based on the change in the position of the object specified according to the time sequence obtained by the lidar.

For example, the preprocessing module 220-6 acquires plane view images composed of only X-axis and Z-axis data from the 3D point cloud data, and connects points where the object is located in each of the plane viewpoint images to move the object. can be identified. In this case, the X-axis data may be width-related data among the 3D point cloud data, and the Z-axis data may be depth-related data among the 3D point cloud data.

The preprocessing module 220-6 uses the movement path of the identified object, and the same object as the object specified from the 3D point cloud data from each of the 2D images successively photographed by a plurality of cameras fixedly installed in the vehicle in which the lidar is installed. can be specified.

In more detail, the pre-processing module 220 - 6 may exclude one or more 2D images that do not match the movement path of the object from among the 2D images continuously photographed by a plurality of cameras from the specific target of the object.

The preprocessing module 220-6 specifies an object of the same type as the object specified from the 3D point cloud data in the 2D image matching the movement path of the object from among the 2D images continuously photographed by the plurality of cameras in the 2D image. You can place a 2D bounding box in advance for

To this end, the pre-processing module 220-6 generates a 2D bounding box in the 2D image based on the optical axis direction of the camera that captured the 2D image matching the movement path of the object and the position at which the object is specified in the 3D point cloud data. You can decide where to place it in advance.

When a signal indicating that the adjustment of the vertex position of the 2D bounding box has been completed is inputted through the user interface (UI) of the input/output unit 210, the preprocessing module 220-6 gives an object specified from the 3D point cloud data. The specified metadata can be given as metadata of the object specified by the 2D bounding box.

For example, the preprocessing module 220 - 6 may assign a tracking ID previously assigned to an object specified from the 3D point cloud data as a tracking ID of the object specified by the 2D bounding box. In addition, the pre-processing module 220 - 6 may include the absolute size identified from the 3D point cloud data and determined by the unique shape of the object in the metadata of the object specified by the 2D bounding box.

Hereinafter, hardware for implementing the logical components of the annotation apparatus 200 as described above will be described in more detail.

5 is a hardware configuration diagram of an annotation apparatus according to an embodiment of the present invention.

5, the annotation device 200 includes a processor 250, a memory 255, a transceiver 260, an input/output device 265, and a data bus. , 270) and may be configured to include a storage (Storage, 275).

The processor 250 may implement the operations and functions of the annotation device 200 based on an instruction according to the software 280a in which the method according to the embodiments of the present invention is implemented residing in the memory 255 . The memory 255 may be loaded with software 280a in which methods according to embodiments of the present invention are implemented. The transceiver 260 may transmit/receive data to and from the training data generating apparatus 100 and the training data verifying apparatus 300 . The input/output device 265 may receive data necessary for the operation of the annotation device 200 , and may output an image, a bounding box, and metadata to be annotated. The data bus 270 is connected to the processor 250 , the memory 255 , the transceiver 260 , the input/output device 265 , and the storage 275 . can play a role.

The storage 275 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 280a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 275 may store the software 280b in which the method according to the embodiments of the present invention is implemented. Also, the storage 275 may store information necessary for performing the method according to the embodiments of the present invention.

According to an embodiment of the present invention, the software 280a and 280b for implementing the automatic object specification method resident in the memory 255 or stored in the storage 275 may be used by the processor 250 for autonomous driving of the vehicle. Receiving 3D point cloud data obtained by a lidar fixedly installed in a vehicle through the transceiver in order to learn artificial intelligence (AI), the processor 250 is X-axis and Y-axis of the received 3D point cloud data and identifying three 2D images in which one of the Z-axis is fixed. setting each 2D temporary bounding box to include a set of , and specifying an object included in the 3D point cloud data by the processor 250 using the 2D temporary bounding box set for each of the three 2D images In order to execute the step of setting the 3D bounding box for

According to another embodiment of the present invention, the software 280a and 280b for implementing the object quality improvement method resident in the memory 255 or stored in the storage 275 may be used by the processor 250 for autonomous driving of the vehicle. Specifying an object included in the 3D point cloud data for each of the 3D point cloud data continuously acquired through the lidar fixedly installed in the vehicle in order to learn the artificial intelligence (AI) in the vehicle, the processor 250 is the identifying a plurality of 3D point cloud data including the same object from among the 3D point cloud data based on the specified object; comparing and selecting one object, and replacing, by the processor 250, points constituting the same object included in the plurality of 3D point cloud data with points constituting the selected one object In order to execute the , it may be a computer program recorded on a recording medium.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the object tracking method in the continuous 3D data resident in the memory 255 or stored in the storage 275 is the processor 250 is the first 3D specifying, for each of the point cloud data, the second 3D point cloud data, and the third 3D point cloud data, one or more objects included in the 3D point cloud data, by the processor 250 , the first 3D point cloud data and the second 3D point cloud data Identifying a common object included in common, the processor 250 calculating the movement speed of the identified common object, the processor 250 based on the calculated movement speed, the third identifying an area in which the common object is estimated to be located in 3D point cloud data, and the processor 250 is located in an area where the common object is estimated to be located among the objects specified from the third 3D point cloud data. In order to execute the step of assigning metadata previously assigned to the common object to the corresponding object, it may be a computer program recorded on a recording medium.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the object tracking method in the continuous 2D data resident in the memory 255 or stored in the storage 275 is the processor 250 is the first 2D specifying, for each of the image, the second 2D image, and the third 2D image, one or more objects included in the 2D image, by the processor 250 being commonly included in the first 2D image and the second image Identifying the object, the processor 250 calculating the movement speed of the identified common object, the processor 250 based on the calculated movement speed, the common object in the third 2D image identifying an area in which the common object is estimated to be located; It may be a computer program recorded on a recording medium in order to execute the step of assigning previously assigned metadata to the .

According to another embodiment of the present invention, the software 280a and 280b for implementing a method of synchronizing 3D data and 2D images residing in the memory 255 or stored in the storage 275 is performed by the processor 250 autonomously of the vehicle. Specifying an object from 3D point cloud data obtained through a lidar fixedly installed in a vehicle in order to learn artificial intelligence (AI) that can be used for driving, wherein the processor 250 is a camera fixedly installed in the vehicle in which the lidar is installed through, specifying the same object as the object from a 2D image taken at the same time as the time at which the 3D point cloud data was obtained, and the processor 250 from the 2D image and the object specified from the 3D point cloud data Adjusting the position or size of a bounding box set in the 3D point cloud data to specify the object, or the position or size of a bounding box set in the 2D image to specify the object, so that the specified objects can correspond to each other In order to execute the , it may be a computer program recorded on a recording medium.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the object tracking method that crosses the boundary of the angle of view resident in the memory 255 or stored in the storage 275 is the processor 250 is the vehicle. In order to learn artificial intelligence (AI) that can be used for autonomous driving, specifying an object from each of a plurality of 2D images taken by a plurality of cameras fixedly installed in a vehicle, the processor 250 is the plurality of cameras specifying, by the processor 250, the same object as the object from 3D point cloud data continuously acquired by the lidar fixedly installed in the same vehicle as the vehicle in which , Identifying a movement path of the object based on In order to do this, it may be a computer program recorded on a recording medium.

And, according to another embodiment of the present invention, the software (280a, 280b) for implementing a method of pre-processing a 2D image using 3D data resident in the memory 255 or stored in the storage 275 is the processor 250 In order to learn artificial intelligence (AI) that can be used for autonomous driving of a vehicle, receiving 3D point cloud data continuously acquired by a lidar fixedly installed in a vehicle through the transceiver, the processor 250 is the specifying an object included in each of the 3D point cloud data according to the temporal sequence obtained by the lidar; based on the change in the position of the specified object according to the temporal sequence, by the processor 250 Identifying a movement path, and the processor 250 using the movement path of the identified object, each of the 2D images successively photographed by a plurality of cameras fixedly installed in the same vehicle as the vehicle in which the lidar is installed In order to execute the step of specifying the same object as the object, it may be a computer program recorded on a recording medium.

More specifically, the processor 250 may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 255 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 260 may include a baseband circuit for processing wired and wireless signals. The input/output device 265 includes an input device such as a keyboard, a mouse, and/or a joystick, and a liquid crystal display (LCD), an organic light emitting diode (OLED) and/or an input device such as a joystick. Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiment included in this specification is implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. Modules reside in memory 255 and may be executed by processor 250 . The memory 255 may be internal or external to the processor 250 , and may be coupled to the processor 250 by various well-known means.

Each component shown in FIG. 5 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present invention is one or more of ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs ( Field Programmable Gate Arrays), a processor, a controller, a microcontroller, a microprocessor, and the like.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored in a recording medium readable through various computer means. can be recorded. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), and a floppy disk. magneto-optical media, such as a disk, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software to perform the operations of the present invention, and vice versa.

Hereinafter, the features of the artificial intelligence learning system according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

6 is an exemplary diagram illustrating a vehicle that collects data for artificial intelligence (AI) machine learning according to an embodiment of the present invention.

As shown in FIG. 6 , the vehicle 10 for collecting data for artificial intelligence (AI) machine learning according to an embodiment of the present invention includes a lidar and a plurality of cameras (camera 1, camera 2, camera). 3, camera 4, camera 5, camera) can be fixedly installed.

The lidar fixedly installed in the vehicle 10 can generate 3D point cloud data corresponding to a three-dimensional image of the vehicle's surroundings by emitting a laser pulse and detecting the light reflected back by objects located around the vehicle. have. Accordingly, the 3D point cloud data obtained by the lidar may include a set of points that reflect the laser pulse emitted by the lidar into a 3D space.

A camera fixedly installed in the vehicle 10 may take a two-dimensional image of the vehicle's surroundings, respectively. 6 shows an embodiment in which six cameras are fixedly installed in the vehicle 10, but the present invention can be implemented based on 2D images captured by fewer than six or more than six cameras. It will be apparent to those of ordinary skill in the art to which this belongs.

7 is an exemplary diagram for explaining a process of automatically specifying an object from 3D point cloud data according to an embodiment of the present invention.

7 , the annotation apparatus 200 according to an embodiment of the present invention may automatically specify the object 20 from 3D point cloud data.

First, the annotation apparatus 200 selects one of the X-axis (ie, the width of the 3D point cloud data), the Y-axis (ie, the height of the 3D point cloud data), and the Z axis (ie, the depth of the 3D point cloud data) of the 3D point cloud data. Three 2D images with fixed (axis) can be identified.

Specifically, the annotation apparatus 200 is a side view image composed only of data of the Y-axis and the Z-axis in which the X-axis is fixed in the 3D point cloud data, and the X-axis and the Z-axis in which the Y-axis is fixed in the 3D point cloud data. A top view image composed of only data and a front view image composed of only data of an X-axis and a Y-axis in which the Z-axis is fixed in the 3D point cloud data may be identified.

The annotation apparatus 200 may identify a set of some points 20-T, 20-S, and 20-F included in the image with respect to each of the three identified 2D images. Then, the annotation device 200 counts the 2D temporary bounding boxes 30-T, 30-S, 30-F so that the identified sets of some points 20-T, 20-S, and 20-F are included. It can be set for each 2D image.

To this end, the annotation apparatus 200 may perform the following for each of the planar view image, the front view image, and the side view image.

The annotation apparatus 200 sets one point located at one end as a reference point with respect to points included in the planar view image, and searches for adjacent points located within a preset adjacent distance on the X-axis based on the reference point. do. When an adjacent point is found, the annotation apparatus 200 may recursively search for a new adjacent point located within the adjacent distance on the X-axis by setting the found adjacent point as a new reference point.

When the adjacent point located within the adjacent distance on the X-axis is not searched, the annotation apparatus 200 searches for the adjacent point located within the adjacent distance on the Z-axis based on the point at which the adjacent point is not searched. When an adjacent point is searched for on the Z-axis, the annotation apparatus 200 may recursively perform a process of searching for the adjacent point on the Z-axis by setting the searched adjacent point as a new reference point.

The annotation apparatus 200 is configured to include both the reference point and the adjacent point found on the X-axis and the reference point and the adjacent point found on the Z-axis when the adjacent point located within the adjacent distance on the Z-axis is not searched. The bounding box 30-T can be set.

In the same manner as in the process of setting the 2D temporary bounding box 30-T in the planar view image as described above, the annotation apparatus 200 targets the points included in the side view image in the Y axis and the Z axis respectively. The process of setting and searching for adjacent points may be recursively performed, and the 2D temporary bounding box 30-S may be set in the side view image so that both the reference point and the adjacent point searched on the Y-axis and Z-axis, respectively, are included.

In addition, the annotation apparatus 200 recursively performs a process of setting a reference point and searching for an adjacent point on the X-axis and the Y-axis, respectively, on the points included in the front view image, and the X-axis and the Y-axis are respectively searched for. The 2D temporary bounding box 30-F may be set in the front view image so that both the reference point and the adjacent point are included.

Then, the annotation apparatus 200 uses the 2D temporary bounding boxes 30-T, 30-S, 30-F set for each of the three 2D images (ie, a planar view image, a side view image, and a front view image). Accordingly, the 3D bounding box 30 for specifying the object 20 included in the 3D point cloud data may be automatically set.

In this case, the 3D bounding box 30 has the 2D temporary bounding box 30-T set in the planar view image as the plane (top) and the bottom (bottom), and the 2D temporary bounding box 30-S set in the side view image. ) as the left side, the right side, and the 2D temporary bounding box 30-F set in the front view image as the front and rear, a hexahedron shape will have

Accordingly, according to embodiments of the present invention, it is possible to automatically specify the object 20 without directly annotating the 3D point cloud data obtained through the lidar fixedly installed in the vehicle 10 .

8 is an exemplary diagram for explaining a process of improving the quality of 3D point cloud data according to an embodiment of the present invention.

As shown in FIG. 8 , the annotation apparatus 200 according to an embodiment of the present invention may improve the quality of continuously acquired 3D point cloud data.

The annotation apparatus 200 may specify an object included in the 3D point cloud data. In addition, the annotation apparatus 200 may identify a plurality of 3D point cloud data including the same object 20 - 1 , 20 - 2 and 20 - 3 from among 3D point cloud data based on the specified object.

The annotation apparatus 200 compares the qualities of the same objects included in a plurality of 3D point cloud data including the same objects 20-1, 20-2, and 20-3 with each other to determine one object with the highest quality. (20-3) can be selected.

Specifically, the annotation apparatus 200 may calculate a volume occupied by objects specified as each of the 3D point cloud data in the 3D point cloud data, and select one object 20 - 3 having the largest calculated volume. . In this case, the annotation apparatus 200 multiplies the width on the X-axis, the height on the Y-axis, and the depth on the Z-axis for the 3D bounding box set to specify the object to determine the volume of the object. can be calculated. When there are a plurality of objects with the largest calculated volume, the annotation apparatus 200 calculates the number of points constituting each of the plurality of objects, and one object 20-3 having the largest number of points can be selected.

When selecting one object based on the calculated volume, the annotation apparatus 200 may preferentially compare volumes of objects having a minimum occlusion ratio of the object to another object.

The annotation device 200 specifies an object to which the same tracking ID as the object specified from the 3D point cloud data is given in the 2D image taken at the same time as the time at which the 3D point cloud data was acquired, and the specified object in the 2D image Volumes of objects having a minimum truncation by the angle of view of the 2D image may be compared preferentially.

The annotation apparatus 200 includes a plurality of 2D images taken at the same time as the time at which the 3D point cloud data was acquired, and the first 2D image and the second 2D image among the plurality of 2D images include different parts of the same object, , if it is possible to derive an object with a cropped ratio of 0 by combining two parts of the object included in the first 2D image and the second 2D image with each other, the quality based on the volume of the object derived by combining the two parts with each other The single best object 20-3 may be selected.

In addition, the annotation apparatus 200 sets the same object 20-1, 20-2, and 20-3 included in a plurality of 3D point cloud data including the same object 20-1, 20-2, and 20-3, respectively. ) may be replaced with points constituting one object 20 - 3 selected as having the best quality.

Accordingly, according to embodiments of the present invention, it is possible to improve the overall quality of 3D point cloud data by replacing data on objects with high-quality data of the same object.

9A to 9C are exemplary diagrams for explaining a process of tracking an object from 3D point cloud data according to an embodiment of the present invention.

9A to 9C , the annotation apparatus 200 according to an embodiment of the present invention may track the same object from continuously acquired 3D point cloud data.

The annotation apparatus 200 may specify one or more objects included in the 3D point cloud data for each of the first 3D point cloud data, the second 3D point cloud data, and the third 3D point cloud data. Here, the first 3D point cloud data may be data acquired chronologically before the second 3D point cloud data, and the second 3D point cloud data may be data acquired chronologically before the third 3D point cloud data.

The annotation apparatus 200 may identify the common object 20 commonly included in the first 3D point cloud data and the second 3D point cloud data. In addition, the annotation apparatus 200 may calculate the movement speed of the identified common object 20 .

Specifically, the annotation apparatus 200 determines the separation distance d between the common object 20 specified from the first 3D point cloud data and the common object 20 specified from the second 3D point cloud data, and the first 3D point cloud The moving speed of the common object 20 may be calculated using a separation time between the data acquisition time t1 and the second 3D point cloud data acquisition time t2.

The annotation apparatus 200 may identify an area 30 - 3 in which the common object 20 is estimated to be located in the third 3D point cloud data, based on the calculated movement speed.

The annotation apparatus 200 may verify whether the common object 20 actually exists in the region 30 - 3 where the common object 20 is estimated to be located in the third 3D point cloud data.

Specifically, the annotation apparatus 200 obtains the area occupied by the common object 20 in the first 2D image taken at the same time as the time t1 at which the first 3D point cloud data was obtained, and the third point cloud data The area where the common object 20 is estimated to be located in the third 2D image taken at the same time as the time t3 is compared with each other, and the common object is actually within the area where the common object 20 is estimated to be located. It can be verified whether an object corresponding to (20) exists. In this case, the first 2D image and the third 2D image may be images taken by a camera fixedly installed in the same vehicle as the vehicle in which the lidar used to obtain the first 3D point cloud data and the third 3D point cloud data is installed. have.

To this end, the annotation apparatus 200 compares the RGB pattern of the region occupied by the common object in the first 2D image with the RGB pattern of the region where the common object is estimated to be located in the third 2D image. Based on this, it may be verified whether an object corresponding to the common object 20 exists in an area where the common object 20 is estimated to be located. Alternatively, the annotation apparatus 200 compares the edge pattern of the area occupied by the common object in the first 2D image with the edge pattern of the area where the common object is estimated to be located in the third 2D image. Based on this, it may be verified whether an object corresponding to the common object 20 exists in an area where the common object 20 is estimated to be located.

In addition, the annotation apparatus 200 performs an object 20 corresponding to an area 30 - 3 in which a common object is estimated to be located among objects specified from the third 3D point cloud data, and the common object 20 . Pre-set metadata can be provided.

Accordingly, according to embodiments of the present invention, after annotating some 3D point cloud data among continuously acquired 3D point cloud data, subsequent 3D point cloud data may be automatically annotated.

10A to 10C are exemplary diagrams for explaining a process of tracking an object from 2D images according to an embodiment of the present invention.

9A to 9C , the annotation apparatus 200 according to an embodiment of the present invention may track the same object from continuously acquired 2D images.

The annotation apparatus 200 may specify, with respect to each of the first 2D image, the second 2D image, and the third 2D image, one or more objects included in the 2D image. Here, the first 2D image may be an image photographed chronologically before the second 2D image, and the second 2D image may be an image photographed chronologically before the third 2D image.

The annotation apparatus 200 may identify the common object 20 commonly included in the first 2D image and the second image. In addition, the annotation apparatus 200 may calculate the movement speed of the identified common object 20 .

Specifically, the annotation apparatus 200 includes the width of the bounding box 30-1 set to specify the common object in the first 2D image and the bounding box 30-2 set to specify the common object in the second 2D image. By comparing the areas of , the separation distance between the common object 20 specified from the first 2D image and the common object 20 specified from the second 2D image may be estimated. The annotation device 200 calculates the separation time between the photographing time t1 of the first image and the photographing time t2 of the second image, and calculates the moving speed of the common object 20 using the separation distance and the separation time. can be calculated.

The annotation apparatus 200 may identify an area 30 - 3 in which the common object 20 is estimated to be located in the third 2D image, based on the calculated movement speed.

The annotation apparatus 200 may verify whether the common object 20 actually exists in the region 30 - 3 where the common object 20 is estimated to be located in the third 2D image.

Specifically, the annotation apparatus 200 determines the area occupied by the common object 20 in the first 3D point cloud data acquired at the same time t1 as the time t1 at which the first 2D image was captured, and the area occupied by the third 2D image. The area where the common object 20 is estimated to be located is compared with each other in the third 3D point cloud data obtained at the same time as the time t3, and the area 30- where the common object 20 is estimated to be located. It can be verified whether an object corresponding to the common object 20 actually exists in 3). In this case, the first 3D point cloud data and the third 3D point cloud data may be point cloud data acquired by a lidar fixedly installed in the same vehicle as the vehicle in which the camera used to photograph the first image and the third image is installed.

To this end, the annotation apparatus 200 determines the number of points included in the 3D bounding box 30-1 set to specify the common object 20 in the first 3D point cloud data, and the number of points included in the third 3D point cloud data. Based on whether the difference in the number of points included in the region 30-3 where the object 20 is estimated to be located is included within a preset quality change range, the common object 20 is estimated to be located It may be verified whether an object corresponding to the common object 20 exists in the region.

Alternatively, the annotation apparatus 200 compares the RGB pattern of the region occupied by the common object in the first 2D image with the RGB pattern of the region where the common object is estimated to be located in the third 2D image. Based on this, it may be verified whether an object corresponding to the common object 20 exists in an area where the common object 20 is estimated to be located. In addition, the annotation apparatus 200 compares the edge pattern of the area occupied by the common object in the first 2D image with the edge pattern of the area where the common object is estimated to be located in the third 2D image. As such, it may be verified whether an object corresponding to the common object 20 exists in an area where the common object 20 is estimated to be located.

And, the annotation apparatus 200 writes the common object 20 with respect to the object 20 corresponding to the region 30-3 where the common object is estimated to be located among the objects specified from the third 2D image. You can assign assigned metadata.

Accordingly, according to embodiments of the present invention, after annotating some 2D images among a plurality of continuously acquired 2D images, subsequent 2D images may be automatically annotated.

11 is an exemplary diagram for explaining a process of performing synchronization between 3D point cloud data and a 2D image according to an embodiment of the present invention.

11 , the annotation apparatus 200 according to an embodiment of the present invention synchronizes an object specified from 3D point cloud data of a lidar fixedly installed in the same vehicle and an object specified from a 2D image of a camera with each other. can do it

The annotation apparatus 200 may specify the object 20 included in the 3D point cloud data. The annotation apparatus 200 may specify the same object 20 as the object specified from the 3D point cloud data from the 2D image.

The annotation apparatus 200 may determine whether the object 20 specified from the 3D point cloud data and the object 20 specified from the 2D image correspond to each other. Then, the annotation device 200 sets the 3D bounding set in the 3D point cloud data to specify the same object 20 so that the object 20 specified from the 3D point cloud data and the object 20 specified from the 2D image can correspond to each other. The position or size of the box 30 - 3D may be adjusted, or the position or size of the 2D bounding box 30 - 2D set in the 2D image may be adjusted to specify the same object 20 .

Specifically, the annotation apparatus 200 sets a border line of the 2D bounding box 30 - 2D set in the 2D image to specify the same object 20 based on a depth value included in the 3D point cloud data. ) can be repositioned. In this case, the annotation apparatus 200 identifies an inflection line in which the depth value included in the 3D point cloud data changes beyond a preset threshold range, and to specify the same object 20 within the 2D image. The position of the boundary line of the 2D bounding box 30 - 2D may be readjusted so that the boundary line of the set 2D bounding box 30 - 2D corresponds to the identified inflection line.

On the other hand, the annotation apparatus 200 determines the volume of the 3D bounding box 30-3D for specifying the same object 20, and the 2D bounding box 30-2D for specifying the same object 20. When smaller than the volume of , the volume of the 2D bounding box 30 - 2D may be reduced such that the volume of the 2D bounding box 30 - 2D is equal to the volume of the 3D bounding box 30 - 3D. In this case, the annotation apparatus 200 converts each line segment constituting the 2D bounding box 30-2D within the size of the chroma subsampling applied in the process of compressing the 2D image to a file of the 2D bounding box 30-2D. It is possible to reduce the volume of the 2D bounding box 30 - 2D by moving it inward.

Accordingly, according to embodiments of the present invention, it is possible to reduce the risk that an error may occur due to inconsistency between objects specified from 3D point cloud data and 2D images acquired and photographed at the same time in the same vehicle.

12 and 13 are exemplary diagrams for explaining a process of tracking an object crossing a boundary of an angle of view according to an embodiment of the present invention.

12 and 13 , the annotation apparatus 200 according to an embodiment of the present invention may track the same object irrespective of the angles of view of cameras that have respectively captured 2D images.

The annotation apparatus 200 may specify the object 20 included in each of the plurality of 2D images. Then, the annotation apparatus 200 may specify the same object 20 as the object specified from the 2D images from the 3D point cloud data.

The annotation apparatus 200 may identify a movement path of the same object 20 based on a change in the position of the same object 20 specified from the 3D point cloud data.

Specifically, when two 2D images obtained by photographing spatially continuous angles of view among a plurality of 2D images, the annotation apparatus 200 includes different parts 20-part1 and 20-part2 of one object, respectively. , it is possible to identify the movement path of the same object 20 .

To this end, the annotation apparatus 200 calculates in advance the sum of the ratio truncated by the angle of view of the first 2D image and the ratio truncated by the angle of view of the second image among two 2D images obtained by photographing spatially continuous angles of view. When it is equal to or greater than the set threshold ratio, it may be determined that the two 2D images include different parts 20-part1 and 20-part2 of one object, respectively.

If the sum of the ratio cropped by the angle of view of the first 2D image and the ratio cropped by the angle of view of the second image is equal to or greater than a threshold ratio, the annotation apparatus 200 determines the RGB value of the object specified from the first 2D image and the second 2D image Based on the RGB value of the object specified from , it can be verified whether the two 2D images each include different parts (20-part1, 20-part2) of one object.

If the sum of the ratio of the cropped by the angle of view of the first 2D image and the ratio of the cropped by the angle of view of the second image is equal to or greater than the threshold ratio, the annotation device 200 is based on the boundary line between the first 2D image and the second 2D image, Based on whether the edge of the object specified from the first 2D image and the edge of the object specified from the second 2D image are continuous, the two 2D images are different parts of one object (20-part1, 20-part2) It can be verified whether each of the

In addition, the annotation apparatus 200 may assign the same metadata to the object 20 specified from each of the plurality of 2D images based on the identified movement path of the same object.

Accordingly, according to embodiments of the present invention, even if one object 20 crosses the boundary between the two 2D images (ie, the boundary of the camera angle of view), the objects 20 individually specified from the two 2D images are mutually exclusive. It can be guaranteed to be specified as the same object 20 .

14 is an exemplary diagram for explaining a process of preprocessing a 2D image using 3D point cloud data according to an embodiment of the present invention.

12 and 13 , the annotation apparatus 200 according to an embodiment of the present invention may preprocess 2D images based on 3D point cloud data.

The annotation apparatus 200 may specify an object included in each of the 3D point cloud data according to a time sequence obtained by the lidar.

If any object is not included in all of the 3D point cloud data, the annotation apparatus 200 converts 2D images captured by a plurality of cameras during the time the 3D point cloud data is acquired by the lidar from a specific target of the object. can be excluded.

The annotation apparatus 200 may identify the movement path of the object 20 based on the change in the position of the object 20 specified according to the temporal sequence obtained by the lidar.

The annotation apparatus 200 uses the movement path of the identified object 20 to be identical to the object specified from the 3D point cloud data from each of the 2D images successively photographed by a plurality of cameras fixedly installed in the vehicle in which the lidar is installed. The object 20 can be specified.

Specifically, the annotation apparatus 200 may exclude one or more 2D images that do not match the movement path of the object 20 from among the 2D images continuously photographed by a plurality of cameras from the specific target of the object.

The annotation apparatus 200 selects an object of the same type as the object 20 specified from the 3D point cloud data in a 2D image matching the movement path of the object 20 among 2D images continuously photographed by a plurality of cameras. A 2D bounding box for specifying within a 2D image can be pre-arranged.

When a signal indicating that the adjustment of the vertex position of the 2D bounding box is completed is inputted through the user interface (UI), the annotation apparatus 200 transmits metadata previously assigned to the object 20 specified from the 3D point cloud data in 2D It can be given as metadata of the object 20 specified by the bounding box.

Hereinafter, as described above, a process of generating data for machine learning of artificial intelligence (AI) according to embodiments of the present invention will be described in more detail.

15 is a flowchart illustrating a method of generating data for artificial intelligence (AI) learning according to an embodiment of the present invention.

As shown in FIG. 15 , the apparatus 100 for designing learning data according to an embodiment of the present invention may design a data structure for artificial intelligence (AI) learning ( S110 ). Specifically, the learning data design device 100 is based on the user's control and the properties of the received project, an ontology for artificial intelligence (AI) learning, a classification system of data for artificial intelligence (AI) learning can be defined

The learning data design apparatus 100 may collect data for artificial intelligence (AI) learning based on the designed data structure (S120). To this end, the learning data design apparatus 100 receives 3D point cloud data and 2D images from the outside, performs web crawling to collect 3D point cloud data and 2D images, or 3D point cloud from an external device. Data and 2D images can be downloaded.

In this case, the 3D point cloud data is data obtained by the lidar fixed in the vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of the vehicle. Such 3D point cloud data may be continuously acquired in time series by the lidar installed in the vehicle. And, the 2D image is an image taken by a plurality of cameras fixedly installed in the vehicle. Such 2D images may be continuously captured in time series by a camera installed in the vehicle.

The learning data design apparatus 100 removes duplicate or extremely similar data from the collected 3D point cloud data and 2D images, and performs purification to de-identify personal information included in the collected 3D point cloud data and 2D images. You can (S130).

The learning data design apparatus 100 may process the collected and refined 3D point cloud data and 2D images through the plurality of annotation apparatuses 200 ( S140 ).

The learning data design apparatus 100 may generate data for artificial intelligence (AI) learning by performing annotation work results from the plurality of annotation apparatuses 200 and packaging the received annotation work results (S150).

In addition, the learning data design apparatus 100 may transmit and deliver the generated artificial intelligence (AI) learning data to the artificial intelligence learning apparatus 400 ( S160 ).

16 is a flowchart illustrating an annotation method according to an embodiment of the present invention.

As shown in FIG. 15 , the annotation apparatus 200 according to an embodiment of the present invention may receive 3D point cloud data and 2D images to be annotated from the training data generating apparatus 100 ( S210 ). .

The annotation apparatus 200 may specify an object to be subjected to machine learning of artificial intelligence (AI) for each of the received 3D point cloud data and the 2D image ( S220 ).

Characteristically, the annotation apparatus 200 according to an embodiment of the present invention may provide various means for more easily performing annotation on 3D point cloud data and 2D images. Specific descriptions of these means are the same as those described with reference to FIGS. 7 to 14 , and thus will not be repeated.

The annotation apparatus 200 may generate metadata for 3D point cloud data and 2D images to be annotated ( S230 ).

The annotation apparatus 200 may generate an annotation work result based on the coordinates and metadata of the object specified from the 3D point cloud data and the 2D images ( S240 ). In this case, the annotation work result may have a JSON file format, but is not limited thereto.

In addition, the annotation apparatus 200 may transmit the generated annotation work result directly to the learning data generating apparatus 100 or via the learning data verifying apparatus 300 ( S250 ).

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention pertains that other modifications based on the technical idea of the present invention can be implemented in addition to the embodiments disclosed herein. It is obvious to those with ordinary knowledge. In addition, although specific terms have been used in the present specification and drawings, these are only used in a general sense to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Training data generator: 100 Annotator: 200
Training data verification device: 300 Artificial intelligence training device: 400
Communication unit: 205 Input/output unit: 210
Storage: 215 Object Specifier: 220
Metadata generator: 225 Result generator: 230
3D Data Object Specific Module: 220-1
2D Image Object Specific Module: 220-2
Quality Management Module: 220-3 Object Tracking Module: 220-4
Synchronization processing module: 220-5 Preprocessing module: 220-6

Claims (10)

The annotation device specifies an object from each of a plurality of 2D images taken by a plurality of cameras fixedly installed in the vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of the vehicle. step;
specifying, by the annotation device, the same object as the object from 3D point group data continuously acquired by a lidar fixedly installed in the same vehicle as the vehicle in which the plurality of cameras are installed;
identifying, by the annotation device, a movement path of the object based on a change in the position of the object specified from the 3D point cloud data; and
adding, by the annotation device, identical metadata to an object specified from each of the plurality of 2D images, based on the identified movement path;
The step of providing the metadata is
Not the relative size occupied by the object in the 2D image, but the absolute size identified from the 3D point cloud data and determined by the unique shape of the object in the metadata of the object specified from each of the plurality of 2D images It is characterized by including the same,
The step of providing the metadata is
Granting the same tracking ID (tracking identifier) to the specified object from each of the plurality of 2D images,
The tracking ID is characterized in that it is a unique identifier of the object given to track the object in the 2D images,
The metadata is
The category of the object specified from the 3D point cloud data or the 2D image, the rate at which the object is cut by the angle of view of the 2D image, the rate at which the object is obscured by other objects or objects, the tracking ID of the object, the time at which the 2D image was taken , Weather conditions on the day the 2D image was taken, file size, image size, copyright holder, resolution, bit value, aperture transmission, exposure time, ISO sensitivity, focal length, aperture opening value, angle of view, white balance, RGB depth, class Includes person, tag, shooting location, type of road, road surface information, traffic jam information,
Re-adjusting, by the annotation device, a position of a border line of a 2D bounding box set in a 2D image to specify the same object based on a depth value included in the 3D point cloud data; and ,
The annotation device identifies an inflection line in which a depth value included in the 3D point cloud data changes beyond a preset threshold range, and a boundary line of a 2D bounding box set to specify the same object in a 2D image Re-adjusting the position of the boundary line of the 2D bounding box to correspond to the identified inflection line;
Re-adjusting, by the annotation device, the position of the boundary line of the 3D bounding box set to specify the same object based on the RGB values included in the 2D image;
The step of re-adjusting the position of the boundary line of the 3D bounding box is,
Edge detection included in the 2D image, and the boundary line of the 3D bounding box set in the 3D point cloud data to specify the same object corresponds to the edge detected from the 2D image. An object tracking method that crosses the boundary of an angle of view, characterized in that readjustment.
The method of claim 1, wherein the step of identifying the movement path comprises:
When two 2D images obtained by photographing spatially continuous angles of view from among the plurality of 2D images each include different parts of one object, it is characterized in that the movement path of the object is identified. How to track objects beyond The method of claim 2, wherein the step of identifying the movement path comprises:
When the sum of the truncated ratio by the angle of view of the first 2D image and the ratio truncated by the angle of view of the second 2D image among the two 2D images obtained by photographing the spatially continuous angles of view is greater than or equal to a preset threshold ratio, the An object tracking method that crosses a boundary of an angle of view, characterized in that it is determined that two 2D images each include different parts of one object. The method of claim 3, wherein the step of identifying the movement path comprises:
When the sum of the ratios is equal to or greater than the threshold ratio, based on the RGB (Red, Green, Blue) value of the object specified from the first 2D image and the RGB value of the object specified from the second 2D image, the two An object tracking method that crosses a boundary of an angle of view, characterized in that it is verified whether a 2D image includes different parts of an object. The method of claim 3, wherein the step of identifying the movement path comprises:
When the sum of the ratios is equal to or greater than the threshold ratio, the edge of the object specified from the first 2D image and the object specified from the second 2D image based on the boundary line between the first 2D image and the second 2D image An object tracking method that crosses a boundary of an angle of view, characterized in that it is verified whether the two 2D images each include different parts of one object based on whether the edges of are continuous. The method of claim 1, wherein the step of identifying the movement path comprises:
When the object specified from the 3D point cloud data crosses the boundary line between two 2D images obtained by taking spatially continuous angles of view among the plurality of 2D images, the movement path of the object is identified, characterized in that , an object tracking method that crosses the boundary of the angle of view. The method of claim 1, wherein specifying the same object comprises:
The plurality of 2D images are arranged in a circle according to the optical axis angle of the camera so that the vehicle in which the lidar and the camera are installed is a virtual circular center point, and only X-axis and Z-axis data in the 3D point cloud data are configured. An object tracking method crossing a boundary of an angle of view, characterized in that the same object is specified by acquiring a top view image and matching the plurality of 2D images arranged in a circle with the plane viewpoint image. memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
specifying, by the processor, an object from each of a plurality of 2D images captured by a plurality of cameras fixedly installed in the vehicle in order to learn artificial intelligence (AI) that can be used for autonomous driving of the vehicle;
specifying, by the processor, the same object as the object from 3D point cloud data continuously acquired by a lidar fixedly installed in the same vehicle as the vehicle in which the plurality of cameras are installed;
identifying, by the processor, a movement path of the object based on a change in the position of the object specified from the 3D point cloud data; and
Giving, by the processor, the same metadata to the object specified from each of the plurality of 2D images, based on the identified movement path;
The step of providing the metadata is
Not the relative size occupied by the object in the 2D image, but the absolute size identified from the 3D point cloud data and determined by the unique shape of the object in the metadata of the object specified from each of the plurality of 2D images It is characterized by including the same,
The step of providing the metadata is
Granting the same tracking ID (tracking identifier) to the specified object from each of the plurality of 2D images,
The tracking ID is characterized in that it is a unique identifier of the object given to track the object in the 2D images,
The metadata is
The category of the object specified from the 3D point cloud data or the 2D image, the rate at which the object is cut by the angle of view of the 2D image, the rate at which the object is obscured by other objects or objects, the tracking ID of the object, the time at which the 2D image was taken , Weather conditions on the day the 2D image was taken, file size, image size, copyright holder, resolution, bit value, aperture transmission, exposure time, ISO sensitivity, focal length, aperture opening value, angle of view, white balance, RGB depth, class Includes person, tag, shooting location, type of road, road surface information, traffic jam information,
Re-adjusting, by the processor, a position of a border line of a 2D bounding box set in a 2D image to specify the same object based on a depth value included in the 3D point cloud data;
The processor identifies an inflection line in which the depth value included in the 3D point cloud data changes beyond a preset threshold range, and the boundary line of the 2D bounding box set to specify the same object in the 2D image is Re-adjusting the position of the boundary line of the 2D bounding box to correspond to the identified inflection line;
Re-adjusting, by the processor, the position of the boundary line of the 3D bounding box set to specify the same object based on the RGB value included in the 2D image;
The step of re-adjusting the position of the boundary line of the 3D bounding box is,
Edge detection included in the 2D image, and the boundary line of the 3D bounding box set in the 3D point cloud data to specify the same object corresponds to the edge detected from the 2D image. A computer program recorded on a recording medium to execute the readjustment.
The method of claim 8, wherein the step of identifying the movement path comprises:
When the object specified from the 3D point cloud data crosses the boundary line between two 2D images obtained by taking spatially continuous angles of view among the plurality of 2D images, the movement path of the object is identified. , a computer program recorded on a recording medium.
The method of claim 8, wherein specifying the same object comprises:
The plurality of 2D images are arranged in a circle according to the optical axis angle of the camera so that the vehicle in which the lidar and the camera are installed is a virtual circular center point, and only X-axis and Z-axis data in the 3D point cloud data are configured. A computer program recorded on a recording medium to acquire a top view image and to specify the same object by matching the plurality of circularly arranged 2D images with the planar view image.

Images