KR102480062B1 - Method for generate training data through 3D object recognition and synthesizing into virtual space, and computer program recorded on record-medium for executing method therefor - Google Patents

Abstract

The present invention proposes a method for generating training data. The method may include the steps of: allowing a training data design device to collect an annotation work object for training artificial intelligence (AI); allowing the training data design device to identify an object from the collected annotation work object and generating a 3D object; and allowing the training data design device to synthesize the generated 3D object into a virtual space to be trained and generate training data.

Description

Method for generating training data through 3D object recognition and synthesizing into virtual space, and computer program recorded on record-medium for executing the method executing method therefor}

The present invention relates to data design for artificial intelligence (AI) learning. More specifically, in the process of generating data for artificial intelligence (AI) learning, the collected work objects are processed into 3D objects, and learning data generation method and execution through 3D object recognition and virtual space synthesis to increase learning efficiency It relates to a computer program recorded on a recording medium in order to do so.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning abilities, reasoning abilities, and perception abilities using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the form of learning data.

In general, the design of artificial intelligence (AI) learning data proceeds in the steps of data structure design, data collection, data refinement, data processing, data expansion, and data verification.

To describe each step in more detail, data structure design is performed through ontology definition, classification system definition, and the like. Data collection is performed by collecting data through direct filming, web crawling, or associations/professional organizations. Data purification is performed by removing redundant data from collected data and de-identifying personal information. Data processing is performed by inputting meta data and performing annotation. Data extension is performed by performing ontology mapping and supplementing or extending the ontology as needed. In addition, data verification is performed by verifying validity according to the set target quality using various verification tools.

Here, data purification refines a large number of collected data ranging from a few thousand to several million. At this time, data purification does not stop at simply removing redundant data, and requires a technology capable of selectively removing data irrelevant to the learning target.

Further, the annotation in the data processing step is performed by processing a bounding box on an object included in the image and inputting attribute information of the object included in the image. Such annotations are also referred to as data labeling. And, a dataset corresponding to the work result of the annotation is calculated in the form of a JSON (Java Script Object Notation) file.

Since such annotation work is performed on a large number of data ranging from a few thousand to several million, annotation work results are also composed of a large number. Therefore, in order to individually verify whether a large number of annotation work results have been correctly performed, a script is used, or a reviewer manually performs the verification.

Among these, when the verifier manually performs the verification, communication between the operator performing the annotation work and the reviewer is very important, but the communication is not smooth, and as a result, errors in the annotation work result cannot be accurately corrected. there were difficulties Accordingly, there is a demand for various means capable of more easily processing a large number of data.

Korean Patent Laid-open Publication No. 10-2020-0042629, ‘Method and apparatus for generating touch-based annotations and images in mobile devices for artificial intelligence learning’, (published on April 24, 2020)

One object of the present invention is to provide a learning data generation method through 3D object recognition to increase learning efficiency by processing collected work objects into 3D objects in the process of generating artificial intelligence (AI) learning data.

Another object of the present invention is to provide a computer program recorded on a recording medium to execute a learning data generation method through 3D object recognition to increase learning efficiency by processing artificial intelligence (AI) collected work objects into 3D objects. will be.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical problem as described above, the present invention proposes a learning data generation method. The method includes a step in which a learning data design device collects annotation work objects for artificial intelligence (AI) learning, and the learning data design device identifies an object from the collected annotation work objects to generate 3D It may include generating an object and generating learning data by synthesizing the created 3D object into a virtual space to be learned by the learning data design device.

More specifically, in the collecting step, the annotation work object may include a 2D image captured by a camera and point cloud data generated through lidar.

In the collecting step, a similarity between objects identified in consecutive 2D images among collected 2D images may be calculated, and if the calculated similarity is lower than a preset threshold, collection may be stopped.

The collecting step calculates a similarity between objects identified in consecutive 2D images among collected 2D images, and when the calculated similarity is lower than a preset threshold, 2D images collected before similarity calculation, The 2D images collected after calculating the similarity can be collected by recognizing them as images different from each other.

The collecting may generate a red, green, blue (RGB) histogram for pixels of objects identified in the continuous 2D image, and calculate a similarity by comparing the generated RGB histograms.

The collecting step divides the continuous 2D image into three images according to RGB (Red, Green, Blue), extracts edges of each divided image, and detects edges in each image. After identifying an enclosure by the image, the similarity can be determined based on the size, location or shape that the identified enclosure occupies within the image.

In the collecting step, when the collection of the work object is completed, a collection ratio for each image group may be calculated in relation to the number of images included in different image groups.

In the generating of the learning data, the calculated collection ratio for each image group may be included in the learning data.

The generating of the 3D object may include detecting an object area existing in the 2D image and extracting RGB (Red, Green, Blue) information for the object area, and points matching the object area among the point cloud data The method may include extracting information and matching the point information and the extracted RGB information.

In the matching step, the point information may be projected onto 2D image coordinates, and the point information and the extracted RGB information may be matched based on the projected image coordinates and the extracted RGB information.

In the generating of the 3D object, the RGB information and the point information may be extracted and matched using artificial intelligence (AI) that has been machine-learned in advance.

In the step of generating the learning data, the annotation work object includes 2D images taken by a plurality of cameras arranged at various angles, and in the step of generating the 3D object, the 2D images taken by the plurality of cameras and , A 3D object may be created through posture information of the plurality of cameras.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute the learning data generation method as described above. The computer program is combined with a computing device including a memory, a transceiver, and a processor that processes instructions resident in the memory, so that the processor learns artificial intelligence (AI). Collecting annotation work objects for the process, generating a 3D object by identifying objects from the collected annotation work objects, and creating a virtual space in which the created 3D object is a learning target It may be a computer program recorded on a recording medium to execute the step of generating learning data by synthesizing it.

Details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, learning efficiency can be increased by identifying objects from the collected annotation work objects to create 3D objects, and creating learning data by synthesizing the created 3D objects in a virtual space to be learned.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 and 2 are configuration diagrams of an artificial intelligence learning system according to various embodiments of the present invention.
3 is a logical configuration diagram of a learning data design device according to an embodiment of the present invention.
4 is a hardware configuration diagram of a learning data design device according to an embodiment of the present invention.
5 is a logical configuration diagram of an apparatus for verifying learning data according to an embodiment of the present invention.
6 is a hardware configuration diagram of an apparatus for verifying learning data according to an embodiment of the present invention.
7 is a flowchart illustrating a method for generating learning data according to an embodiment of the present invention.
8 is a flowchart illustrating a machine learning method for behavior estimation according to an embodiment of the present invention.
9 is a flowchart for explaining a feedback method according to an embodiment of the present invention.
10 is a flowchart for explaining a feedback method according to another embodiment of the present invention.
11 is an exemplary diagram for explaining a vehicle collecting data for artificial intelligence (AI) machine learning according to an embodiment of the present invention.
12 is an exemplary diagram for explaining a process for extracting a key point according to an embodiment of the present invention.
13 is an exemplary diagram for explaining feedback information according to an embodiment of the present invention.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in this specification, and are excessively inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as 'consisting of' or 'having' should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. On the other hand, when a component is referred to as "'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents or substitutes other than the accompanying drawings.

On the other hand, data purification refines a large number of collected data ranging from a few thousand to several million. At this time, data purification does not stop at simply removing redundant data, and requires a technology capable of selectively removing data irrelevant to the learning target.

Further, the annotation in the data processing step is performed by processing a bounding box on an object included in the image and inputting attribute information of the object included in the image. Such annotations are also referred to as data labeling. And, a dataset corresponding to the work result of the annotation is calculated in the form of a JSON (Java Script Object Notation) file.

Since such annotation work is performed on a large number of data ranging from a few thousand to several million, annotation work results are also composed of a large number. Therefore, in order to individually verify whether a large number of annotation work results have been correctly performed, a script is used, or a reviewer manually performs the verification.

Among these, when the verifier manually performs the verification, communication between the operator performing the annotation work and the reviewer is very important, but the communication is not smooth, and as a result, errors in the annotation work result cannot be accurately corrected. there were difficulties Accordingly, there is a demand for various means capable of more easily processing a large number of data.

In order to overcome these limitations, the present invention proposes various means for verifying annotated work products, feeding back the verification results, and increasing learning efficiency.

1 and 2 are configuration diagrams of an artificial intelligence learning system according to various embodiments of the present invention.

As shown in FIG. 1, the artificial intelligence learning system according to an embodiment of the present invention includes a learning data design device 100, one or more annotation devices 200-1, 200-2, ..., 200-n; 200 , It may be configured to include a learning data verification device 300 and an artificial intelligence learning device 400.

In addition, as shown in FIG. 2, the artificial intelligence learning system according to another embodiment of the present invention includes one or more annotation devices (200-a, 200-b, ..., 200-m; 200) and a learning data verification device ( 300-a, 300-b, ..., 300-m; 300) consisting of a plurality of groups (Group-a, Group-b ..., Group-m), the learning data generating device 100 and the artificial intelligence learning device ( 400) may be configured.

Since the components of the artificial intelligence learning system according to various embodiments are merely functionally distinct elements, two or more components are integrated and implemented in the actual physical environment, or one component is implemented in the actual physical environment. may be implemented separately from each other.

Each component is described, and the learning data design device 100 is a device that can be used to design and generate data for machine learning of artificial intelligence (AI).

As such, the learning data design device 100 is basically a device that is distinguished from the learning data verification device 300, but in an actual physical environment, the learning data design device 100 and the learning data verification device 300 are integrated into one device. It may be integrated and implemented.

Specifically, the learning data design device 100 may receive attributes of a project related to artificial intelligence (AI) learning from the artificial intelligence learning device 400 . The learning data design device 100 is based on the user's control and the properties of the received project, designing a data structure for artificial intelligence (AI) learning, refining collected data, processing data, expanding data, and verification can be performed.

First of all, the learning data design device 100 may design a data structure for artificial intelligence (AI) learning. For example, the learning data design device 100 has an ontology for artificial intelligence (AI) learning and a data classification system for artificial intelligence (AI) learning based on the properties of the user's control and the received project. can define

The learning data design device 100 may collect data for artificial intelligence (AI) learning based on the designed data structure. To this end, the learning data design device 100 receives 3D point cloud data and 2D images from the outside, collects 3D point cloud data and 2D images by performing web crawling, or collects 3D point cloud data from a device of an external institution. Data and 2D images can be downloaded.

Here, the 3D point cloud data is data obtained by a lidar fixedly installed in the vehicle. A lidar fixed to a vehicle may generate 3D point cloud data corresponding to a 3D image of the surroundings of the vehicle by emitting laser pulses and detecting light reflected by objects located around the vehicle. That is, the point cloud constituting the 3D point cloud data refers to a set of points that reflect laser pulses emitted by LIDAR into a 3D space.

And, the 2D image is an image taken by a plurality of cameras fixedly installed in the vehicle. For autonomous driving, a plurality of cameras are fixedly installed in one vehicle to take two-dimensional images of the surroundings of the vehicle, respectively. For example, six cameras may be installed in one vehicle, but is not limited thereto.

The learning data design apparatus 100 may remove redundant or extremely similar data from among the collected 3D point cloud data and 2D images. The learning data design apparatus 100 may de-identify personal information included in the collected 3D point cloud data and 2D images.

The learning data design device 100 may distribute and transmit the collected and refined 3D point cloud data and 2D images to a plurality of annotation devices 200 . In this case, the learning data design device 100 may distribute 3D point cloud data and 2D images according to a pre-allocated amount for the operator (ie, labeler) of the annotation device 200.

The learning data design device 100 may receive annotation work results directly from the annotation device 200 or receive annotation work results and inspection results from the learning data verification device 300 .

The learning data design device 100 may generate AI learning data by packaging the received annotation work result. And, the learning data design device 100 may transmit the generated artificial intelligence (AI) learning data to the artificial intelligence learning device 400 .

Characteristically, the learning data design device 100 collects annotation work objects for artificial intelligence (AI) learning, identifies objects from the collected annotation work objects, creates 3D objects, and generates 3D objects. Learning data may be created by synthesizing 3D objects into a virtual space to be learned.

In addition, the learning data design device 100 collects annotation work objects for artificial intelligence (AI) learning, identifies objects from the collected annotation work objects to create 3D objects, and generates 2D images for the created 3D objects. Extracts a plurality of key points, compares the extracted key points with the human body posture model stored in the human body posture model data set built in advance, and extracts the human body posture model that matches the extracted key point among the stored human body posture model data sets to estimate the behavior of a 3D object.

Meanwhile, the specific configuration and operation of the learning data design device 100 will be described with reference to FIGS. 3 and 4 later.

The learning data generation device 100 having such characteristics transmits and receives data with the annotation device 200, the learning data verification device 300, and the artificial intelligence learning device 400, and performs calculations based on the transmitted and received data. Any device that can do this is acceptable. For example, the learning data generating device 100 may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto.

With the following configuration, the annotation device 200 is a local computing device that can be used to perform annotation work on 2D images or 3D point cloud data distributed by the learning data design device 100. All or part of the annotation device 200 may be a device in which an annotation worker performs annotation work through a clouding service.

Specifically, the annotation device 200 may output one 2D image or 3D point cloud data to be annotated to a display from among 2D images or 3D point cloud data received from the learning data design device 100.

The annotation device 200 may select a tool according to a signal input from a user through an input/output device. Here, the tool is a tool for setting a bounding box specifying one or more objects included in 2D image or 3D point cloud data.

The annotation device 200 may receive coordinates according to the selected tool through an input/output device. Also, the annotation device 200 may specify an object included in the 2D image or 3D point cloud data by setting a bounding box based on the input coordinates. Here, the bounding box is an area for specifying an object to be learned by artificial intelligence (AI) among objects included in the image. Such a bounding box may have a rectangle or cube shape, but is not limited thereto.

For example, the annotation device 200 receives two coordinates through an input/output device, and bounds the input two coordinates based on a rectangle having the coordinates of the upper left vertex and the coordinates of the lower right vertex in the 2D image. By setting a box, an object included in a 2D image can be specified. In this case, the two coordinates may be set by the user inputting one type of input signal twice (eg, mouse click) or by the user inputting two types of input signal once (eg, mouse drag). It may, but is not limited thereto.

The annotation device 200 may generate 2D image or 3D point cloud data to be annotated, or metadata for a set object according to a signal input from a user through an input/output device. Here, the metadata is 3D point cloud data or 2D image, and data for describing an object specified from the 3D point cloud data or 2D image. Such metadata includes the category of the object specified from 3D point cloud data or 2D image, the rate at which the object is cut by the angle of view, the rate at which the object is obscured by other objects or objects, the tracking ID of the object, the time the image was taken, the image It may include, but is not limited to, the weather conditions of the day on which the photo was taken, file size, image size, copyright holder, resolution, bit value, aperture transmission, exposure time, ISO sensitivity, focal length, aperture value, angle of view, white Balance, RGB depth, class name, tag, shooting location, road type, road surface information, or traffic jam information may be further included.

The annotation device 200 may generate an annotation work result based on an object set from 2D image or 3D point cloud data and generated metadata. In this case, the annotation work result may have a JSON (Java Script Object Notation) file format, but is not limited thereto. The annotation device 200 may transmit the generated annotation work result to the learning data design device 100 . Also, the annotation device 200 may transmit 2D image or 3D point cloud data in which an object is set to the learning data design device 100 for verification in addition to the generated annotation work result.

The annotation device 200 according to an embodiment of the present invention does not transmit annotation work results and 2D image or 3D point cloud data in which objects are set to the learning data design device 100, and input/output constituting the annotation device 200 Control data of the device may be transmitted to the learning data design device 100.

Here, the control data of the input/output device is data in which one or more signals input by the user to control the input/output device are time-sequentially stored in the process of the annotation device 200 performing an annotation operation on 2D image or 3D point cloud data. It can be. Here, the user may be referred to as a worker, performer, labeler, or data labeler, but is not limited thereto.

For example, when the annotation device 200 is driven by an operating system according to an event-driven architecture, one or more signals included in the control data of the input/output device may be sent to the annotation device 200. It may be an event message generated by the operating system in response to the control of the input/output device. Also, the annotation device 200 may generate control data of the input/output device by duplicating a system queue in which event messages generated by the operating system are stored in a first-in-first-out structure.

As a more specific example, when the operating system of the annotation device 200 corresponds to Windows, the control data of the input/output device includes WM_LBUTONDOWN generated in response to a mouse left button click, WM_KEYDOWN generated in response to a keyboard input, and the like Event messages may be included.

The annotation device 200 having the characteristics described above may be any device capable of transmitting and receiving data to and from the learning data design device 100 and performing calculations based on the transmitted and received data. For example, the annotation device 200 may be a stationary computing device such as a desktop, workstation, or server, or a smart phone, laptop, tablet, phablet, or portable multimedia player. (Portable Multimedia Player, PMP), personal digital assistants (PDAs), or e-book readers (E-book reader).

With the following configuration, the learning data verification device 300 is a device that can be used to verify artificial intelligence (AI) learning data. That is, the learning data verification device 300 verifies whether the annotation task result generated by the annotation device 200 meets a pre-set target quality or whether the annotation task result is valid for artificial intelligence (AI) learning. It is a device that can

Specifically, the learning data verification device 300 may receive an annotation job result from the annotation device 200 . Here, the annotation work result may include coordinates of an object specified from 3D point cloud data and 2D images, and metadata about the image or object. The metadata of the annotation work result includes the category of the specified object, the rate at which the object is truncated by the angle of view of the 2D image (truncation), the rate at which the object is occluded by other objects or objects (occlusion), and the tracking ID of the object (tracking). identifier), the time the image was taken, weather conditions on the day the image was taken, etc. may be included, but is not limited thereto. Such an annotation work result may have a JSON (Java Script Object Notation) file format, but is not limited thereto.

The learning data verification device 300 may inspect the received annotation work result. To this end, the learning data verification device 300 may perform inspection using a script for annotation work results. Here, the script is a code for verifying whether or not the target quality previously set for the annotation work result meets or whether the data is valid.

In addition, the learning data verification device 300 may directly perform inspection by a reviewer and transmit the inspection result input from the reviewer to the annotation device 200 .

In addition, the learning data verification device 300 may transmit annotation work results and verification results received from the annotation devices 200 to the learning data design device 100 .

Characteristically, the learning data verification device 300 receives an annotation task result for artificial intelligence (AI) learning, performs verification of the annotation task result, and when the validation result is determined to be non-conforming, in advance about the cause of non-conformity. Among the feedback information included in the constructed feedback pool, feedback information corresponding to the reason for the determined non-conformity may be transmitted to the annotation device 200 .

Meanwhile, the specific configuration and operation of the learning data verification device 300 will be described with reference to FIGS. 5 and 6 later.

The learning data verification device 300 having the characteristics described above is any device capable of transmitting and receiving data to and from the annotation device 200 and the learning data design device 100 and performing calculations based on the transmitted and received data. Any device may be acceptable. For example, the learning data verification device 300 may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto.

With the following configuration, the artificial intelligence learning device 400 is a device that can be used for machine learning of artificial intelligence (AI).

As such, the artificial intelligence learning device 400 may be any device capable of transmitting and receiving data to and from the learning data design device 100 and performing calculations using the transmitted and received data. For example, the artificial intelligence learning device 400 may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto.

As described above, the learning data design device 100, one or more annotation devices 200, the learning data verification device 300, and the artificial intelligence learning device 400 are a security line that directly connects devices, a public wired communication network Alternatively, data may be transmitted and received using a network in which one or more of the mobile communication networks are combined.

For example, public wired communication networks may include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). It may be, but is not limited thereto. In addition, in the mobile communication network, Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile telecommunication may be included, but is not limited thereto.

3 is a logical configuration diagram of a learning data design device according to an embodiment of the present invention.

As shown in FIG. 3, the learning data design device 100 according to an embodiment of the present invention includes a communication unit 105, an input/output unit 110, a storage unit 115, a work object collection unit 120, and a 3D It may be configured to include an object generator 125, an action estimator 130, and a learning data generator 135.

Since the components of the learning data design device 100 are merely functionally distinct elements, two or more components are integrated and implemented in an actual physical environment, or one component is mutually exclusive in an actual physical environment. It could be implemented separately.

Describing each component, the communication unit 105 may transmit/receive data with the learning data collection device (not shown), the learning data verification device 300, and the artificial intelligence learning device 400.

Specifically, the communication unit 105 may receive 3D point cloud data and 2D images from the learning data collection device. The communication unit 105 may receive 3D point cloud data and 2D images individually or collectively from the learning data collection device.

And, the communication unit 105 may generate learning data and transmit it to the artificial intelligence learning device 400 .

With the following configuration, the input/output unit 110 may receive a signal from a user through a user interface (UI) or output an operation result to the outside. The input/output unit 110 may output collected annotation work objects, generated 3D objects, and learning data obtained by synthesizing 3D objects in a virtual space.

The storage unit 115 may store 3D point cloud data and 2D images received through the communication unit 105 . Also, the storage unit 115 may store generated 3D objects and learning data obtained by synthesizing 3D objects in a virtual space.

With the following configuration, the work object collection unit 120 collects annotation work objects for artificial intelligence (AI) learning through the communication unit 105 .

Here, the work object collection unit 120 may calculate a similarity between objects identified in consecutive 2D images among the collected 2D images, and stop collecting when the calculated similarity is lower than a preset threshold. . That is, in the process of collecting work objects, the work object collection unit 120 may refine data through the similarity of consecutive 2D images in order to remove overlapping or very similar work objects.

In addition, the work object collection unit 120 calculates a similarity between objects identified in consecutive 2D images among the collected 2D images, and when the calculated similarity is lower than a preset threshold, the 2D collected prior to similarity calculation The images and the 2D images collected after calculating the similarity can be collected by recognizing them as images different from each other. That is, the work object collection unit 120 may recognize that the learning object has been changed when the similarity of the consecutive 2D images among the consecutive 2D images is lower than a preset threshold value. For example, while collecting 2D images of the movement of a person located in a store, if the similarity of successive 2D images is lower than a set threshold, it is determined that the object in the 2D image has changed to another person, and collection is stopped. It can be recognized as a separate learning object and recorded as a changed learning object or recorded as separate data.

At this time, the work object collection unit 120 may generate a red, green, blue (RGB) histogram for pixels of objects identified in the continuous 2D image, and calculate a similarity by comparing the generated RGB histograms. Here, the RGB histogram is a graph representing the brightness distribution of each primary color (RGB) in an image. For example, in an RGB histogram, the horizontal axis indicates the brightness level of a color, and the vertical axis indicates the number of pixels allocated to the brightness level of a color. The more pixels there are, the brighter and darker the color can be expressed. As such, the similarity calculation unit 115 may calculate the similarity by comparing color saturation and gradation state, white balance tendency, etc. of consecutive 2D images through the RGB histogram.

In addition, the work object collection unit 120 divides the continuous 2D image into three images according to RGB (Red, Green, Blue), extracts the edge of each divided image, and After identifying an enclosure by the extracted edge, the similarity may be determined based on the size, position, or shape of the identified enclosure in the image.

In addition, when the collection of work objects is completed, the work object collection unit 120 may calculate a collection ratio for each image group related to the number of images included in different image groups. Here, the collection ratio for each image group may be included in the learning data to increase the learning efficiency of the artificial intelligence learning device 400 . For example, when the work object collection unit 120 includes person A, person B, and person C in the collected 2D image, "person A 60%, person B 20%, person C 20%", The collection rate for each image group can be recorded together. Here, the collection rate for each image group may be recorded as metadata of each 2D image or as separate data.

With the following configuration, the 3D object creation unit 125 may generate a 3D object by identifying objects from the collected annotation work objects.

Specifically, the 3D object generator 125 may detect an object area existing in the 2D image and extract RGB (Red, Green, Blue) information from the detected object area. Thereafter, the 3D object generator 125 may extract a point matching the object area from among the 3D point cloud data and generate a 3D object by matching the point information with the extracted RGB information. At this time, the 3D object generator 120 projects the point information onto the 2D image coordinates, and generates a 3D object by matching the point information and the extracted RGB information based on the projected image coordinates and the extracted RGB information. can Here, the 3D object generating unit 125 may extract and match the RGB information and the point information using artificial intelligence (AI), which has been machine-learned in advance.

In addition, in the case of 2D images captured by a plurality of cameras arranged at various angles, the 3D object generation unit 125 identifies the type of 3D object appearing in the 2D image by using machine learning artificial intelligence in advance, , After determining the postures of a plurality of cameras observing the 3D object, the coordinates of the 3D object may be estimated based on the recognized postures of the cameras. In addition, the 3D object generator 125 may learn the artificial intelligence model with the coordinates of the estimated 3D object, the type of the 3D object, and the posture information of the camera.

With the following configuration, the action estimator 130 extracts a plurality of key points of the 2D image for the 3D object generated by the 3D object generator 125, and converts the extracted key points into pre-constructed human body posture model data. Compared with the human body posture model stored in the set, the behavior of the 3D object can be estimated by extracting a human body posture model that matches the key point extracted from the stored human body posture model data set. That is, the behavior estimator 130 may generate skeleton data of the 3D object and compare the stored skeleton data to estimate the behavior of the 3D object.

In this case, the skeleton data may include one or more key points corresponding to positions of points (eg, joints, etc.) that are reference points for changes in body shape, posture, or direction of the object. Specifications such as the number of one or more key points constituting the skeleton data, the number of the key points, the connection relationship, and the position to be located are defined in advance according to the property of the object.

For example, in the case of skeleton data based on a 3D human pose model, it consists of 16 key points connected according to the main skeleton of the human body, key point 1 is the left hip, key point 2 is the left knee, Key point 3 is the left foot, key point 4 is the right hip, key point 5 is the right knee, key point 6 is the right foot, key point 7 is the center of the torso, key point 8 is the upper body, key point 9 is the neck, key point 10 is the center of the head, key point 11 is the right shoulder, key point 12 is the right elbow, key point 13 is the right hand, key point 14 is the left shoulder, key point 15 is the left elbow, key point 16 is the left hand, and each position is It is defined in advance.

The action estimator 130 may predict a human body posture model matched with the extracted key points using artificial intelligence (AI) that has been machine-learned in advance.

In detail, the action estimator 130 may extract key points of the 3D object in units of predetermined frames. In this case, the action estimator 130 may predict a joint position based on a Residual Network (ResNet) model, and extract a key point by connecting the predicted joint position to fit the human skeleton. At this time, the action estimator 130 may extract key points based on the 2D image and 3D point cloud data.

Thereafter, the action estimator 130 may generate an integrated key point by integrating key points extracted in predetermined frame units. At this time, the action estimator 130 may perform a preprocessing process for tracking the skeleton and adjust coordinates to process 2D images of different sizes.

Thereafter, the action estimator 130 integrates the extracted key points, removes key points including the head, eyes, and ears, and deletes the frame when there is no key point on the neck or thigh. That is, the action estimator 130 may delete key points that are not useful for recognition or delete unnecessary 2D images.

In addition, the action estimator 130 compares key points between consecutive first frames and second frames, and if it is determined that the key points extracted in the second frame are missing, the action estimator 130 uses the key points of the first frame A key point of the second frame may be added. Meanwhile, details of adding the missing key point will be described later with reference to FIG. 12 .

Thereafter, the action estimator 130 may extract feature data from the generated integrated key point. Here, the feature data may include at least one of a window size, height between the uppermost key point and the lowermost key point, body velocity, normalized joint positions, joint velocities, joint angles, and arm and leg lengths. there is. For example, the height between the uppermost key point and the lowermost key point means the length from the neck to the thigh based on the window size, and can be used to normalize all feature data. The speed of the body may be calculated based on the position of the neck of the previous frame from the position of the neck of the current frame. The velocities of the joints may be calculated based on the joint positions of the previous frame from the positions of the joints of the current frame.

In addition, the action estimator 130 may reduce the dimension of the extracted feature data by using Principal Component Analysis (PCA). That is, the action estimator 130 may reduce the dimension by creating a representative principal component by a linear combination of the measured feature data. In addition, the action estimator 130 may machine-learn artificial intelligence through the extracted feature data.

With the following configuration, the learning data generation unit 135 may perform action classification on a work object in frame units through the output human body posture model, and generate learning data according to the action classification. Here, the learning data generation unit 135 may generate a frame classified as a behavior to be learned as learning data according to the behavior classification. In addition, the learning data generation unit 135 may control the number of frames to be generated as learning data for each frame classified according to the behavior classification. In addition, the learning data generation unit 135 may generate learning data by synthesizing the generated 3D object into a virtual space to be learned.

4 is a hardware configuration diagram of a learning data design device according to an embodiment of the present invention.

As shown in FIG. 5, the learning data design device 100 includes a processor 150, a memory 155, a transceiver 160, an input/output device 165, and a data bus. (Bus, 170) and storage (Storage, 175) can be configured.

The processor 150 may implement operations and functions of the learning data design device 100 based on instructions according to the software 180a in which the learning data generation method is implemented resident in the memory 155 . Software 180a in which a learning data generation method is implemented may be loaded in the memory 155 . The transceiver 160 may transmit and receive data with the annotation device 200 , the learning data verification device 300 , and the artificial intelligence learning device 400 . The input/output device 165 may receive data necessary for the operation of the learning data design device 100 and output images and structure templates. The data bus 170 is connected to the processor 150, the memory 155, the transceiver 160, the input/output device 165, and the storage 175, and is a movement path for transferring data between each component. role can be fulfilled.

The storage 175 may store an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 180a in which the learning data generation method is implemented. The storage 175 may store software 180b in which the learning data generation method is implemented. In addition, the storage 175 may store a database 185 necessary for performing a method of generating skeleton data. Here, the database 185 may include and store standardized structure templates for each type of object, but is not limited thereto.

According to an embodiment of the present invention, the software 180a, 180b for implementing the method of generating learning data resident in the memory 155 or stored in the storage 175 is provided by the processor 150 using artificial intelligence (AI). ) Collecting annotation work objects for learning, the processor 150 identifying objects from the collected annotation work objects and generating a 3D object, and the processor 150 learning the created 3D object It may be a computer program recorded on a recording medium in order to execute a step of generating learning data by synthesizing in a target virtual space.

In addition, according to one embodiment of the present invention, the software 180a, 180b for implementing the behavior estimation method resident in the memory 155 or stored in the storage 175 includes the processor 150, artificial intelligence, AI) collecting annotation work objects for learning, the processor 150 identifying objects from the collected annotation work objects and generating a 3D object, and the processor 150 generating a 3D object Extracting a plurality of key points of the 2D image for the 2D image, comparing the extracted key points with human body posture models stored in a pre-constructed human body posture model data set, and matching the extracted key points among the stored human body posture model data sets In order to execute the step of estimating the behavior of the 3D object by extracting the human body posture model, it may be a computer program recorded on a recording medium.

More specifically, the processor 150 may include an Application-Specific Integrated Circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 155 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 160 may include a baseband circuit for processing wired/wireless signals. The input/output device 165 includes an input device such as a keyboard, a mouse, and/or a joystick, and a Liquid Crystal Display (LCD), an Organic LED (OLED), and/or a liquid crystal display (LCD). Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiments included in this specification are implemented as software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. A module may reside in memory 155 and be executed by processor 150 . The memory 155 may be internal or external to the processor 150 and may be connected to the processor 150 by various well-known means.

Each component shown in FIG. 4 may be implemented by various means, eg, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), FPGAs ( Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored on a recording medium readable through various computer means. can be recorded. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks), floptical It includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media such as a floptical disk, and ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to operate as one or more pieces of software to perform the operations of the present invention, and vice versa.

5 is a logical configuration diagram of an apparatus for verifying learning data according to an embodiment of the present invention.

Referring to FIG. 5 , the learning data verification device 300 includes a communication unit 305, an input/output unit 310, a storage unit 315, a response generation unit 320, a verification execution unit 325, and a feedback generation unit 330. ).

Since the components of the learning data verification device 300 are merely functionally distinct elements, two or more components are integrated and implemented in an actual physical environment, or one component is mutually exclusive in an actual physical environment. It could be implemented separately.

Describing each component, the communication unit 305 may transmit/receive data with one or more of the annotation device 200 and the artificial intelligence learning device 400.

Specifically, the communication unit 305 may receive annotation work results from the annotation device 200 . Here, the annotation work result is data including information of an image as a target of the annotation work, coordinate information input from an annotation worker, and attribute information of an object specified by the coordinate information. For example, the annotation work result may be data expressed in JSON file format with information of an image as a target of the annotation work, coordinate information input from an annotation worker, and attribute information of an object specified by the coordinate information.

And, the communication unit 305 may transmit the verified and packaged annotation work result to the artificial intelligence learning device 400 .

With the following configuration, the input/output unit 310 may receive a signal from a reviewer through a user interface (UI) or output a calculated result to the outside.

Here, the reviewer means a person who verifies the annotation work result. Such a reviewer may be referred to as a verifier, performer, inspector, etc., but is not limited thereto.

Also, the input/output unit 310 may transmit verification results and feedback information to the annotation device 200 . Also, the input/output unit 310 may receive a query about an annotation job from a worker working on an annotation job result, and transmit a response to the inquiry to the annotation device 200 .

With the following configuration, the storage unit 315 may store a pre-built feedback pool for non-conformity reasons. Here, the feedback pool may include at least one of a reviewer's comment and a work image that match the determined nonconformity reason. Also, the feedback pool may include at least one of a reviewer's comment on the reason for non-conformity, a question received from a worker, and a response to the question. This feedback information may be configured in text or image form.

In the following configuration, the response generating unit 320 receives a query on an annotation job from a worker who works on an annotation job result from the annotation device 200, and through feedback information included in a feedback pool stored in the storage unit 315. Can respond to queries.

Here, the response generation unit 320 uses artificial intelligence (AI) that has been machine-learned in advance based on natural language processing (NLP), comments from reviewers on reasons for non-conformance, questions received from operators, and responses to questions. At least one of them may be analyzed, and a response to the query may be generated and provided. In this case, the response generation unit 320 may provide a response by emphasizing an error region in an image matching a result of an annotation work performed by an operator, along with a text-type response.

With the following configuration, the verification unit 330 may receive a verification result from a reviewer who reviews the annotation work result, and a comment corresponding to the determined non-conformance reason when the verification result is determined to be unsuitable.

Here, the verification unit 330 first transmits the comments received from the reviewer to the annotation device 200, receives the rewritten annotation work result with reference to the comment, and re-verifies the rewritten annotation work result. can be done

With the following configuration, the feedback generation unit 330 may transmit feedback information together with comments received from reviewers.

In addition, the feedback generation unit 330 may extract feedback information corresponding to the determined non-conformance reason from among feedback information included in the feedback pool based on the comments received from the reviewer.

Here, the feedback generation unit 330 analyzes the reason for non-conformity using artificial intelligence (AI), which has been machine-learned in advance, based on natural language processing (NLP), and is non-conforming. A comment on the reason can be created and included in the feedback information. However, it is not limited thereto, and the feedback generating unit 330 is based on natural language processing (NLP) and uses machine-learned artificial intelligence (AI) in advance to match the reviewer's comments and tasks with the reason for non-conformity stored in the feedback pool. Feedback information may be included by analyzing at least one of the images and generating a comment on the reason for non-conformance. At this time, the feedback generating unit 330 may transmit the feedback information in the form of a chatbot (chatter robot).

6 is a hardware configuration diagram of an apparatus for verifying learning data according to an embodiment of the present invention.

As shown in FIG. 6, the learning data verification device 300 includes a processor 350, a memory 355, a transceiver 360, an input/output device 365, and a data bus. (Bus, 370) and storage (Storage, 375) can be configured.

The processor 350 may implement operations and functions of the learning data verification apparatus 300 based on instructions according to the software 380a in which the verification method resides in the memory 355 . Software 380a implementing the feedback method may be loaded in the memory 355 . The transceiver 360 may transmit and receive data with the annotation device 200 and the artificial intelligence learning device 400 . The input/output device 365 may receive data necessary for the operation of the learning data verification device 300, reviewer comments, and the like, and output feedback information extracted from a feedback pool. The data bus 370 is connected to the processor 350, the memory 355, the transceiver 360, the input/output device 365, and the storage 375, and is a moving path for transferring data between each component. role can be fulfilled.

The storage 375 may store an application programming interface (API), a library file, a resource file, and the like required for execution of the software 380a in which the feedback method is implemented. The storage 375 may store software 380b in which the feedback method is implemented. Also, the storage 375 may store information necessary for performing the feedback method. In particular, the storage 375 may be configured to include a database 385 to store the feedback pool.

According to an embodiment of the present invention, the software (380a, 380b) for implementing the feedback method resident in the memory 355 or stored in the storage 375 allows the processor 350 to perform artificial intelligence (AI) learning. Receiving an annotation work result for the process, the processor 350 verifying the annotation work result, and if the processor 350 determines that the validation result is non-conformity, the reason for non-conformity is established in advance. In order to execute the step of transmitting the feedback information corresponding to the reason for the nonconformity determined among the feedback information included in the feedback pool, it may be a computer program recorded on a recording medium.

More specifically, the processor 350 may include an Application-Specific Integrated Circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 355 may include read-only memory (ROM), random access memory (RAM), flash memory, a memory card, a storage medium, and/or other storage devices. The transceiver 360 may include a baseband circuit for processing wired/wireless signals. The input/output device 365 includes an input device such as a keyboard, a mouse, and/or a joystick, and a Liquid Crystal Display (LCD), an Organic LED (OLED), and/or a liquid crystal display (LCD). Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiments included in this specification are implemented as software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. A module may reside in memory 355 and be executed by processor 350 . The memory 355 may be internal or external to the processor 350 and may be connected to the processor 350 by various well-known means.

Each component shown in FIG. 6 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), FPGAs ( Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored on a recording medium readable through various computer means. can be recorded. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks), floptical It includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, such as a floptical disk, and ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to operate as one or more pieces of software to perform the operations of the present invention, and vice versa.

7 is a flowchart illustrating a method for generating learning data according to an embodiment of the present invention.

Referring to FIG. 7 , the learning data design device 100 according to an embodiment of the present invention collects annotation work objects for artificial intelligence (AI) learning (S110).

Here, the learning data design apparatus 100 calculates a similarity between objects identified in consecutive 2D images among the collected 2D images, and stops collection when the calculated similarity is lower than a preset threshold. . That is, in the process of collecting work objects, the work object collection unit 120 may refine data through the similarity of consecutive 2D images in order to remove overlapping or very similar work objects.

In addition, the learning data design device 100 calculates the similarity between objects identified in consecutive 2D images among the collected 2D images, and when the calculated similarity is lower than a preset threshold, the 2D data collected before calculating the similarity The images and the 2D images collected after calculating the similarity can be collected by recognizing them as images different from each other.

In this case, the learning data design apparatus 100 may generate a red, green, blue (RGB) histogram for pixels of objects identified in the continuous 2D image, and compare the generated RGB histograms to calculate a degree of similarity.

In addition, the learning data design device 100 divides continuous 2D images into three images according to RGB (Red, Green, Blue), extracts edges of each divided image, and within each image After identifying an enclosure by the extracted edge, the similarity may be determined based on the size, position, or shape of the identified enclosure in the image.

In addition, when the collection of work objects is completed, the learning data design device 100 may calculate a collection ratio for each image group related to the number of images included in image groups with different learning objects. Here, the collection ratio for each image group can be included in the learning data to increase the learning efficiency of the artificial intelligence learning device 400.

Next, the learning data design device 100 may generate a 3D object by identifying objects from the collected annotation work objects (S120).

Specifically, the learning data design apparatus 100 may detect an object area existing in the 2D image and extract RGB (Red, Green, Blue) information from the detected object area. Thereafter, the learning data design apparatus 100 may extract a point matching the object region from among the 3D point cloud data and generate a 3D object by matching the point information with the extracted RGB information. At this time, the learning data design device 100 projects the point information onto the 2D image coordinates and creates a 3D object by matching the point information and the extracted RGB information based on the projected image coordinates and the extracted RGB information. can Here, the learning data design device 100 may extract and match the RGB information and the point information using artificial intelligence (AI) that has been machine-learned in advance.

In addition, in the case of a 2D image taken by a plurality of cameras arranged at various angles, the learning data design device 100 identifies the type of 3D object appearing in the 2D image by using machine learning artificial intelligence in advance, , After determining the postures of a plurality of cameras observing the 3D object, the coordinates of the 3D object may be estimated based on the recognized postures of the cameras. In addition, the 3D learning data design device 100 may learn the artificial intelligence model with the coordinates of the estimated 3D object along with the type of the 3D object and the posture information of the camera.

Next, the learning data design device 100 extracts a plurality of key points of the 2D image for the 3D object generated in step S120, and uses the extracted key points as a human body posture model stored in a previously built human body posture model data set. Compared to , the behavior of the 3D object can be estimated by extracting a human body posture model that matches the key point extracted from the stored human body posture model data set.

Here, the learning data design device 100 may predict a human body posture model matched with the extracted key points using artificial intelligence (AI) that has been machine-learned in advance. Meanwhile, information for machine learning of artificial intelligence will be described later with reference to FIG. 8 .

Then, the learning data design device 100 may perform action classification on a work object in frame units through the output human body posture model, and generate learning data according to the action classification (S140).

Here, the learning data design apparatus 100 may generate a frame classified as a behavior to be learned as learning data according to the behavior classification. In addition, the learning data design apparatus 100 may control the number of frames to be generated as learning data for each frame classified according to the behavior classification. In addition, the learning data design apparatus 100 may generate learning data by synthesizing the generated 3D object into a virtual space to be learned.

8 is a flowchart illustrating a machine learning method for behavior estimation according to an embodiment of the present invention.

Referring to FIG. 8 , the learning data design apparatus 100 may extract key points of a 3D object in units of predetermined frames (S121). In this case, the learning data design apparatus 100 may predict joint positions based on a Residual Network (ResNet) model, and extract key points by connecting the predicted joint positions to match the human skeleton. In this case, the learning data design apparatus 100 may extract key points based on the 2D image and 3D point cloud data.

Next, the learning data design apparatus 100 may generate an integrated key point by integrating key points extracted in units of predetermined frames (S122). At this time, the learning data design apparatus 100 may perform a preprocessing process for tracking the skeleton and adjust coordinates to process 2D images of different sizes.

Here, the learning data design device 100 integrates the extracted key points, removes key points including the head, eyes, and ears, and deletes the frame when there is no key point on the neck or thigh. That is, the learning data design device 100 may delete key points that are not useful for recognition or delete unnecessary 2D images.

In addition, the learning data design apparatus 100 compares key points between consecutive first frames and second frames, and when it is determined that the key points extracted in the second frames are missing, the key points of the first frames are used. Thus, the key point of the second frame can be added.

Next, the learning data design device 100 may extract feature data from the generated integrated key point (S124). Here, the feature data may include at least one of a window size, height between the uppermost key point and the lowermost key point, body velocity, normalized joint positions, joint velocities, joint angles, and arm and leg lengths. there is. For example, the height between the uppermost key point and the lowermost key point means the length from the neck to the thigh based on the window size, and can be used to normalize all feature data. The speed of the body may be calculated based on the position of the neck of the previous frame from the position of the neck of the current frame. The velocities of the joints may be calculated based on the joint positions of the previous frame from the positions of the joints of the current frame.

Here, the learning data design device 100 may reduce the dimension of the extracted feature data by using Principal Component Analysis (PCA). That is, the learning data design apparatus 100 may reduce the dimension by creating a representative principal component by linear combination of the measured feature data.

And, the learning data design device 100 may perform artificial intelligence machine learning through the extracted feature data (S124).

9 is a flowchart for explaining a feedback method according to an embodiment of the present invention.

First, the learning data verification apparatus 300 may receive an inquiry about an annotation job from a worker working on an annotation job result from the annotation apparatus 200 (S210).

Next, the learning data verification apparatus 300 may perform a response to the query through the feedback information included in the feedback pool (S220).

Here, the learning data verification device 300 is based on natural language processing (NLP) and uses artificial intelligence (AI) that has been machine-learned in advance to respond to comments from reviewers on reasons for non-conformance, queries received from operators, and queries received from operators. At least one of the responses may be analyzed, and a response to the query may be generated and provided. At this time, the learning data design device 100 may provide a response by emphasizing an error region in an image matching a result of an annotation work performed by an operator along with a response in the form of text.

And, the learning data verification device 300 may transmit the generated response to the annotation device 200 (S230).

10 is a flowchart for explaining a feedback method according to another embodiment of the present invention.

First, the learning data verification apparatus 300 may receive a work result from a worker performing an annotation job by the annotation apparatus 200 (S340).

Next, the learning data verification device 300 verifies the work result received in step S340 (S340). That is, the learning data verification apparatus 300 may receive a verification result from a reviewer who reviews the annotation work result and, if the verification result is determined to be unsuitable, a comment corresponding to the determined unsuitable reason.

Here, the learning data verification device 300 first transmits the comments input from the reviewer to the annotation device 200, receives the rewritten annotation work product with reference to the comment, and re-verifies the rewritten annotation work product. can be performed.

Next, the learning data verification apparatus 300 may transmit the verification result in step S360, the suitability result when the work object is suitable, to the operator through the annotation apparatus 200 (S370).

Here, the learning data verification device 300 may generate feedback information when the verification result in step S360 is that the work object is unsuitable (S380). That is, the learning data verification device 300 analyzes the reason for non-conformity using artificial intelligence (AI), which has been machine-learned in advance, based on natural language processing (NLP), You can create comments about the reason for nonconformity. However, it is not limited thereto, and the learning data verification device 300 is based on natural language processing (NLP), using artificial intelligence (AI) that has been machine-learned in advance, and matches the reviewer's comments and At least one of the work images may be analyzed, a comment on a reason for nonconformity may be generated, and feedback information may be included.

Then, the learning data verification device 300 may transmit feedback information together with the comments received from the reviewer (S390). At this time, the learning data verification device 300 may transmit the feedback information in the form of a chatter robot.

11 is an exemplary diagram for explaining a vehicle collecting data for artificial intelligence (AI) machine learning according to an embodiment of the present invention.

The learning data design device 100 according to an embodiment of the present invention identifies objects from the collected annotation work objects, creates 3D objects, and synthesizes the created 3D objects into a virtual space to be learned to generate learning data This can increase the learning efficiency. At this time, the learning data design apparatus 100 may create a 3D object through a 2D image obtained through a camera and 3D point cloud data obtained through LIDAR.

For example, as shown in FIG. 11 , a vehicle 10 that collects data for artificial intelligence (AI) machine learning according to an embodiment of the present invention includes lidar, a plurality of cameras 1, camera 2, camera 3, camera 4, camera 5, camera) can be fixedly installed.

A lidar fixed to the vehicle 10 emits a laser pulse, detects light reflected by objects located around the vehicle, and generates 3D point cloud data corresponding to a 3D image of the surroundings of the vehicle. there is. Accordingly, 3D point cloud data obtained by lidar may include a set of points that reflect laser pulses emitted by lidar into a 3-dimensional space.

A camera fixedly installed in the vehicle 10 may take two-dimensional images of the surroundings of the vehicle. 12 shows an embodiment in which six cameras are fixedly installed in the vehicle 10, but the present invention can be implemented based on 2D images captured by less than or more than six cameras. It will be obvious to those skilled in the art

12 is an exemplary diagram for explaining a process for extracting a key point according to an embodiment of the present invention.

The learning data design device 100 according to an embodiment of the present invention identifies objects from the collected annotation work objects, creates 3D objects, estimates the behavior of the created 3D objects, classifies them according to the behaviors, and selectively learns By generating data, learning efficiency can be increased.

Here, in the learning data design device 100, the action estimator 130 compares key points between consecutive first frames and second frames and determines that the key points extracted in the second frame are missing. , key points of the second frame may be added using the key points of the first frame.

For example, as shown in FIG. 12 , since the previous frame (prev_frame) has no occluded parts, it can be confirmed that all key points (point a) are extracted. However, it can be confirmed that the missing key point (point b) is generated in the current frame (curr_frame). That is, if the object is a person, if the profile is taken based on the camera, if the arm is covered by the torso or if the legs cross each other, a key point may not be extracted from the covered area. there is.

Accordingly, the learning data design apparatus 100 may track the key point (point a) of the previous frame (prev_frame), estimate the obscured keypoint (point b), and add the corresponding key point to the current frame (curr_frame). .

13 is an exemplary diagram for explaining feedback information according to an embodiment of the present invention.

The learning data verification apparatus 300 according to an embodiment of the present invention verifies the annotated work result, and if it is determined to be non-conforming as a result of performing the verification, the feedback pool included in the pre-established feedback pool By transmitting the feedback information corresponding to the reason for the determined nonconformity among the feedback information, it is possible to improve the accuracy of the annotation work by facilitating communication between the operator and the reviewer.

For example, as shown in (a) of FIG. 13 , in the process of performing an annotation work on a work object including a cup as an object, the annotation operator omits the handle of the cup and creates a bounding box. When is set, the learning data verification device 300 may search for answer data from the feedback pool based on keywords such as cups or handles, and transmit it to the annotation device 200. At this time, the learning data verification device 300 may deliver a description in the form of a text, or may provide the guide image including the same object to the annotation device 200 as shown in (b).

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention belongs that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. It is self-evident to those skilled in the art. In addition, although specific terms have been used in the present specification and drawings, they are only used in a general sense to easily explain the technical content of the present invention and help understanding of the present invention, but are not intended to limit the scope of the present invention. Accordingly, the foregoing detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be selected by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Learning data design unit: 100 Annotator unit: 200
Learning data verification device: 300 AI learning device: 400
Communication unit: 105, 305 I/O unit: 110, 310
Storage unit: 115, 315 Work object collection unit: 120
3D object creation unit: 125 behavior estimation unit: 130
Learning data generator: 135 Response generator: 320
Verification Execution Unit: 325 Feedback Generation Unit: 330

Claims (10)

Collecting, by a learning data generating device, an annotation work object for artificial intelligence (AI) learning;
generating, by the learning data generation device, a 3D object by identifying an object from the collected annotation work objects; and
Characterized in that the learning data generation device comprises generating learning data by synthesizing the generated 3D object into a virtual space to be learned,
At the stage of collecting
The annotation work object is characterized in that it includes a 2D image captured by a camera and 3D point cloud data generated through Lidar,
The collecting step is
Among the collected 2D images, a similarity between objects identified in consecutive 2D images is calculated, and when the calculated similarity is lower than a preset threshold, the 2D images collected before similarity calculation and the similarity collected after similarity calculation are calculated. It is characterized by recognizing 2D images as images with different learning objects and recording that the learning objects have been changed,
At the stage of generating training data
The annotation work object includes 2D images captured by a plurality of cameras arranged at various angles,
The step of creating the 3D object is
Identifying the type of 3D object appearing in the 2D image using machine learning artificial intelligence in advance, and generating a 3D object through the 2D image captured by the plurality of cameras and the posture information of the plurality of cameras Characterized in that, learning data generation method.
The method of claim 1, wherein the collecting step
A similarity between objects identified in consecutive 2D images among the collected 2D images is calculated, and the collection is stopped when the calculated similarity is lower than a preset threshold.
The method of claim 1, wherein the collecting step
Characterized in that generating a red, green, blue (RGB) histogram for the pixels of the object identified in the continuous 2D image, and calculating a similarity by comparing the generated RGB histogram, learning data generation method.
The method of claim 1, wherein the collecting step
The continuous 2D image is divided into three images according to RGB (Red, Green, Blue), edge detection of each divided image, and enclosure by the edge extracted from each image ), and then determining the similarity based on the size, position, or shape of the identified closed area occupied in the image.
The method of claim 1, wherein the collecting step
Characterized in that, when the collection of the work object is completed, a collection ratio for each image group related to the number of images included in different image groups (groups) with learning objects is calculated.
The method of claim 5, wherein generating the learning data
Characterized in that the calculated collection rate for each image group is included in the learning data, learning data generation method.
The method of claim 1, wherein generating the 3D object
detecting an object area existing in the 2D image and extracting RGB (Red, Green, Blue) information for the object area;
And extracting point information matching the object region from among the 3D point cloud data, and matching the point information with the extracted RGB information.
memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
Collecting, by the processor, annotation work objects for artificial intelligence (AI) learning;
generating, by the processor, a 3D object by identifying an object from the collected annotation work objects; and
The processor includes the step of synthesizing the created 3D object into a virtual space to be learned to generate learning data,
At the stage of collecting
The annotation work object is characterized in that it includes a 2D image captured by a camera and 3D point cloud data generated through Lidar,
The collecting step is
Among the collected 2D images, a similarity between objects identified in consecutive 2D images is calculated, and when the calculated similarity is lower than a preset threshold, the 2D images collected before similarity calculation and the similarity collected after similarity calculation are calculated. It is characterized by recognizing 2D images as images with different learning objects and recording that the learning objects have been changed,
At the stage of generating training data
The annotation work object includes 2D images captured by a plurality of cameras arranged at various angles,
The step of creating the 3D object is
Identifying the type of 3D object appearing in the 2D image using machine learning artificial intelligence in advance, and generating a 3D object through the 2D image captured by the plurality of cameras and the posture information of the plurality of cameras Characterized in that, a computer program recorded on a recording medium.
The method of claim 8, wherein the collecting step
A computer program recorded on a recording medium, characterized in that for generating a red, green, blue (RGB) histogram for the pixels of the object identified in the continuous 2D image, and calculating a similarity by comparing the generated RGB histogram.
The method of claim 8, wherein the collecting step
The continuous 2D image is divided into three images according to RGB (Red, Green, Blue), edge detection of each divided image, and enclosure by the edge extracted from each image ), and then determining the degree of similarity based on the size, position or shape occupied by the identified closed area in the image.

Images