KR102310588B1 - Method of generating skeleton data for artificial intelligence learning, and computer program recorded on record-medium for executing method thereof - Google Patents

Abstract

Disclosed is a method for generating skeleton data of an object included in an image when annotating data for training artificial intelligence (AI). According to the present invention, the method comprises the following steps: when an image which is a target of an annotation work for training AI is loaded, identifying a structure template to be utilized for generation of skeleton data; outputting the identified structure template by overlaying the identified structure template on the image; moving positions of one or more key points included in the structure template under control of an operator; and generating skeleton data corresponding to an object included in the image on the basis of position coordinates of the moved key point and a connection relationship between the key points. In this case, the skeleton data is data related to a three-dimensional skeleton of an object for identifying the body shape, pose, or direction of the object included in the image and the structure template can be a data structure having a predefined number of key points according to properties of the object and a connection relationship between the predefined key points.

Description

A method of generating skeleton data for artificial intelligence learning and a computer program recorded on a recording medium to execute the same {Method of generating skeleton data for artificial intelligence learning, and computer program recorded on record-medium for executing method thereof}

The present invention relates to data design for artificial intelligence (AI) learning. More specifically, in annotating data for artificial intelligence (AI) learning, it relates to a method of generating skeleton data of an object included in an image, and a computer program recorded in a recording medium for executing the method.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning ability, reasoning ability, and perception ability using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the form of data for learning.

In general, the design of data for artificial intelligence (AI) learning proceeds in the steps of data structure design, data collection, data purification, data processing, data expansion, and data verification.

To describe each step in more detail, the design of the data structure is made through the definition of an ontology, a definition of a classification system, and the like. The collection of data is made by collecting data through direct shooting, web crawling, or association/professional organizations. Data purification is performed by removing duplicate data from the collected data and de-identifying personal information. Data processing is performed by inputting metadata and performing annotations. Data expansion is performed by performing ontology mapping and supplementing or extending the ontology as needed. And, the verification of the data is performed by verifying the validity according to the set target quality using various verification tools.

In general, the annotation of the data processing step is performed by processing a bounding box on an object included in an image, and inputting property information of the object that has been processed with the bounding box. Such annotations are also referred to as data labeling. And, the dataset corresponding to the work result of the annotation is calculated in the form of a JSON (Java Script Object Notation) file.

On the other hand, in the case of simply identifying the properties of an object included in an image, annotation through bounding box processing is sufficient. However, in order to identify the body shape, pose, or direction of the object included in the image, skeleton data of the object included in the image is required.

The skeleton data may be configured to include one or more key points corresponding to positions of points (eg, joints, etc.) that are reference points for changes in body shape, posture, or direction of an object. As for the one or more key points constituting the skeleton data, standards such as the number, connection relationship, and point to be located are defined in advance according to the properties of the object.

For example, in the case of skeleton data according to a 3D human pose model, it consists of 16 key points connected according to the main skeleton of the human body, key point 1 is the left hip, key point 2 is the left knee, Key point 3 is the left foot, key point 4 is the right hip, key point 5 is the right knee, key point 6 is the right foot, key point 7 is the center of the torso, key point 8 is the upper body, key point 9 is the neck, key point 10 is the center of the head, key point 11 is the right shoulder, key point 12 is the right elbow, key point 13 is the right hand, key point 14 is the left shoulder, key point 15 is the left elbow, key point 16 is the left hand, each position is It is predefined.

Therefore, in order to generate skeleton data in the image annotation process for artificial intelligence (AI) learning, the worker performing the annotation identifies the specification of the skeleton data corresponding to the property of the object, and the key point according to the identified specification is It is not omitted, and there is a difficulty in performing an annotation so that the key point is located at the correct point.

Republic of Korea Patent Publication No. 10-2018-0122247, 'A device and method for generating machine learning data and annotations using skeleton information extracted from heterogeneous sensors', (published on November 12, 2018)

One object of the present invention is to provide a method for generating skeleton data of an object included in an image in annotating data for artificial intelligence (AI) learning.

Another object of the present invention is to provide a computer program recorded on a recording medium to execute a method capable of generating skeleton data of an object included in an image in annotating data for artificial intelligence (AI) learning.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical task as described above, the present invention proposes a method for generating skeleton data of an object included in an image in annotating data for artificial intelligence (AI) learning. The method is a structure template that can be utilized to generate skeleton data (skeleton data) when the image that is the target of the annotation work for artificial intelligence (AI) learning is loaded (loading) identifying a; outputting the identified structure template by overlaying it on the image; moving the positions of one or more key points included in the structure template under the control of an operator; and generating skeleton data corresponding to an object included in the image based on the position coordinates of the moved key point and a connection relationship between the key points. In this case, the skeleton data is data related to a three-dimensional skeleton of an object for identifying the body shape, pose, or direction of the object included in the image, and the structure template is based on the properties of the object. Accordingly, it may be a data structure having a predefined number of key points and a connection relationship between the predefined key points.

More specifically, the step of identifying the structure template includes a structure template standardized for each type of object according to the preset properties of the project related to the artificial intelligence (AI) learning, the properties of the image, or the properties of the worker. One structure template can be identified from the database.

Furthermore, in the step of identifying the structure template, one structure template may be identified from the database according to a size, position or shape occupied by an object to which the structure template is to be applied in the image.

In the overlaying and outputting step, the structure template may be overlaid on the image so that one or more preset reference key points among the key points included in the structure template are positioned on the preset feature points. In this case, the feature point may be a point defined in advance according to a position of a joint from which a bone diverges in the skeleton of the object, or a position of a body organ with minimal movement among the body organs of the object.

In the step of overlaying and outputting, the structure template is applied to a third artificial intelligence machine learned using a dataset consisting of the size, position and shape of the object and the position of the feature point. The position of the feature point at which the reference key point is to be located may be identified by querying based on the size, position, or shape that the object occupies in the image.

In the step of moving the position of the key point, among the key points included in the structure template, a key point that can correspond to an object included in the image and a key point that cannot correspond to an object included in the image are different from each other. It can be output through a user interface (UI).

The method may further include setting attribute information including body shape, posture, or direction information for the object after moving the position of the key point. In this case, the generating of the skeleton data includes the attribute information to generate the skeleton data, and the setting of the attribute information is based on the positional coordinates of the moved key point and the connection relationship between the key points. After identifying the body shape, posture, or direction information to be proposed to the worker, the identified body shape, posture, or direction information to be suggested may be output through a user interface (UI).

In the step of moving the position of the key point, the three-dimensionally rotated shape of the structure template may be output through a user interface (UI) under the control of the operator.

In addition, moving the position of the key point includes a first edge connecting between the first key point and the second key point included in the structure template, based on the position coordinates of the moved key point; , when the angle between the second trunk line connecting the second key point and the third key point is out of a preset threshold angle range, it is recognized through a user interface (UI) that an error exists in the position movement of the key point. You can also print

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute a method for generating skeleton data. The computer program includes a memory; input output device; and a processor for processing instructions resident in the memory. Then, the computer program, the processor, when the image to be annotated for artificial intelligence (AI) learning is loaded into the memory, identifying a structure template that can be utilized to generate the skeleton data; outputting, by the processor, the identified structure template over the image through the input/output device; moving, by the processor, the position of one or more key points included in the structure template according to the control of the operator input through the input/output device; and generating, by the processor, the skeleton data corresponding to the object included in the image based on the positional coordinates of the moved key point and the connection relationship between the key points. have.

Specific details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, in generating skeleton data for an object included in an image for artificial intelligence (AI) learning, by using a template in which the number and connection relationship of key points are standardized, skeleton data Key points that should be included in the .

In addition, according to embodiments of the present invention, by setting a key point corresponding to a part of an object that cannot be clearly identified on the image to be distinguished, the operator more clearly confirms the skeleton data according to the three-dimensional skeleton of the object, Annotations can be performed.

As a result, according to embodiments of the present invention, it is possible to generate skeleton data that can accurately learn the body shape, posture, or direction of an object included in an image.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 to 3 are block diagrams of an artificial intelligence learning system according to various embodiments of the present invention.
4 is a logical configuration diagram of an annotation device according to an embodiment of the present invention.
5 is a hardware configuration diagram of an annotation apparatus according to an embodiment of the present invention.
6 to 10 are exemplary views for explaining a process of generating skeleton data according to an embodiment of the present invention.
11 to 14 are exemplary views for explaining a process of generating skeleton data for consecutive images according to an embodiment of the present invention.
15 is a flowchart illustrating a method for generating skeleton data according to an embodiment of the present invention.
16 is a flowchart illustrating a structure template identification method using a pre-performed annotation result according to an embodiment of the present invention.

It should be noted that technical terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in this specification should be interpreted in the meaning generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise defined in this specification, and excessively inclusive. It should not be construed in the meaning of a human being or in an excessively reduced meaning. In addition, when the technical terms used in the present specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be understood by being replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “consisting of” or “having” should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are included. It should be construed that it may not, or may further include additional components or steps.

Also, terms including ordinal numbers such as first, second, etc. used herein may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but another component may exist in between. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents, or substitutes other than the accompanying drawings.

On the other hand, as described above, in the skeleton data, specifications such as the number of key points, connection relationships, and points to be located are determined in advance according to the properties of the object so that the body type, posture, or direction of the object included in the image can be identified. It is defined. Therefore, in order to generate skeleton data in the image annotation process for artificial intelligence (AI) learning, the worker performing the annotation identifies the specification of the skeleton data corresponding to the property of the object, and the key point according to the identified specification is It is not omitted, and there is a difficulty in performing an annotation so that the key point is located at the correct point.

In order to overcome this difficulty, the present invention can generate skeleton data of an object included in an image by using a template of a standardized structure, or by using a result of an annotation previously performed on a continuous image. We would like to suggest the means available.

1 to 3 are block diagrams of an artificial intelligence learning system according to various embodiments of the present invention.

As shown in Figure 1, the artificial intelligence learning system according to an embodiment of the present invention is one or more annotation devices (100-1, 100-2, 100-3, ..., 100-n; 100) and artificial intelligence learning Device 300 may be included.

As shown in FIG. 2 , the artificial intelligence learning system according to another embodiment of the present invention may be configured to additionally include the learning data design apparatus 200 in addition to the annotation apparatus 100 and the artificial intelligence learning apparatus 300 . .

In addition, as shown in Fig. 3, the artificial intelligence learning system according to another embodiment of the present invention is a network in which the annotation apparatus 100, the learning data design apparatus 200, and the artificial intelligence learning apparatus 300 are disclosed. They can also be connected to each other through a public network). In this case, a part of the annotation apparatus 100 may be a device for performing annotation by a clouding service.

As such, since the components of the AI learning system according to various embodiments are merely functionally distinct elements, two or more components are integrated with each other in the actual physical environment, or one component is the actual physical environment. may be implemented separately from each other.

Each component will be described. The annotation apparatus 100 is a device that can be used to annotate an image provided from the learning data design apparatus 200 or the artificial intelligence learning apparatus 300 .

In particular, when the annotation apparatus 100 according to the present invention generates skeleton data for an object included in an image, a template of a standardized structure is utilized or an annotation previously performed on a continuous image is used. It has features that can use the results.

As such, the annotation apparatus 100 can transmit and receive data to and from the learning data design apparatus 200 or the artificial intelligence learning apparatus 300, and any apparatus capable of performing an operation using the transmitted and received data may be permitted. have.

For example, the annotation device 100 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto, and a smart phone, such as laptops, tablets, phablets, Portable Multimedia Players (PMPs), Personal Digital Assistants (PDAs), or E-book readers. It may be any of the mobile computing devices.

A detailed configuration and operation of the annotation apparatus 100 as described above will be described later with reference to FIGS. 4 to 16 .

With the following configuration, the learning data design device 200 is a device that can be used to design and generate data for artificial intelligence (AI) learning. As such, the learning data designing apparatus 200 is basically a device distinct from the artificial intelligence learning apparatus 300, but may be implemented by being integrated into the artificial intelligence learning apparatus 300 in an actual physical environment.

Specifically, the learning data design apparatus 200 may receive the properties of the project related to artificial intelligence (AI) learning from the artificial intelligence learning apparatus 300 . The learning data design device 200 performs design of a data structure for artificial intelligence (AI) learning, purification of collected data, processing of data, expansion of data, and verification of data, based on the user's control and the properties of the project. can be done

In particular, the learning data design apparatus 200 may transmit an image to be annotated to the annotation apparatus 100 for data processing for artificial intelligence (AI) learning. The learning data design apparatus 200 may receive an annotation work result from the annotation apparatus 100 . In this case, the annotation operation result may have a JSON (Java Script Object Notation) file format. Alternatively, the learning data design apparatus 200 may generate a JSON file based on the received result after receiving an annotation work result in a format different from JSON. Then, the learning data design apparatus 200 may inspect the received or generated JSON file, package it, and transmit it to the artificial intelligence learning apparatus 300 .

As such, the learning data design device 200 may be any device capable of transmitting and receiving data to and from the annotation device 100 and the artificial intelligence learning device 300 and performing an operation using the transmitted/received data. have. For example, the learning data design apparatus 200 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

With the following configuration, the artificial intelligence learning device 300 is a device that can be used to perform machine learning of artificial intelligence (AI) based on data for artificial intelligence (AI) learning.

Specifically, the artificial intelligence learning apparatus 300 may receive a packaged JSON file directly from the annotation apparatus 100 or from the learning data design apparatus 200 . And, the artificial intelligence learning apparatus 300 may perform machine learning of artificial intelligence (AI) using the received JSON file.

As such, the artificial intelligence learning apparatus 300 can transmit and receive data to and from the annotation apparatus 100 or the learning data design apparatus 200, and any apparatus may be permitted as long as it can perform an operation using the transmitted and received data. have. For example, the artificial intelligence learning apparatus 300 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

As described above, at least one annotation device 100, learning data design device 200, and artificial intelligence learning device 300 are a combination of at least one of a secure line, a public wired communication network, or a mobile communication network directly connected between the devices. Data can be transmitted and received using the network.

For example, public wired networks include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). However, it is not limited thereto. In addition, the mobile communication network includes Code Division Multiple Access (CDMA), Wideband CDMA, WCDMA, High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile communication may be included, but are not limited thereto.

Hereinafter, the configuration of the annotation apparatus 100 as described above will be described in more detail.

4 is a logical configuration diagram of an annotation device according to an embodiment of the present invention.

As shown in FIG. 4 , the annotation apparatus 100 includes a communication unit 105 , an input/output unit 110 , a storage unit 115 , a template providing unit 120 , an annotation working unit 125 , and a skeleton generating unit 130 . ) may be included.

As such, the components of the annotation apparatus 100 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is separated from each other in the actual physical environment. could be implemented.

When each component is described, the communication unit 105 may transmit/receive data to and from the learning data design apparatus 200 and the artificial intelligence learning apparatus 300 .

Specifically, the communication unit 105 may receive one or more images from the learning data design apparatus 200 or the artificial intelligence learning apparatus 300 .

Here, the image is an image to be subjected to annotation work for artificial intelligence (AI) learning. As such, according to the data processing plan designed by the learning data design device 200 or the artificial intelligence learning device 300, the images are individually received, or a plurality of images are collectively received. can receive

The communication unit 105 may transmit an annotation operation result including the skeleton data generated by the skeleton generating unit 130 to the learning data design apparatus 200 or the artificial intelligence learning apparatus 300 .

Here, the skeleton data is data related to a three-dimensional skeleton of an object for identifying the body shape, pose, or direction of the object included in the image.

As such, the skeleton data may be configured to include one or more key points corresponding to positions of points (eg, joints, etc.) that are reference points for changing the body shape, posture, or direction of the object. One or more key points included in the skeleton data refer to points that are reference points for changes in body shape, posture, or direction of an object. In addition, standards such as the number, connection relationship, and location of one or more key points constituting the skeleton data are defined in advance according to the properties of the object.

For example, in the case of skeleton data according to a 3D human pose model, it may be composed of 16 key points connected according to the main skeleton of the human body. And, out of 16 key points, key point 1 is the left hip, key point 2 is the left knee, key point 3 is the left foot, key point 4 is the right hip, key point 5 is the right knee, key point 6 is the right foot, the height Point 7 is the center of the torso, key point 8 is the upper body, key point 9 is the neck, key point 10 is the center of the head, key point 11 is the right shoulder, key point 12 is the right elbow, key point 13 is the right hand, key point 14 is Left shoulder, key point 15 is the left elbow, key point 16 is the left hand, each position is predefined.

In addition, the communication unit 150 may receive a project property, an image property, or an operator property from the learning data design device 200 or the artificial intelligence learning device 300 .

Here, the properties of the project include the learning purpose of the project related to learning of artificial intelligence (AI), the learning period, the number of images required for learning, the properties of objects to be identified in the images, standards related to the object's skeleton data, etc. However, the present invention is not limited thereto.

The properties of the image include the file name of the image, size (width, height), resolution, bit level, compression format, recording device name, exposure time, ISO sensitivity, focal length, aperture opening value, location coordinates (GPS latitude, longitude) of the image. , shooting time, etc. may be included, but is not limited thereto.

The attributes of the operator may include, but are not limited to, the name of the operator, an identification number, the amount of work allocated, the cost according to the operation, and evaluation of the operation result.

With the following configuration, the input/output unit 110 may input a signal from an operator through a user interface (UI) or output the calculated result to the outside.

Here, the worker means a person who performs the annotation work. Such an operator may be referred to as a user, performer, labeler, or data labeler, but is not limited thereto.

Specifically, the input/output unit 110 may output an image to be annotated. In particular, the input/output unit 110 may output the structure template by overlaying it on the image.

Here, the structure template refers to a data structure having a pre-defined number of key points according to object properties and a connection relationship between the pre-defined key points. As such, the structure template can finally be used to generate the skeleton data.

The input/output unit 110 may receive a control signal for moving a position of a key point to be included in the skeleton data (ie, a key point included in the structure template) from an operator.

The input/output unit 110 may receive a control signal for setting attribute information of an object to be included in the skeleton data from an operator.

Here, the attribute information of the object may include, but is not limited to, the type of the object, and information about the body shape, posture, or direction of the object.

In addition, the input/output unit 110 may output a three-dimensionally rotated shape of the structure template under the control of the operator.

With the following configuration, the storage unit 115 may store data required for annotation work.

Specifically, the storage unit 115 may store the image received through the communication unit 105 . The storage unit 115 may store the properties of the project, the properties of the images, or the properties of the workers received through the communication unit 105 .

The storage unit 115 may temporarily store a structure template including a key point whose position is moved according to a control signal input through the input/output unit 110 . The storage unit 115 may temporarily store the properties of the object input through the input/output unit 110 .

In particular, the storage unit 115 may store a database in which structure templates standardized for each type of object are provided.

With the following configuration, the template providing unit 120 provides a structure template that can be utilized to generate skeleton data when an image that is a target of an annotation operation for artificial intelligence (AI) learning is loaded into memory. have.

According to an embodiment of the present invention, the template providing unit 120 may identify the structure template by being limited to the current image to be annotated.

Specifically, the template providing unit 120 may identify one structure template from the database of the storage unit 115 according to the project property, the image property, or the worker property set in advance through the communication unit 105 . .

For example, the template providing unit 120 may identify one structure template from the database of the storage unit 115 according to the size, position, or shape occupied by the object to which the structure template is to be applied in the image.

In addition, the template providing unit 120 may output the identified structure template by overlaying it on the image through the input/output unit 110 .

In particular, in overlaying the structure template on the image, the template providing unit 120 may position one or more preset reference key points among key points included in the structure template on the preset feature points.

Here, the feature point may be a point defined in advance according to a position of a joint from which a bone diverges in the skeleton of the object or a position of a body organ with minimal movement among the body organs of the object.

For example, in the case of a structure template according to a 3D human body posture model consisting of 16 key points, the key point 10 may be located at the center of the head of the object.

In order to identify the position of the feature point where the reference key point is to be located, the template providing unit 120 performs machine learning using a dataset consisting of the size, position and shape of the object and the position of the feature point. For the artificial intelligence (AI) of , it is possible to identify the position of the feature point where the reference key point is to be located by querying based on the size, position, or shape occupied by the object to which the structure template is applied in the image.

Unlike the above, according to another embodiment of the present invention, the template providing unit 120 may identify the structure template by using the result of the annotation previously performed on the continuous image.

Specifically, when the first image, which is the object of the annotation operation, is loaded, the template providing unit 120 may identify the first image and the second image that is temporally continuously photographed.

To this end, the template providing unit 120 identifies, as a second image, an image previously annotated by a worker performing an annotation operation on the first image among images related to a project including the first image as the object of the annotation operation. can do. That is, the second image may be an image previously annotated by the operator among images belonging to the same project as the first image.

If the number of images on which the operator has previously been annotated is plural, the template providing unit 120, based on the properties of the image, has the same photographing location coordinates and the photographing device name of the first image among the images on which the operator has previously performed annotations. image can be identified. In addition, the template providing unit 120 may identify, as the second image, an image having the smallest photographing time and time difference of the first image among images having the same photographing location coordinates and photographing device name of the first image.

After identifying the second image, the template providing unit 120 may identify the same object as the object included in the second image from among the objects included in the first image.

To this end, the template providing unit 120 determines the object included in the first image and the object included in the second image based on the similarity of the size, position, or shape occupied by the object in the first image and the second image. It can be determined whether they correspond to the same object.

In this case, the template providing unit 120 may perform image processing on the first image and the second image in order to determine the similarity of the size, position, or shape of the object.

For example, the template providing unit 120 may divide each of the first image and the second image into three images according to RGB (Red, Green, Blue). The template providing unit 120 may extract an edge of each image divided into three images (edge detection). In more detail, the template providing unit 120 may use either a Laplacian of Gaussian (LoG) algorithm or a Difference of Gaussian (DoG) algorithm for edge extraction of each image.

When using the LoG algorithm, the template providing unit 120 may use a Gaussian filter to remove noise existing in the image. The template providing unit 120 may apply a Laplacian filter to the image from which the noise has been removed. In addition, the template providing unit 120 may extract an edge by detecting a zero crossing in the image to which the Laplacian filter is applied.

When using the DoG algorithm, the template providing unit 120 generates two Gaussian masks having different variances from the image. The template providing unit 120 subtracts the other mask from the generated one mask. Then, the template providing unit 120 may extract the edge by applying the subtracted mask to the image.

The template providing unit 120 may identify one or more enclosures by edges extracted from within each image. In this case, the template providing unit 120 may first process each image with binarization in order to clarify whether the edge region is closed.

The template providing unit 120 may determine the similarity between the object in the first image and the object in the second image based on the size, position, or shape occupied by the identified closed region in the image.

After identifying the same object, the template providing unit 120 may identify a structure template to be applied to the object included in the first image based on the skeleton data annotated on the same object included in the second image.

In addition, the template providing unit 120 may output the identified structure template by overlaying it on the first image through the input/output unit 110 .

In particular, in overlaying the structure template identified through the second image on the first image, the template providing unit 120 may automatically correct the position coordinates of the structure template to correspond to the object included in the first image.

Specifically, a difference value between the position of the object included in the second image and the position of the object included in the first image may be calculated. Then, the template providing unit 120 may change the position of the key point included in the structure template in response to the calculated difference value, and then overlay it on the first image and output it.

In order to verify whether the position coordinates of the automatically corrected structure template are correct corrections, the template providing unit 120 is configured to provide the first element included in the structure template based on the position coordinates of the key points changed in response to the calculated difference value. It is determined whether the angle between the first edge connecting the key point and the second key point and the second edge connecting the second key point and the third key point is out of a preset threshold angle range. can

As a result of the determination, when the angle between the first trunk line and the second trunk line is within the critical angle range, the template providing unit 120 may determine that the position coordinates of the structure template are correct. Alternatively, when the angle between the first trunk and the second trunk is out of the critical angle range, the template providing unit 120 may use the first key point or The position of the third key point may be changed again.

Furthermore, in changing the position of the key point included in the structure template, the template providing unit 120 includes a size, position or shape occupied by the object in the first image, the size occupied by the object in the second image, Based on the result of comparing the position or shape, it is possible to determine whether to apply the difference value to all keypoints included in the structure template or to apply the difference value only to some keypoints included in the structure template.

That is, when the object in the first image is moved while maintaining the shape of the object in the second image to some extent, the template providing unit 120 automatically corrects the positional coordinates of all key points included in the structure template in a batch. can Alternatively, when the object in the first image is moved because the shape of the object in the second image is changed, the template providing unit 120 may automatically correct only the position coordinates of some key points included in the structure template.

With the following configuration, the annotation working unit 125 may perform annotation to move the position of one or more key points included in the structure template provided by the template providing unit 120 .

Specifically, the annotation working unit 125 may move the position of the key point of the output structure template by overlaying it on the image through the template providing unit 120 according to the control signal input through the input/output unit 110 . .

In order to verify whether the positional movement of the key point is correct, the annotation working unit 125 performs an interval between the first key point and the second key point included in the structure template based on the position coordinates of the moved key point. It may be determined whether an angle between a connecting first edge and a second edge connecting between the second key point and the third key point is out of a preset threshold angle range.

As a result of the determination, when the angle between the first trunk line and the second trunk line is within the threshold angle range, the annotation working unit 125 may determine that the position movement of the key point is correct. On the other hand, when the angle between the first trunk line and the second trunk line is out of the critical angle range, the annotation working unit 125 indicates that there is an error in the position movement of the key point in the user interface (UI) of the input/output unit 110 . It can be output through

According to an embodiment of the present invention, the annotation working unit 125 may provide a function of distinguishing a part that can be clearly identified on an image from a part that cannot be clearly identified on an image among a portion of an object.

Specifically, the annotation work unit 125 selects a key point that can correspond to an object included in an image among key points included in the structure template and a key point that cannot correspond to an object included in the image to the input/output unit 110 . can be output to different user interfaces (UIs).

According to another embodiment of the present invention, the annotation working unit 125 may provide a function for the operator to check whether the key point included in the structure template is located at the correct point.

In detail, the annotation working unit 125 may output a three-dimensionally rotated shape of the structure template through the user interface (UI) of the input/output unit 110 under the control of the operator.

When the position movement of the key point as described above is completed, the annotation work unit 125 may set attribute information on the object included in the image according to the operator's control signal input through the input/output unit 110 . Here, the attribute information of the object may include, but is not limited to, the type of the object, and information about the body type, posture, or direction of the object.

According to another embodiment of the present invention, the annotation working unit 125 may preemptively suggest the attribute information on the object before the attribute information on the object is set by the operator.

Specifically, the annotation work unit 125 may identify body shape, posture, or direction information to be suggested to the operator based on the positional coordinates of the moved key point and the connection relationship between the key points. In addition, the annotation work unit 125 may output the identified body shape, posture, or direction information to be proposed through the user interface (UI) of the input/output unit 110 .

With the following configuration, the skeleton generating unit 130 may generate skeleton data based on the result of the annotation work performed by the annotation work unit 125 .

Specifically, the skeleton generator 130 may generate skeleton data corresponding to the object included in the image based on the positional coordinates of the key points included in the structure template and the connection relationship between the key points. In this case, the skeleton data may be generated by including attribute information of the object.

In addition, the skeleton generating unit 130 may transmit an annotation operation result including the generated skeleton data to the learning data design apparatus 200 or the artificial intelligence learning apparatus 300 through the communication unit 105 . In this case, the annotation operation result may have a JSON (Java Script Object Notation) file format, but is not limited thereto.

Hereinafter, hardware for implementing the logical components of the annotation apparatus 100 as described above will be described in more detail.

5 is a hardware configuration diagram of an annotation apparatus according to an embodiment of the present invention.

5, the annotation device 100 includes a processor 150, a memory 155, a transceiver 160, an input/output device 165, and a data bus. , 170) and storage (Storage, 175).

The processor 150 may implement the operation and function of the annotation apparatus 100 based on a command according to the software 180a in which the method for generating skeleton data residing in the memory 155 is implemented. The memory 155 may be loaded with software 180a in which the method for generating skeleton data is implemented. The transceiver 160 may transmit/receive data to and from the learning data design apparatus 200 and the artificial intelligence learning apparatus 300 . The input/output device 165 may receive data necessary for the operation of the annotation device 100 , and may output an image and a structure template. The data bus 170 is connected to the processor 150 , the memory 155 , the transceiver 160 , the input/output device 165 and the storage 175 . can play a role.

The storage 175 may store an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 180a in which the method for generating the skeleton data is implemented. The storage 175 may store the software 180b in which the method for generating skeleton data is implemented. In addition, the storage 175 may store the database 185 necessary for performing the method for generating skeleton data. Here, the database 185 may include and store standardized structure templates for each type of object, but is not limited thereto.

According to the first embodiment of the present invention, the software 180a, 180b for implementing the method for generating skeleton data resident in the memory 155 or stored in the storage 175 is the processor 150 artificial intelligence (AI) learning. When the image to be annotated for is loaded into the memory 155, a step of identifying a structure template that can be used for generation of skeleton data, the processor 150 overlays the structure template on the image to the input/output device 165 outputting through, the processor 150 moving the position of one or more key points included in the structure template under the control of the operator input through the input/output device 165, and the processor 150 moving the key It may be a computer program recorded on a recording medium to execute the step of generating skeleton data corresponding to the object included in the image based on the positional coordinates of the points and the connection relationship between the key points.

According to the second embodiment of the present invention, the software 180a, 180b for implementing the method for generating skeleton data resident in the memory 155 or stored in the storage 175 is the processor 150 artificial intelligence (AI) learning. When the first image, which is the subject of the annotation operation for Identifying the same object as the object included in the second image, the processor 150 identifying the structure template to be applied to the object included in the first image based on the skeleton data annotated on the object included in the second image Step, the processor 150 may be a computer program recorded on the recording medium to execute the step of overlaying the identified structure template on the first image and outputting it through the input/output device 165 .

More specifically, the processor 150 may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 155 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 160 may include a baseband circuit for processing wired and wireless signals. The input/output device 165 includes an input device such as a keyboard, a mouse, and/or a joystick, and a liquid crystal display (LCD), an organic light emitting diode (OLED) and/or an input device such as a joystick. Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiment included in this specification is implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. The module resides in the memory 155 and may be executed by the processor 150 . The memory 155 may be internal or external to the processor 150 , and may be connected to the processor 150 by various well-known means.

Each component shown in FIG. 5 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present invention is one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs ( Field Programmable Gate Arrays), a processor, a controller, a microcontroller, a microprocessor, and the like.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored in a recording medium readable through various computer means. can be recorded. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), and a floppy disk. magneto-optical media, such as a disk, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software to perform the operations of the present invention, and vice versa.

Hereinafter, the features of the artificial intelligence learning system according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

6 to 10 are exemplary views for explaining a process of generating skeleton data according to an embodiment of the present invention.

Referring to FIG. 6 , in the annotation apparatus 100 of the artificial intelligence learning system according to an embodiment of the present invention, when an image 10 to be annotated for artificial intelligence (AI) learning is loaded into a memory, the image A structure template 30 for generating the skeleton data 40 of one or more objects 20 included in 10 may be identified.

Here, the structure template 30 is a data structure having a predefined number of key points 31 according to the properties of the object 20 and a connection relationship 33 between the predefined key points 31 . it means.

Referring to FIG. 7 , the annotation apparatus 100 may overlay the identified structure template 30 on the object 20 included in the image 10 and output it.

In this case, the annotation apparatus 100 may position one or more preset reference key points 31-sp among the key points 31 included in the structure template 30 above the preset feature points 31-sp. have.

Here, the feature point 31 -sp may be a predefined point according to a position of a joint where bones are branched in the skeleton of the object 20 or a position of a body organ with minimal movement among the body organs of the object.

Referring to FIG. 8 , the annotation device 100 moves (moves from 31-old to 31-new) the position of one or more key points 31 included in the structure template 30 under the control of the operator. can be performed.

In particular, the annotation apparatus 100 may provide a function of distinguishing a part that can be clearly identified on the image 10 from a part that cannot be clearly identified on the image 10 among a portion of the object 20 .

For example, the annotation apparatus 100 expresses or shades the color of the key point 31 corresponding to a portion that cannot be clearly identified on the image 10 with a different color, so that it can be clearly identified on the image 10 . It is possible to provide a function that can be distinguished from the key point 31 corresponding to the available part.

Referring to FIG. 9 , the annotation apparatus 100 may verify whether the positional movement of the key point is a correct movement.

Specifically, the annotation apparatus 100 connects between the first key point 31-1 and the second key point 31-2 included in the structure template 30 based on the position coordinates of the moved key point. The angle d between the first trunk line 33-1 and the second trunk line 33-2 connecting between the second key point 31-2 and the third key point 31-3 is It can be determined whether it is outside the threshold angle range set in .

As a result of the determination, when the angle d between the first trunk line 33-1 and the second trunk line 33-2 is within the critical angle range, the annotation apparatus 100 may determine that the position movement of the key point is correct. have. On the other hand, if the angle d between the first trunk line 33-1 and the second trunk line 33-2 is out of the critical angle range, the annotation device 100 has an error in the position movement of the key point. can be output through a user interface (UI).

In addition, the annotation apparatus 100 may set attribute information on the object 20 included in the image 10 under the control of the operator. Here, the attribute information of the object 20 may include information about the type of the object, body shape, posture, or direction of the object, but is not limited thereto.

Referring to FIG. 10 , the annotation apparatus 100 includes the location coordinates of the key point 31 included in the structure template 30 , the connection relationship 33 of the key point 31 , and the skeleton including object property information. Data 40 may be generated.

In addition, the annotation apparatus 100 may transmit an annotation operation result including the generated skeleton data 40 to the learning data design apparatus 200 or the artificial intelligence learning apparatus 300 .

According to the embodiment of the present invention as described above, in generating the skeleton data 40 for the object 20 included in the image 10 for artificial intelligence (AI) learning, the key point 31 By using the template 30 in which the number and connection relationship 33 are standardized, the key point 31 to be included in the skeleton data 40 is not omitted, and the connection relationship 33 between the key points 31 ) can be set correctly.

In addition, according to an embodiment of the present invention, by setting the key point 31 corresponding to a part of the object 20 that cannot be clearly identified on the image 10 to be distinguishable, the operator can It becomes possible to more clearly check the skeleton data according to the dimensional skeleton and to perform annotations.

As a result, according to an embodiment of the present invention, it is possible to generate the skeleton data 40 that can accurately learn the body shape, posture, or direction of the object 20 included in the image 10 .

11 to 14 are exemplary views for explaining a process of generating skeleton data for consecutive images according to an embodiment of the present invention.

Referring to FIG. 11 , in the annotation apparatus 100 of the artificial intelligence learning system according to an embodiment of the present invention, when the first image 10, which is the object of the annotation work for artificial intelligence (AI) learning, is loaded into the memory, , it is possible to identify the first image 10 and the second image 10-old that is temporally continuously photographed.

Specifically, the annotation apparatus 100 selects an image previously annotated by a worker performing an annotation operation on the first image 10 among images related to a project including the first image 10 as the object of the annotation operation. It can be identified by the second image 10-old. That is, the second image 10-old may be an image previously annotated by an operator among images belonging to the same project as the first image 10 .

If there are a plurality of images on which the operator has previously annotated, the annotation apparatus 100 determines the photographing location coordinates of the first image 10 and the photographing apparatus from among the images on which the operator has previously performed annotations, based on the properties of the image. The same person can identify the same image. In addition, the annotation apparatus 100 selects an image having the smallest difference between the photographing time and the photographing time of the first image 10 among the images in which the photographing location coordinates of the first image 10 and the photographing apparatus name are the same as the second image (10-old). can be identified as

Referring to FIG. 12 , the annotation apparatus 100 may identify the same object as the object 20-old included in the second image from among the objects 20 included in the first image 10 .

Specifically, the annotation apparatus 100 determines the first image 10 and the second image 10-old based on the similarity of the size, position, or shape occupied by the objects 20 and 20-old in the first image 10 and the second image 10-old. It may be determined whether the object 20 included in the image 10 and the object 20-old included in the second image 10-old correspond to the same object.

In this case, in order to determine the similarity of the objects 20 and 20-old, the annotation apparatus 100 performs edge extraction, binarization, and closed region identification with respect to the first image 10 and the second image 10-old. of image processing may be performed.

13 and 14 , the annotation apparatus 100 performs an annotation on the first image 10 based on the skeleton data 40 annotated on the same object 20-old included in the second image 10-old. ) to be applied to the object 20 included in the structure template 30 can be identified.

In addition, the process of performing the annotation of the first image 10 based on the identified structure template 30 is the same as that described with reference to FIGS. 7 to 10 , and thus the description thereof will not be repeated.

As described above, according to the embodiment of the present invention, in generating the skeleton data 40 for the object 20 included in the image 10 for artificial intelligence (AI) learning, the continuous image 10 -old) enables faster annotation by utilizing the previously performed annotation result.

Therefore, according to an embodiment of the present invention, skeleton data 40 for learning the experience, posture, or direction of the objects 20-old and 20 respectively included in a plurality of consecutively photographed images 10-old and 10 . can be created more quickly and accurately.

Hereinafter, the operation of the annotation apparatus 100 as described above will be described in more detail.

15 is a flowchart illustrating a method for generating skeleton data according to an embodiment of the present invention.

In describing the method for generating skeleton data according to an embodiment of the present invention with reference to FIG. 15 , the same description as described with reference to FIGS. 6 to 10 will not be repeated.

Referring to FIG. 15 , the annotation apparatus 100 may load an image to be subjected to an annotation operation for artificial intelligence (AI) learning into a memory under the control of the operator ( S100 ).

When the image is loaded into the memory, the annotation apparatus 100 may identify a structure template that can be used to generate the skeleton data ( S200 ).

Specifically, according to an embodiment of the present invention, the annotation apparatus 100 may identify one structure template from a database according to a property of a project, a property of an image, or a property of a worker.

Alternatively, according to another embodiment of the present invention, the annotation apparatus 100 may identify a structure template by using a result of an annotation previously performed on a continuous image. This process will be described later with reference to FIG. 16 .

The annotation apparatus 100 may overlay the identified structure template on the image and output it (S300).

In particular, in overlaying the structure template on the image, the annotation apparatus 100 may position one or more preset reference key points among key points included in the structure template on the preset feature points.

Here, the feature point may be a point defined in advance according to a position of a joint from which a bone diverges in the skeleton of the object or a position of a body organ with minimal movement among the body organs of the object.

As described above, by pre-positioning a reference key point from among the key points included in the structure template, it is possible to improve the convenience of the operator's annotation work.

The annotation apparatus 100 may move the position of one or more key points included in the structure template under the control of the operator ( S400 ). Then, the annotation apparatus 100 may set attribute information on the object included in the image under the control of the operator ( S500 ).

The annotation apparatus 100 may generate skeleton data based on location coordinates of key points included in the structure template, connection relationships between key points, and property information of objects ( S600 ). In addition, the annotation apparatus 100 may transmit an annotation operation result including the generated skeleton data to the learning data design apparatus 200 or the artificial intelligence learning apparatus 300 .

16 is a flowchart illustrating a structure template identification method using a pre-performed annotation result according to an embodiment of the present invention.

In describing a method of generating skeleton data for a continuous image according to an embodiment of the present invention with reference to FIG. 16 , the same description as described with reference to FIGS. 11 to 14 will not be repeated.

Referring to FIG. 16 , when the first image to be annotated is loaded into the memory, the annotation apparatus 100 may identify the first image and the second image temporally continuously photographed ( S210 ).

The annotation apparatus 100 may identify the same object as the object included in the second image among the objects included in the first image (S220).

In addition, the annotation apparatus 100 may identify a structure template applicable to the object included in the first image based on the skeleton data annotated on the same object included in the second image ( S230 ).

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention pertains that other modifications based on the technical idea of the present invention can be implemented in addition to the embodiments disclosed herein. It is obvious to those with ordinary knowledge. In addition, although specific terms have been used in the present specification and drawings, these are only used in a general sense to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Annotation devices: 100-1, 100-2, 100-3, … , 100-n; 100
Learning data design device: 200 Artificial intelligence learning device: 300
Communication unit: 105 Input/output unit: 110
Storage unit: 115 Template supply unit: 120
Annotation Workspace: 125 Skeleton Creator: 130

Claims (10)

Step of identifying, by the annotation device, a structure template for generating skeleton data of an object included in an image to be subjected to an annotation operation for artificial intelligence (AI) learning ;
outputting, by the annotation device, the identified structure template overlaid on the image;
moving, by the annotation device, a position of one or more key points included in the structure template according to an operator's control; and
generating, by the annotation device, skeleton data corresponding to an object included in the image based on the positional coordinates of the moved key point and a connection relationship between the moved key points,
The skeleton data is data related to a three-dimensional skeleton of an object for identifying the body shape, pose, or direction of the object included in the image,
The method for generating skeleton data, characterized in that the structure template is a data structure having a predefined number of key points according to the properties of the object and a connection relationship between the predefined key points. The method of claim 1 , wherein identifying the structure template comprises:
A method for generating skeleton data, characterized in that one structure template is identified from a database provided with structure templates standardized for each type of object. 3. The method of claim 2, wherein identifying the structure template comprises:
and identifying one structure template from the database according to a size, position or shape occupied by an object to which the structure template is to be applied in the image. The method of claim 1 , wherein the overlaid outputting comprises:
Overlaying the structure template on the image so that one or more preset reference key points among the key points included in the structure template are located on the preset feature points,
The method for generating skeleton data, characterized in that the feature point is a point defined in advance according to a position of a joint from which a bone diverges in the skeleton of an object or a position of a body organ with minimal movement among the body organs of the object. 5. The method of claim 4, wherein the overlaid outputting comprises:
With respect to the third artificial intelligence machine-learned using a dataset consisting of the size, position and shape of the object and the position of the feature point, the object to which the structure template is to be applied occupies in the image. A method for generating skeleton data, characterized in that the position of the feature point at which the reference key point is to be located is identified by querying based on the size, position or shape of the present. The method of claim 1, wherein moving the position of the key point comprises:
Among the key points included in the structure template, a key point that can correspond to an object included in the image and a key point that cannot correspond to an object included in the image are output to different user interfaces (UI) A method for generating skeleton data, characterized in that. The method of claim 1, wherein after moving the position of the key point,
The method further comprising the step of setting, by the annotation device, attribute information including body shape, posture, or direction information about the object,
The step of generating the skeleton data is
Generates the skeleton data by including the attribute information,
The step of setting the attribute information is
After identifying the body shape, posture, or direction information to be suggested to the worker based on the position coordinates of the moved key point and the connection relationship between the moved key points, the identified body shape, posture or direction information to be proposed A method for generating skeleton data, characterized in that output through a user interface (UI). The method of claim 1, wherein moving the position of the key point comprises:
According to the control of the operator, the structure template is characterized in that the output through a user interface (UI) a three-dimensional rotated shape, skeleton data generation method. The method of claim 1, wherein moving the position of the key point comprises:
Based on the coordinates of the position of the moved key point, a first edge connecting the first key point and the second key point included in the structure template, and the second key point and the third key point When the angle between the connecting second trunks is out of a preset critical angle range, the method for generating skeleton data, characterized in that outputting an error in the position movement of the key point through a user interface (UI). memory;
input output device; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
identifying, by the processor, a structure template for generating skeleton data of an object included in an image to be annotated for artificial intelligence (AI) learning;
overlaying, by the processor, the identified structure template on the image and outputting it through the input/output device;
moving, by the processor, the position of one or more key points included in the structure template according to the control of the operator input through the input/output device; and
Execute, by the processor, generating skeleton data corresponding to the object included in the image based on the positional coordinates of the moved key point and the connection relationship between the moved key points,
The skeleton data is data related to a three-dimensional skeleton of an object for identifying the body type, posture, or direction of the object included in the image,
The structure template is a data structure having a predefined number of key points according to the properties of the object and a connection relationship between the predefined key points, the computer program recorded on the recording medium.

Images