KR20220060722A - Image data labelling apparatus and method thereof - Google Patents

Abstract

An image data labeling apparatus including a memory to store an artificial neural network and a processor includes an image collecting unit, an image processing unit, and an object labeling unit. The image collecting unit collects first image data. The image processing unit generates first fake image data based on the first image data. The object labeling unit trains the artificial neural network based on the first image data and the first fake data, classifies a pixel region in the first image data, based on the trained artificial neural network, and performs labeling for an object, based on the classified pixel region. When the image collecting unit collects second image data, the object labeling unit re-trains the artificial neural network, in which a feature and a weight of the artificial neural network, which are determined when learning the first image data, are alternatively provided, while employing, as input values, second fake image data generated based on the first image data, the fake image data, the second image data, and the second image data.

Description

Image data labeling learning method and apparatus {IMAGE DATA LABELLING APPARATUS AND METHOD THEREOF}

This embodiment relates to an image data labeling learning method and apparatus for performing active learning in data labeling for collecting, classifying, and processing a large amount of data.

In general, the data labeling task is to assign a specific value to the training data before the modeling work of machine learning (ML) or the modeling work of deep learning (DL). Although this data labeling work is the basis of the AI (Artificial Intelligence) industry, it is recognized as a key factor for achieving the development of the AI industry in companies. Conventional data labeling has a disadvantage in that it is costly and takes a long time to be performed by a person. In addition, even if automatic data labeling is performed, at least thousands or tens of thousands of images are required to classify images. Labeling such a large amount of data is costly, including time, and as more images are required for labeling, consistent and accurate labeling becomes difficult.

The present invention is to solve the problems described above, and requires the minimum data required for data labeling operation used as preprocessing, and at the same time enables the reduction of the labeling operation time, and at the same time increases the labeling accuracy. It relates to a method and apparatus for learning image data labeling.

According to an embodiment of the present invention for solving the above problems, an image data labeling learning apparatus including a memory and a processor storing an artificial neural network includes: an image collecting unit for collecting first image data; an image processing unit generating first fake image data based on the first image data; and training the artificial neural network based on the first image data and the first fake image data, classifying a pixel region in the first image data based on the learned artificial neural network, and based on the divided pixel region and an object labeling unit for labeling the object, and when the image collecting unit collects second image data, the object labeling unit includes the first image data, the fake image data, the second image data, and the second image data. Using the second fake image data generated by the second image data as an input value, the artificial neural network in which the feature values and the weights of the artificial neural network determined when the first image data is learned are alternately trained again.

The image processing unit may generate fake image data in which annotation coordinate data for an object included in the first image data is maintained.

The image processing unit may generate a plurality of the fake image data including the annotation coordinate data.

When learning the artificial neural network based on the second image data, the object labeling unit may train the artificial neural network using the first image data and a partial region of the first fake image data as input values.

The object labeling unit may perform an automatic image search through the Internet network, determine the type of the object determined based on the automatic image search, and label the object based on the determined type of the object. there is.

The object labeling unit may classify a pixel area in the first image data based on color data and depth data included in the first image data, and perform labeling on the object based on the divided pixel area.

It may further include; a labeling inspection unit that calculates the accuracy of the labeling given by the object labeling unit based on the category information of the object stored in advance, and adjusts the feature value and the weight of the artificial neural network based on the calculated accuracy. .

According to the present invention, the data required for the data labeling operation is required to a minimum, and at the same time, it is possible to shorten the operation time of the labeling, and at the same time, it is possible to increase the accuracy of the labeling.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram for explaining a function of an image data labeling learning method and an apparatus according to an embodiment.
2 is a diagram illustrating an example of a result of performing labeling by dividing an area of image data according to an embodiment.
3 is a view for explaining each configuration of the disclosed image data labeling learning apparatus and the operation of the configuration.
4 and 5 are flowcharts for a method for learning labeling of image data.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims.

In the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. have meaning Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in an embodiment of the present invention, an ideal or excessively formal meaning is not interpreted as

The shapes, sizes, proportions, angles, numbers, etc. disclosed in the drawings for explaining the embodiments of the present invention are illustrative and the present invention is not limited to the illustrated matters. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. When 'including', 'having', 'consisting', etc. mentioned in this specification are used, other parts may be added unless 'only' is used. When a component is expressed in the singular, cases including the plural are included unless otherwise explicitly stated.

In interpreting the components, it is construed as including an error range even if there is no separate explicit description.

In the case of a description of the positional relationship, for example, when the positional relationship of two parts is described as 'on', 'on', 'on', 'beside', etc., 'right' Alternatively, one or more other parts may be positioned between two parts unless 'directly' is used.

Reference to an element or layer “on” another element or layer includes any intervening layer or other element directly on or in the middle of another element. Like reference numerals refer to like elements throughout.

The size and thickness of each component shown in the drawings are illustrated for convenience of description, and the present invention is not necessarily limited to the size and thickness of the illustrated component.

Each feature of the various embodiments of the present invention may be partially or wholly combined or combined with each other, and as those skilled in the art will fully understand, technically various interlocking and driving are possible, and each embodiment may be implemented independently of each other, It may be possible to implement together in a related relationship.

Hereinafter, the working principle and embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a view for explaining the function of the image data labeling learning method and the apparatus according to an embodiment.

The image data labeling apparatus 100 recognizes an object in the collected image 101 . Referring to FIG. 1 , the object recognized in the collected image 101 may be a person. The image data labeling apparatus 100 generates an image 102 in which a box (bounding box) is created in a recognized object, and an image 103 in which a labeling that can insert information about the recognized object is generated. , labeling the image data.

The image that has been labeled is an input value that is input to machine learning (ML) or deep learning (DL). The machine learning or deep learning 200 may output actions or results required for autonomous driving or other various AI industries by using the information included in the labeling.

On the other hand, as for the image shown in FIG. 1, the input image is a part of the game image, but the image data is not limited thereto, and various image data defined in pixels is sufficient, and is not limited thereto.

2 is a diagram illustrating an example of a result of performing labeling by dividing an area of image data according to an embodiment.

In order to label image data, it is necessary to perform image segmentation. Image segmentation is distinguished from AI classification or object detection.

Specifically, classification is a problem of determining what kind of photo or object the image data represents when given image data. Therefore, even if a plurality of image data including different cats is input, the classification should classify the plurality of image data as image data of a cat.

Object detection not only classifies objects, but also finds where objects are located in image data through a bounding box. In other words, object detection should perform localization.

Image segmentation is to determine a class for each object by segmenting an area within the image data. Specifically, image segmentation includes semantic segmentation and panoptic segmentation shown in FIG. 2 .

If there are multiple people, trees, and buildings in image data, semantic segmentation classifies classes into 'people', 'trees' and 'buildings', and recognizes multiple people as the same class from trees and buildings in the background. will be. Therefore, in semantic segmentation, a plurality of people are recognized as one and the same class (purple), not as individual objects.

Panoptic segmentation can distinguish 'people' of the same class as different people according to their location. That is, panoptic segmentation performs object recognition by determining whether there is an object or no object for each pixel in the image data.

In the disclosed embodiment, the image data labeling method may perform panoptic segmentation. To this end, the image data labeling method creates a bounding box that classifies which class a pixel value in the image data belongs to, records the classified class of each pixel in a pixel unit in a prediction mask, or detects a feature point in the image. (Keypoint detection) or DensePose and Semantic Segmentation, which map pixels on image data to a 3D surface, can be used comprehensively.

The disclosed image data labeling method generates labeling by inserting information into an object divided by panoptic segmentation, and performs image annotation as shown in FIG. 2 .

On the other hand, in order to perform such a data labeling method, a large amount of image data required for the above-described method is required, and data necessary for learning the artificial neural network used must be collected. In the image data labeling method described below, a method for minimizing data required for learning and a method for increasing the accuracy of the data labeling method despite the minimum data will be described in detail.

3 is a view for explaining each configuration of the disclosed image data labeling learning apparatus and the operation of the configuration.

The disclosed image data labeling learning apparatus 100 includes, in hardware, a memory (not shown) for storing image data and various programs, and a processor (not shown) for operating the stored programs. In particular, the disclosed image data labeling learning apparatus 100 stores the artificial neural network required for learning, the processor trains the artificial neural network, and the learned artificial neural network is again stored in the memory. If image data is collected, the processor generates (infers) a label on the image data through the trained artificial neural network.

On the other hand, the image data labeling learning apparatus 100 may express the above-described hardware configuration in software. Referring to FIG. 3 , the image data labeling learning apparatus 100 includes an image collection unit 110 , an image processing unit 120 , an object labeling unit 130 , and a labeling inspection unit 140 .

Specifically, the image collecting unit 110 collects image data. The image data may be data composed of pixel values, as described above with reference to FIG. 1 . The image collection unit 110 may receive image data from the outside through a wired/wireless communication interface, or may use pre-stored image data.

The image processing unit 120 generates a plurality of fake image data based on the collected image data. A plurality of fake images may be generated based on the image data collected by the image collecting unit 110 . For example, the image processing unit 120 may generate fake image data obtained by changing weather conditions. However, even if the image processing unit 120 generates the fake image data, the annotation coordinate data is maintained in the original image data.

When the image processing unit 120 generates at least one or more fake image data, the object labeling unit 130 uses the collected image data and the fake image as input values to train the artificial neural network.

Specifically, the object labeling unit 130 uses the image data and the generated plurality of fake image data as training data of the artificial neural network. Also, the object labeling unit 130 may use only a partial region of the image data and a partial region of the fake image data as training data.

The object labeling unit 130 may accumulate image data and use it as input data. For example, a game image as shown in FIG. 3 may be input as image data (hereinafter, first image data). The object labeling unit 130 may generate a plurality of fake image data, but the accuracy of the artificial neural network may be low only with such fake image data. Therefore, after labeling the first image data, the object labeling unit 130 stores the first image data in the memory without discarding it.

Afterwards, the object labeling unit 130 may receive an image other than the game image as image data (hereinafter referred to as second image data). When the image collecting unit 110 collects the second image data, the object labeling unit 103 generates the first image data, the fake image data of the first image data, the second image data, and the second image data. An artificial neural network is trained using the second fake image data as an input value. In addition, the object labeling unit 130 re-learns the artificial neural network by alternating the feature values and weights of the artificial neural network determined during learning of the first image data, so that the artificial neural network performs labeling with minimal data. can learn to do it.

Meanwhile, the artificial neural network used by the object labeling unit 130 may vary. For example, the neural network used by the object labeling unit 130 may include a Convolution Neural Network (CNN). The CNN can be composed of a feature learning that generates input image data as a feature map including feature values and a neuron network that labels the generated feature map as an input value. Here, the neuron network may be composed of a multi-layer perceptron, which multiplies an input value by a weight, adds a bias, and then transfers the transformed input value to a transfer function. or activation function). That is, the perceptron outputs only the transformed input value that satisfies the threshold value of the transfer function as an output value.

When the object labeling unit 130 trains the artificial neural network based on image data and fake image data, by alternately using the above-described feature values and weights of the artificial neural network, the artificial neural network can perform labeling with minimal data. can be taught to

On the other hand, the artificial neural network used by the object labeling unit 130 may not necessarily be a CNN, and may include various embodiments such as a fully convolutional model (FCN), which is a modification of CNN. In addition, the object labeling unit 130 may perform labeling through an artificial neural network including a neural network structure other than the perceptron necessarily including feature values or weights.

Referring back to FIG. 3 , the object labeling unit 130 differentiates (localization, 131) a pixel region in the image data collected through the learned artificial neural network, and on the basis of the divided pixel region, the object included in the image data. perform labeling.

Methods for the object labeling unit 130 to perform labeling may vary. The object labeling unit 130 may perform labeling through the depth data 132 , the color data 134 , and the automatic image search 133 included in the image data while using the Panoptic Segmentation described above in FIG. 2 .

Specifically, the object labeling unit 130 may distinguish a background and an object in the image data based on the depth data 132 and the color data 134 . For example, if the image data is data including temporally continuous frames, the object labeling unit 130 warps previous image data based on current image data as a reference. The object labeling unit 130 may fuse the converted previous image and the current image based on a probability algorithm, estimate depth data in the image data, and estimate what is a background model that is distinguished from the object.

When the object and the background are distinguished, the object labeling unit 130 may perform an automatic image search 133 to collect information on the object. For example, the object labeling unit 130 may use as labeling information data connected to categories while users upload images to the Internet network.

When the object is determined based on the above-described various data (135), the object labeling unit 130 performs labeling for each object (136).

The labeling method used by the object labeling unit 130 may be various in addition to the above-described method. That is, the disclosed image data labeling learning apparatus 100 processes image data to accumulate fake image data and data necessary for learning of an artificial neural network, and when learning only artificial neurons, the feature values and weights used for previous learning results are alternated It is enough to use it, and there is no limit to how to perform labeling.

After the object labeling unit 130 performs object labeling, the labeling inspection unit 140 reviews the labeled image data.

Specifically, the labeling inspection unit 140 calculates the accuracy of the labeling (141). For example, the labeling inspection unit 140 may pre-store the labeling value input by the user for the collected image data, and compare it with the labeling result generated by the object labeling unit 130 . In addition, the labeling inspection unit 140 may perform an automatic image search through the Internet network, and may compare it with the labeling generated by the object labeling unit 130 based on the type of object derived from the automatic image search result. The calculated accuracy may correspond to an error rate of the artificial neural network, which may be used to adjust features and weights of the artificial neural network.

4 and 5 are flowcharts for a method for learning labeling of image data. In order to avoid overlapping descriptions, they will be described together below.

The image data labeling learning method according to the disclosed embodiment collects first image data ( 300 ).

Here, the first image data means a set of representative images on which labeling is performed. That is, the second image data is an image that can be collected by the image collecting unit 110 after the labeling inspection of the first image data is finished.

The image data labeling learning method generates a plurality of first fake image data ( 310 ).

The fake image data may be generated based on the first image data, but the annotation coordinate data is maintained in order to label the object.

The image data labeling learning method learns a pre-stored artificial neural network based on the first image data and the first fake image data ( 320 ).

A method for learning the artificial neural network will be described with reference to FIG. 5 (A).

In the image data labeling learning method, first image data is input to the learned artificial neural network ( 330 ).

The artificial neural network included in the object labeling unit 130 classifies regions based on pixel values of the first image data and performs object labeling ( 340 ).

In an embodiment, the image data labeling learning method classifies objects through localization and panoptic segmentation. In another embodiment, the image data labeling learning method may classify a pixel area in the first image data based on color data and depth data, and perform labeling on an object based on the divided pixel area.

When an object is identified, the image data labeling learning method inserts information into the classified object. In an embodiment, the image data labeling learning method performs automatic image search through an Internet network, determines the type of object determined based on automatic image search, and performs labeling on the object based on the determined type of object can do.

When the labeling is performed, the image data labeling learning method adjusts the labeling check and the artificial neural network ( 350 ).

Specifically, the image data labeling learning method calculates the labeling accuracy given by the object labeling unit 130 based on the category information of the object stored in advance, and adjusts the feature value and weight of the artificial neural network based on the calculated accuracy. The accuracy of the neural network can be improved.

Referring to FIG. 5 , the image data labeling learning method trains an artificial neural network by accumulating first image data and fake image data in addition to using the first image data and fake image data as input values.

In the image data labeling learning method, after collecting the second image data ( 400 ), second fake image data generated based on the second image data is generated ( 410 ). In addition, the image data labeling learning method may use the first image data and the first fake image data as learning data of the second image data ( 411 ). Also, in the image data labeling learning method, the artificial neural network is trained again by alternating the feature values and weights used when learning the artificial neural network using the first image data ( 412 ).

When the artificial neural network is trained (420), as shown in FIG. 4, the image data labeling learning method inputs second image data to the artificial neural network (430), classifies regions based on pixel values in the second image data, and object labeling is performed (440).

In the image data labeling learning method, when labeling is performed, the labeling included in the second image data is inspected, and the artificial neural network is adjusted based on the calculated accuracy ( S450 ).

Through this, the image data labeling learning method can efficiently perform pre-processing that requires a large amount of data, time and cost through minimal input data and labeling inspection.

Although embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made within the scope without departing from the technical spirit of the present invention. . Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: image data labeling learning device 110: image collection unit
120: image processing unit 130: object labeling unit
140: labeling inspection unit 200: ML, DL

Claims (7)

In an image data labeling learning apparatus comprising a memory storing an artificial neural network and a processor operating the artificial neural network,
an image collecting unit for collecting first image data;
an image processing unit generating first fake image data based on the first image data; and
training the artificial neural network based on the first image data and the first fake image data, classifying a pixel area in the first image data based on the learned artificial neural network, and based on the divided pixel area Including; an object labeling unit for performing labeling on the object;
When the image collection unit collects second image data, the object labeling unit receives second fake image data generated by the first image data, the fake image data, the second image data, and the second image data as input values. , an image data labeling learning apparatus for re-learning an artificial neural network in which feature values and weights of the artificial neural network determined during learning of the first image data are alternated.
The method of claim 1,
The image processing unit,
Image data labeling learning apparatus for generating fake image data in which annotation coordinate data for an object included in the first image data is maintained.
3. The method of claim 2,
The image processing unit,
Image data labeling learning apparatus for generating a plurality of the fake image data including the annotation coordinate data.
The method of claim 1,
The object labeling unit,
When the artificial neural network is learned based on the second image data, the image data labeling learning apparatus for learning the artificial neural network using the first image data and a partial region of the first fake image data as input values.
The method of claim 1,
The object labeling unit,
Image data labeling learning apparatus for performing an automatic image search through an Internet network, determining the type of the determined object based on the automatic image search, and performing labeling on the object based on the determined type of the object. The method of claim 1,
The object labeling unit,
Image data labeling learning apparatus for classifying a pixel area in the first image data based on color data and depth data included in the first image data, and performing labeling on an object based on the divided pixel area.
The method of claim 1,
Image data further comprising: a labeling inspection unit that calculates the labeling accuracy given by the object labeling unit based on the category information of the object stored in advance, and adjusts the feature value and the weight of the artificial neural network based on the calculated accuracy Labeling learning device.

Images