KR102340998B1 - Auto labeling method and system - Google Patents

Abstract

Disclosed are an auto labeling method and a system. According to one embodiment of the present invention, an auto labeling method performed by an auto-labeling system includes the steps of: generating an auto labeling model using a data set uploaded from an operator; performing auto labeling on the data set uploaded from the operator using the generated auto labeling model; selecting data to be labeled from the uploaded data set based on a result predicted through the auto labeling performed; and re-learning the auto labeling model using the data set on which the data inspection and re-labeling of the selected data have been completed. Therefore, the accuracy of auto-labeling and the accuracy of the model are improved.

Description

AUTO LABELING METHOD AND SYSTEM

The description below relates to auto-labeling technology.

Recently, due to the development of artificial intelligence, various high-performance algorithms are being developed. At the same time, interest in data labeling, which is the data set creation required for artificial intelligence learning, is also rapidly attracting attention.

Data labeling refers to the task of collecting, classifying, and processing data required for machine learning, and it is a very important task that occupies more than 80% of the training data construction time. The current data labeling method is mainly in the form of crowdsourcing, and this method generates data by labeling data collected by several people. Since a low number of people generate data, it is cumbersome to investigate all the labeling data, and since there is a possibility that there is a possibility that there is a difference between each person about the labeling data, it can affect the model accuracy when learning the artificial intelligence model. .

Accordingly, there is a need for a labeling technique that supplements the problems of data labeling and can be easily used even by beginners who do not have much knowledge in machine learning.

It is possible to provide an auto-labeling method and system that performs prediction on a data set using a pre-learning model, and improves the accuracy of auto-labeling and the accuracy of the model through inspection and correction of the predicted data.

It is possible to provide an auto-labeling method and system for gradually reducing user intervention through iterative re-learning of the auto-labeling model.

The auto-labeling method performed by the auto-labeling system includes: generating an auto-labeling model using a data set uploaded from an operator; performing auto-labeling on the data set uploaded from the worker using the generated auto-labeling model; selecting data to be labeled from the uploaded data set based on a result predicted through the auto-labeling performed; and re-learning the auto-labeling model using a data set on which data inspection and re-labeling of the selected data have been completed.

The selecting may include deriving an image label including label name, accuracy, and location data for each label included in each image through the auto-labeling performed, and using the derived image label to obtain data within a preset accuracy range delivering the set to the operator; And as the label name and location data included in the data set of the preset accuracy range are modified by the operator, inspection and re-labeling are completed, and storing the log information corrected through the completed inspection and re-labeling in a database may include

In the selecting step, as the auto-labeling is repeatedly performed, it is determined whether a predicted result is included in a preset accuracy range, and based on the predicted result, data included in a preset accuracy range is used as a weakness image. and collecting a plurality of learning indicators including the type of the selected weakness image and the class data type in which the correction has occurred in the selected data to be labeled.

In the re-learning, auto-labeling is re-performed on weak data, which is part data of the completed data set or the completed data set, using the re-learned auto-labeling model, and predicted through the re-performed auto-labeling. Comparing the result and the labeled data set from the operator, and deriving a guide for the quantitative learning index of the retrained auto-labeling model and data to be additionally uploaded through the comparison.

The re-learning includes deriving accuracy based on the correct answer error rate of the result predicted through the re-performed auto-labeling and the data set labeled from the operator, and provides an additional data guide including the derived accuracy, , presents an example of the selected weak image type, displays the color distribution of the weak image, displays the type of class that the current auto-labeling model is difficult to predict, recommends a class to be added preferentially, and currently It may include the step of presenting the correct answer for each class of the auto-labeling model, and the distance to the correct answer.

In the performing step, clusters for each class are created through distribution of work files of the same or similar properties using a cluster model in order to distribute the auto-labeling completed data to each worker, and the generated clusters for each class are generated. It may include the step of distributing to each worker.

The auto-labeling system includes: a model generator for generating an auto-labeling model using a data set uploaded from an operator; a labeling performing unit for performing auto-labeling on the data set uploaded from the worker using the generated auto-labeling model; a data selection unit for selecting data to be labeled from the uploaded data set based on a result predicted through the auto-labeling performed; and a re-learning unit for re-learning the auto-labeling model by using the data set on which the data inspection and re-labeling of the selected data have been completed.

Through Auto-Labeling and Easy-Labeling, it is possible to reduce the working time of the operator performing data labeling and increase the consistency.

Updating the auto-labeling model can improve the labeling and performance of the labeling model.

Through iterative re-learning of the auto-labeling model, user intervention can be gradually reduced.

1 is a diagram for explaining data labeling.
FIG. 2 is a diagram for explaining an operation of generating an initial model for auto-labeling, according to an embodiment.
3 is a view for explaining the general operation of the auto labeling system according to an embodiment.
4 is a diagram for explaining a manual labeling method.
5 is a diagram for explaining an auto-labeling operation according to an embodiment.
6 is a view for explaining an inspection operation of auto-labeling, according to an embodiment.
7 is a diagram for explaining an accelerating operation of auto labeling according to an embodiment.
8 and 9 are diagrams for explaining an operation of providing a guide for additional data according to an embodiment.
10 is a diagram for explaining an operation of selecting data for each worker for improving work speed, according to an embodiment.
11 is a block diagram illustrating a configuration of an auto labeling system according to an embodiment.
12 is a flowchart illustrating an auto labeling method in an auto labeling system according to an embodiment.
13 is a diagram for explaining the accuracy of auto labeling according to an embodiment.
14 is an example illustrating a result screen of data annotation according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining data labeling.

In general, as a method of labeling image data, a method of generating a mask by hitting a square box on a target class (detection) or painting a color (segmentation) is representative as shown in FIG. 1 according to the purpose of learning . In more detail, for example, if a "person" in a photo/video is divided into a box (Bounding-box), a data set for a detection model is created, Polygon), a data set for a segmentation model can be created when it is distinguished from the background and other objects by coloring with a brush. Through this, the artificial intelligence can learn to distinguish between the "person" area, the background, or other objects in the input photo/video.

However, since users (workers) individually perform labeling, if many users perform labeling, data consistency may be inconsistent, and consequently, it may affect the training of the artificial intelligence model. In addition, in order to train artificial intelligence, data labeling is the most effective for training an artificial intelligence model because the artificial intelligence can accurately classify the object only when there are as few as 1,000 to as many as hundreds of thousands of data sets of these photos/videos. It takes a lot of time.

Accordingly, in the embodiment, in order to solve the above problems, an auto-labeling operation for reducing the user's convenience and working time in the labeling operation will be described.

FIG. 2 is a diagram for explaining an operation of generating an initial model for auto-labeling, according to an embodiment.

The data set 201 may be uploaded to the auto-labeling system. As the data set is uploaded, a label may be registered (eg, a class (human, bicycle, etc.) that AI predicts). In this case, when registering a label, a category of the corresponding label may be selected (whether the corresponding label is a type close to an animal-human or a type close to an object-industrial field object, etc.). For auto-labeling, a sample selection algorithm can be applied with unsupervised learning. When a data set is uploaded, images can be classified into a plurality of (n) types through a clustering model (unsupervised learning). For example, in the case of a data set in which dogs, cats, and people are mixed, images may be classified as dogs, cats, and people. Manual labeling may be performed by each operator 203 on a portion of the screening data for each cluster in the screening data set 202 for each cluster classified through the clustering model. As the manual labeling progresses, the obtained data sets may be merged 204 to generate an initial model 210 .

For example, in each set classified through the clustering model, initial manual labeling is performed by each worker for 10 to 30 sheets, and the data sets subjected to the initial manual labeling are merged to generate an initial model. In this case, transfer learning may proceed to the BaseModel based on the category set in the data set. It is possible to provide high learning effect with a low learning time by performing transfer learning on a model of a similar category that is already owned. Through this, generation of the initial model may be completed.

Referring to FIG. 4 , a manual labeling operation will be described. In manual labeling, the data set is distributed to a plurality of workers, respectively, and when labeling is completed from a plurality of workers using the decomposed data set to a plurality of workers, the data set with labeling completed from a small number of inspectors is inspected, and the inspector According to the opinion of (203), the data set can be completed as re-labeling including the label position, class, etc. proceeds. In this case, as the relabeled data set is collected through a physical method, a relabeled data set may be obtained.

3 is a view for explaining the general operation of the auto labeling system according to an embodiment.

The auto-labeling system predicts the input vision data once using the auto-labeling model 300, and then inspects and corrects the data predicted by the operator (user) to improve the accuracy of auto-labeling and further improve the auto-labeling model. accuracy can be improved. In this case, the auto-labeling model 300 is a pre-trained model, and may include a pre-trained model (described as 'SOTA' model in FIG. 3) or a model (initial model) generated by a user.

The auto-labeling system can reduce the working time of the operator performing data labeling and increase the consistency through the auto-labeling and easy-labeling functions. The auto labeling system can perform inspection at the same time as labeling.

Auto labeling can include two functions. Detect all classes in images captured in a general environment (e.g., images of roads, images of people passing by, etc.) Alternatively, using an artificial intelligence model created by the user (eg, the initial model described in FIG. 2 ) in an image (eg, product defect detection, locating a handshake, etc.) taken in a specific environment A predefined class can be detected. In this case, the class may be designated by a user (worker).

The easy labeling may serve to detect a predicted value for one class. A class in the data set or a predefined class is detected through a pre-trained model (SOTA model) or a user-generated model (initial model).

In the uploaded data set, the class can be detected through SOTA or an initial model created by the user. In this case, the detected class guarantees accuracy above a threshold that can be recommended to a user. The labeled data may include class information in the form of a purpose (classification, detection, segmentation) of the predicted data. The operator 203 may verify the detected class information in the manner of classification error, false positive, false negative, and IoU correction according to the type of purpose. The verified information may be stored as a labeling data set. In this case, the data corrected through verification may be used as a re-learning data set to improve the accuracy of the auto-labeling model 300 .

5 is a diagram for explaining an auto labeling operation according to an embodiment.

Data sets can be uploaded to the auto-labeling system by a small number of operators. The auto-labeling system may perform auto-labeling on the uploaded data set. In this case, the inspection may proceed simultaneously with the labeling progress. The auto-labeling system can select the data to be labeled among the uploaded data sets using the auto-labeling model (review role). In other words, some data to be labeled may be selected. Verification of the data selected by a small number of workers, and correction of label positions and classes may proceed. Thereafter, as the labeling is completed, the labeled data set may be completed through automatic merging of the labeled data. The auto-labeling system can retrain the auto-labeling model using the completed data set.

In this way, the work method using auto-labeling can be switched from the method in which the existing inspector completely checks whether the label is correct in a large amount of data and requests correction again, to a method in which most of the role of the inspector is automatically performed by artificial intelligence. have. Accordingly, the intervention of the inspector can be gradually reduced according to the iterative re-learning of the auto-labeling model.

6 is a view for explaining an inspection operation of auto labeling according to an embodiment.

As the result of the data set is predicted through the auto-labeling model, the inspection operation may proceed. Data sets can be uploaded to the auto-labeling system. Data sets can be fed into the auto-labeling model. Prediction of data can be performed through the auto-labeling model. As the prediction is performed on the data set using the auto-labeling model, a result may be obtained. In this case, the operator may be instructed to inspect the weakness data included in the preset accuracy range for each data. For example, the accuracy range may be set between 0 and 50 by the user/operator. Inspection and re-labeling of the weakness data included in the preset accuracy range from each operator 203 may be performed. As the inspection and re-labeling of the weak data included in the preset accuracy range is completed, the auto-labeling model may be re-trained through the obtained data set. In this case, the data set not included in the preset accuracy range may be inputted into the auto-labeling model and re-learned without re-labeling. As such, the weakness data refers to some of the input data input to the auto-labeling model. As the auto-labeling model is retrained using weak data, the learning rate is high, and the auto-labeling model can be optimized quickly.

Alternatively, the accuracy may be set to a preset value (eg, 50). The operator may be instructed to inspect the weakness data included below the preset accuracy for each data. Inspection and re-labeling of weakness data corresponding to less than a preset accuracy from each worker 203 may be performed. The auto-labeling model may be re-trained through the acquired data set as the inspection and re-labeling of the weak data included below the preset accuracy is completed. As such, the weakness data refers to some of the input data input to the auto-labeling model. As the auto-labeling model is retrained using weak data, the learning rate is high, and the auto-labeling model can be optimized quickly.

First, as the data set is predicted through the auto-labeling model, an image label including a label name, accuracy, location information, etc. for each label included in each image may be derived. Using the derived image label, only a relatively low-accuracy data set can be delivered to the operator (a weakness in which the model is relatively difficult to predict). The operator can complete the inspection and re-labeling by modifying the label name, location, etc. in the data. In this case, the modification log may be stored in the database. The auto-labeling model can be retrained using the data set that has been re-labeled by the operator.

By repeatedly performing this process, it is possible to improve the accuracy of auto-labeling and to minimize the amount of data for a labeling object that a human being works with.

7 is a diagram for explaining an accelerating operation of auto labeling according to an embodiment.

In the embodiment, a strategy for accelerating auto-labeling will be described. The auto-labeling system can shorten and reduce the efficiency of the labeling operation and the working time through the repetition of auto-labeling. The auto-labeling system can provide a guide to quantitative learning indicators of auto-labeling models and additional datasets for accelerating learning.

As the auto-labeling system performs data labeling through repetitive auto-labeling, it continuously collects two learning indicators for inspection. In this case, the two learning indicators may include a weak image type (accuracy-based image type) and a class data type in which correction has occurred.

8 is a diagram for explaining an operation of providing a guide for additional data, according to an embodiment.

The auto-labeling system can retrain the auto-labeling model through the corrected data set as the operator inspects and re-labels the data after auto-labeling. Thereafter, the modified data set may be input to the retrained auto-labeling model. The auto-labeling system can re-perform auto-labeling on the corrected data set using the retrained auto-labeling model. As the auto-labeling system re-performs auto-labeling, the re-predicted result can be compared with the data set labeled by the inspector. At this time, the auto-labeling system uses quantitative learning indicators of the auto-labeling model at the present time and additional data to be uploaded (additional data) by using the change in the accuracy of the weak image type and the correct answer error of the data labeled by the operator (correction data). ) can be derived. Additional data focusing on weaknesses can be uploaded to the data set by the administrator.

More specifically, referring to FIG. 9 , the auto-labeling system may compare existing weakness data and operator-labeled data (data in which correction occurs) with the results of the retrained auto-labeling model. For example, the auto-labeling system can compare the location of a particular class in the data, the class name, the class type, and the color/brightness of an image. The auto-labeling system can provide the quantitative learning indicators of the current auto-labeling model, the weak image type, the class distribution of the additional data, etc. as a guide for the additional data. The auto-labeling system may derive the accuracy through the correct error rate of the corrected data and the result data of the retrained auto-labeling model, and may provide a guide for additional data including the derived accuracy. The auto-labeling system may provide an example of the weak image type and display the color distribution of the weak image. For example, the auto-labeling system may store information about the weak point image by type. The auto-labeling system may provide an example of a weak point image type based on information about the stored weak point image. This allows the operator to further supplement similar types of images. The auto-labeling system can indicate the type of class that the current auto-labeling model is difficult to predict, and recommend a class to be added first. The auto-labeling system can present the correct answer for each class of the current auto-labeling model, and the distance from the correct answer.

10 is a diagram for explaining an operation of selecting data for each worker for improving work speed, according to an embodiment.

The more frequent the use of tools such as class change and position correction during labeling work, the faster the work speed decreases, and it can cause deviations and decreased accuracy when working for a long time. The auto labeling system minimizes repetitive tasks such as clicks and UX changes, and data sorting technology can be applied to improve the professionalism of each worker.

The auto-labeling system uses a cluster model to distribute the auto-labeling completed data to each worker, and creates a cluster for each class by distributing work files with the same or similar properties, and distributes the generated clusters for each class to each worker. can do. Through this, it is possible to improve work efficiency by distributing images of the same or similar class and same/similar shape to each worker as much as possible to be concentrated.

11 is a block diagram illustrating a configuration of an auto labeling system according to an embodiment, and FIG. 12 is a flowchart illustrating an auto labeling method in the auto labeling system according to an embodiment.

The processor of the auto labeling system 100 may include a model generating unit 1110 , a labeling performing unit 1120 , a data selecting unit 1130 , and a re-learning unit 1140 . These may be expressions of different functions performed by the processor according to a control instruction provided by the program code stored in the auto-labeling system of the processor. The processor and components of the processor may control the auto labeling system to perform steps 1210 to 1240 included in the auto labeling method of FIG. 12 . In this case, the processor and the components of the processor may be implemented to execute instructions according to the code of the operating system included in the memory and the code of at least one program.

The processor may load the program code stored in the file of the program for the auto-labeling method into the memory. For example, when a program is executed in the auto-labeling system, the processor may control the auto-labeling system to load the program code from the program file into the memory according to the control of the operating system. At this time, each of the model generating unit 1110, the labeling performing unit 1120, the data selecting unit 1130, and the re-learning unit 1140 executes a command of a corresponding part of the program code loaded in the memory to perform subsequent steps ( 1210 to 1240) may be different functional representations of the processor.

In operation 1210, the model generator 1110 may generate an auto-labeling model using the data set uploaded by the operator.

In step 1220, the labeling performing unit 1120 may perform auto-labeling on the data set uploaded by the operator using the generated auto-labeling model. The labeling performing unit 1120 generates a cluster for each class through distribution of work files of the same or similar properties using a cluster model in order to distribute the auto-labeling completed data to each worker, and recalls the generated clusters for each class. It can be distributed to each worker.

In operation 1230, the data selector 1130 may select data to be labeled from the uploaded data set based on the result predicted through the auto-labeling performed. The data selection unit 1130 derives an image label including label name, accuracy, and location data for each label included in each image through the performed auto-labeling, and uses the derived image label to set a data set within a preset accuracy range. can be transmitted to the worker. The data sorting unit 1130 completes the inspection and re-labeling as the label name and location data included in the data set of the preset accuracy range are corrected by the operator, and records the log information corrected through the completed inspection and re-labeling to the database. can be saved The data selection unit 1130 determines whether a predicted result is included in a preset accuracy range as the auto-labeling is repeatedly performed, and selects data included in a preset accuracy range as a weakness image based on the predicted result. And, it is possible to collect a plurality of indicators including the type of the selected weakness image and the class data type in which the correction occurred in the selected data to be labeled.

In step 1240, the re-learning unit 1140 may re-learn the auto-labeling model by using the data set for which data inspection and re-labeling of the selected data have been completed. The re-learning unit 1140 re-performs auto-labeling on weak data, which is a partial data of a completed data set or a completed data set, using the re-learned auto-labeling model, and obtains results predicted through the re-performed auto-labeling and the operator. By comparing the labeled data sets, it is possible to derive the quantitative learning indicators of the retrained auto-labeling model and guides for the data to be additionally uploaded. The re-learning unit 1140 derives accuracy based on the error rate of the correct answer predicted through the data set labeled from the operator and the re-performed auto-labeling, provides an additional data guide including the derived accuracy, and selects Presents examples of weak image types, displays the color distribution of weak images, displays the types of classes that the current auto-labeling model is difficult to predict, recommends classes to be added first, and the current auto-labeling model You can present the correct answer for each class, and the distance from the correct answer.

In this case, steps 1220 to 1240 may be repeatedly performed.

13 is a diagram for explaining the accuracy of auto labeling according to an embodiment.

The auto-labeling system can reduce the amount of work by learning the auto-labeling and auto-labeling model by applying the user's change history and weakness data.

The auto-labeling system can preferentially learn the types of data with many changes based on the revision (change history) of the inspector. The auto-labeling system can perform weakness-based learning on the auto-labeling model. When performing auto-labeling, an accuracy range can be established. For example, when an accuracy range between 0 and 50 is set, the data set corresponding to the accuracy range between 0 and 50, which is confusing for the auto-labeling model, can be learned preferentially.

The auto-labeling acceleration strategy selects and recommends the data that the model needs to learn first through automatic data analysis, so that the auto-labeling model is trained at a faster rate, thereby maximizing the efficiency of auto-labeling.

14 is an example illustrating a result screen of data annotation according to an embodiment.

Performance evaluation of auto labeling and easy labeling can be performed. Evaluation may be verified using Average Precision (AP) for detection and Mean Intersection over Union (mIoU) for segmentation.

For example, the data set used for the experiment to perform the performance evaluation may be PascalVoC2012, MSCoco, and Google ImageNet data, which are the data sets used for the performance evaluation of the auto-labeling model. The data set is all composed of image data, and annotation data for each class may be included for each image.

For the accuracy evaluation of the labeling result, in general, in the case of detection, AP may be used, and in the case of segmentation, mIoU may be used. To calculate AP, we first calculate IoU. IoU is calculated using the predicted boundary box and ground truth, and can be expressed as Equation 1.

Equation 1:

In the embodiment, if the IoU is 0.5 or more, it may be determined that it is correctly detected, and if it is less than 0.5, it may be determined that the IoU is incorrectly detected.

In addition, the performance can be verified by measuring the labeling speed suggested in the embodiment with the operator. For example, in order to verify the performance, 5 experimental groups of 10 people each with 50 experimenters may be configured and evaluated. In the experiment, 50 experimenters annotated 100 random images for each data set three times, auto-labeling the same image, and using easy labeling for a single class to measure accuracy and speed. Through the experiment, it can be confirmed that the speed of labeling and the accuracy of labeling are higher than that of the operator.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (7)

In the auto labeling method performed by the auto labeling system,
generating an auto-labeling model using a data set uploaded from an operator;
performing auto-labeling on the data set uploaded from the worker using the generated auto-labeling model;
selecting data to be labeled from the uploaded data set based on a result predicted through the auto-labeling performed; and
Re-learning the auto-labeling model using the data set on which the data inspection and re-labeling of the selected data have been completed
including,
The performing step is,
Creating a cluster for each class through distribution of work files of the same or similar properties using a cluster model to distribute the auto-labeling completed data to each worker, and distributing the generated cluster for each class to each worker
including,
The re-learning step is
Using the re-learned auto-labeling model, auto-labeling is re-performed on the weak data that is the completed data set or part of the completed data set, and the results predicted through the re-performed auto-labeling and the data labeled from the operator Compare the sets, derive a guide for the quantitative learning index of the retrained auto-labeling model and additional data to be uploaded through the comparison, and perform the re-performed auto-labeling with the labeled data set from the operator Derives accuracy based on the correct error rate of the predicted result, provides an additional data guide including the derived accuracy, presents an example of the selected weak image type, and displays the color distribution of the weak image, Indicate the type of class that the current auto-labeling model is difficult to predict, recommend the class to be added first, and present the correct answer for each class of the current auto-labeling model, and the distance to the correct answer
An auto-labeling method comprising The method of claim 1,
The selecting step is
Deriving an image label including label name, accuracy, and location data for each label included in each image through the auto-labeling performed, and using the derived image label to deliver a data set of a preset accuracy range to the operator step; and
Inspection and re-labeling are completed as the label name and location data included in the data set of the preset accuracy range are modified by the operator, and storing log information corrected through the completed inspection and re-labeling in a database
An auto-labeling method comprising The method of claim 1,
The selecting step is
As the auto-labeling is repeatedly performed, it is determined whether a predicted result is included in a preset accuracy range, based on the predicted result, data included in a preset accuracy range is selected as a weakness image, and the selected An auto-labeling method comprising the step of collecting a plurality of learning indicators including the type of weakness image and the class data type in which the correction has occurred in the selected data to be labeled. delete delete delete In the auto labeling system,
a model generator for generating an auto-labeling model using a data set uploaded from an operator;
a labeling performing unit for performing auto-labeling on the data set uploaded from the worker using the generated auto-labeling model;
a data selection unit for selecting data to be labeled from the uploaded data set based on a result predicted through the auto-labeling performed; and
A re-learning unit for re-learning an auto-labeling model using a data set that has completed data inspection and re-labeling for the selected data
including,
The labeling performing unit,
Creating a cluster for each class through distribution of work files of the same or similar properties using a cluster model to distribute the auto-labeling completed data to each worker, and distributing the generated cluster for each class to each worker including,
The re-learning unit,
Using the re-learned auto-labeling model, auto-labeling is re-performed on the weak data that is the completed data set or part of the completed data set, and the results predicted through the re-performed auto-labeling and the data labeled from the operator Compare the sets, derive a guide for the quantitative learning index of the retrained auto-labeling model and additional data to be uploaded through the comparison, and perform the re-performed auto-labeling with the labeled data set from the operator Derives accuracy based on the correct error rate of the predicted result, provides an additional data guide including the derived accuracy, presents an example of the selected weak image type, and displays the color distribution of the weak image, It displays the type of class that the current auto-labeling model is difficult to predict, recommends a class to be added first, and presents the correct answer for each class of the current auto-labeling model, and the distance to the correct answer.
Auto labeling system.

Images