KR102245896B1 - Annotation data verification method based on artificial intelligence model and system therefore - Google Patents

Abstract

Disclosed are an annotation data verification method based on an artificial intelligence model and a system thereof. The annotation data verification method according to an embodiment of the present invention comprises: a step of selecting a preset number of annotation data for verification from among annotation data of a population; a step of detecting an annotation error image from the annotation data for verification by using each of a plurality of pre-trained artificial intelligence models; a step of determining any one artificial intelligence model from the artificial intelligence models using the number of the annotation error images detected by each of the artificial intelligence models and the number of ground-truth annotation error images of the annotation data for verification; and a step of detecting an annotation error image from the annotation data of the population by using the determined artificial intelligence model. Accordingly, the quality of annotation data can be improved.

Description

Annotation data verification method based on artificial intelligence model and system therefore}

The present invention relates to an annotation data verification technology based on an artificial intelligence model, and more specifically, to an annotation data verification method and system capable of improving the quality of annotation data by verifying the annotation data based on an artificial intelligence model. .

In recent years, deep learning has been applied to various recognitions such as face recognition, full body recognition, posture recognition, voice recognition, object recognition, and data mining. In particular, researches on combining deep learning learning networks with object recognition that recognize specific objects from images are being actively conducted in various ways.

To learn artificial intelligence, images (or data) are collected, annotations are performed on the collected images, and learning is conducted using annotated images. Then, the image used for learning is loaded, and an annotation for the designated block is generated by designating a block with a mouse or the like in the loaded image.

However, in the prior art, since the corresponding user for generating the annotation performs an annotation on each of all images, it is time-consuming and expensive, and the quality of the data annotated by the user is eventually reconfirmed by a person, resulting in a quality error. Since there is no choice but to find, correct, and improve, a lot of resources are spent on verification.

Embodiments of the present invention provide an annotation data verification method and system capable of improving the quality of annotation data by verifying annotation data based on an artificial intelligence model.

An annotation data verification method according to an embodiment of the present invention includes: selecting a preset number of annotation data for verification among annotation data of a population; Detecting an annotation error image from the verification annotation data using each of a plurality of pre-learned artificial intelligence models; Using the number of annotation error images detected by each of the plurality of artificial intelligence models and the number of ground-truth annotation error images of the verification annotation data, one of the plurality of artificial intelligence models Determining an intelligence model; And detecting an annotation error image from the annotation data of the population by using the determined artificial intelligence model.

In the selecting step, the annotation data for verification may be selected from the population's annotation data so that a feature distribution including a level of difficulty in identification for each object and a distribution of the number of objects for each image is maintained with respect to the annotation data of the population.

Further, the annotation data verification method according to an embodiment of the present invention is based on a quality measurement result including a false positive rate, a non-detection rate, an error rate and an accuracy score for each object of the ground-truth annotation data for the verification annotation data. It may further include the step of estimating the quality of the annotation data of the population based on.

In the determining step, each of the plurality of artificial intelligence models, which is applied to each object, is determined, and the step of detecting an annotation error image from the annotation data of the population is performed by using the artificial intelligence model determined for each object. The annotation error image may be detected from the population's annotation data.

In the detecting step, an object box that is less than a preset reference Intersection over Union (IoU) is compared with a box for each object predicted using an artificial intelligence model determined for each object and a box for each object in the annotation data of the population as an error. By determining, an error inclusion level for each image may be predicted, and the annotation error image may be detected through the predicted error inclusion level.

An annotation data verification system according to an embodiment of the present invention includes: a selection unit for selecting a preset number of annotation data for verification among annotation data of a population; Any one of the plurality of artificial intelligence models using the number of annotation error images detected by each of a plurality of pre-learned artificial intelligence models and the number of ground-truth annotation error images of the verification annotation data A decision unit that determines the artificial intelligence model of the model; And a detection unit for detecting an annotation error image from the annotation data of the population by using the determined artificial intelligence model.

The selection unit may select the annotation data for verification from the population's annotation data so that a feature distribution including a level of difficulty in identification for each object and a distribution of the number of objects for each image is maintained with respect to the annotation data of the population.

Further, the annotation data verification system according to an embodiment of the present invention is based on a quality measurement result including a false positive rate, an undetected rate, an error rate and an accuracy score for each object of the ground-truth annotation data for the verification annotation data. It may further include an estimation unit for estimating the quality of the annotation data of the population on the basis of.

The determination unit may each determine an artificial intelligence model applied for each object among the plurality of artificial intelligence models, and the detection unit may detect the annotation error image from the annotation data of the population using the artificial intelligence model determined for each object. have.

The detection unit compares the box for each object predicted using the artificial intelligence model determined for each object and the box for each object in the annotation data of the population, and determines an object box less than a preset reference Intersection over Union (IoU) as an error. , It is possible to predict an error inclusion level for each image, and detect the annotation error image through the predicted error inclusion level.

According to embodiments of the present invention, quality of annotation data may be improved by verifying annotation data based on an artificial intelligence model.

According to embodiments of the present invention, by verifying the annotation data based on an artificial intelligence model, it is possible to quantify the improvement target (input resource and cost calculation) compared to the quality improvement through the selection of an existing object to be worked on, It is possible to perform an effective annotation data quality improvement task compared to the input cost.

According to embodiments of the present invention, quality improvement of an object including a large number of errors, that is, an image, is performed first, so that the efficiency of the overall data quality improvement operation can be improved.

1 shows an exemplary diagram for explaining an annotation data verification method according to an embodiment of the present invention.
2 shows an exemplary diagram for explaining a quality estimation process using annotation data for verification.
3 shows an exemplary diagram for explaining a process of measuring the quality of the actual measurement of the annotation data for verification.
4 and 5 show exemplary diagrams for explaining a process of determining an artificial intelligence model.
6 shows an exemplary diagram for explaining a process of selecting final quality verification data from an annotation data of a population.
7 shows the configuration of an annotation data verification system according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, and common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to those who have, and the invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments, and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, actions and/or elements in which the recited component, step, operation and/or element is Or does not preclude additions.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same elements in the drawings, and duplicate descriptions for the same elements are omitted.

Embodiments of the present invention make it a gist to improve the quality of annotation data by verifying annotation data based on an artificial intelligence model.

1 shows an exemplary diagram for explaining an annotation data verification method according to an embodiment of the present invention.

1, the annotation data verification method according to an embodiment of the present invention selects verification annotation data for evaluating and selecting the performance of each of the artificial intelligence models from the entire annotation data, that is, the annotation data of the population (or Selection), and the first step of estimating the quality of the population through ground-truth annotation quality measurement and measurement of the selected annotation data for verification, using each of a plurality of pre-learned artificial intelligence models. An annotation error image is detected from the verification annotation data selected in the first step, and the number of annotation error images detected by each of the plurality of artificial intelligence models and the number of ground-truth annotation error images of the verification annotation data A third step of detecting the final annotation error image from the population's annotation data using the second step of determining any one of the plurality of AI models and the AI model determined in the second step. Includes.

The first step is to select the verification annotation data from the population's annotation data so that the feature distribution including the level of difficulty in identifying each object and the distribution of the number of objects per image is maintained. The quality of the population's annotation data can be estimated based on the quality measurement results including the false positive rate, non-detection rate, error rate, and accuracy score for each object acquired through actual measurement.

The second step is to determine the verification annotation data, that is, an artificial intelligence model corresponding to each object included in the population's annotation data, through the performance evaluation of artificial intelligence models using the verification annotation data. The artificial intelligence model of can be applied to the third stage.

The third step is to automatically annotate the population's annotation data using an artificial intelligence model determined for each object, and compare the box for each object predicted by the automatic annotation and the box for each object in the population's annotation data to be a preset standard. By determining the object box below Intersection over Union (IoU) as an error, the level of error inclusion for each image is predicted, and the annotation error image is detected through the predicted error inclusion level to determine or select the final quality verification data.

The method of the present invention will be described in detail with reference to FIGS. 2 to 6 as follows.

2 shows an exemplary diagram for explaining a quality estimation process using annotation data for verification.

As shown in FIG. 2, the first step is to select data for verification through random sampling from the annotation data of the population, that is, all the annotation data, and the selected verification data must have a feature distribution of the entire annotation data. For example, the verification annotation data maintains the level of difficulty in identifying each object of the entire annotation data, e.g., Easy, Moderate, and Hard distribution, and the number of objects per image distribution, e.g., the number of bicycles, fences, persons, bikes, bridges The distribution must be maintained. In this case, if the number of sampling should include a certain level of the total annotation data, the difficulty of identifying each object will be known to those skilled in the art, and thus a detailed description thereof will be omitted.

Since the verification annotation data sampled in the first step is data for estimating the error rate of the entire annotation data, the size of the verification annotation data can be calculated using Equation 1 below.

[Equation 1]

Here, N denotes the size of the population, e denotes the tolerance (permissible error rate between the population and the sample), and z denotes the z value of the confidence level (the number of times a specified interval is entered when 100 samples are taken, e.g. , 95% means 95 out of 100 are included in the corresponding section and 5 can be out of the specified section), and P can mean the proportion of the expected population. The size of the annotation data for verification may be determined in consideration of realistic conditions including time and manpower. As an example, the verification annotation data may sample 3000 images, and each object may include 30 or more images.

When the verification annotation data is selected, the measured quality of the verification annotation data is measured as shown in FIG. 3. An application developed for checking the quality of annotations, for example,'DQMiner for AI Data', allows a person to directly perform work, thereby storing the quality check result of the annotation data for verification, and through this, It provides quality measurement results including false positive rate, non-detection rate and accuracy score for each object. At this time, DQMiner for AI Data can evaluate the result for each object as'no error','category error','no object', and'area error', and in the case of'area error', it is scored from 1 to 3 points. You can assess your level. When the measured quality of the validation annotation data is measured, the quality level of the population is estimated from the measured quality measurement results.

At this time, the quality of the population can be estimated with a 95% confidence interval using the following <Equation 2>.

[Equation 2]

Here, since the false positive rate, the non-detection rate, and the error rate are error rates, they can be estimated through rate estimation, and the accuracy score (in the case of a region error) can be estimated by numerical estimation because the average error value appears. Of course, the estimation interval may vary according to the number of annotation data for verification.

4 and 5 show exemplary diagrams for explaining a process of determining an artificial intelligence model.

4 and 5, the process of determining an artificial intelligence model is a process of selecting a target for model performance comparison among GT data, a process of learning three object detection models, and a model-based result and GT comparison. Including the process of doing.

In the process of selecting a target for model performance comparison among GT data, random data is selected from the previously identified GT data (images) and compared with the performance of various types of algorithms learned by deep learning. At this time, since there are several deep learning algorithms to be applied, the object can be extracted at a level (cost, time) that can be iteratively verified.

Here, a comparison criterion can be selected through random selection, and the comparison criterion is the quality status of the images by grasping the quality status of the selected images by randomly selecting a certain number of images from the verification annotation data. Can be grasped.

The process of learning the three Object Detection models is to learn preset algorithms such as Faster R-CNN, YOLO, EfficientDet, etc. using the training data. Here, the training data, that is, the training data, may select a well-known data set while having the same characteristics and purpose of the data set to be checked. And, each artificial intelligence model may be learned using transfer learning. Since transfer learning is known to those skilled in the art, a detailed description will be omitted.

In other words, the learning of the three Object Detection models is an artificial intelligence model using an object detection algorithm using the quality-checked data, for example, the first artificial intelligence model, the second artificial intelligence model, and the third artificial intelligence model. , Objects are detected from image data selected from random selection for each artificial intelligence model. In addition, the actual GT annotation content and the annotation content to which the deep learning model is applied are compared, and a certain number of images are selected based on the images containing a large number of object errors found through comparison with the model results, and the actual quality status of the corresponding images. To grasp.

In the process of performing model-based results and GT comparison, prediction performance can be compared in various ways, and object accuracy checks for each algorithm including the quality check result data and visual verification can be performed if the results are different.

At this time, the corresponding process compares the quality error status table created in the process of learning the three object detection models, and verifies the validity of the artificial intelligence model through comparison with the status table of random extraction, and the status between artificial intelligence models. Table comparison allows you to determine the best performing model. For example, a model with excellent overall performance may be selected, or a model with excellent performance for each object may be selected. For example, as shown in FIG. 4, the detection probability is the highest through the error image detection probability of the first artificial intelligence model, the error image detection probability of the second artificial intelligence model, and the error image detection probability of the third artificial intelligence model. 2 You can select an artificial intelligence model.

6 shows an exemplary diagram for explaining a process of selecting final quality verification data from an annotation data of a population.

6, the process of selecting the final quality verification data from the annotation data of the population is an artificial intelligence model selected through the above-described process, for example, an artificial intelligence model selected for each object of the population. By comparing the box of the object-specific model predicted through the artificial intelligence model for each object by applying it to the annotation data (total annotation data) and the annotation box worked by human (GT), an object box less than the standard IoU is errored. Judged by Through this, the level of error inclusion for each image can be predicted. Through this process, the final quality verification data is selected centering on the images that are expected to have high annotation errors.

By selecting the final quality verification data that is expected to have high errors, it is possible to quantify the cost and schedule for quality improvement work, and to ensure high efficiency of quality improvement work.

As described above, the method according to an embodiment of the present invention can improve the quality of the annotation data by verifying the annotation data based on the artificial intelligence model.

Specifically, the method according to an embodiment of the present invention predicts the expected value of the existence of errors of each annotation by verifying the annotation data based on an artificial intelligence model, and through this, it is predicted that an annotation error will exist strongly from the corresponding artificial intelligence model. If the data (object) is selected as the quality improvement target, it is possible to quantify the improvement target (input resource and cost calculation) compared to the quality improvement through the selection of a random target to be worked on, and the annotation data quality is more efficient than the input cost. Improvement can be done.

In addition, since the method according to an embodiment of the present invention first improves the quality of an object including many errors, that is, an image, it is possible to increase the efficiency of the overall data quality improvement operation.

7 illustrates a configuration of an annotation data verification system according to an embodiment of the present invention, and illustrates a conceptual configuration of a system that performs the method of FIGS. 1 to 6.

Referring to FIG. 7, an annotation data verification system 700 according to an embodiment of the present invention includes a selection unit 710, an estimation unit 720, a determination unit 730, and a detection unit 740.

The selection unit 710 selects a preset number of annotation data for verification among the annotation data of the population.

Here, the selection unit 710 may select the annotation data for verification from the population's annotation data so that a feature distribution including a level of identification difficulty per object and a distribution of the number of objects per image with respect to the population annotation data is maintained.

The estimating unit 720 is based on the quality of the annotation data of the population based on the quality measurement result including the false positive rate, non-detection rate, error rate and accuracy score for each object of the ground-truth annotation data for the verification annotation data. Estimate

The determination unit 730 uses the number of annotation error images detected by each of a plurality of pre-learned artificial intelligence models and the number of ground-truth annotation error images for verification. The artificial intelligence model of any one of the models is determined.

Here, the determination unit 730 may each determine an artificial intelligence model applied for each object among the plurality of artificial intelligence models.

The detection unit 740 detects an annotation error image from the annotation data of the population using the determined artificial intelligence model.

Here, when an artificial intelligence model is determined for each object by the determination unit 730, the detection unit 740 may detect an annotation error image from the annotation data of the population by using the artificial intelligence model determined for each object.

The detection unit 740 compares the box for each object predicted using an artificial intelligence model determined for each object and the box for each object in the annotation data of the population, and determines an object box less than or equal to a preset reference IoU as an error, including errors for each image. The level may be predicted, and an annotation error image may be detected through the predicted error inclusion level.

Although the description of the system of the present invention shown in FIG. 7 is omitted, each component constituting FIG. 7 may include all the contents described in FIGS. 1 to 6, which is a person skilled in the art. It is self-evident to

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to operate as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations can be made from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (10)

Selecting a predetermined number of annotation data for verification from among the annotation data of the population by the selection unit;
Detecting an annotation error image from the verification annotation data using each of a plurality of artificial intelligence models previously learned by a detection unit;
The determination unit uses the number of annotation error images detected by each of the plurality of artificial intelligence models and the number of ground-truth annotation error images of the verification annotation data to determine which one of the plurality of artificial intelligence models. Determining one artificial intelligence model; And
Detecting an annotation error image from the population's annotation data using the determined artificial intelligence model in the detection unit
Including,
The determining step
Each of the plurality of artificial intelligence models is determined to be applied to each object,
The step of detecting an annotation error image from the annotation data of the population
Detecting the annotation error image from the annotation data of the population using an artificial intelligence model determined for each object,
The detecting step
By comparing the box for each object predicted using the artificial intelligence model determined for each object and the box for each object in the annotation data of the population, and determining an object box less than a preset reference IoU (Intersection over Union) as an error, An annotation data verification method for predicting an error inclusion level and detecting the annotation error image through the predicted error inclusion level.
The method of claim 1,
The selecting step
An annotation data verification method, characterized in that the annotation data for verification is selected from the annotation data of the population so that a feature distribution including a level of identification difficulty for each object and a distribution of the number of objects for each image is maintained with respect to the annotation data of the population.
The method of claim 1,
Estimating the quality of the annotation data of the population based on a quality measurement result including a false positive rate, a false positive rate, an error rate, and an accuracy score for each object of the ground-truth annotation data for the verification annotation data.
Annotation data verification method further comprising a.
delete delete A selection unit for selecting a preset number of annotation data for verification among the annotation data of the population;
Any one of the plurality of artificial intelligence models using the number of annotation error images detected by each of a plurality of pre-learned artificial intelligence models and the number of ground-truth annotation error images of the verification annotation data A decision unit that determines the artificial intelligence model of the model; And
A detection unit that detects an annotation error image from the annotation data of the population using the determined artificial intelligence model
Including,
The decision part
Each of the plurality of artificial intelligence models is determined to be applied to each object,
The detection unit
Detecting the annotation error image from the annotation data of the population using an artificial intelligence model determined for each object,
The detection unit
By comparing the box for each object predicted using the artificial intelligence model determined for each object and the box for each object in the annotation data of the population, and determining an object box less than a preset reference IoU (Intersection over Union) as an error, An annotation data verification system for predicting an error inclusion level and detecting the annotation error image through the predicted error inclusion level.
The method of claim 6,
The selector
An annotation data verification system, characterized in that the annotation data for verification is selected from the annotation data of the population so that a feature distribution including a level of difficulty in identification for each object and a distribution of the number of objects for each image is maintained with respect to the annotation data of the population.
The method of claim 6,
An estimator for estimating the quality of the annotation data of the population based on a quality measurement result including a false positive rate, a non-detection rate, an error rate and an accuracy score for each object of the ground-truth annotation data for the verification annotation data
Annotation data verification system, characterized in that it further comprises.
delete delete

Images