KR20240045414A - Method and device for quality inspection and automatic annotation of training data based on deep learning ensemble - Google Patents

Abstract

A method and device for deep learning ensemble-based learning data quality inspection and automatic annotation are disclosed. A learning data management method for quality inspection and automatic annotation includes receiving learning data consisting of source data and annotation data for the source data; performing a syntax correctness check through syntax analysis of the annotation data; Performing validation through a deep learning ensemble artificial intelligence model on training data without syntax errors according to the syntax analysis results; And it may include annotating objects recognized in the source data through the deep learning ensemble artificial intelligence model.

Description

Method and device for deep learning ensemble-based learning data quality inspection and automatic annotation {METHOD AND DEVICE FOR QUALITY INSPECTION AND AUTOMATIC ANNOTATION OF TRAINING DATA BASED ON DEEP LEARNING ENSEMBLE}

The explanation below concerns techniques for checking the quality of training data.

Recently, as research on artificial intelligence has become active in the field of computer science, several algorithms related to deep learning techniques that mimic the human learning system are being developed. Accordingly, many software is adopting various algorithms related to deep learning.

Deep learning is a form of artificial intelligence developed from artificial neural networks that learns data using information input and output layers similar to neurons in the brain. It organizes algorithms into layers to create an artificial neural network that can learn on its own and make intelligent decisions. The purpose is to

Deep learning ensemble is a technique that improves prediction ability by combining multiple learning models. The types of ensemble learning can be divided into voting, bagging, boosting, and stacking.

There may be many things that are important in learning, but which of them to learn from is an important issue. If you learn with good and correct information, the learning efficiency and results will be better than if you do not.

What is also important in machine learning is what data it learns from. Several methods are being studied for configuring (or generating) learning data.

1. Korean Patent Publication No. 10-2019-0044814 (Data generation and data construction method for deep learning learning, publication date: May 2, 2019)

Provides technology to automate quality inspection and annotation of learning data using deep learning ensemble models.

A learning data management method executed on a computer device, wherein the computer device includes at least one processor configured to execute computer-readable instructions included in a memory, and the learning data management method includes: , receiving learning data consisting of source data and annotation data for the source data; performing, by the at least one processor, a syntax correctness check through parsing of the annotation data; and performing validation through a deep learning ensemble artificial intelligence model on training data without syntax errors according to a syntax analysis result, by the at least one processor.

According to one aspect, the step of performing the syntax correctness check includes performing syntax analysis on an annotation of the corresponding type through a type loader in which a dataset for object detection is set as an annotation definition. may include.

According to another aspect, the deep learning ensemble artificial intelligence model may be a model learned with a pair of the source data and the annotation data through ensemble learning using K-fold cross validation.

According to another aspect, the learning data management method may further include annotating objects recognized in the source data through the deep learning ensemble artificial intelligence model by the at least one processor.

According to another aspect, the annotating step includes generating a bounding box type annotation on an object recognized in the source data; And it may include transmitting the generated annotation along with the source data to a local storage or remote storage.

According to embodiments of the present invention, the manual work rate can be dramatically reduced and the cost of generating training data can be reduced by automating quality inspection and annotation of training data using a deep learning ensemble model.

1 is a block diagram for explaining an example of the internal configuration of a computer device according to an embodiment of the present invention.
Figure 2 shows the configuration of a deep learning ensemble-based learning data quality inspection system in one embodiment of the present invention.
Figure 3 shows the configuration of a deep learning ensemble-based learning data automatic annotation system in one embodiment of the present invention.
Figure 4 is an example diagram illustrating a learning method of a deep learning ensemble artificial intelligence model in one embodiment of the present invention.
Figure 5 is a flowchart showing an example of a learning data management method including quality inspection and automatic annotation that can be performed by a computer device according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Embodiments of the present invention relate to quality inspection and automatic annotation technology for learning data.

Embodiments including those specifically disclosed herein can automate quality inspection and annotation for learning data using a deep learning ensemble model.

1 is a block diagram showing an example of a computer device according to an embodiment of the present invention. For example, a learning data management system according to embodiments of the present invention may be implemented by the computer device 100 shown in FIG. 1.

As shown in FIG. 1, the computer device 100 is a component for executing the learning data management method according to embodiments of the present invention, including a memory 110, a processor 120, a communication interface 130, and It may include an input/output interface 140.

The memory 110 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, non-perishable large-capacity recording devices such as ROM and disk drives may be included in the computer device 100 as a separate permanent storage device that is distinct from the memory 110. Additionally, an operating system and at least one program code may be stored in the memory 110. These software components may be loaded into the memory 110 from a computer-readable recording medium separate from the memory 110. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 110 through the communication interface 130 rather than a computer-readable recording medium. For example, software components may be loaded into memory 110 of computer device 100 based on computer programs installed by files received over network 160.

The processor 120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 120 by the memory 110 or the communication interface 130. For example, processor 120 may be configured to execute received instructions according to program code stored in a recording device such as memory 110.

The communication interface 130 may provide a function for the computer device 100 to communicate with other devices through the network 160. For example, a request, command, data, file, etc. generated by the processor 120 of the computer device 100 according to a program code stored in a recording device such as memory 110 is transmitted to the network ( 160) and can be transmitted to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device 100 through the communication interface 130 of the computer device 100 via the network 160. Signals, commands, data, etc. received through the communication interface 130 may be transmitted to the processor 120 or memory 110, and files, etc. may be stored in a storage medium (as described above) that the computer device 100 may further include. It can be stored as a permanent storage device).

The communication method is not limited, and may include not only a communication method utilizing communication networks that the network 160 may include (e.g., mobile communication network, wired Internet, wireless Internet, and broadcasting network), but also short-distance wired/wireless communication between devices. there is. For example, the network 160 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , may include one or more arbitrary networks such as the Internet. Additionally, the network 160 may include any one or more of network topologies including a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, etc. Not limited.

The input/output interface 140 may be a means for interfacing with the input/output device 150. For example, input devices may include devices such as a microphone, keyboard, camera, or mouse, and output devices may include devices such as displays and speakers. As another example, the input/output interface 140 may be a means for interfacing with a device that integrates input and output functions, such as a touch screen. The input/output device 150 may be configured as a single device with the computer device 100.

Additionally, in other embodiments, computer device 100 may include fewer or more components than those of FIG. 1 . However, there is no need to clearly show most prior art components. For example, the computer device 100 may be implemented to include at least a portion of the input/output device 150 described above, or may further include other components such as a transceiver, a camera, various sensors, and a database.

Hereinafter, specific embodiments of a method and device for deep learning ensemble-based learning data quality inspection and automatic annotation will be described.

The computer device 100 according to this embodiment provides training data quality inspection and automatic annotation to clients through a dedicated application installed on the client or access to a web/mobile site related to the computer device 100. Management services can be provided. The computer device 100 may be configured with a learning data management system implemented on a computer. For example, the learning data management system may be implemented in the form of a program that operates independently, or may be implemented in the form of an in-app of a specific application to enable operation on the specific application.

The processor 120 of the computer device 100 may be implemented as a component for performing the following learning data management method. Depending on the embodiment, components of the processor 120 may be selectively included in or excluded from the processor 120. Additionally, depending on the embodiment, components of the processor 120 may be separated or merged to express the functions of the processor 120.

The processor 120 and the components of the processor 120 can control the computer device 100 to perform steps included in the learning data management method below. For example, the processor 120 and its components may be implemented to execute instructions according to the code of an operating system included in the memory 110 and the code of at least one program.

Here, the components of the processor 120 may be expressions of different functions performed by the processor 120 according to instructions provided by program codes stored in the computer device 100.

The processor 120 may read necessary instructions from the memory 110 where instructions related to controlling the computer device 100 are loaded. In this case, the read command may include an command for controlling the processor 120 to execute steps that will be described later.

Steps included in the learning data management method to be described later may be performed in an order different from the order shown, and some of the steps may be omitted or additional processes may be included.

The learning data management service according to the present invention includes a learning data loading function, a function to check the syntactic accuracy of the learning data, an automatic annotation tool through deep learning ensemble artificial intelligence model construction and learning, a validation function of the learning data, and a diversity check function of the learning data. etc. can be provided.

Figure 2 shows the configuration of a deep learning ensemble-based learning data quality inspection system in one embodiment of the present invention. The learning data quality inspection system 200 according to embodiments of the present invention may be implemented by the computer device 100 shown in FIG. 1.

Referring to FIG. 2, the learning data quality inspection system 200 can provide a learning data quality inspection function for using industrial safety artificial intelligence source data as learning data for an object recognition model.

The learning data quality inspection system 200 may include a type loader 210, a syntax analyzer 220, and a data validity analyzer 230.

Crowd workers, who participate in tasks such as collection, purification, processing, and quality control of data necessary for artificial intelligence learning, annotate the source data and annotate the learning data by accessing a dedicated application or web/mobile site. You can upload.

The type loader 210 may preset definitions for annotation data. The type loader 210 can set a dataset for object detection, for example, PASCAL VOC, MS COCO, JSON, etc., as an annotation definition.

The parser 220 can receive input data annotated on the source data by a crowd worker along with the source data.

The syntax analyzer 220 may receive source data and annotation data as learning data. Source data may refer to image data captured from camera images at industrial sites.

The syntax analyzer 220 may receive training data loaded from the crowd worker and perform syntax accuracy and data diversity checks through parsing annotations of the type set in the type loader 210.

The syntax analyzer 220 can visualize and provide syntactic accuracy data and diversity distribution information as a result of annotation syntax analysis through a statistical view.

The syntax analyzer 220 may request the crowd worker to correct syntax errors in annotations that deviate from the settings of the type loader 210, that is, contain syntax errors.

The syntax analyzer 220 may transmit the pair of source data and annotation data received as learning data to the data validity analyzer 230.

The data validity analyzer 230 can perform validation on learning data through a deep learning ensemble artificial intelligence model. The data validation analyzer 230 can perform a validation check on learning data that does not have syntax errors in the annotation by parsing the annotation data.

In this embodiment, a deep learning ensemble artificial intelligence model is trained through a K-fold cross validation setting and the overall average of accuracy between source data and annotation data is above a certain level (e.g., 95%). can be built. The data validity analyzer 230 can perform validation checks on learning data, such as whether the annotations on the source data are properly labeled and whether the annotation coordinates are correct, through a deep learning ensemble artificial intelligence model.

The deep learning ensemble artificial intelligence model will be explained again below.

Figure 3 shows the configuration of a deep learning ensemble-based learning data automatic annotation system in one embodiment of the present invention. The learning data automatic annotation system 300 according to embodiments of the present invention may be implemented by the computer device 100 shown in FIG. 1.

Referring to FIG. 3, the learning data automatic annotation system 300 can provide an automatic annotation function for industrial safety artificial intelligence source data.

In this embodiment, it is possible to build a deep learning ensemble artificial intelligence model whose overall average accuracy between source data and annotation data is above a certain level (for example, 95%) through learning through a K-fold cross-validation setting. The annotation engine 310 can automatically annotate objects recognized in source data through a deep learning ensemble artificial intelligence model.

In detail, the learning data automatic annotation system 300 may include an annotation engine 310 for object recognition and automatic annotation.

In the embodiment, for convenience of explanation, the operation of performing annotation using test image information for industrial safety artificial intelligence recognition captured at an industrial site will be described as an example.

The annotation engine 310 can acquire test images for industrial safety artificial intelligence recognition at industrial sites captured by a camera. For example, a camera may be installed on a forklift at an industrial site, and test images for industrial safety artificial intelligence recognition at an industrial site may be captured through the camera installed on the forklift.

The annotation engine 310 can capture the acquired test image information for industrial safety artificial intelligence recognition, and use an object detection algorithm (e.g., YOLO DarkNet) from the captured test image information for industrial safety artificial intelligence recognition to identify objects. Information can be detected.

The annotation engine 310 may receive test image information for industrial safety artificial intelligence recognition, including detected object information, as input to a learning model for object recognition. The annotation engine 310 can recognize and annotate objects from test image information for industrial safety artificial intelligence recognition that includes object information detected using a learning model for object recognition. In other words, the annotation engine 310 recognizes an object from test image information for industrial safety artificial intelligence recognition including detected object information, and annotates the recognized object with an MS COCO type, which is one of the bounding box types. can be created.

Alternatively, the annotation engine 310 may receive detected object information as input to a learning model for object recognition. The annotation engine 310 can recognize and annotate objects from detected object information using a learning model for object recognition. In other words, the annotation engine 310 can recognize an object from detected object information and create an annotation of the MS COCO type on the recognized object.

The annotation engine 310 may obtain coordinate information about the area of the recognized object and then annotate the area of the recognized object. The annotation engine 310 includes person, face, upper body, lower body, forklift, battery car, etc. for the recognized object area. You can create an annotation of the bounding box type.

More specifically, for example, the annotation engine 310 may primarily annotate identification information about the recognized object and then secondarily annotate detailed information about the recognized object. When the recognized object is a person, the annotation engine 310 can primarily annotate the recognized object as a person and secondarily annotate detailed information about the person (e.g., upper body, lower body, etc.). You can. Alternatively, if the recognized object is an object, the annotation engine 310 may primarily annotate the recognized object as a forklift and secondarily annotate detailed information about the forklift (e.g., wheels, steering wheel, etc.) Annotation is possible. Additionally, after a bounding box type annotation is created in the annotation engine 310, the annotation may be modified through a confirmation process by the user.

The annotation engine 310 may store captured images and annotation data locally. At this time, the local may be an embedded-based electronic device, and the captured image and annotation data may be updated whenever new image and annotation data are stored locally.

The annotation engine 310 can transmit the captured image and annotation data to a remote storage. For example, the annotation engine 310 may transmit captured images and annotation data to a remote storage through wireless or wired communication. The remote storage may store the captured image and annotation data delivered from the annotation engine 310.

Figure 4 is an example diagram illustrating a learning method of a deep learning ensemble artificial intelligence model in one embodiment of the present invention.

In the present invention, a model for learning data quality inspection and automatic annotation can be built through ensemble modeling based on artificial intelligence model analysis.

You can select at least one of the ensemble learning types and build a model for learning data quality inspection and automatic annotation through ensemble learning of that type. Referring to Figure 4, after selecting an artificial intelligence model (①), initial learning can be performed for the selected artificial intelligence model using a small data set consisting of pairs of source data and annotation data (②). After validating the initial learning results of the artificial intelligence model, if there are no abnormalities, main learning using a large data set can be performed and validation and inference work on the learning results can be carried out sequentially. You can tune various hyperparameters, such as merging artificial intelligence models, changing part of the model (layer change, LR scheduler change, GPU acceleration, etc.), and loss function change (③). Through the above processes (①, ②, ③), the artificial intelligence model with optimal performance can be selected as the final model based on the learning results based on the decision rule of the artificial intelligence model (④).

Figure 5 is a flowchart showing an example of a learning data management method including quality inspection and automatic annotation that can be performed by a computer device according to an embodiment of the present invention.

Referring to FIG. 5, in step S501, the processor 120 may preset a definition for annotation data in a type loader for object detection-based annotation.

In step S502, the processor 120 may check the syntax accuracy and data diversity of the annotation data by parsing the annotation of the type set in the type loader for the annotation data by the crowd worker.

In step S503, the processor 120 may parse the annotation data and perform validation on learning data that does not have problems with the annotation syntax using a deep learning ensemble artificial intelligence model. The processor 120 can build a deep learning ensemble artificial intelligence model with an overall average accuracy of 95% or more between source data and annotation data through learning through the inspection results of step S502 and K-fold cross-validation settings, Through this, you can perform validation of the learning data.

The processor 120 may visualize and provide syntactic accuracy and data diversity test results for the training data, as well as validation results for the training data.

In step S504, the processor 120 can automate annotations for objects recognized in the source data through a deep learning ensemble artificial intelligence model learned with training data consisting of pairs of source data and annotation data.

The processor 120 can visualize and provide automatic annotation results, and can store source data and annotation data as annotation results in a storage (local or remote).

As such, according to embodiments of the present invention, the manual work rate can be dramatically reduced and the cost of generating training data can be reduced by automating quality inspection and annotation of learning data using a deep learning ensemble model.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general-purpose or special-purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for the purpose of being interpreted by or providing instructions or data to the processing device. there is. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. At this time, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (5)

In a learning data management method executed on a computer device,
The computer device includes at least one processor configured to execute computer-readable instructions contained in a memory,
The learning data management method is,
Receiving learning data consisting of source data and annotation data for the source data, by the at least one processor;
performing, by the at least one processor, a syntax correctness check through parsing of the annotation data; and
Performing validation through a deep learning ensemble artificial intelligence model on training data without syntax errors according to a syntax analysis result, by the at least one processor.
A learning data management method including. According to paragraph 1,
The step of performing the syntax correctness check is,
A step of performing syntax analysis on annotations of the corresponding type through a type loader in which a dataset for object detection is set as an annotation definition.
A learning data management method including. According to paragraph 1,
The deep learning ensemble artificial intelligence model is a model learned with a pair of the source data and the annotation data through ensemble learning using K-fold cross validation.
A learning data management method characterized by: According to paragraph 1,
The learning data management method is,
Annotating, by the at least one processor, an object recognized in the source data through the deep learning ensemble artificial intelligence model.
A learning data management method further comprising: According to clause 4,
The annotation step is,
Creating a bounding box type annotation on an object recognized in the source data; and
Transferring the generated annotation together with the source data to a local storage or remote storage.
A learning data management method including.

Images