KR20230053263A - Method of providing annotation review feedback and a computer program recorded on a recording medium to execute the method - Google Patents

Abstract

Disclosed is a method of providing annotation review feedback efficiently in a process of annotating a plurality of pieces of artificial intelligence (AI) learning data. The method may comprise: a step of receiving, by an annotation device, a command indicating that a worker has a question through a user interface (UI) in an annotation work process for AI learning; a step of generating, by the annotation device, question data corresponding to the input command; a step of searching, by the annotation device, a database of a learning data design device pre-set based on the question data; and a step of converting, by the annotation device, response data searched from the database into a natural language to provide the same to the worker through a chatting unit.

Description

Method of providing annotation review feedback and a computer program recorded on a recording medium to execute the method

The present invention relates to data design for artificial intelligence (AI) learning. More specifically, it relates to a method for efficiently providing annotation review feedback in the process of annotating a large number of artificial intelligence (AI) learning data and a computer program recorded on a recording medium to execute the method.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning abilities, reasoning abilities, and perception abilities using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the form of learning data.

In general, the design of artificial intelligence (AI) learning data proceeds in the steps of data structure design, data collection, data refinement, data processing, data expansion, and data verification.

To describe each step in more detail, data structure design is performed through ontology definition, classification system definition, and the like. Data collection is performed by collecting data through direct filming, web crawling, or associations/professional organizations. Data purification is performed by removing redundant data from collected data and de-identifying personal information. Data processing is performed by inputting meta data and performing annotation. Data extension is performed by performing ontology mapping and supplementing or extending the ontology as needed. In addition, data verification is performed by verifying validity according to the set target quality using various verification tools.

In general, annotation in the data processing step is performed by processing a bounding box for an object included in an image and inputting property information of the object included in the image. Such annotations are also referred to as data labeling. And, a dataset corresponding to the work result of the annotation is calculated in the form of a JSON (Java Script Object Notation) file.

Since such annotation work is performed on a large number of data ranging from a few thousand to several million pieces of data, a large number of workers perform the annotation work. In addition, many annotation workers try to solve many questions they have in the process of performing a large number of annotation tasks assigned to them through Q&A.

However, questions that annotation workers with normal thinking skills have are often the same or similar to each other, and it is inefficient for a manager to respond in detail to all inquiries individually received from numerous annotation workers.

In addition, data processing rules that must be commonly observed are established in the process of performing annotation work by numerous workers. These data processing rules may be determined in various ways according to the learning goal and learning methodology of artificial intelligence (AI). For example, data processing rules include the order of setting bounding boxes for specifying objects, setting methods and tools to be used, input types of property information assigned to specified objects, input rules, and Formats that can be input may be determined in various combinations.

However, it cannot be determined that annotation workers with normal thinking ability will accurately follow data processing rules, and it is common for many annotation workers to generate the same or similar errors.

Korean Patent Laid-open Publication No. 10-2020-0042629, ‘Method and apparatus for generating touch-based annotations and images in mobile devices for artificial intelligence learning’, (published on April 24, 2020)

One object of the present invention is to efficiently provide annotation review feedback in the process of annotating a large number of artificial intelligence (AI) learning data.

Another object of the present invention is to provide a computer program recorded on a recording medium to execute a method capable of efficiently providing question and answering.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical problem as described above, we propose an annotation review feedback efficiently in the process of annotating a large number of AI learning data. The method includes the steps of receiving a command from an operator through a user interface (UI) in an annotation work process for artificial intelligence (AI) learning by an annotation device, informing that a question mark exists; generating, by the annotation device, query data in response to the input command; Searching, by the annotation device, a database of previously set learning data design devices based on the query data; and converting, by the annotation device, response data retrieved from the database into natural language and providing the result to the worker through a chatting unit.

If it is determined that the question mark input from the job is a question mark for the same reason, the annotation device may additionally search for a basic usage method for the annotation job in the database search step.

In addition, the step of searching the database searches for questions of workers accumulated and existing in the chatting unit, and the step of providing the questions to the operator analyzes the frequency of questions and question type patterns in relation to the questions received, You can provide response data.

In the searching of the database, a similarity to an image to be annotated is retrieved, and in the providing to the operator, a highlight may be displayed on the image and provided as response data.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute the method as described above. The computer program may include a memory; transceiver; And combined with a computing device configured to include a processor for processing instructions residing in the memory, the processor, in the process of annotation (annotation) for artificial intelligence (Artificial Intelligence, AI) learning, a user interface ( Receiving a command notifying that a question exists from a worker through a user interface (UI); generating, by the processor, query data in response to the input command; searching, by the processor, a database of previously set learning data design devices based on the query data; and converting, by the processor, the response data retrieved from the database into natural language and providing the result to the worker through a chatting unit; it may be a computer program recorded on a recording medium.

Details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, with respect to questions that workers have in the process of performing annotation work on a plurality of images for artificial intelligence (AI) learning, automatic response through a database in which other workers' questions and answers are accumulated By doing so, it is possible to reduce the number of responses that the manager has to answer directly.

That is, according to the embodiments of the present invention, Q&A for questions that workers have can be processed like Frequently Asked Questions (FAQ).

In particular, the present invention, in the process of pointing out missing points as a result of the reviewer's review, selects similar data from the repository as well as the reviewer's points out, processes it in natural language, and provides additional information to the user about the wrong issue through a chatting system. It has the effect of providing effective feedback.

In addition, when an operator makes an inquiry, the present invention determines and finds a similar missing point in the storage and automatically performs an answer, so there is an effect of enabling more efficient feedback.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in this specification, and are overly inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. On the other hand, when a component is referred to as “directly connected” or “directly connected” to another component, it should be understood that no other component exists in the middle.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents or substitutes other than the accompanying drawings.

The data collected for artificial intelligence learning consists of very large numbers taken in real time, not just a few. There is a lot of burden for the learning data collection device to transmit a lot of raw data to the data collection server for the collected data. In addition, since the collection device generally collects raw data through real-time shooting, among the collected raw data, similar data with low artificial intelligence learning efficiency may be included, thereby highlighting the efficiency of data management.

An artificial intelligence learning system according to an embodiment of the present invention may include one or more annotation devices and an artificial intelligence learning device.

An artificial intelligence learning system according to another embodiment of the present invention may be configured to additionally include a learning data design device in addition to the annotation device and the artificial intelligence learning device.

In addition, in the artificial intelligence learning system according to another embodiment of the present invention, the annotation device, the learning data design device, and the artificial intelligence learning device may be connected to each other through a public network. In this case, a part of the annotation device may be a device that performs annotation by a clouding service.

Since the components of the artificial intelligence learning system according to various embodiments are merely functionally distinct elements, two or more components are integrated and implemented in the actual physical environment, or one component is implemented in the actual physical environment. may be implemented separately from each other.

Each component is described, and the annotation device is a device that can be used to annotate an image provided from a learning data design device or an artificial intelligence learning device.

In particular, the annotation apparatus according to the present invention has a feature of providing various conveniences to the operator in the process of setting a bounding box and inputting object attribute information in relation to annotation work.

Any device capable of transmitting and receiving data to and from a learning data design device or an artificial intelligence learning device and performing calculations using the transmitted and received data may be used as the annotation device.

For example, the annotation device may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto, and is not limited to, a smart phone or a laptop. ), tablets, phablets, portable multimedia players (PMPs), personal digital assistants (PDAs) or e-book readers. can be any one of them.

With the following configuration, the learning data design device is a device that can be used to design and generate artificial intelligence (AI) learning data. Such a learning data design device is basically a device that is distinguished from an artificial intelligence learning device, but may be implemented by being integrated with an artificial intelligence learning device in a real physical environment.

Specifically, the learning data design device may receive attributes of a project related to artificial intelligence (AI) learning from the artificial intelligence learning device. The learning data design device can design data structures for artificial intelligence (AI) learning, refine collected data, process data, expand data, and verify data based on the user's control and the attributes of the project. there is.

In particular, the learning data design device may transmit an image to be annotated to the annotation device for data processing for artificial intelligence (AI) learning. The learning data design device may receive an annotation work result from the annotation device. In this case, the annotation work result may have a JSON (Java Script Object Notation) file format. Alternatively, the learning data design device may generate a JSON file based on the received result after receiving a result of annotation work in a format different from JSON. In addition, the learning data design device inspects the received or generated JSON file, packages it, and transmits it to the artificial intelligence learning device.

As such, the learning data design device may be any device capable of transmitting and receiving data to and from the annotation device and the artificial intelligence learning device and performing calculations using the transmitted and received data. For example, the learning data design device may be any one of a desktop, a workstation, or a fixed computing device such as a server, but is not limited thereto.

With the following configuration, the artificial intelligence learning device is a device that can be used to perform machine learning of artificial intelligence (AI) based on data for artificial intelligence (AI) learning.

Specifically, the artificial intelligence learning device may receive a JSON file packaged directly from the annotation device or from the learning data design device. And, the artificial intelligence learning device may perform artificial intelligence (AI) machine learning using the received JSON file.

Such an artificial intelligence learning device may be any device capable of transmitting and receiving data to and from an annotation device or learning data design device and performing calculations using the transmitted and received data. For example, the artificial intelligence learning device may be any one of a desktop, a workstation, or a fixed computing device such as a server, but is not limited thereto.

As described above, one or more annotation devices, learning data design devices, and artificial intelligence learning devices may transmit and receive data using a network in which one or more of a security line, a common wired communication network, or a mobile communication network directly connected between devices is combined. there is.

For example, public wired communication networks may include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). It may be, but is not limited thereto. In addition, in the mobile communication network, Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile telecommunication may be included, but is not limited thereto.

In addition, the annotation device may include a communication unit, an input/output unit, a storage unit, a bounding box setting unit, an object property setting unit, and a result generation unit.

Since the components of the annotation device are merely functionally distinct elements, two or more components may be integrated and implemented in an actual physical environment, or one component may be implemented separately from each other in an actual physical environment. There will be.

The communication unit may transmit/receive data with the learning data design device and the artificial intelligence learning device.

Specifically, the communication unit may receive one or more images from the learning data design device or the artificial intelligence learning device.

Here, the image is an image that is a target of annotation work for artificial intelligence (AI) learning. According to the data processing plan designed by the learning data design device or the artificial intelligence learning device, images to be annotated may be individually received or a plurality of images may be collectively received.

The communication unit may transmit the work result of the annotation to the learning data design device or the artificial intelligence learning device.

Here, the work result may include the coordinates of the bounding box set under the operator's control and property information of the object. In addition, the work result may have a JSON file format, but is not limited thereto.

And, the communication unit may receive project properties, image properties, or worker properties from the learning data design device or the artificial intelligence learning device.

Here, the properties of the project may include the learning purpose, learning period, number of images required for learning, object properties to be identified in images, bounding box setting rules, etc. for a project related to artificial intelligence (AI) learning. It is not limited to this.

Image properties include image file name, image size (width, height), resolution, bit level, compression format, shooting device name, exposure time, ISO speed, focal length, aperture value, and shooting location coordinates (GPS latitude, longitude). , shooting time, etc. may be included, but is not limited thereto.

The worker's attributes may include, but are not limited to, the worker's name, identification number, assigned work amount, cost according to the work, work result evaluation, and the like.

In addition, the bounding box setting rule is a rule that a worker must follow in a process of setting a bounding box for an object in an image according to a project property, an image property, or a worker property. These bounding box setting rules may include, but are not limited to, a minimum size value of an area and a maximum number of pixels separated from an object's outline.

With the following configuration, the input/output unit may input a signal from an operator through a user interface (UI) or output calculated results to the outside.

Here, the worker means a person who performs annotation work. Such an operator may be referred to as a user, performer, labeler, or data labeler, but is not limited thereto.

Specifically, the input/output unit may output an image to be an annotation work. The input/output unit may receive a control signal for setting a bounding box from an operator. In addition, the input/output unit may overlay and output the bounding box on the image.

Here, the bounding box is an area for specifying an object to be learned by artificial intelligence (AI) among objects included in the image. Such a bounding box may have a rectangle or polygon shape, but is not limited thereto.

The input/output unit may output a list of recommendation information that can be used as object attribute information. Also, the input/output unit may receive a control signal for setting object attribute information from an operator.

Here, the proposed information list is a list of information that is composed of information that is likely to be included in object attribute information and can be utilized as object attribute information in the process of performing annotation by a worker.

In addition, object attribute information is information for specifying the attribute of an object that is an artificial intelligence (AI) learning target. Such object attribute information includes information on the type of annotation, class name (class), classification items (tags), truncated state of the object, major classification, subclassification, or upper level (instance upper). may be, but is not limited thereto.

With the following configuration, the storage unit can store data required for annotation work.

Specifically, the storage unit may store an image received through the communication unit. The storage unit may store project properties, image properties, or worker properties received through the communication unit.

The storage unit may temporarily store the location and size of the bounding box set according to the control signal input through the input/output unit. The storage unit may temporarily store attributes of objects input through the input/output unit.

With the following configuration, the bounding box setting unit may set a bounding box for specifying an object to be an artificial intelligence (AI) learning target among objects included in an image. In particular, the bounding box setting unit according to the present invention automatically enlarges or reduces the size of the bounding box according to the shape of the object included in the image so that the bounding box primarily set by the operator has a size corresponding to the outline of the object. It has features that can

First of all, the bounding box setting unit may set a partial region in an image, which is an object of annotation work for learning, as a bounding box according to control of an operator input through an input/output unit.

Specifically, the bounding box setting unit receives two coordinates from the operator through the input and output unit, and sets the bounding box based on a rectangle having the input two coordinates as the coordinates of the upper left vertex and the coordinates of the lower right vertex in the image. can In this case, the two coordinates may be set by the operator inputting one type of input signal twice (eg, mouse click) or by the operator inputting two types of input signals once (eg, mouse drag). It may, but is not limited thereto.

Alternatively, the bounding box setting unit may receive three or more line segments from an operator through an input/output unit, and may set a bounding box based on a polygon composed of the input three or more line segments. In this case, three or more line segments may be input by an operator at one time using a polygon tool or by inputting three or more segments by an operator using a straight line tool, but are not limited thereto.

Next, the bounding box setting unit may identify an object that is completely included in the inner region of the bounding box set by the operator's control or spans one or more line segments among a plurality of line segments constituting the bounding box.

Specifically, the bounding box setting unit may identify all or some of the objects included in the inner region of the bounding box set under the control of the operator. To this end, the bounding box setting unit may perform image processing on an area inside the bounding box.

For example, the bounding box setting unit may divide the inner region of the bounding box into three images according to RGB (Red, Green, Blue). The bounding box setting unit may extract an edge of each image divided into three images. More specifically, the bounding box setting unit may use either a Laplacian of Gaussian (LoG) algorithm or a Difference of Gaussian (DoG) algorithm to extract an edge of each image.

When using the LoG algorithm, the bounding box setting unit may remove noise existing in the image using a Gaussian filter. The bounding box setting unit may apply a Laplacian filter to the noise-removed image. The bounding box setting unit may extract an edge by detecting zerocrossing in the image to which the Laplacian filter is applied.

When using the DoG algorithm, the bounding box setting unit creates two Gaussian masks with different variances from the image. The bounding box setting unit subtracts another mask from one created mask. The bounding box setting unit may extract an edge by applying the subtracted mask to the image.

The bounding box setting unit may identify one or more enclosures by edges extracted from each image. In this case, the bounding box setting unit may first process binarization on each image in order to clarify whether the edge area is closed. And, the bounding box setting unit may identify the identified closed area as an object.

If there are a plurality of objects identified in the inner region of the bounding box, the bounding box setting unit may select only one object from among the identified plurality of objects according to a preset priority. In this case, the priority may be a criterion set according to the size, position or shape of the object occupying the image.

Next, the bounding box setting unit determines whether the identified object spans one or more line segments among a plurality of line segments constituting the bounding box, based on continuity between pixels included in the inner area of the bounding box and pixels in the outer area of the bounding box. can determine whether Here, continuity may be determined based on whether a difference in color, saturation, or brightness between two adjacent pixels is within a preset range.

The bounding box setting unit may determine that the object is completely included in the inner region of the bounding box when the identified object does not cross any line segment among a plurality of line segments constituting the bounding box.

Characteristically, according to an embodiment of the present invention, when it is determined that the object is completely included in the inner region of the bounding box, the bounding box setting unit sets the size of the bounding box so that the object closely fits the inner region of the bounding box. can be scaled down

In one embodiment, the bounding box setting unit sets the bounding box until the distance from the nearest point existing on the outline of the object is within the preset maximum number of pixels for each of the plurality of line segments constituting the bounding box. The size of the bounding box can be reduced by moving the segment towards the center.

Characteristically, according to one embodiment of the present invention, when the bounding box setting unit determines that the object spans one or more line segments among a plurality of line segments constituting the bounding box, the size of the bounding box is such that the object is completely included in the bounding box. can be enlarged. After the size expansion of the bounding box is performed, the bounding box setting unit may reduce the size of the bounding box so that the object completely fits within the inner region of the bounding box.

In one embodiment, when the object spans one or more line segments among a plurality of line segments constituting the bounding box, the bounding box setting unit moves the object towards the outside of the bounding box until the object is completely included in the inner area of the bounding box. The size of the bounding box can be enlarged by moving the spanned line segment.

In another embodiment, the bounding box setting unit, when the object spans one or more line segments among a plurality of line segments constituting the bounding box, sets the center of the bounding box to the line segment the object spans until the object is completely included in the inner region of the bounding box. By moving in the direction of , the size of the bounding box can be enlarged in all directions.

In another embodiment, the bounding box setting unit divides the bounding box into a plurality of detailed areas when the object spans one or more line segments among a plurality of line segments constituting the bounding box, and then the object is completely included in the inner area of the bounding box. , it is possible to enlarge only the size of the detailed region over which the object spans among the divided detailed regions. In this case, the size of the detailed region may be determined in advance according to the resolution of the image or the resolution of the object to be identified. Also, the bounding box enlarged according to the shape of the detailed region may have a polygonal shape.

In another embodiment, the bounding box setting unit, when the object spans one or more line segments among a plurality of line segments constituting the bounding box, until the object is completely included in the inner region of the merged bounding box, outside the line segment the object spans A process of newly setting an additional area in the area and merging the newly set additional area and the bounding box may be repeatedly performed.

Next, the object property setting unit may filter a pre-created list of recommended information.

Specifically, the object property setting unit includes information related to the type of object that cannot be set by the work tool used by the operator in the process of setting the bounding box (ie, object irrelevant information) can be removed.

As an embodiment, when a worker uses a tool capable of designating a rectangular area in the process of setting a bounding box, the object property setting unit removes information related to the type of object corresponding to a living organism from a list of previously created recommendation information. can do.

In another embodiment, when a worker uses a tool capable of designating a skeleton structure of an object in the process of setting a bounding box, the object property setting unit is related to the type of object that does not have a skeleton from a list of pre-created recommended information. Information can also be removed.

The object property setting unit may remove information irrelevant to the object from the list of recommendation information and sort remaining information.

As an embodiment, the object property setting unit may arrange the information included in the list of recommended information according to a pre-set sorting criterion based on the position or size occupied by the bounding box in the image.

In another embodiment, the object property setting unit compares the shape of the object included in the bounding box with the basic shape provided in advance for each type of object, and sorts the information included in the list of recommended information based on the similarity of the two shapes. may be

Next, the object property setting unit may output the list of recommended information prepared and filtered to the input/output unit. The object property setting unit may receive a control signal for setting object property information from an operator through an input/output unit.

When one piece of information is selected from the list of recommended information under the control of an operator, the object property setting unit may provide feedback according to the type of object corresponding to the selected information.

In one embodiment, the object property setting unit reflects the color or transparency set differently according to the type of object corresponding to the selected information, and sets a user interface (UI) related to the area inside the bounding box or the area inside the object. can be changed

With the following configuration, the output generation unit may generate annotation work results and transmit them to the learning data design device or the artificial intelligence learning device.

Specifically, the result generation unit determines the location and size of the expanded or reduced bounding box under the control of the operator, and when the property information of the object is determined under the control of the operator based on the list of recommended information, the location of the determined bounding box is determined. And it is possible to create an annotation work result including coordinates according to the size and determined attribute information. Such a work result may have a JSON file format, but is not limited thereto.

In addition, the output generation unit may transmit the generated annotation work result to the learning data design device or the artificial intelligence learning device through the communication unit.

Hereinafter, hardware for implementing the above-described logical components of the annotation device will be described in more detail.

The annotation device may include a processor, a memory, a transceiver, an input/output device, a data bus, and a storage.

The processor may implement operations and functions of the annotation device based on instructions according to software in which the annotation method resides in memory. Software in which the annotation method is implemented may be loaded in the memory. The transceiver may transmit and receive data to and from the learning data design device and the artificial intelligence learning device. The input/output device may receive data necessary for the operation of the annotation device and output a list of images, bounding boxes, and recommendation information. The data bus is connected to the processor, memory, transceiver, input/output device, and storage, and may serve as a movement path for transferring data between each component.

The storage may store an application programming interface (API), a library file, a resource file, and the like required for execution of software in which the annotation method is implemented. The storage may store software 180b in which the annotation method is implemented. Also, the storage may store information necessary for performing the annotation method. In particular, the storage may include a database 185 for storing an image to be annotated, a project attribute, an image attribute, a worker attribute, and a worker's work log.

According to the first embodiment of the present invention, software (180a, 180b) for implementing an annotation method resident in a memory or stored in a storage enables a processor to learn artificial intelligence (AI) under the control of an operator input through an input/output device. Setting a partial region in the image that is a target of annotation work for the bounding box as a bounding box, the processor identifying an object completely included in the inner region of the bounding box or spanning one or more line segments among a plurality of line segments constituting the bounding box. step of expanding or reducing the size of the bounding box so that the processor has a size corresponding to the outline of the identified object, and the position of the bounding box expanded or reduced by the operator's control input through the input/output device by the processor and if the size is confirmed, it may be a computer program recorded on a recording medium to execute a step of generating an annotation work result including coordinates according to the location and size of the determined bounding box.

According to the second embodiment of the present invention, the software (180a, 180b) for implementing the annotation method resident in the memory or stored in the storage enables the processor to learn artificial intelligence (AI) according to the operator's control input through the input/output device. Setting a region in the image that is the target of annotation work for the bounding box, creating a list of recommendation information that can be used as attribute information of an object corresponding to the bounding box set by the processor, and When the property information of an object is determined under the control of an operator based on the list of recommended information, records are recorded to execute the step of generating the work result of the annotation including coordinates according to the location and size of the bounding box and the determined property information. It may be a computer program recorded on a medium.

More specifically, the processor may include an Application-Specific Integrated Circuit (ASIC), other chipsets, logic circuits, and/or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver may include a baseband circuit for processing wired/wireless signals. Input/output devices include input devices such as keyboards, mice, and/or joysticks, liquid crystal displays (LCDs), organic LEDs (OLEDs), and/or active organic An image output device such as an active matrix OLED (AMOLED) or the like may include a printing device such as a printer or a plotter.

When the embodiments included in this specification are implemented as software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. A module resides in memory and can be executed by a processor. The memory may be internal or external to the processor and may be coupled with the processor in a variety of well-known means.

Each component may be implemented by various means, eg, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), FPGAs ( Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored on a recording medium readable through various computer means. can be recorded. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks), floptical It includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, such as a floptical disk, and ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to operate as one or more pieces of software to perform the operations of the present invention, and vice versa.

An annotation device of an artificial intelligence learning system according to an embodiment of the present invention transmits a control signal for specifying one object included in an image from an operator when an image to be an annotation work for AI learning is loaded. can be input.

The annotation device may set a partial region in the image as a bounding box according to a control signal input from a person in charge.

A rectangular shape having two coordinates input from a vendor as an upper left vertex and a lower right vertex can be set as a bounding box, but the bounding box that can be set according to the present invention may be various, such as polygons and skeletons, in addition to rectangles.

The annotation device may identify an object that is completely included in the inner region of the bounding box or spans one or more line segments among a plurality of line segments constituting the bounding box.

And, when it is determined that the identified object spans one or more line segments among a plurality of line segments constituting the bounding box, the annotation device may enlarge the size of the bounding box so that the object is included in the bounding box.

As an embodiment, the annotation device may enlarge the size of the bounding box by moving a line segment spanning the object toward the outside of the bounding box until the object is completely included in the inner region of the bounding box.

In another embodiment, the annotation device may move the center of the bounding box in the direction of the line segment where the object spans and expand the size of the bounding box in all directions until the object is completely included in the inner region of the bounding box.

As another embodiment, after dividing the bounding box into a plurality of detailed regions, the annotation device may enlarge only the size of the detailed region spanning the object among the divided detailed regions until the object is completely included in the inner region of the bounding box. .

In another embodiment, the annotation device repeatedly sets an additional area outside the line segment where the object spans and merges the newly set additional area with the bounding box until the object is completely included in the inner area of the bounding box. You may.

If the object does not cross any of the plurality of line segments constituting the bounding box, the annotation device determines that the object is completely contained in the inner area of the bounding box, and determines that the object is completely included in the inner area of the bounding box. size can be reduced.

In one embodiment, the annotator is the center of the bounding box until the distance from the nearest point existing on the outline of the object is within a preset maximum number of pixels for each of a plurality of line segments constituting the bounding box. You can reduce the size of the bounding box by moving the line towards .

According to the embodiment of the present invention as described above, in performing annotation on each of a plurality of images for artificial intelligence (AI) learning, the size of the bounding box is automatically enlarged or enlarged according to the shape of an object included in the image. By being reduced, the operator can more easily set the bounding box, and the work result of the annotation can have a more uniform quality.

An annotation device according to an embodiment of the present invention may set a bounding box based on coordinates input by an operator.

As an embodiment, the annotation device may set a bounding box based on a rectangle having two coordinates input from an operator as coordinates of an upper left vertex and a coordinate of a lower right vertex in an image. As another embodiment, the annotation device may set a bounding box based on a polygon composed of three or more line segments input from an operator.

The annotation device may identify all or part of objects included in the area inside the set bounding box. To this end, the annotation device may perform image processing such as edge extraction, binarization, and closed region identification on the inner region of the bounding box.

The annotation device may determine whether there are a plurality of objects included in the inner region of the bounding box. As a result of the determination, if there are a plurality of objects identified in the inner region of the bounding box, the annotation device may select only one object from among the identified plurality of objects according to a preset priority.

The annotation device may determine whether the identified object is completely included in the inner region of the bounding box. To this end, the annotation device determines whether the identified object spans one or more line segments among a plurality of line segments constituting the bounding box, based on continuity between pixels included in the inner area of the bounding box and pixels in the outer area of the bounding box. can judge

As a result of the determination, when it is determined that the object spans one or more line segments among a plurality of line segments constituting the bounding box, the annotation device may enlarge the size of the bounding box so that the object is completely included in the bounding box.

As an embodiment, the annotation device may enlarge the size of the bounding box by moving a line segment spanning the object toward the outside of the bounding box until the object is completely included in the inner region of the bounding box.

In another embodiment, the annotation device may move the center of the bounding box in the direction of the line segment where the object spans and expand the size of the bounding box in all directions until the object is completely included in the inner region of the bounding box.

As another embodiment, after dividing the bounding box into a plurality of detailed regions, the annotation device may enlarge only the size of the detailed region spanning the object among the divided detailed regions until the object is completely included in the inner region of the bounding box. .

In another embodiment, the annotation device repeatedly sets an additional area outside the line segment where the object spans and merges the newly set additional area with the bounding box until the object is completely included in the inner area of the bounding box. You may.

As a result of the determination, when it is determined that the object is completely included in the inner region of the bounding box, the annotation device may reduce the size of the bounding box so that the object closely fits the inner region of the bounding box.

In one embodiment, the annotator is the center of the bounding box until the distance from the nearest point existing on the outline of the object is within a preset maximum number of pixels for each of a plurality of line segments constituting the bounding box. By moving the line segment toward , the size of the bounding box can be reduced.

Hereinafter, a method for specifying an object area and detecting an object using a central point according to various embodiments of the present invention as described above and a computer program recorded on a recording medium for executing the method will be described in detail.

In sports-related learning, ball detection is also important in recognizing human motions, and objects close to various prototypes exist in sports images.

In the present invention, when the circular detection function is activated, a circular box is created, and when the box is dragged and dropped on the ball position, it is enlarged until it is larger than the ball, and the center point is corrected. Here, the center point correction can also be manually corrected by the operator.

Regarding object region designation and object detection methods using center points, when an operator creates a first center point, an annotation device creates a first radius including all objects, and a second center point based on an outline within the first radius. After creating the 2nd radius based on the 2nd center point, the final object can be extracted.

The operator may create only the first center point and specify the object, and the operator may proceed to the creation of the first radius and designate the first radius according to the operator's intention.

By extending this method, objects can be extracted by designating 1P when the boundaries are relatively clear for objects of various shapes as well as close-to-circular objects.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention belongs that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. It is self-evident to those skilled in the art. In addition, it should be noted that technical terms used in the present specification and drawings are used only to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in this specification, and are overly inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. On the other hand, when a component is referred to as “directly connected” or “directly connected” to another component, it should be understood that no other component exists in the middle.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents or substitutes other than the accompanying drawings.

On the other hand, as described above, since the annotation work is performed for a large number of data ranging from a few thousand to several million pieces of data, a large number of workers perform the annotation work. In addition, many annotation workers try to solve many questions they have in the process of performing a large number of annotation tasks assigned to them through Q&A. However, questions that annotation workers with normal thinking skills have are often the same or similar to each other, and it is inefficient for a manager to respond in detail to all inquiries individually received from numerous annotation workers.

In addition, data processing rules that must be commonly observed are established in the process of performing annotation work by numerous workers. These data processing rules may be determined in various ways according to the learning goal and learning methodology of artificial intelligence (AI). However, it cannot be determined that annotation workers with normal thinking ability will accurately follow data processing rules, and it is common for many annotation workers to generate the same or similar errors.

In order to overcome these limitations, the present invention intends to propose means for efficiently providing question and answering and partially controlling the operator's input in the process of performing annotation work by the operator.

An artificial intelligence learning system according to an embodiment of the present invention may include at least one annotation device, a learning data design device, and an artificial intelligence learning device.

In addition, in the artificial intelligence learning system according to another embodiment of the present invention, the annotation device, the learning data design device, and the artificial intelligence learning device may be connected to each other through a public network. In this case, a part of the annotation device may be a device that performs annotation by a clouding service.

Since the components of the artificial intelligence learning system according to various embodiments are merely functionally distinct elements, two or more components are integrated and implemented in the actual physical environment, or one component is implemented in the actual physical environment. may be implemented separately from each other.

Each component is described, and the annotation device is a device that can be used to annotate an image provided from a learning data design device or an artificial intelligence learning device.

In particular, the annotation device according to the present invention can efficiently provide answers to questions that a worker (ie, a user of the annotation device) has in the process of performing an annotation task, and limit inputs related to errors easily caused by the operator. It has features that can be

Any device capable of transmitting and receiving data to and from the learning data design device and the artificial intelligence learning device, respectively, and performing calculations using the transmitted and received data may be used as the annotation device.

For example, the annotation device may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto, and is not limited to, a smart phone or a laptop. ), tablets, phablets, portable multimedia players (PMPs), personal digital assistants (PDAs) or e-book readers. can be any one of them.

With the following configuration, the learning data design device is a device that can be used to design and generate artificial intelligence (AI) learning data. Such a learning data design device is basically a device that is distinguished from an artificial intelligence learning device, but may be implemented by being integrated with an artificial intelligence learning device in a real physical environment.

Specifically, the learning data design device may receive attributes of a project related to artificial intelligence (AI) learning from the artificial intelligence learning device. The learning data design device can design data structures for artificial intelligence (AI) learning, refine collected data, process data, expand data, and verify data based on the user's control and the attributes of the project. there is.

In more detail, the learning data design device may transmit an image to be annotated to the annotation device for data processing for artificial intelligence (AI) learning. The learning data design device may receive an annotation work result from the annotation device. In this case, the annotation work result may have a JSON (Java Script Object Notation) file format. Alternatively, the learning data design device may generate a JSON file based on the received result after receiving a result of annotation work in a format different from JSON. In addition, the learning data design device inspects the received or created JSON file, packages it, and transmits it to the artificial intelligence learning device.

Characteristically, the learning data design apparatus according to the present invention may include a database, which is an external storage in which questions and answers of a plurality of workers performing annotation work are accumulated and stored. Here, the question and answer is the data that the user of the annotation device (i.e., the operator) has inquired about the questions in the process of performing the annotation work, and the response data that the user (i.e., manager) of the learning data design device has answered. can be configured.

When query data is received from the annotation device, the learning data design device may retrieve response data matching keywords of the received query data from the database. And, the learning data design device may transmit the searched response data to the annotation device.

If response data matching the keyword of the annotation device does not exist in the database, the learning data design device may receive a message requesting generation of response data matching the keyword from the annotation device. In this case, after outputting the query data received from the annotation device, the learning data design device may receive response data corresponding to the query data from the user. The learning data design device may match response data input from the user with keywords and store them in a database, and then transmit the corresponding response data to the annotation device.

The learning data design device may identify guide information related to the same keyword when more than a preset threshold number of query data including the same keyword is received. And, the learning data design device may distribute the identified guide information to all annotation devices performing annotation work.

In this case, the guide information may be previously set information corresponding to the goal and learning methodology of artificial intelligence (AI) learning so that workers can refer to it in each process of annotation work. The annotation work process in which this guide information is classified and set includes a process of loading an image to be an annotation work, a process of selecting a tool by the operator, a process of inputting coordinates using the tool, and a process of inputting coordinates using the tool. It can be distinguished by including a process of setting a bounding box by and a process of inputting attribute information of an object specified by the bounding box.

Also, the learning data design device may receive control disregard data from the annotation device. Here, the control ignoring data is data in which control signals input by the user of the annotation device in response to the input restriction pattern are accumulated and stored when the user (ie, the operator) of the annotation device ignores the input restriction pattern and performs control.

The learning data design device may determine or readjust a job evaluation grade of a user (ie, a worker) of the corresponding annotation device based on the control signal accumulated and stored in the received control disregard data.

Here, the job evaluation grade of the worker is a grade in which the worker's ability to perform the annotation work is evaluated. Such a job evaluation grade of the worker may be used to determine the type and amount (load) of work to be assigned to the worker, or pay to be paid to the worker, but is not limited thereto.

As such, the learning data design device may be any device capable of transmitting and receiving data to and from the annotation device and the artificial intelligence learning device and performing calculations using the transmitted and received data. For example, the learning data design device may be any one of a desktop, a workstation, or a fixed computing device such as a server, but is not limited thereto.

With the following configuration, the artificial intelligence learning device is a device that can be used to perform machine learning of artificial intelligence (AI) based on data for artificial intelligence (AI) learning.

Specifically, the artificial intelligence learning device may receive a packaged JSON file from the learning data design device. And, the artificial intelligence learning device may perform artificial intelligence (AI) machine learning using the received JSON file.

As such, the artificial intelligence learning device may be any device capable of transmitting and receiving data to and from the learning data design device and performing calculations using the transmitted and received data. For example, the artificial intelligence learning device may be any one of a desktop, a workstation, or a fixed computing device such as a server, but is not limited thereto.

As described above, one or more annotation devices, learning data design devices, and artificial intelligence learning devices may transmit and receive data using a network in which one or more of a security line, a common wired communication network, or a mobile communication network directly connected between devices is combined. there is.

For example, public wired communication networks may include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). It may be, but is not limited thereto. In addition, in the mobile communication network, Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile telecommunication may be included, but is not limited thereto.

Hereinafter, the configuration of the annotation device as described above will be described in more detail.

The annotation device may include a communication unit, an input/output unit, a storage unit, an annotation operation unit, a quality management unit, and a result generation unit.

Since the components of the annotation device are merely functionally distinct elements, two or more components may be integrated and implemented in an actual physical environment, or one component may be implemented separately from each other in an actual physical environment. There will be.

The communication unit may transmit/receive data to and from the learning data design device.

Specifically, the communication unit may receive one or more images from the learning data design device. Here, the image is an image that is a target of annotation work for artificial intelligence (AI) learning. According to the data processing plan designed by the learning data design device or the artificial intelligence learning device, images to be annotated may be individually received or a plurality of images may be collectively received.

The communication unit may receive project properties, image properties, or worker properties from the learning data design device. Here, the properties of the project may include the learning purpose, learning period, number of images required for learning, object properties to be identified in images, bounding box setting rules, etc. for a project related to artificial intelligence (AI) learning. It is not limited to this. Image properties include image file name, image size (width, height), resolution, bit level, compression format, shooting device name, exposure time, ISO speed, focal length, aperture value, and shooting location coordinates (GPS latitude, longitude). , shooting time, etc. may be included, but is not limited thereto. The worker's attributes may include, but are not limited to, the worker's name, identification number, assigned work amount, cost according to the work, work result evaluation, and the like.

The communication unit may transmit the query data generated by the quality management unit to the learning data design device. Here, the query data is data generated to inquire about questions that a user (ie, a worker) has in the process of performing an annotation task.

The communication unit may receive response data from the learning data design device. Here, the response data is data generated to solve questions that the user (ie, the operator) has in the process of performing the annotation work.

The communication unit may transmit a message requesting generation of response data matching the keyword to the learning data design device under the control of the quality management unit.

The communication unit may receive guide information from the learning data design device. Here, the guide information may be information set in advance corresponding to the goal and learning methodology of artificial intelligence (AI) learning so that the operator can refer to it in each process of the annotation work.

The communication unit may transmit the control disregard data to the learning data design device under the control of the quality management unit. Here, the control ignoring data is data in which control signals input by the user of the annotation device in response to the input restriction pattern are accumulated and stored when the user (ie, the operator) of the annotation device ignores the input restriction pattern and performs control.

And, the communication unit may transmit the work result of the annotation generated by the annotation operation unit to the learning data design device. Here, the work result may include the coordinates of the bounding box set under the operator's control and property information of the object. In addition, the work result may have a JSON file format, but is not limited thereto.

With the following configuration, the input/output unit may input a signal from an operator through a user interface (UI) or output calculated results to the outside.

Here, the worker means a person who performs annotation work. Such an operator may be referred to as a user, performer, labeler, or data labeler, but is not limited thereto.

Specifically, the input/output unit may output an image to be an annotation work. The input/output unit may receive a control signal for setting a bounding box from an operator. In addition, the input/output unit may overlay and output the bounding box on the image.

Here, the bounding box is an area for specifying an object to be learned by artificial intelligence (AI) among objects included in the image. Such a bounding box may have a rectangle or polygon shape, but is not limited thereto.

The input/output unit may receive a control signal for setting object attribute information from an operator. Here, also, the property information of the object is information for designating the property of the object to be an artificial intelligence (AI) learning target. Such object attribute information includes information on the type of annotation, class name (class), classification items (tags), truncated state of the object, major classification, subclassification, or upper level (instance upper). may be, but is not limited thereto.

The input/output unit may output response data retrieved from the database of the learning data design device based on the query data. The input/output unit may output guide information that a worker can refer to in each process of annotation work.

The input/output unit may deactivate a part of the user interface (UI) for receiving a control signal from the operator or ignore part of the control signal input from the operator through the user interface (UI) according to the control of the quality control unit.

With the following configuration, the storage unit can store data required for annotation work.

Specifically, the storage unit may store an image received through the communication unit. The storage unit may store project properties, image properties, or worker properties received through the communication unit.

With the following configuration, the annotation work unit can perform annotation work on data for artificial intelligence (AI) learning.

Specifically, the annotation task unit may load an image to be an annotation task into memory. The annotation work unit may select a tool according to an operator's control signal input through the input/output unit. Here, the tool is a tool for setting a bounding box that specifies one or more objects included in the image.

The annotation work unit may receive coordinates through the selected tool according to a control signal input through the input/output unit. The annotation work unit may set a bounding box for specifying an object in the image based on the input coordinates. Here, the bounding box is an area for specifying an object to be learned by artificial intelligence (AI) among objects included in the image. Such a bounding box may have a rectangle or polygon shape, but is not limited thereto.

For example, the annotation work unit receives two coordinates from the operator through the input/output unit, and sets the bounding box based on a rectangle having the input two coordinates as the coordinates of the upper left vertex and the lower right vertex in the image. can In this case, the two coordinates may be set by the operator inputting one type of input signal twice (eg, mouse click) or by the operator inputting two types of input signals once (eg, mouse drag). It may, but is not limited thereto.

Also, the annotation work unit may set property information of the object specified by the bounding box according to the control signal input through the input/output unit. Here, the property information of the object is information for designating the property of an object to be an artificial intelligence (AI) learning target. Such object attribute information includes information on the type of annotation, class name (class), classification items (tags), truncated state of the object, major classification, subclassification, or upper level (instance upper). may be, but is not limited thereto.

With the following configuration, the quality management unit may manage the quality of work in the process of performing annotation work through the annotation work unit.

Characteristically, in order to improve the quality of annotation work, the quality control unit according to an embodiment of the present invention efficiently provides answers to questions that workers have in the process of performing annotation work, Relevant inputs can be restricted.

First of all, a process of providing a response to a question raised by a quality management unit in the process of performing an annotation work by a worker according to an embodiment of the present invention will be described.

Specifically, the quality management unit may receive a command notifying the existence of a question mark from an operator through a user interface (UI) of the input/output unit in the annotation work process for artificial intelligence (AI) learning of the annotation work unit.

For example, the quality management unit may receive a command notifying the existence of a question from an operator through a button icon included in the user interface (UI) or a preset shortcut key, but is not limited thereto. no.

The quality control unit may generate query data in response to a command notifying that a question mark is input from an operator.

Here, the query data is data generated to inquire about questions that the operator has in the process of performing the annotation work. Such query data may include keywords, contents, and situation data, but is not limited thereto.

The keyword included in the query data is a word that can represent a question that a worker has. The content is a detailed description of the question the worker has. And, the situation data is data about the situation in which the operator has questions.

The quality control unit may receive input of keywords and contents from a worker through a user interface (UI) of the input/output unit. However, the quality management unit may receive only content input from a worker and automatically determine keywords.

More specifically, the quality control unit may identify the progress state of the work according to the type of control signal input from the operator in the annotation work process.

Here, the annotation work process is a process of loading an image to be annotated, a process of selecting a tool by an operator, a process of inputting coordinates using a tool, a process of setting a bounding box by coordinates, and a process of setting an object specified by the bounding box. It can be distinguished by including the attribute information input process.

For example, if the type of control signal input from the operator corresponds to a command to select a menu icon for loading an image or to search for a location of an image stored in a storage device, the quality control unit proceeds with the annotation work. It may be determined that the state corresponds to an image loading process.

If the type of control signal input from the operator corresponds to a command to move the input cursor toward the tool bar after image loading, the quality control unit determines that the progress of the annotation work corresponds to the tool selection process. can do.

If the type of control signal input from the operator corresponds to a command to move the input cursor within the image after the tool is selected, the quality control unit may determine that the progress state of the annotation work corresponds to the process of inputting coordinates.

If the type of control signal input from the operator corresponds to a command to input the last coordinate among the coordinates that can be input using the selected tool, the quality control unit determines that the progress of the annotation work corresponds to the setting process of the bounding box. can judge

And, if the type of control signal input from the operator in the quality management unit corresponds to a command to move the input cursor toward the property input window after the bounding box is set, the progress of the annotation work corresponds to the process of inputting object property information can be judged to be

The type of control signal for identifying the work progress state described above is only an example, and the fact that the work progress state can be identified according to the type of other control signals is a common knowledge in the art to which the present invention belongs. It will be self-evident to those who have it.

After identifying the progress status of the annotation task, the quality management unit may determine one keyword based on the identified task progress status.

As described above, when a keyword is input or determined, the quality management unit may determine situation data.

According to an embodiment of the present invention, the quality control unit may further include at least one of information of an image to be annotated, coordinate information input using a tool, and attribute information of an object to determine situation data.

According to another embodiment of the present invention, the quality control unit may link one or more event messages according to the occurrence time of the event (linked list). To this end, the operating system of the annotation device may correspond to an event driven architecture type operating system that detects events and processes data or performs operations according to the generated events. In addition, the event message may correspond to a system message generated by controlling an input/output device of a worker, selectively extracted from a message queue managed by an operating system of an annotation device. there is. In addition, the quality control unit may determine situation data including event messages connected according to the occurrence time of the event.

According to another embodiment of the present invention, the quality management unit may connect one or more captured images according to the precedence of the capturing time. Here, the captured image may correspond to an image obtained by capturing a screen output by an input/output device of an annotation device according to a previously set capture cycle for each process of the annotation work process. And, the quality control unit may determine situation data including the connected captured images according to the precedence and precedence of the capture time.

The quality control unit may generate query data by including the input content, the input or determined keyword, and the determined situation data.

Next, the quality management unit may search the database of the learning data design device based on the generated query data.

According to an embodiment of the present invention, the quality control unit may search for response data matching keywords of the query data among response data prepared to respond to queries of third workers with respect to data stored in the database of the learning data design device. there is. Here, the third workers refer to persons who have been allocated annotation work from the same project as the project in which the annotation work of the annotation device transmitting the query data is derived.

According to another embodiment of the present invention, the quality management unit may search for response data matching keywords of the query data among response data created in the annotation process for the third project with respect to data stored in the database of the learning data design device. there is. Here, the third project means a project for detecting a class of the same type as a class to be detected from an image by learning artificial intelligence (AI) based on the annotation work result of the annotation device.

When response data is retrieved from the database as a result of the database search as described above, the quality management unit may output the response data retrieved from the database through the user interface (UI) of the input/output unit.

In addition, when response data matching the keyword of the query data is not searched from the database, the quality management unit may transmit a message requesting generation of response data matching the keyword of the query data to the learning data design device.

Meanwhile, a process of limiting an input related to an error easily caused by a quality control unit according to another embodiment of the present invention will be described.

Specifically, the quality control unit identifies the progress of the work according to the type of control signal input from the operator through the user interface (UI) of the input/output unit in the annotation work process for artificial intelligence (AI) learning of the annotation work unit. . The process of identifying the progress of the work of the quality management unit is the same as described above, so it will not be described repeatedly.

The quality control unit may determine an input limiting pattern based on the progress state of the identified task. Here, the input restriction pattern is set for each annotation work process because there is a possibility of including errors in the result of the annotation work, and may include a combination of one or more control signals.

According to an embodiment of the present invention, the input restriction pattern may be set based on a result of performing verification using a script on a result of an annotation job previously performed by an operator. Here, the script may be a code provided for verifying the validity of data or whether or not a pre-set target quality is met for a pre-executed annotation work result.

According to another embodiment of the present invention, the input restriction pattern may be set based on a result of verifying annotation work results performed by third workers. Here, the third workers refer to persons who have been allocated annotation work from the same project as the project in which the annotation work of the annotation device transmitting the query data is derived.

For example, if the progress of the identified work corresponds to the image loading process, the quality control unit may be included in the image according to the attributes of a project related to artificial intelligence (AI) learning set in advance, the attributes of an image, or the attributes of a worker. You can determine the types of objects that are predicted to be absent. The quality management unit may include a control signal related to a tool for only objects of the determined type in the input limiting pattern.

As another example, when the progress state of the identified work corresponds to a process of selecting a tool by a worker, the quality control unit may identify the type of input/output device receiving a control signal from the worker. The quality control unit may include in the input limiting pattern control signals related to tools that cannot be controlled using the identified type of input/output device.

As another example, when the progress state of the identified work corresponds to a process of inputting coordinates using a tool, the quality control unit relates to an area in which coordinates cannot be input with the corresponding tool according to a method of using a tool preset in correspondence with the selected tool. A control signal can be included in the input limiting pattern.

As another example, when the progress state of the identified work corresponds to the process of inputting attribute information of an object, the quality management unit allocates the object specified by the bounding box according to the type of tool selected by the operator to set the bounding box. Can identify types of classes that cannot be As a specific example, when a tool selected by the operator to set the bounding box corresponds to a skeleton tool, the quality management unit may identify a type of class corresponding to a non-living body that cannot be specified by the skeleton tool. And, the quality management unit may include a control signal related to the type of the identified class in the input limiting pattern.

As described above, when the input limiting pattern is determined, the quality control unit may limit the control signal corresponding to the input limiting pattern among the control signals input from the operator through the user interface (UI) of the input/output unit.

According to one embodiment of the present invention, the quality control unit selectively disables only the user interface (UI) capable of receiving the control signal corresponding to the input restriction pattern among the user interfaces (UI) of the input/output unit for receiving the control signal from the operator. can make it

According to another embodiment of the present invention, the quality control unit may ignore a control signal corresponding to an input restriction pattern among control signals input from an operator through a user interface (UI) of an input/output unit. The quality management unit may generate control disregard data by accumulating and storing control signals input from the operator in response to the input restriction pattern. And, the quality control unit may transmit the generated control disregard data to the learning data design device through the communication unit.

With the following configuration, the result generation unit may generate a work result of annotation and transmit it to the learning data design device.

Specifically, the result generation unit may generate an annotation work result including coordinates according to the location and size of the bounding box set through the annotation operation unit and set attribute information. Such a work result may have a JSON file format, but is not limited thereto.

In addition, the output generation unit may transmit the generated annotation work result to the learning data design device through the communication unit.

Hereinafter, hardware for implementing the above-described logical components of the annotation device will be described in more detail.

The annotation device may include a processor, a memory, a transceiver, an input/output device, a data bus, and a storage.

The processor may implement operations and functions of the annotation device based on instructions according to software in which the annotation method resides in memory. Software in which the annotation method is implemented may be loaded in the memory. The transceiver may transmit and receive data to and from the learning data design device and the artificial intelligence learning device. The input/output device may receive data required for operation of the annotation device and output images, response data, and guide information. The data bus is connected to the processor, memory, transceiver, input/output device, and storage, and may serve as a movement path for transferring data between each component.

The storage may store an application programming interface (API), a library file, a resource file, and the like required for execution of software in which the annotation method is implemented. The storage may store software in which the annotation method is implemented. Also, the storage may store information necessary for performing the annotation method. In particular, the storage may include a database for storing an image that is a target of annotation work, a project property, an image property, a worker property, and the like.

According to the first embodiment of the present invention, the software (180a, 180b) resident in the memory or stored in the storage has a question from the operator through the input/output device during the process of annotation work for artificial intelligence (AI) learning by the processor. A step of receiving a command indicating that The process of generating query data in response to an input command, the process of searching a database of a previously set learning data design device based on the query data, and the process of processing the searched response data from the database through the input/output device. It may be a computer program recorded on a recording medium to execute the outputting step.

According to the second embodiment of the present invention, software residing in memory or stored in storage identifies the progress state of a task according to the type of a control signal input from an operator during an annotation task process for artificial intelligence (AI) learning by a processor. The step of determining an input restriction pattern based on the progress state of the identified task by the processor, and the step of the processor limiting the control signal corresponding to the input restriction pattern among the control signals input from the operator through the input/output device. It may be a computer program recorded on a recording medium to do so.

More specifically, the processor may include an Application-Specific Integrated Circuit (ASIC), other chipsets, logic circuits, and/or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver may include a baseband circuit for processing wired/wireless signals. Input/output devices include input devices such as keyboards, mice, and/or joysticks, liquid crystal displays (LCDs), organic LEDs (OLEDs), and/or active organic An image output device such as an active matrix OLED (AMOLED) or the like may include a printing device such as a printer or a plotter.

When the embodiments included in this specification are implemented as software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. A module resides in memory and can be executed by a processor. The memory may be internal or external to the processor and may be coupled with the processor in a variety of well-known means.

Each component may be implemented by various means, eg, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), FPGAs ( Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored on a recording medium readable through various computer means. can be recorded. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks), floptical It includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, such as a floptical disk, and ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to operate as one or more pieces of software to perform the operations of the present invention, and vice versa.

Hereinafter, features of the artificial intelligence learning system according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

An annotation device of an artificial intelligence learning system according to an embodiment of the present invention can efficiently provide answers to questions that a worker has in the process of performing an annotation task.

More specifically, the annotation device of the artificial intelligence learning system according to an embodiment of the present invention may receive a command notifying that a doubt exists from an operator through a user interface.

For example, the annotation device may receive a command notifying that a question mark exists from an operator through a button icon included in a user interface, but is not limited thereto.

The annotation device may generate query data in response to a command input from an operator notifying that a question mark exists. To this end, the annotation device may output a popup window and receive input of keywords and content from an operator through the outputted popup window. Alternatively, the annotation device may receive only content input from an operator and automatically determine keywords.

More specifically, the annotation device may identify the progress state of the work according to the type of the control signal input from the operator during the annotation work process. Here, the annotation work process is a process of loading an image to be annotated, a process of selecting a tool by an operator, a process of inputting coordinates using a tool, a process of setting a bounding box by coordinates, and a process of setting an object specified by the bounding box. It can be distinguished by including the attribute information input process. And, the annotation device may determine one keyword based on the progress state of the identified task.

Once keywords are entered or determined, the annotation device can determine contextual data.

According to an embodiment of the present invention, the annotation device may determine context data by further including one or more of information of an image to be annotated, coordinate information input using a tool, and property information of an object.

According to another embodiment of the present invention, the annotation device may determine context data including one or more event messages connected according to the occurrence time of the event.

To this end, the operating system of the annotation device may correspond to an operating system of an event-driven architecture that senses an event and processes data or performs an operation according to the generated event. Also, the event message may correspond to a system message selectively extracted from a message queue managed by the operating system of the annotation device and generated by controlling the input/output device of the operator.

According to another embodiment of the present invention, the annotation device may determine situational data including one or more captured images connected according to the sequence of capture time. Here, the captured image may correspond to an image obtained by capturing a screen output by an input/output device of an annotation device according to a previously set capture cycle for each process of the annotation work process.

In addition, the annotation device may generate query data by including the input content, the input or determined keyword, and the determined situation data.

The annotation device may search the database of the learning data design device based on the generated query data. In this way, when searching the database of the learning data design device, the annotation device may search for response data matching keywords of the query data among response data created to respond to the query of the third workers. Alternatively, the annotation device may search for response data matching keywords of the query data among response data created in the annotation process for the third project.

Here, the third workers refer to persons who have been allocated annotation work from the same project as the project in which the annotation work of the annotation device transmitting the query data is derived. And, the third project refers to a project for detecting a class of the same type as a class to be detected from an image by learning artificial intelligence (AI) based on the annotation work result of the annotation device.

Also, the annotation device may output response data retrieved from the database of the learning data search device through a user interface.

Therefore, the artificial intelligence (AI) learning system according to an embodiment of the present invention addresses questions that workers have in the process of performing annotation work on a plurality of images for AI learning, and other workers' By automatically responding through a database in which questions and answers are accumulated, the number of responses that the manager has to answer directly can be reduced.

That is, according to an embodiment of the present invention, Q&A for questions that workers have can be processed like Frequently Asked Questions (FAQ).

In addition, the artificial intelligence (AI) learning system according to an embodiment of the present invention can point out a missing point if the reviewer's review results are inappropriate, and in this process, it selects similar data from the repository as well as the reviewer's points It can do natural language processing and provide additional information to the user about wrong issues through a chat system.

Specifically, in case of rejection, the reason for rejection can be delivered as a commentary, and if there is a reason for rejection even after reworking after passing the commentary, it can be regarded as a case where the user did not understand the commentary or additional explanation is needed, so it is an error comes out and gives feedback, the reviewer simply catches the points for errors and rejects them, selects similar data from the storage, and natural language-based AI intervenes to add additional explanations about the errors and chat through the chat system. You can provide an accurate explanation to the operator.

In addition, the artificial intelligence (AI) learning system according to an embodiment of the present invention can perform an answer by determining and finding a similar missing point in the storage when an operator makes an inquiry.

Specifically, in the case of a chatting system, workers' questions accumulate after a certain amount of time, and answers can be automatically provided to workers by analyzing the frequency and question type patterns of the accumulated questions.

Information provided by the repository includes reviewers' feedback on work, questions from users, and answers to questions.

Also, images of similar tasks can be provided when answering, and similar data can be selected from storage, appropriate answers and commentaries can be extracted, and provided through a chat system through natural language conversion.

In the case of selecting and providing similar data from the repository, not only text information (existing feedback, questions, and answers), but also work images obtained from similar data can be provided, and in selecting similar data from the repository, feedback or question Data to be provided to the user can be selected based on the similarity of the image as well as the keyword search of the image.

When the reviewer gives feedback, image information such as highlighting the part in a specific object can be provided together, and the error part can be visually expressed (in the image) and provided along with text.

As described above, the annotation review feedback providing method may be performed in parallel while a worker performs an annotation task.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention belongs that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. It is self-evident to those skilled in the art. In addition, although specific terms have been used in the present specification and drawings, they are only used in a general sense to easily explain the technical content of the present invention and help understanding of the present invention, but are not intended to limit the scope of the present invention. Accordingly, the foregoing detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be selected by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (5)

Annotation device, in the process of annotation (annotation) for artificial intelligence (Artificial Intelligence, AI) learning, through a user interface (User Interface, UI) step of receiving a command notifying the existence of a question mark from the operator;
generating, by the annotation device, query data in response to the input command;
Searching, by the annotation device, a database of previously set learning data design devices based on the query data; and
The method of providing annotation review feedback comprising: converting, by the annotation device, response data retrieved from the database into natural language and providing the result to the operator through a chatting unit.
According to claim 1,
The annotation device,
If it is determined that the question input from the task is a question for the same reason, in the step of searching the database, a basic usage method for the annotation task is additionally searched.
According to claim 1,
The step of searching the database is,
Retrieve questions of workers accumulated in the chatting unit;
The step of providing the worker,
A method for providing feedback for annotation review, characterized in that providing response data by analyzing question frequency and question type patterns in relation to the input questions.
According to claim 3,
The step of searching the database is,
Retrieve the similarity of the image to be annotated,
The step of providing the worker,
A method of providing feedback for annotation reviews, characterized in that a highlight is displayed on the image and provided as response data.
memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
Receiving, by the processor, a command notifying that a question exists from an operator through a user interface (UI) in an annotation work process for artificial intelligence (AI) learning;
generating, by the processor, query data in response to the input command;
searching, by the processor, a database of previously set learning data design devices based on the query data; and
The processor converts the response data retrieved from the database into natural language and provides it to the worker through a chatting unit; a computer program recorded on a recording medium to execute.