KR102310598B1 - Method of creating data for machine learning using metadata, and computer program recorded on record-medium for executing method thereof - Google Patents

Abstract

The present invention proposes a method of generating artificial intelligence (AI) training data that meets a specific purpose from big data by using metadata. The method includes: distributing, by a training data generation device, a plurality of images, which are targets of an annotation work; specifying, by a plurality of annotation devices, coordinates of an object included in each of the distributed images, generating coordinates in the image of the specified object and metadata for the image, and generating an annotation work result to include the generated metadata; and receiving, by the training data generation device, a plurality of annotation work results, classifying the annotation work results based on the metadata, and generating artificial intelligence (AI) training data.

Description

Method of creating data for machine learning using metadata, and computer program recorded on record-medium for executing method thereof

The present invention relates to data design for artificial intelligence (AI) learning. More specifically, a method capable of generating data for artificial intelligence (AI) learning that meets a specific purpose from big data using metadata, and a computer program recorded on a recording medium to execute the same is about

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning ability, reasoning ability, and perception ability using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the form of data for learning.

In general, the design of data for artificial intelligence (AI) learning proceeds in the stages of data structure design, data collection, data purification, data processing, data expansion, and data verification.

To describe each step in more detail, the design of the data structure is made through the definition of an ontology, a definition of a classification system, and the like. The collection of data is made by collecting data through direct shooting, web crawling, or association/professional organizations. Data purification is performed by removing duplicate data from the collected data and de-identifying personal information. Data processing is performed by performing annotations and inputting metadata. Data expansion is performed by performing ontology mapping and supplementing or extending the ontology as needed. And, the verification of the data is performed by verifying the validity according to the set target quality using various verification tools.

Data processing is also referred to as data labeling. More specifically, the annotation operation of data processing is performed by the operator processing the object included in the image with a bounding box. Here, the bounding box is an area for specifying the position and shape of an object included in the image.

In addition, the metadata input operation of data processing is performed by an operator inputting image information, copyright information, shooting condition information, annotated object information, environment information, and the like. For example, the information of the image may include a file name, a file size, and an image size. Copyright information may include information about image copyright holders. The shooting condition information may include resolution, bit value, aperture transmission amount, exposure time, ISO sensitivity, focal length, aperture opening value, angle of view, white balance, and RGB depth. The object information may include annotation type, coordinate value, area size, class name, tag item, truncation status, large classification, medium classification, small classification, and instance information. In addition, the environment information may include a shooting date and time, a shooting location, and weather information.

The number and types of information that can be input in the metadata input process of data processing may vary depending on the purpose of artificial intelligence (AI) learning. For example, in the case of artificial intelligence (AI) learning data for performing automatic driving, the environmental information will further include the type of road on which the image was taken, road surface information, and traffic congestion information. can

As described above, data processing is performed for a large number of data ranging from a few thousand to a maximum of several million pieces of data. Accordingly, various means for more easily processing a large number of data and more easily verifying the processed data are required.

Korean Patent Application Laid-Open No. 10-2020-0042629, ‘Method for generating touch-based annotations and images of mobile devices for artificial intelligence learning, and device therefor’, (published on April 24, 2020)

One object of the present invention is to provide a method capable of generating data for artificial intelligence (AI) learning that meets a specific purpose from big data using metadata.

Another object of the present invention is to provide a computer program recorded on a recording medium to execute a method capable of generating data for artificial intelligence (AI) learning that meets a specific purpose from big data.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical task as described above, the present invention proposes a method for generating data for artificial intelligence (AI) learning that meets a specific purpose from big data using metadata. The method includes: distributing, by the learning data generating apparatus, a plurality of images to be subjected to an annotation operation; A plurality of annotation devices specify coordinates of an object included in each image with respect to the plurality of distributed images, and generate coordinates in the image of the specified object and metadata for the image and generating an annotation work result by including the generated metadata; and receiving, by the learning data generating device, the plurality of annotation work results, classifying the plurality of annotation work results based on the metadata, and generating artificial intelligence (AI) learning data. can do.

In the step of generating the data for artificial intelligence (AI) learning, big data is constructed using the results of the annotation work classified based on the metadata, and when a requested value is input from the outside, the big data is mined. (mining) or filtering (filtering) to extract the result of the annotation work corresponding to the request value, and packaging the extracted result of the annotation work to generate the artificial intelligence (AI) learning data. In this case, the request value may be a value indicating the type of information to be included in the artificial intelligence (AI) learning data.

For example, the big data includes a hash table consisting of a plurality of buckets according to the type of metadata, and each bucket includes a link to an annotation work result. It may be configured to include more than one slot. And, generating the data for artificial intelligence (AI) learning is searching for one or more slots by substituting the requested value into a hash function set in advance corresponding to the hash table, and searching for one or more slots. One or more annotations can be extracted through the included link.

In the step of generating the data for artificial intelligence (AI) learning, when the loading density of the hash table exceeds a preset threshold density in the process of building the big data, the link included in the slot is maintained. In one state, the capacity of the hash table may be increased by doubling the number of buckets according to the type of metadata and then re-adjusting the home bucket of the slot.

In the step of generating the data for artificial intelligence (AI) learning, when the type of information indicated by the request value is not included in the metadata, the data structure is such that the type of information that is not included in the metadata is included. After redesigning the data structure, the plurality of images may be redistributed to the plurality of annotation devices.

In the step of generating the data for artificial intelligence (AI) learning, the big data is divided into a plurality of layers, and based on the job evaluation grade given to the user of the annotation device in the process of building the big data, the The layer to which the annotation work result belongs is determined, and the annotation work result may be extracted from each layer according to a ratio corresponding to the request value input from the outside.

On the other hand, in the step of generating the annotation work result, when the plurality of images correspond to images continuously photographed by the same camera, environment information previously given to the previously photographed image among the plurality of images is followed by Metadata for the captured image may be generated by including the captured image as environment information. In this case, the environment information may include information on the photographing date and time of the image, the photographing location, weather information, road type, road surface, and traffic congestion.

In the step of generating the result of the annotation work, when there are a plurality of previously photographed images, environment information previously given for an image including an object to which the same tracking identifier is assigned among the plurality of images It can be included as environmental information of the image taken by trailing. In this case, the tracking ID may be a unique identifier of an object assigned to track the object.

The step of generating the annotation work result is based on a result of comparing the background area occupied by the object to which the same ID is given for each of the previously photographed image and the subsequently photographed image with each other, based on the result of comparing the previously photographed image with each other. It may be determined whether the environment information of the photographed image is applicable as the environment information of the photographed image by following the environment information given to the photographed image.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute the verification method as described above. The computer program includes a memory; transceiver; and a processor for processing instructions resident in the memory. The computer program may further include: distributing, by the processor, a plurality of images to be annotated through the transceiver; receiving, by the processor, the plurality of annotation work results through the transceiver, wherein the annotation work results include coordinates of an object in the image for each of the plurality of images and metadata for the images; and a computer program recorded on a recording medium in order to execute, by the processor, generating data for artificial intelligence (AI) learning by classifying the plurality of annotation work results based on the metadata.

Specific details of other embodiments are included in the detailed description and drawings.

According to conventional means, when data for artificial intelligence (AI) learning is requested from a customer who wants to learn artificial intelligence (AI), thereafter, data structure design, data collection, data purification, data processing, It took a relatively long time to expand data and verify data, and to generate and deliver artificial intelligence (AI) learning data to customers based on the results.

In contrast, according to embodiments of the present invention, in response to the needs of customers who want to learn artificial intelligence (AI), big data built in advance is simply mined or filtered for artificial intelligence (AI) learning. By generating data and directly delivering the generated AI training data, the period required to generate and deliver AI training data can be greatly shortened.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 and 2 are block diagrams of an artificial intelligence learning system according to various embodiments of the present invention.
3 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.
4 is a logical configuration diagram of an annotation device according to an embodiment of the present invention.
5 is a hardware configuration diagram of an annotation apparatus according to an embodiment of the present invention.
6 is a signal flow diagram illustrating a method of generating data for artificial intelligence (AI) learning according to an embodiment of the present invention.
7 is a conceptual diagram for explaining a concept of generating data for artificial intelligence (AI) learning according to an embodiment of the present invention.
8 is an exemplary diagram for explaining the concept of a tracking ID according to an embodiment of the present invention.
9 is an exemplary diagram for explaining a user interface (UI) for checking a tracking ID according to an embodiment of the present invention.
10 and 11 are exemplary views for explaining a process of determining environmental information in a continuous image according to some embodiments of the present invention.
12 and 13 are exemplary diagrams for explaining a process of automatically determining environment information according to some embodiments of the present invention.
14 is an exemplary diagram for explaining a user interface (UI) for reviewing metadata according to an embodiment of the present invention.
15 is a flowchart illustrating an operation process of an annotation apparatus according to an embodiment of the present invention.

It should be noted that technical terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in this specification should be interpreted in the meaning generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise defined in this specification, and excessively inclusive. It should not be construed in the meaning of a human being or in an excessively reduced meaning. In addition, when the technical terms used in the present specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be understood by being replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “consisting of” or “having” should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are included. It should be construed that it may not, or may further include additional components or steps.

Also, terms including ordinal numbers such as first, second, etc. used herein may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but another component may exist in between. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents, or substitutes other than the accompanying drawings.

The processing of images to generate data for artificial intelligence (AI) learning is performed with a large number of images ranging from a few thousand to a maximum of several million images. Therefore, according to the conventional means, it took a relatively long time to generate and deliver artificial intelligence (AI) learning data to the customer after receiving a request from a customer to learn artificial intelligence (AI).

In order to solve this difficulty, the present invention intends to propose various means for more easily processing a large number of images and more easily verifying the processed data.

1 and 2 are block diagrams of an artificial intelligence learning system according to various embodiments of the present invention.

As shown in Figure 1, the artificial intelligence learning system according to an embodiment of the present invention is a learning data generating device 100, one or more annotation devices (200-1, 200-2, ..., 200-n; 200) , the learning data verification apparatus 300 and the artificial intelligence learning apparatus 400 may be included.

In addition, as shown in FIG. 2, the artificial intelligence learning system according to another embodiment of the present invention includes one or more annotation devices 200-a, 200-b, ..., 200-m; 200 and a learning data verification device ( 300-a, 300-b, …, 300-m; 300) a plurality of groups (Group-a, Group-b …, Group-m), a learning data generating apparatus 100 and an artificial intelligence learning apparatus ( 400) may be included.

As such, since the components of the AI learning system according to various embodiments are merely functionally distinct elements, two or more components are integrated with each other in the actual physical environment, or one component is the actual physical environment. may be implemented separately from each other.

Description of each component, the learning data generating apparatus 100 is a device that can be used to design and generate artificial intelligence (AI) learning data. As such, the learning data generating apparatus 100 is basically a device distinct from the learning data verifying apparatus 300, but in an actual physical environment, the learning data generating apparatus 100 and the learning data verifying apparatus 300 are one device. It may be integrated and implemented.

Characteristically, the learning data generating apparatus 100 according to an embodiment of the present invention preemptively performs artificial intelligence (AI) prior to receiving a request for artificial intelligence (AI) learning data from the artificial intelligence learning apparatus 400 . Big data that can generate learning data can be built.

And, when the learning data generating device 100 receives a request value according to the customer's needs from the artificial intelligence learning device 400, it mines or filters the pre-built big data to perform artificial intelligence (AI) The data for learning may be generated, and the generated data for artificial intelligence (AI) learning may be transmitted to the artificial intelligence learning apparatus 400 .

The training data generating apparatus 100 having such a characteristic transmits and receives data to and from the annotation apparatus 200, the learning data verification apparatus 300, and the artificial intelligence learning apparatus 400, and performs an operation based on the transmitted and received data Any device that can do it is allowed. For example, the training data generating apparatus 100 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

A detailed configuration and operation of the learning data generating apparatus 100 as described above will be described later with reference to FIG. 3 .

With the following configuration, the annotation device 200 is a device that can be used to annotate the image distributed by the training data generating device 100 .

Characteristically, the annotation apparatus 200 according to an embodiment of the present invention tracks a specified object from each image (ie, an image to be annotated) distributed by the learning data generating apparatus 100 . You can give a tracking ID (tracking identifier) that can be done. Also, the annotation apparatus 200 may automatically give environment information to each image distributed by the learning data generating apparatus 100 .

Any device capable of transmitting and receiving data to and from the training data generating device 100 and the training data verifying device 300, and performing an operation based on the transmitted/received data, may use the annotation device 200 having these characteristics. may be permitted. For example, the annotation device 100 may be a desktop, a stationary computing device such as a workstation or a server, or a smart phone, a laptop, a tablet, a phablet, or a portable multimedia player. It may be any one of a portable computing device such as a Portable Multimedia Player (PMP), Personal Digital Assistants (PDA), or an E-book reader.

In addition, all or a part of the annotation apparatus 200 may be a device in which an annotator performs an annotation through a clouding service.

A detailed configuration and operation of the annotation apparatus 200 will be described later with reference to FIGS. 4 and 5 .

With the following configuration, the learning data verification device 300 is a device that can be used to verify data for artificial intelligence (AI) learning. That is, the learning data verification apparatus 300 verifies whether the annotation operation result generated by the annotation apparatus 200 meets a preset target quality or whether the annotation operation result is valid for artificial intelligence (AI) learning. It is a device that can

Specifically, the training data verification apparatus 300 may receive an annotation work result from the annotation apparatus 200 . Here, the annotation work result may include the coordinates of the object specified by the annotation operator, and metadata about the image or object. Metadata of the result of annotation work may include image information, copyright information, shooting condition information, specified object information, and environment information, but is not limited thereto. Such an annotation operation result may have a JSON (Java Script Object Notation) file format, but is not limited thereto.

The training data verification apparatus 300 may inspect the received annotation work result. To this end, the learning data verification apparatus 300 may perform inspection using a script on the annotation work result. Here, the script is a code for verifying whether a preset target quality is met or whether data is valid with respect to the results of the annotation work.

Characteristically, the learning data verification apparatus 300 according to embodiments of the present invention may provide a user interface (UI) that can more easily inspect the tracking ID and environment variables of objects included in the annotation work result. .

Specifically, when the training data verification apparatus 300 outputs a user interface (UI) capable of verifying metadata of an annotation work result, a crop image of an object to which the same tracking ID is assigned from images included in the image group. After collecting the cropped images, the cropped images collected on the user interface (UI) may be sequentially arranged and output according to the shooting order of the images.

Here, the image group is a set of annotated images. As such, the image group may be formed for each annotation device 200 or for each project. For example, an image group consists of images continuously captured in time series by the same camera as the camera that captured the image to be annotated, and other images placed adjacent to the camera that captured the image to be annotated. Images taken spatially and continuously by a camera may be included, and the present invention is not limited thereto.

And, the crop image is an image in which a part of the image to be annotated is cut based on the coordinates set for specifying the object by the annotation.

In addition, the learning data verification apparatus 300 compares the pattern of the RGB (Red, Green, Blue) values of the objects included in each of the crop images and the pattern of the edge with respect to the crop images of the objects to which the same tracking ID is given. In cross-contrast, it is possible to check whether it is correct that the same tracking ID is assigned to each object.

On the contrary, the learning data verification apparatus 300 cross-contrasts the pattern of the RGB values and the pattern of the edges of the objects to which different tracking IDs are given, targeting the crop images of the objects to which different tracking IDs are given, It is also possible to check whether it is correct that different tracking IDs have been assigned to them.

In this case, the pattern of RGB values means a pattern for a ratio of RGB values to pixels in an image and an arrangement order of RGB values. And, the pattern of the edge means a pattern for the number of pixels included in the enclosure closed by the edge extracted from the image and the relative position where the edge is located in the image.

And, when it is determined that the learning data verification device 300 is not correct that the same tracking ID is given, or it is determined that it is incorrect that different tracking IDs are given, the tracking ID determined to be incorrect on the user interface (UI) is Information about the given object can be output.

In this way, the learning data verification apparatus 300 outputs a user interface (UI) in which crop images of a plurality of objects to which the same tracking ID is assigned are continuously arranged, and the result of self-inspection whether the tracking ID of the objects is correctly assigned. By outputting the included user interface (UI), the user of the learning data verification apparatus 300 can easily check and inspect a plurality of objects determined to be identical at a glance.

Meanwhile, the learning data verification apparatus 300 outputs a user interface (UI) capable of verifying the metadata of the annotation work result, a figure having a preset color corresponding to a value included in each metadata. ), each of the images included in the image group may be graphicized and included in the user interface (UI).

For example, the training data verification apparatus 300 may express each of the images included in the image group as one line having a color corresponding to a value included in the metadata. The learning data verification apparatus 300 may configure a bar-shaped graphic by continuously arranging each line segment in the width direction of the line segment according to the shooting sequence of the images. In addition, the learning data verification apparatus 300 may output the configured bar-shaped graphic by including it in the user interface (UI).

In this way, as the training data verification apparatus 300 outputs a user interface (UI) in which a plurality of images are simplified graphically according to the value of the metadata, the user of the training data verification apparatus 300 can display the metadata of the plurality of images. can be easily identified at a glance, and images with specific metadata values can be intuitively identified differently from other images.

In addition, the training data verification apparatus 300 may transmit the annotation work result and the inspection result received from the annotation apparatuses 200 to the training data generating apparatus 100 .

The training data verification apparatus 300, having the above-described characteristics, transmits/receives data to and from the annotation apparatus 200 and the training data generation apparatus 100, and any device capable of performing an operation based on the transmitted/received data. A device may be acceptable. For example, the training data verification apparatus 300 may be any one of a desktop, a workstation, or a stationary computing device such as a server, but is not limited thereto.

With the following configuration, the artificial intelligence learning device 400 is a device that can be used to develop artificial intelligence (AI).

Specifically, the artificial intelligence learning apparatus 400 transmits a request value including a requirement that the artificial intelligence (AI) learning data must satisfy to the learning data generating apparatus 100 in order for the artificial intelligence (AI) to achieve the development purpose. can be transmitted The artificial intelligence learning apparatus 400 may receive data for artificial intelligence (AI) learning from the learning data generating apparatus 100 . In addition, the artificial intelligence learning apparatus 400 may machine-lear an artificial intelligence (AI) to be developed by using the received artificial intelligence (AI) learning data.

As such, the artificial intelligence learning apparatus 400 may be any device as long as it is capable of transmitting and receiving data to and from the learning data generating apparatus 100 and performing an operation using the transmitted/received data. For example, the artificial intelligence learning apparatus 400 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

As described above, the training data generating device 100 , one or more annotation devices 200 , the training data verification device 300 , and the artificial intelligence learning device 400 are a secure line directly connecting the devices and a public wired communication network. Alternatively, data may be transmitted/received using a combined network of one or more of the mobile communication networks.

For example, public wired networks include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). However, it is not limited thereto. In addition, the mobile communication network includes Code Division Multiple Access (CDMA), Wideband CDMA, WCDMA, High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile communication may be included, but are not limited thereto.

Hereinafter, the configuration of the learning data generating apparatus 100 as described above will be described in more detail.

3 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.

As shown in FIG. 3 , the learning data generating apparatus 300 includes a communication unit 105 , an input/output unit 110 , a storage unit 115 , a data structure design unit 120 , a data collection and refinement unit 125 , and data It may be configured to include a processing unit 130 , a big data management unit 135 , and a learning data generation unit 140 .

As such, the components of the learning data generating apparatus 100 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is implemented with each other in the actual physical environment. It may be implemented separately.

When each component is described, the communication unit 105 may transmit/receive data to and from one or more of the annotation apparatus 200 , the learning data verification apparatus 300 , and the artificial intelligence learning apparatus 400 .

Specifically, the communication unit 105 may transmit a plurality of images to be annotated to the annotation apparatus 200 . The communication unit 105 may receive the annotation work result and the inspection result from the learning data verification apparatus 300 .

The communication unit 105 may receive a request value from the artificial intelligence learning apparatus 400 . Here, the required value is a value related to the requirements that the artificial intelligence (AI) learning data must satisfy in order for the artificial intelligence (AI) to achieve the development purpose. That is, the required value is a value indicating the type of information to be included in the artificial intelligence (AI) learning data. In addition, the communication unit 105 may transmit the artificial intelligence (A) learning data to the artificial intelligence learning apparatus 400 .

With the following configuration, the input/output unit 110 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 110 may receive a control signal for designing a data structure for artificial intelligence (AI) learning from a user. The input/output unit 110 may receive data (ie, images) for artificial intelligence (AI) learning from the outside. The input/output unit 110 may receive a quota for distributing annotation tasks to the plurality of annotation apparatuses 200 from a user.

The input/output unit 110 may output state information of big data managed by the big data management unit 135 . In addition, the input/output unit 110 may output a request value received from the artificial intelligence learning apparatus 400 through the communication unit 105 .

With the following configuration, the storage unit 115 may store data for generating data for artificial intelligence (AI) learning.

Specifically, the storage unit 115 may store a data structure designed by the data structure design unit 120 . The storage unit 115 may store data (ie, images) for artificial intelligence (AI) learning collected by the data collection and refinement unit 125 . The storage unit 115 may store an annotation work result received from the annotation apparatus 200 or the training data verification apparatus 300 .

In addition, the storage unit 115 may store information on big data that is generated and managed by the big data management unit 135 . The storage unit 115 according to another embodiment of the present invention may store information on big data generated and managed by the big data management unit 135 through an external storage device.

With the following configuration, the data structure design unit 120 may design a data structure for artificial intelligence (AI) learning.

Specifically, the data structure design unit 120 may design a data structure for artificial intelligence (AI) learning based on the user's control input through the input/output unit 110 . For example, the data structure design unit 120 may define an ontology for artificial intelligence (AI) learning and a classification system for data for artificial intelligence (AI) learning based on the user's control. In this case, the user of the learning data generating apparatus 100 may perform control so that the structure of data for artificial intelligence (AI) learning has versatility in consideration of the needs of main business partners or customers.

With the following configuration, the data collection and refinement unit 125 may collect and refine data for artificial intelligence (AI) learning based on the data structure designed by the data structure design unit 120 .

Specifically, the data collection and refinement unit 125 may receive an image from the outside through the input/output unit 110 based on the designed data structure. The data collection and refinement unit 125 may collect images by crawling the web through the communication unit 105 . Also, the data collection and refinement unit 125 may download an image from a device of an external organization that holds image data.

The data collection and refinement unit 125 may remove duplicate or extremely similar images from among the collected images. In addition, the data collection and refinement unit 125 may de-identify personal information included in the collected image.

With the following configuration, the data processing unit 130 may process the collected and refined data (ie, an image to be annotated) using the annotation apparatus 200 .

Specifically, the data processing unit 130 distributes the collected and refined data (ie, images to be annotated) to the plurality of annotation devices 200 based on the quota set through the input/output unit 110 . can be transmitted.

The data processing unit 130 may receive the annotation work result directly from the annotation apparatus 200 through the communication unit 105 , or receive the annotation work result and the inspection result from the learning data verification apparatus 300 .

With the following configuration, the big data management unit 135 may build and manage big data related to a plurality of annotation work results based on the metadata.

Specifically, the big data management unit 135 performs a plurality of annotation operations received from the annotation apparatus 200 or the learning data verification apparatus 300 through the data processing unit 130 based on metadata included in the result of the annotation operation. The results can be classified.

Here, the metadata included in the annotation work result may include image information, copyright information, shooting condition information, annotated object information, and environment information, but is not limited thereto. The image information included in the metadata may include a file name, a file size, and an image size. Copyright information may include information about image copyright holders. The shooting condition information may include resolution, bit value, aperture transmission amount, exposure time, ISO sensitivity, focal length, aperture opening value, angle of view, white balance, and RGB depth. The object information may include annotation type, coordinate value, area size, class name, tag item, truncation status, large classification, medium classification, small classification, and instance information. The environmental information may include shooting date and time, shooting location, weather information, road type, road surface information, and traffic jam information. Also, the number and type of information that may be included in the metadata is not limited thereto.

The big data management unit 135 may construct big data using a plurality of annotation work results classified based on metadata. The big data management unit 135 may include a plurality of results of annotation work received after the big data has been previously constructed in the previously constructed big data.

As such, the big data constructed by the big data management unit 135 may include a hash table. A hash table included in big data may be composed of a plurality of buckets according to the type of metadata. Each of the plurality of buckets constituting the hash table may be configured to include one or more slots. In addition, each slot included in the bucket may include a link to an annotation work result.

On the other hand, when the load density of the hash table exceeds a preset threshold density in the process of building big data, the big data management unit 135 maintains the link included in each slot while maintaining the metadata. By doubling the number of buckets according to the type of , only the home bucket of each slot is readjusted, the capacity of the hash table can be increased.

Alternatively, the big data constructed by the big data management unit 135 may be divided into a plurality of layers. In this case, the big data management unit 135 determines that the result of the annotation work generated by the annotation device 200 belongs to the job evaluation grade given to the user of the annotation device 200 in the process of building the big data. It is also possible to determine the layer of big data and add the corresponding annotation work result to the determined layer of big data.

With the following configuration, the learning data generation unit 140 may generate data for artificial intelligence (AI) learning by using the big data built by the big data management unit 135 .

Specifically, when a request value is input or received from the outside through the input/output unit 110 or the communication unit 105, the learning data generation unit 140 mines or filters the constructed big data to obtain the required value. It is possible to extract the result of the annotation work corresponding to .

Here, the required value is a value related to the requirements that the artificial intelligence (AI) learning data must satisfy in order for the artificial intelligence (AI) to achieve the development purpose. That is, the required value is a value indicating the type of information to be included in the artificial intelligence (AI) learning data.

For example, the learning data generating unit 140 may search for one or more slots by substituting a requested value into a hash function set in advance corresponding to a hash table of big data. The training data generator 140 may extract one or more annotation work results through links respectively included in one or more discovered slots.

Alternatively, the learning data generating unit 140 may extract an annotation work result from each layer of the big data composed of a plurality of layers according to a ratio set in advance in response to the request value.

In addition, the learning data generator 140 may generate data for artificial intelligence (AI) learning by packaging the extracted annotation work result. In addition, the learning data generating unit 140 may transmit the generated artificial intelligence (AI) learning data to the artificial intelligence learning apparatus 400 through the communication unit 105 .

On the other hand, when the type of information indicated by the request value is not included in the metadata of the annotation work result, the learning data generation unit 140 may include a type of information not included in the metadata in the metadata. The data structure may be redesigned through the data structure design unit 120 . In addition, the training data generator 140 may redistribute a plurality of images to the plurality of annotation apparatuses 200 through the data processing unit 130 to re-perform the annotation operation.

Hereinafter, the configuration of the annotation apparatus 200 as described above will be described in more detail.

4 is a logical configuration diagram of an annotation device according to an embodiment of the present invention.

As shown in FIG. 4 , the annotation device 200 includes a communication unit 205 , an input/output unit 210 , a storage unit 215 , an object specifying unit 220 , a metadata generation unit 225 , and a result generation unit ( 230) may be included.

As such, the components of the annotation apparatus 200 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is separated from each other in the actual physical environment. could be implemented.

When each component is described, the communication unit 205 may transmit/receive data to and from the learning data generating apparatus 100 and the learning data verifying apparatus 300 .

Specifically, the communication unit 205 may receive a plurality of images distributed by the learning data generating apparatus 100 . Here, the image is an image to be subjected to annotation work for artificial intelligence (AI) learning. Such a plurality of images may be received individually or collectively.

In addition, the communication unit 205 may transmit the annotation work result to the training data verification apparatus 300 . Such an annotation work result may have a JSON file format, but is not limited thereto.

With the following configuration, the input/output unit 210 may receive a signal from a user through a user interface (UI) or may output an operation result to the outside. Here, the user means a person who performs the annotation work. As such, a user may be referred to as an operator, performer, labeler, or data labeler, but is not limited thereto.

Specifically, the input/output unit 210 may output an image to be annotated. The input/output unit 110 may receive a control signal for setting a bounding box from a user. In addition, the input/output unit 110 may output the image by overlaying the bounding box on the image.

In addition, the input/output unit 110 may receive a control signal for setting metadata of an image from a user.

With the following configuration, the storage unit 215 may store data required for annotation work.

Specifically, the storage unit 215 may store the image received through the communication unit 205 . Also, the storage unit 215 may store the properties of the project, the properties of the image, or the properties of the user received through the communication unit 205 .

With the following configuration, the object specifying unit 220 may specify an object included in an image to be annotated.

Specifically, the object specifying unit 220 may load an image to be annotated into a memory. The object specifying unit 220 may select a tool according to a user's control signal input through the input/output unit 210 . Here, the tool is a tool for setting a bounding box that specifies one or more objects included in an image.

The object specifying unit 220 may receive coordinates through the selected tool according to a control signal input through the input/output unit 210 . The object specifying unit 220 may specify an object included in the image by setting a bounding box based on the input coordinates.

Here, the bounding box is an area for specifying an object to be subjected to artificial intelligence (AI) learning among objects included in the image. As such, the bounding box may have a rectangular or polygonal shape, but is not limited thereto.

For example, the object specifying unit 220 receives two coordinates from the user through the input/output unit 210, and has the input two coordinates as the coordinates of the upper left vertex and the coordinates of the lower right vertex in the image. By setting the bounding box based on the rectangle, you can specify the object included in the image.

In this case, the two coordinates may be set by the user inputting one type of input signal twice (eg, clicking the mouse), or setting the user by inputting two types of input signals once (eg, dragging the mouse). However, the present invention is not limited thereto.

With the following configuration, the metadata generator 225 may generate metadata for an image to be annotated.

Here, the metadata may include image information, copyright information, shooting condition information, annotated object information, and environment information, but is not limited thereto. The image information included in the metadata may include a file name, a file size, and an image size. Copyright information may include information about image copyright holders. The shooting condition information may include resolution, bit value, aperture transmission amount, exposure time, ISO sensitivity, focal length, aperture opening value, angle of view, white balance, and RGB depth. The object information may include annotation type, coordinate value, area size, class name, tag item, truncation status, large classification, medium classification, small classification, and instance information. The environmental information may include shooting date and time, shooting location, weather information, road type, road surface information, and traffic jam information. Also, the number and type of information that may be included in the metadata is not limited thereto.

The metadata generator 225 according to an embodiment of the present invention may set specific values to be included in the metadata according to a control signal input from the user. Without resorting to it, it is possible to automatically determine or assign the tracking ID and environment information of the object. In other words, the metadata generating unit 225 may generate metadata by including the coordinates in the image of the specified object, an automatically assigned tracking ID, and automatically determined environment information.

Specifically, the metadata generating unit 225 may automatically assign a tracking ID to the object specified by the object specifying unit 220 . Here, the tracking ID is a unique identifier of an object given to track an object from an image group. An image group is a set of annotated images. For example, an image group consists of images continuously captured in time series by the same camera as the camera that captured the image to be annotated, and other images placed adjacent to the camera that captured the image to be annotated. Images taken spatially and continuously by a camera may be included, and the present invention is not limited thereto.

The metadata generator 225 according to an embodiment of the present invention may assign a tracking ID based on the color of the object. Specifically, the metadata generator 225 may identify a pattern of RGB values for a region occupied by an object in the image. In addition, the metadata generating unit 225 may generate an identifier corresponding to the identified RGB value pattern and assign it as a tracking ID of the object specified by the object specifying unit 220 . Here, the RGB value pattern refers to a pattern for the ratio of the RGB values to the pixels and the arrangement order of the RGB values.

In this case, before generating a new identifier, the metadata generating unit 225 determines the RGB value pattern of the objects specified from the images included in the image group and the object specified by the object specifying unit 220 . A similar object can be searched for by comparing the identified RGB value patterns with each other. In addition, when a similar object is searched for from images included in the image group, the metadata generator 225 assigns the tracking ID assigned to the searched similar object as the tracking ID of the object specified by the object specifying unit 220 . can do.

The metadata generator 225 according to another embodiment of the present invention may assign a tracking ID based on the shape of the object. Specifically, the metadata generator 225 may extract an edge of the image (edge detection). The metadata generating unit 225 may identify only the edge of the object specified by the object specifying unit 220 from all edges of the extracted image. The metadata generating unit 225 may generate an identifier corresponding to the pattern of the edge of the identified object, and may provide it as a tracking ID of the object specified by the object specifying unit 220 . Here, the edge pattern means a pattern for the number of pixels included in the area closed by the edge extracted from the image and the relative position where the edge is located in the image.

In this case, the metadata generating unit 225 identifies an edge pattern for objects specified from images included in the image group and an object specified by the object specifying unit 220 before generating a new identifier. A similar object can be searched for by comparing the pattern of the edge of the object with each other. In addition, when a similar object is searched for from images included in the image group, the metadata generator 225 assigns the tracking ID assigned to the searched similar object as the tracking ID of the object specified by the object specifying unit 220 . can do.

The metadata generator 225 according to another embodiment of the present invention may assign a tracking ID based on a symbol or character included in an object. Specifically, the metadata generator 225 may identify whether a symbol or character is included in the area occupied by the object in the image. The metadata generating unit 225 targets objects specified from images included in the image group, and similar objects including the same symbols or characters as the symbols or characters identified from the object specified by the object specifying unit 220 . can be searched for. In addition, when a similar object is searched for from images included in the image group, the metadata generator 225 assigns the tracking ID assigned to the searched similar object as the tracking ID of the object specified by the object specifying unit 220 . can do.

Meanwhile, the metadata generator 225 may automatically determine environment information on an image to be annotated. Here, the environment information means information related to a place or time point at which an image was captured by the camera. Such environmental information may include information about the photographing date and time, photographing location, weather information, road type, road surface, and traffic congestion of the image, but is limited thereto. doesn't happen

The metadata generating unit 225 according to an embodiment of the present invention compares an existing image with an image to be annotated with each other, and compares the existing image with environment information given to the existing image to the environment of the image to be annotated. information can be applied.

More specifically, the metadata generator 225 compares the existing image with the image to be annotated with each other, and determines whether the environment information given to the existing image can be applied as the environment information of the image to be annotated. can determine whether Here, the existing image refers to an image taken before an image to be annotated by the same camera as the camera that photographed the image to be annotated.

If there are a plurality of images taken before the image to be annotated, the metadata generating unit 225 is given the same tracking ID as the object specified by the object specifying unit 220 among the plurality of images. You can select an image that contains an existing object as an existing image.

On the other hand, when there are a plurality of images taken prior to an image to be annotated, the metadata generator 225 sets the image to be annotated from among the plurality of images and the image closest to the shooting time to the existing image. You can also choose an image.

Thereafter, the metadata generating unit 225 may identify a background area in which the area occupied by the object to which the same tracking ID as the object of the existing image is assigned is removed from the image to be annotated. The metadata generator 225 may identify a background area in which the area occupied by the object given the same tracking ID as the object of the image to be annotated is removed from the selected existing image.

In addition, the metadata generating unit 225 converts the environment information previously given to the existing image to the object of the annotation based on the result of comparing the background area of the image to be annotated with the background area of the existing image. It can be determined whether the image is applicable to the environment information of the image.

That is, the metadata generating unit 225 generates an area occupied by an object to which the same ID is assigned with respect to each of the previously photographed image (ie, the existing image) and the subsequent photographed image (the image to be annotated). It is to compare the background regions from which . is removed, and based on the comparison result, it is determined whether the environment information given to the previously photographed image can be applied as the environmental information of the photographed image by trailing it.

As an example of the specific determination, the metadata generator 225 may generate an existing image based on a pattern of RGB values for a background region of an image to be annotated and a pattern of RGB values for a background region of an existing image. It may be determined whether the environment information previously given for the .

On the other hand, the metadata generating unit 225 generates a preset environment for an existing image based on an edge pattern for a background region of an image to be annotated and an edge pattern for a background region of an existing image. It may be determined whether the information is applicable as environmental information of an image to be annotated.

Alternatively, the metadata generator 225 calculates an average brightness value or an average gamma value for a background area of an image to be annotated, and an average brightness value or an average brightness value for a background area of an existing image. Based on the average gamma value, it may be determined whether environment information previously given for an existing image is applicable as environment information of an image to be annotated.

In addition, when it is determined that the environment information given to the existing image can be applied as the environment information of the image to be annotated, the metadata generator 225 performs an annotation operation on the environment information given to the existing image. Metadata can be created by including it as the environment information of the target image.

In summary, when a plurality of images correspond to images continuously photographed by the same camera, the metadata generating unit 225 follows the environment information provided for the previously photographed image among the plurality of images. Metadata for the captured image can be automatically generated by including it as the environment information of the image and trailing it.

On the other hand, when it is determined that the metadata generator 225 cannot apply the environment information previously given to the existing image as the environment information of the image to be annotated, one or more set in the image to be annotated. Environment information on an image to be annotated may be automatically determined based on the characteristics of the reference point.

More specifically, the metadata generator 225 may set one or more reference points in an image to be annotated. Many images used in machine learning have a relatively similar composition so that artificial intelligence (AI) can achieve a specific purpose. The reference point according to an embodiment of the present invention may be one or more points in the image set in response to the learning purpose of artificial intelligence (AI). In addition, the metadata generator 225 may determine environment information of an image to be annotated based on the color, brightness, or gamma of each of the one or more set reference points.

The metadata generator 225 according to an embodiment of the present invention may extract an edge of an image to be annotated, and divide the image into a plurality of regions based on the extracted edge. The metadata generator 225 may set one or more reference points for each of a plurality of divided regions. The metadata generator 225 may identify the background area by removing the area occupied by the object from the image to be annotated based on the extracted edge. The metadata generator 225 may determine the directionality of the object in the image based on a class assigned to the object. For example, when the class assigned to the object is a vehicle, the metadata generator 225 may determine the direction in which the wheels of the object are heading in the image as the ground. The metadata generator 225 may identify a ground area and an air area from among a plurality of areas included in the identified background area based on the determined directionality of the object. Here, the ground area is an area closest to the ground among a plurality of areas divided by an edge. The aerial area is the area furthest from the ground among a plurality of areas separated by an edge. In addition, the metadata generator 225 determines road surface information constituting the environment information based on the pixel of the reference point set in the ground area, and generates the environment information based on the pixel of the reference point set in the air area. It is possible to determine the weather information to configure.

The metadata generator 225 according to another embodiment of the present invention may set one or more reference points for each of four line segments constituting a border of an image to be annotated. The metadata generator 225 may determine the directionality of the object in the image based on a class assigned to the object. The metadata generator 225 may identify a ground line segment and an aerial segment from among the four line segments based on the determined directionality of the object. Here, the ground line segment is the line segment closest to the ground among the four line segments constituting the edge of the image. The aerial segment is the furthest segment from the ground among the four segments constituting the edge of the image. In addition, the metadata generator 225 determines road surface information constituting environmental information based on pixels of reference points set in the ground segment, and weather information composing environment information based on pixels at reference points set in the aerial segment. can be decided

Meanwhile, the metadata generating unit 225 sets a reference point at the same position for each of the image to be annotated and the existing image included in the image group, and the pixel of the reference point set in the image to be annotated and the existing image. In contrast to the pixels of the reference point set in the same way in the image, if the color, brightness, and gamma values of the pixels are within the preset allowable ranges, it is determined that the image to be annotated is the environment information preset for the existing image. may be If the difference between the color, brightness, and gamma value of the pixel of the reference point set to be the same in the image to be annotated and the existing image is out of the allowable range, the metadata generator 225 sets the environment information manually. A message indicating that it should be performed may be output through a user interface (UI).

With the following configuration, the result generating unit 230 may generate an annotation work result based on the metadata generated by the metadata generating unit 225 . In this case, the annotation work result may have a JSON file format, but is not limited thereto.

In addition, the result generating unit 230 may directly transmit the generated annotation work result to the learning data generating apparatus 100 through the communication unit 205 , or may transmit it to the learning data verifying apparatus 300 .

Hereinafter, hardware for implementing the logical components of the annotation apparatus 200 as described above will be described in more detail.

5 is a hardware configuration diagram of an annotation apparatus according to an embodiment of the present invention.

5, the annotation device 200 includes a processor 250, a memory 255, a transceiver 260, an input/output device 265, and a data bus. , 270) and may be configured to include a storage (Storage, 275).

The processor 250 may implement the operations and functions of the annotation device 200 based on an instruction according to the software 280a in which the method according to the embodiments of the present invention is implemented residing in the memory 255 . The memory 255 may be loaded with software 280a in which methods according to embodiments of the present invention are implemented. The transceiver 260 may transmit/receive data to and from the training data generating apparatus 100 and the training data verifying apparatus 300 . The input/output device 265 may receive data necessary for the operation of the annotation device 200 , and may output an image, a bounding box, and metadata to be annotated. The data bus 270 is connected to the processor 250 , the memory 255 , the transceiver 260 , the input/output device 265 , and the storage 275 . can play a role.

The storage 275 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 280a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 275 may store the software 280b in which the method according to the embodiments of the present invention is implemented. Also, the storage 275 may store information necessary for performing the method according to the embodiments of the present invention.

According to an embodiment of the present invention, the software (280a, 280b) for implementing the tracking ID grant and verification method resident in the memory 255 or stored in the storage 275 is the processor 250 artificial intelligence (AI) Specifying an object included in an image to be annotated for learning, the processor 250 assigning a tracking ID to the specified object, and the processor 250 the image of the specified object It may be a computer program recorded on a recording medium to execute the step of generating metadata for the image by including the coordinates and the given tracking ID.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the metadata generation and verification method resident in the memory 255 or stored in the storage 275 is the processor 250 artificial intelligence (AI) The step of specifying an object included in an image to be annotated for learning, the processor 250 compares the existing image with the image, and sets the environment information previously given for the existing image as the environment information of the image. determining whether or not it is applicable, and, when the environment information previously given for the existing image is applicable as the environment information of the image, by the processor 250, the coordinates in the image of the specified object and the existing image It may be a computer program recorded on a recording medium in order to execute the step of generating metadata for the image by including the given environment information.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the automatic generation method of metadata resident in the memory 255 or stored in the storage 275 is the processor 250 artificial intelligence (AI). specifying an object included in an image to be annotated for learning; the processor 250 sets one or more reference points in the image, and based on the color, brightness, or gamma of pixels of the set reference point In order to execute the steps of determining the environment information for the image, and the processor 250 generating the metadata for the image by including the coordinates in the image of the specified object and the determined environment information. It can be a written computer program.

More specifically, the processor 250 may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 255 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 260 may include a baseband circuit for processing wired and wireless signals. The input/output device 265 includes an input device such as a keyboard, a mouse, and/or a joystick, and a liquid crystal display (LCD), an organic light emitting diode (OLED) and/or an input device such as a joystick. Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiment included in this specification is implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. Modules reside in memory 255 and may be executed by processor 250 . The memory 255 may be internal or external to the processor 250 , and may be coupled to the processor 250 by various well-known means.

Each component shown in FIG. 5 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present invention is one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs ( Field Programmable Gate Arrays), a processor, a controller, a microcontroller, a microprocessor, and the like.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored in a recording medium readable through various computer means. can be recorded. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), and a floppy disk. magneto-optical media, such as a disk, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. Such hardware devices may be configured to operate as one or more software to perform the operations of the present invention, and vice versa.

Hereinafter, the features of the artificial intelligence learning system according to various embodiments of the present invention as described above will be described in detail with reference to the drawings.

6 is a signal flow diagram illustrating a method of generating data for artificial intelligence (AI) learning according to an embodiment of the present invention.

As shown in FIG. 6 , the apparatus 100 for generating learning data of an artificial intelligence learning system according to an embodiment of the present invention may design a data structure for artificial intelligence (AI) learning ( S105 ).

Specifically, the learning data generating apparatus 100 may define an ontology for artificial intelligence (AI) learning and a classification system of data for artificial intelligence (AI) learning based on the user's control. In this case, the user of the learning data generating apparatus 100 may perform control so that the structure of data for artificial intelligence (AI) learning has versatility in consideration of the needs of main business partners or customers.

The learning data generating apparatus 100 may collect and refine data for artificial intelligence (AI) learning based on the designed data structure (S110).

Specifically, the learning data generating apparatus 100 may receive an image from the outside, collect images by performing web crawling, or download an image from an external device holding image data.

In addition, the learning data generating apparatus 100 may remove duplicate or extremely similar images from among the collected images, and de-identify personal information included in the images.

The learning data generating apparatus 100 may distribute and transmit the collected and refined data (ie, an image to be annotated) to the plurality of annotation apparatuses 200 ( S115 ).

Each of the plurality of annotation devices 200 of the artificial intelligence learning system according to an embodiment of the present invention may specify an object included in an image to be annotated and received from the learning data generating device 100 ( S120).

Specifically, the annotation apparatus 200 may receive coordinates through a tool according to a user's control signal, set a bounding box based on the input coordinates, and specify an object included in the image. Here, the bounding box is an area for specifying an object to be subjected to artificial intelligence (AI) learning among objects included in the image.

The annotation apparatus 200 may generate metadata for an image to be annotated ( S125 ).

Characteristically, the annotation apparatus 200 according to an embodiment of the present invention may set specific values to be included in metadata according to a control signal input from a user, but the tracking ID and environment of the object without relying on the user's control signal Information can be automatically determined or given.

The annotation apparatus 200 may generate an annotation work result based on the generated metadata, and transmit the generated annotation work result to the learning data verification apparatus 300 ( S130 ).

The learning data verification apparatus 300 of the artificial intelligence learning system according to an embodiment of the present invention may inspect the results of the annotation work received from the annotation apparatuses 200 ( S135 ).

Specifically, the learning data verification apparatus 300 may perform the inspection using a script on the annotation work result. Here, the script is a code for verifying whether a preset target quality is met or whether data is valid with respect to the results of the annotation work.

Characteristically, the learning data verification apparatus 300 according to an embodiment of the present invention may provide a user interface (UI) that can more easily inspect the tracking ID and environment variables of the object included in the annotation work result. .

The training data verification apparatus 300 may transmit the annotation work result and the inspection result to the training data generating apparatus 100 ( S140 ).

The learning data generating apparatus 100 may build big data on a plurality of annotation work results based on the metadata of the annotation work results ( S145 ).

After the big data is constructed as described above, the learning data generating apparatus 100 may receive a data request including a requested value from the artificial intelligence learning apparatus 400 ( S150 ). Here, the required value is a value related to the requirements that the artificial intelligence (AI) learning data must satisfy in order for the artificial intelligence (AI) to achieve the development purpose. That is, the required value is a value indicating the type of information to be included in the artificial intelligence (AI) learning data.

The training data generating apparatus 100 may extract an annotation work result corresponding to a requested value by mining or filtering the constructed big data. The learning data generating apparatus 100 may generate data for artificial intelligence (AI) learning by packaging the extracted annotation work result ( S155 ).

Then, the learning data generating apparatus 100 may transmit the generated artificial intelligence (AI) learning data to the artificial intelligence learning apparatus 400 (Sa160).

The artificial intelligence learning apparatus 400 may use the artificial intelligence (AI) learning data received from the learning data generating apparatus 100 to machine-lear the artificial intelligence (AI) to be developed (S165).

7 is a conceptual diagram for explaining a concept of generating data for artificial intelligence (AI) learning according to an embodiment of the present invention.

According to conventional means, when data for artificial intelligence (AI) learning is requested from a customer who wants to learn artificial intelligence (AI), thereafter, data structure design, data collection, data purification, data processing, Data for data expansion and data validation were performed, and the results were packaged to generate data for artificial intelligence (AI) learning. That is, according to conventional means, after a request is received from a customer who wants to learn artificial intelligence (AI), it takes a relatively long time to generate and deliver artificial intelligence (AI) learning data to the customer.

As shown in Figure 7, the artificial intelligence learning system according to the embodiment of the present invention is the artificial intelligence (AI) learning data from the artificial intelligence learning device (400, that is, a customer who wants to learn artificial intelligence (AI)) Before the request is received, the big data 110 capable of preemptively generating data for artificial intelligence (AI) learning may be built in advance in the learning data generating apparatus 100 .

Then, the learning data generating apparatus 100 receives a request value according to the customer's needs from the artificial intelligence learning apparatus 400, it simply mines or filters the big data built in advance, and then the artificial intelligence (AI) learning data 410 ), and the generated artificial intelligence (AI) learning data 410 may be directly transmitted to the artificial intelligence learning apparatus 400 .

Accordingly, according to embodiments of the present invention, it is possible to greatly shorten the period from when a customer's request is received until the data for artificial intelligence (AI) learning is generated and delivered.

8 is an exemplary view for explaining the concept of a tracking ID (tracking identifier) according to an embodiment of the present invention. And, Figure 9 is an exemplary diagram for explaining a user interface (UI) for checking the tracking ID according to an embodiment of the present invention.

As shown in FIG. 8 , the annotation apparatus 200 according to an embodiment of the present invention provides a tracking ID for an object (20a, 20b, …) specified from an image (10a, 10b, …) to be annotated. can be given

Here, the tracking ID is a unique identifier of an object assigned to track an object from a set of annotated images (ie, an image group, 10a, 10b, ...). For example, the tracking ID includes images continuously shot in time-series by the same camera as the camera that shot the image to be annotated, or images that are placed adjacent to the camera that shot the image to be annotated. It may be an identifier assigned to track a specific object from images continuously taken spatially by another camera.

The annotation apparatus 200 according to embodiments of the present invention determines the object of the annotation operation based on the color of the object (pattern of RGB value), the shape of the object (pattern of edge), or symbols or characters included in the object. A tracking ID can be automatically assigned to each specified object (20a, 20b, ...) from the images (10a, 10b, ...).

Specifically, the annotation apparatus 200 may identify a pattern of RGB values for an area occupied by the object 20b in the image 10b to be annotated. The annotation apparatus 200 generates a pattern of RGB values for objects specified from images included in the image group and a pattern of RGB values for objects 20b specified from the image 10b to be annotated. The similar objects 20a may be searched for by comparing them with each other. And, the annotation apparatus 200 may give the tracking ID given to the found similar object 20a as the tracking ID of the object 20b specified from the image 10b to be annotated.

Alternatively, the annotation apparatus 200 may identify an edge pattern for the object 20b in the image 10b to be annotated. The annotation apparatus 200 compares the edge pattern for the objects specified from the images included in the image group and the edge pattern for the object 20b specified from the image 10b to be annotated with each other. Thus, the similar object 20a can be searched. And, the annotation apparatus 200 may give the tracking ID given to the found similar object 20a as the tracking ID of the object 20b specified from the image 10b to be annotated.

Alternatively, the annotation apparatus 200 may identify whether a symbol or character is included in the area occupied by the object 20b in the image 10b to be annotated. The annotation apparatus 200 may search for the similar object 20a including the same symbol or character as the identified symbol or character for the specified objects from the images included in the image group. And, the annotation apparatus 200 may give the tracking ID given to the found similar object 20a as the tracking ID of the object 20b specified from the image 10b to be annotated.

On the other hand, as shown in FIG. 9 , the learning data verification apparatus 300 according to an embodiment of the present invention may provide a user interface (UI) that can easily check whether the tracking ID of the object is correctly assigned. .

Specifically, the training data verification apparatus 300 may identify whether an object to which the same tracking ID is assigned from images included in the image group exists. The training data verification apparatus 300 may collect cropped images (10a-cropped, 10b-cropped, 10c-cropped, 10d-cropped) of the objects 20a, 20b, 20c, 20d to which the same tracking ID is given. have. Then, the learning data verification apparatus 300 sequentially arranges and outputs the cropped images 10a-cropped, 10b-cropped, 10c-cropped, and 10d-cropped collected on the user interface (UI) according to the shooting sequence of the image. can do.

Therefore, according to embodiments of the present invention, it is possible to provide a means for identifying the identity of an object for each image captured by a plurality of cameras, and to track an object that appears identically in a plurality of images do.

10 and 11 are exemplary views for explaining a process of determining environmental information in a continuous image according to some embodiments of the present invention.

As shown in FIGS. 10 and 11 , the annotation apparatus 200 according to an embodiment of the present invention may provide environment information for the continuous images (10-pre, 10-cur, ...).

Here, the environment information means information related to a place or time point at which an image was captured by the camera. For example, the environmental information may include information about the shooting date and time of the image, the shooting location, weather information, the type of road, the road surface, and the traffic congestion. Not limited.

The annotation apparatus 200 according to an embodiment of the present invention compares an existing image (10-pre) with an image to be annotated (10-cur) with each other, and assigns a pre-existing image (10-pre) to each other. The selected environment information can be applied as the environment information of the image (10-cur) to be annotated.

Specifically, the annotation device 200 removes the area occupied by the object (the object to which the same tracking ID as the object of the existing image is given) from the image (10-cur) that is the object of the annotation operation, and removes the background area 30b. can be identified. The annotation device 200 can identify the background area 30a from the existing image (10-pre) by removing the area occupied by the object (the object to which the same tracking ID as the object of the image to be annotated) is removed. have.

The annotation apparatus 200 compares the background area 30b of the image (10-cur) to be annotated with the background area (30a) of the existing image (10-pre) with each other based on the result of comparing the existing image ( The environment information given for 10-pre) can be applied as environment information of the image (10-cur) to be annotated.

Referring to FIG. 10 , the annotation apparatus 200 generates a pattern of RGB values for a background area 30b of an image 10-cur to be annotated. Based on the pattern of RGB values for the background area 30a of the existing image (10-pre), the environment information given to the existing image (10-pre) is annotated on the image (10-cur) It can be judged whether it is applicable with the environmental information of

Referring to FIG. 11 , the annotation apparatus 200 provides an edge pattern with respect to a background area 30b of an image 10-cur to be annotated. Based on the edge pattern for the background area 30a of the existing image 10-pre, the environment information previously given for the existing image 10-pre is applied to the image (10-cur) to be annotated. It can be determined whether it is applicable with environmental information.

12 and 13 are exemplary diagrams for explaining a process of automatically determining environment information according to some embodiments of the present invention.

As shown in FIGS. 12 and 13 , the annotation apparatus 200 according to an embodiment of the present invention applies environment information previously given to an existing image as environment information of an image 10 to be annotated. If this is not possible, based on the characteristics of one or more reference points (50-air, 50-ground) set in the image 10 that is the object of the annotation operation, environmental information about the image 10 that is the object of the annotation operation can decide

Many images used in machine learning have a relatively similar composition so that artificial intelligence (AI) can achieve a specific purpose. Accordingly, the reference points 50-air and 50-ground according to an embodiment of the present invention may be one or more points set in response to the learning purpose of artificial intelligence (AI).

Referring to FIG. 12 , the annotation apparatus 200 may extract an edge of an image 10 to be annotated, and divide the image 10 into a plurality of regions based on the extracted edge. The annotation apparatus 200 may set one or more reference points for each of a plurality of divided regions.

The annotation apparatus 200 may determine the direction of the object 20 in the image 10 based on the class assigned to the object 20 . For example, when the class assigned to the object 20 is a vehicle, the direction in which the wheels of the object 20 are heading in the image 10 may be determined as the ground.

The annotation apparatus 200 may identify the ground area 30-ground and the air area 30-air from among the plurality of areas based on the determined directionality of the object 20 . Here, the ground area 30-ground is an area closest to the ground among a plurality of areas divided by an edge. The air area 30-air is an area farthest from the ground among a plurality of areas separated by an edge.

Then, the annotation apparatus 200 determines road surface information constituting the environment information based on the pixels of the 50-ground set in the ground area 30-ground, and the aerial area 30-air ) may be determined based on the pixel of the reference point 50-air set within the weather information constituting the environment information.

Referring to FIG. 13 , the annotation apparatus 200 may set one or more reference points for each of four line segments constituting an edge of an image 10 to be annotated. The annotation apparatus 200 may determine the direction of the object 20 in the image 10 based on the class assigned to the object 20 .

The annotation apparatus 200 may identify a ground line segment 40-ground and an aerial segment 40-air from among the four line segments based on the determined directionality of the object 20 . Here, the ground line segment 40-ground is the line segment closest to the ground among the four line segments constituting the edge of the image 10 . The aerial line segment 40-air is the line segment furthest from the ground among the four line segments constituting the edge of the image 10 .

The annotation apparatus 200 determines the road surface information constituting the environment information based on the pixels of the 50-ground set within the ground line segment 40-ground, and determines the reference point (40-air) set within the aerial segment 40-air. 50-air), it is possible to determine the weather information constituting the environment information based on the pixels.

14 is an exemplary diagram for explaining a user interface (UI) for reviewing metadata according to an embodiment of the present invention.

As shown in FIG. 14 , the training data verification apparatus 300 according to an embodiment of the present invention may provide a user interface (UI) for easily inspecting metadata of an annotation work result.

Specifically, the learning data verification apparatus 300 includes the images 10-a, 10-b, 10-c, and 10-d, 10-e, 10-f, ...) may be graphicized and included in the user interface (UI).

For example, the training data verification apparatus 300 may convert each of the images 10-a, 10-b, 10-c, 10-d, 10-e, 10-f, ...) included in the image group as metadata. It can be expressed as one line segment having a color corresponding to the value included in . The learning data verification apparatus 300 may configure the graphic 60 in the form of a bar by continuously arranging each line segment in the width direction of the line segment according to the shooting sequence of the images. In addition, the learning data verification apparatus 300 may output the configured bar-shaped graphic 60 by including it in the user interface (UI).

Accordingly, according to an embodiment of the present invention, metadata of a plurality of images can be easily checked at a glance, and an image having a specific metadata value different from other images can be intuitively identified.

15 is a flowchart illustrating an operation process of an annotation apparatus according to an embodiment of the present invention.

Referring to FIG. 15 , the annotation apparatus 200 may receive an image to be annotated from the training data generating apparatus 100 ( S210 ).

The annotation apparatus 200 may specify an object included in an image to be annotated ( S220 ). Specifically, the annotation apparatus 200 selects a tool according to a user's control signal, receives coordinates through the selected tool, and sets a bounding box based on the input coordinates to specify an object included in the image. .

The annotation apparatus 200 may automatically give a tracking ID to the specified object (S230). The process of automatically assigning the tracking ID by the annotation device 200 is the same as that described with reference to FIG. 8 , and thus the description thereof will not be repeated.

The annotation apparatus 200 may automatically determine environment information on an image to be annotated ( S240 ). The process of automatically determining the environment information by the annotation apparatus 200 is the same as that described with reference to FIGS. 10 to 13 , and thus the description thereof will not be repeated.

The annotation apparatus 200 may generate metadata by including the coordinates in the image of the specified object, an automatically assigned tracking ID, and automatically determined environment information (S250).

In addition, the annotation apparatus 200 may generate an annotation work result based on the generated metadata, and transmit the generated annotation work result to the learning data verification apparatus 300 .

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention pertains that other modifications based on the technical idea of the present invention can be implemented in addition to the embodiments disclosed herein. It is obvious to those with ordinary knowledge. In addition, although specific terms have been used in the present specification and drawings, these are only used in a general sense to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Training data generator: 100 Annotator: 200
Training data verification device: 300 Artificial intelligence training device: 400
Communication unit: 105 Input/output unit: 110
Storage: 115 Data structure design: 120
Data collection and refinement unit: 125 Data processing unit: 130
Big data management unit: 135 Learning data generation unit: 140
Communication unit: 205 Input/output unit: 210
Storage: 215 Object Specifier: 220
Metadata generator: 225 Result generator: 230

Claims (10)

distributing, by the learning data generating apparatus, a plurality of images to be subjected to an annotation operation;
A plurality of annotation devices specify coordinates of an object included in each image with respect to the plurality of distributed images, and coordinates in the image of the specified object and metadata for the image generating an annotation work result by including the generated metadata; and
Receiving, by the learning data generating apparatus, the plurality of annotation work results, classifying the plurality of annotation work results based on the metadata, and generating artificial intelligence (AI) learning data. and
The step of generating the annotation work result is
When the plurality of images correspond to images successively photographed by the same camera, environmental information previously given for an image photographed previously among the plurality of images is included as environmental information of the image photographed as the trailing image. to create metadata for the captured image,
When there are a plurality of previously photographed images, the environment information of the photographed image follows the environment information previously given for an image including an object to which the same tracking identifier is assigned among the plurality of images. include,
The tracking ID is a method for generating data for artificial intelligence learning, characterized in that it is a unique identifier of an object given to track the object. The method of claim 1, wherein the generating of the artificial intelligence (AI) learning data comprises:
Big data is constructed using the results of the annotation work classified based on the metadata, and when a request value is input from the outside, the big data is mined or filtered to be added to the request value. Extracting a corresponding annotation work result, and packaging the extracted annotation work result to generate the artificial intelligence (AI) learning data,
The request value is a value indicating a type of information to be included in the data for artificial intelligence (AI) learning, characterized in that it is a data generation method for artificial intelligence (AI) learning. The method of claim 2, wherein the big data is
and a hash table consisting of a plurality of buckets according to the type of metadata, and each of the buckets includes one or more slots including a link to an annotation work result. is composed by
The step of generating the data for artificial intelligence (AI) learning is
Searching for one or more slots by substituting the request value into a hash function set in advance corresponding to the hash table, and extracting one or more annotation work results through a link included in the searched one or more slots A method of generating data for artificial intelligence learning. The method of claim 3, wherein the generating of the artificial intelligence (AI) learning data comprises:
When the loading density of the hash table exceeds a preset threshold density in the process of building the big data, the number of buckets according to the type of metadata while maintaining the link included in the slot A method for generating data for artificial intelligence learning, characterized in that the capacity of the hash table is increased by re-adjusting the home bucket of the slot after doubling the . The method of claim 2, wherein the generating of the artificial intelligence (AI) learning data comprises:
When the type of information indicated by the request value is not included in the metadata, the data structure is redesigned to include the type of information not included in the metadata, and then the plurality of A method for generating data for artificial intelligence learning, characterized in that redistribution of the plurality of images to an annotation device. The method of claim 2, wherein the generating of the artificial intelligence (AI) learning data comprises:
The big data is divided into a plurality of layers, and in the process of constructing the big data, a layer to which the annotation work result belongs is determined based on the job evaluation grade given to the user of the annotation device, and the external The method for generating data for artificial intelligence learning, characterized in that extracting the annotation work result from each layer according to a ratio corresponding to the requested value input from the . The method of claim 1, wherein the generating of the annotation work result comprises:
Based on the result of comparing the background area occupied by the object to which the same ID is given for each of the previously photographed image and the subsequently photographed image with each other, a predetermined environment for the previously photographed image A method of generating data for artificial intelligence learning, characterized in that it is determined whether the information can be applied as environmental information of the photographed image by trailing the information. memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
distributing, by the processor, a plurality of images to be annotated through the transceiver;
The processor specifies the coordinates of an object included in each image with respect to the plurality of distributed images received through the transceiver, and coordinates in the image of the specified object and metadata for the image generating (metadata) and generating an annotation work result by including the generated metadata; and
For executing, by the processor, generating data for artificial intelligence (AI) learning by classifying the plurality of annotation work results based on the metadata,
The step of generating the annotation work result is
When the plurality of images correspond to images successively photographed by the same camera, environment information previously given for an image photographed previously among the plurality of images is included as environmental information of the image photographed after the trailing image. to create metadata for the captured image,
The step of generating the annotation work result is
When there are a plurality of previously photographed images, the environment information of the photographed image follows the environment information previously given for an image including an object to which the same tracking identifier is assigned among the plurality of images. characterized by including,
The tracking ID is a computer program recorded on a recording medium, characterized in that it is a unique identifier of the object given to be able to track the object. The method of claim 8, wherein the generating of the artificial intelligence (AI) learning data comprises:
Big data is constructed using the results of the annotation work classified based on the metadata, and when a request value is input from the outside, the big data is mined or filtered to be added to the request value. Extracting a corresponding annotation work result, and packaging the extracted annotation work result to generate the artificial intelligence (AI) learning data,
The request value is a computer program recorded on a recording medium, characterized in that the value indicates a type of information to be included in the artificial intelligence (AI) learning data. The method of claim 9, wherein the big data is
and a hash table consisting of a plurality of buckets according to the type of metadata, and each of the buckets includes one or more slots including a link to an annotation work result. is composed by
The step of generating the data for artificial intelligence (AI) learning is
Searching for one or more slots by substituting the request value into a hash function set in advance corresponding to the hash table, and extracting one or more annotation work results through a link included in the searched one or more slots A computer program recorded on a recording medium.

Images