KR102187741B1 - Metadata crowd sourcing system and method - Google Patents

Abstract

A metadata crowdsourcing system and method for constructing high quality metadata are disclosed. A metadata crowdsourcing system according to an embodiment includes: a video metadata tagging unit that extracts images for each scene section from video data, and extracts and stores metadata for each image; Generates a problem with the image and metadata stored in the video metadata tagging unit, receives an answer to the generated problem from the user, calculates the user's reliability and accuracy of the problem type, and accumulates and manages the accuracy of the metadata. And a crowdsourcing processor for updating metadata accordingly; A service interface providing unit for presenting a problem generated by the crowdsourcing processing unit to a plurality of unspecified users and receiving an answer to the problem; And a metadata learning model optimization unit updating and optimizing metadata stored in the video metadata tagging unit according to the metadata accuracy.

Description

Metadata crowd sourcing system and method}

The present invention relates to a metadata crowdsourcing system and method for constructing high quality metadata.

With the development of deep learning technology, research and services applying deep learning are in progress in various fields. In particular, the image recognition field using deep learning is greatly improved compared to the performance before deep learning, and is continuously developing.

However, metadata information automatically extracted by applying a publicly disclosed deep learning model is difficult to utilize without modification due to errors such as incorrect tagging and missing information. Therefore, it is common to manually create high-quality metadata by humans, but this requires a lot of manpower, time, and cost.

On the other hand, crowdsourcing is a method of devising ideas and solving problems that individuals, groups, or companies need through the participation of an unspecified number of people. Crowdsourcing has been used in various academic, industrial, social, and economic fields, and among them, reCaptcha v1, reCaptcha v2, reCaptcha v3, and Amazon Mechanical Turk are representative metadata tagging tools using crowdsourcing.

However, the metadata tagging technology using such crowdsourcing still has various problems, such as problems that humans cannot recognize or copyright problems.

According to an embodiment, a metadata crowdsourcing system and method for converting automatically tagged metadata into high-quality metadata using collective intelligence are proposed.

A metadata crowdsourcing system according to an embodiment includes: a video metadata tagging unit that extracts images for each scene section from video data, and extracts and stores metadata for each image; Generates a problem with the image and metadata stored in the video metadata tagging unit, receives an answer to the generated problem from the user, calculates the user's reliability and accuracy of the problem type, and accumulates and manages the accuracy of the metadata. And a crowdsourcing processor for updating metadata accordingly; A service interface providing unit for presenting a problem generated by the crowdsourcing processing unit to a plurality of unspecified users and receiving an answer to the problem; And a metadata learning model optimization unit updating and optimizing metadata stored in the video metadata tagging unit according to the metadata accuracy.

The video metadata tagging unit includes media management including registration and deletion of the video data, and extraction of video sections for each scene, and metadata including object recognition, background recognition, face recognition, and caption generation from the extracted images for each scene section. Extraction can be performed.

The service interface providing unit may be implemented as a Java script to present a problem and receive an answer through a pop-up window, and a design of a user interface may be provided as a cascading style sheet.

User reliability is determined based on a selection degree of metadata having a high reliability among user input values for the presented problem, and the problem type accuracy may be determined based on a selection degree of metadata having a low reliability among user input values.

The metadata accuracy may be calculated by reflecting the number of previously accumulated correct answers and the number of correct answers to the corresponding question in the user reliability and the problem type accuracy.

In addition, the learning model optimization unit may optimize the metadata by applying transfer learning or active learning.

Meanwhile, a metadata crowdsourcing method according to another embodiment includes: a video metadata tagging step of extracting an image for each scene section from video data, extracting metadata for each image, and storing and managing the metadata; Generates a problem with the stored image and metadata, receives the answer to the generated problem from the user, calculates the user's reliability and accuracy of the problem type, and accumulates and manages the accuracy of the metadata through this, and stores the metadata accordingly. A crowdsourcing process step of updating; And a metadata learning model optimization step of updating and optimizing the stored metadata according to the metadata accuracy.

The video metadata tagging step includes media management including registration and deletion of the video data, and video section extraction for each scene, and metadata including object recognition, background recognition, face recognition, and caption generation in the extracted scene section images. You can perform the extraction.

User reliability is determined based on a selection degree of metadata having a high reliability among user input values for the presented problem, and the problem type accuracy may be determined based on a selection degree of metadata having a low reliability among user input values.

The metadata accuracy may be calculated by reflecting the number of previously accumulated correct answers and the number of correct answers to the corresponding question in the user reliability and the problem type accuracy.

In addition, in the learning model optimization step, the metadata may be optimized by applying transfer learning or active learning.

According to the present invention, it is possible to improve the problem of problems that humans cannot recognize of the conventional system and to construct a comprehensive set of correct answers. In addition, by automating all processes, difficult preparation procedures can be omitted and the quality of newly created data can be managed. Furthermore, it is possible to gradually improve the performance of an existing deep learning model without human intervention by using newly created data.

In addition, it is possible to improve the quality of metadata that is automatically generated through the process of optimizing learning models based on transfer learning and active learning by utilizing high-quality metadata generated by user participation.

1 is a configuration diagram of a metadata crowdsourcing system according to an embodiment of the present invention;
2 is a view for explaining the operation of a video metadata tagging unit according to an embodiment of the present invention;
3 is a diagram illustrating an example of a video metadata tagging result;
4 is a diagram illustrating an example of a pop-up screen generated by a service interface providing unit according to an embodiment of the present invention;
5 is a diagram for explaining a processing procedure in a crowd sourcing processing unit according to an embodiment of the present invention;
6 is a diagram for explaining a detailed configuration and operation of a metadata learning model optimization unit according to an embodiment of the present invention, and
7 is a flowchart of a metadata crowdsourcing method according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to those who have it, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same components throughout the specification.

In describing embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted, and terms to be described later are in the embodiments of the present invention. These terms are defined in consideration of the function of the user and may vary according to the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout this specification.

Combinations of each block of the attached block diagram and each step of the flowchart may be executed by computer program instructions (execution engine), and these computer program instructions are executed on a processor of a general purpose computer, special purpose computer or other programmable data processing equipment. As it may be mounted, its instructions executed by the processor of a computer or other programmable data processing equipment generate means for performing the functions described in each block of the block diagram or each step of the flowchart.

These computer program instructions may also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular manner, so that the computer-usable or computer-readable memory It is also possible to produce an article of manufacture containing instruction means for performing the functions described in each block of the block diagram or each step of the flow chart.

In addition, since computer program instructions can be mounted on a computer or other programmable data processing equipment, a series of operation steps are performed on a computer or other programmable data processing equipment to create a computer-executable process. It is also possible for the instructions to perform the data processing equipment to provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

In addition, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing specified logical functions, and in some alternative embodiments mentioned in the blocks or steps. It should be noted that it is also possible for functions to occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, and the blocks or steps may be performed in the reverse order of a corresponding function as necessary.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention exemplified below may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. Embodiments of the present invention are provided to more completely describe the present invention to those of ordinary skill in the art.

1 is a block diagram of a metadata crowdsourcing system according to an embodiment of the present invention.

The metadata crowdsourcing system includes a video metadata tagging unit 110, a crowdsourcing processing unit 120, a service interface providing unit 130, and a metadata learning model optimization unit 140.

The video metadata tagging unit 110 extracts images for each scene section from video data, extracts metadata for each image, and stores and manages them. To this end, media management including registration and deletion of video data and extraction of video sections in units of scenes, and metadata including object recognition, background recognition, face recognition, and caption generation are performed from the extracted scene section images.

The crowdsourcing processing unit 120 generates a problem with the image and metadata stored in the video metadata tagging unit 110, receives an answer to the generated problem from the user, and calculates the reliability of the user and the accuracy of the problem type, Through this, the accuracy of the metadata is accumulated and managed, and the metadata is updated accordingly. Here, the user reliability is determined based on a selection degree of metadata having a high reliability among user input values for a presented problem, and the problem type accuracy may be determined based on a selection degree of metadata having a low reliability among user input values. In addition, the metadata accuracy may be calculated by reflecting the number of previously accumulated correct answers and the number of correct answers to the corresponding question in the user reliability and problem type accuracy.

The service interface providing unit 130 presents the problem generated by the crowdsourcing processing unit 120 to an unspecified number of users and receives an answer to the problem. To this end, it is implemented in Java script, so that problems can be presented and answers can be input through a pop-up window, and the design of the user interface can be provided with a cascading style sheet.

The metadata learning model optimization unit 140 updates and optimizes the metadata stored in the video metadata tagging unit 110 according to the metadata accuracy. For example, it is possible to optimize metadata by applying transfer learning or active learning. Hereinafter, each component will be described in detail with reference to FIGS. 2 to 7.

2 is a diagram illustrating an operation of a video metadata tagging unit according to an embodiment of the present invention.

The video metadata tagging unit 110 includes a media management function including registration and deletion of a video, extraction of a scene section, and a metadata extraction function including object recognition, background recognition, face recognition, and caption generation by extracting an image from a video video. Perform. In this case, for the generation of video metadata, for example, a deep learning-based object/background/face recognition technology may be used, and a single image-based caption generation technology may be used.

To explain the operation process in more detail, when a user registers a video through a web-based user interface, the video metadata tagging server 210 calls the scene section extraction server 220 to obtain an image at 10 fps from the video. And the extracted images sequentially, for example, a faster RCNN-based object recognition server learned with the COCO data set, a CNN-based background recognition server learned with the MIT Places database, and a CNN-based face recognition learned with the FACENET data set. The server is sent to the metadata extraction server 230 consisting of an Attention LSTM-based caption generation server that combines the CNN and RNN learned with the COCO data set, extracts the metadata, and stores it in the database 240. The extracted information is visually displayed.

3 is a diagram illustrating an example of a video metadata tagging result.

Referring to FIG. 3, it can be seen that a list is displayed for each scene section, and the tagging result in the representative image for each section is divided into persons, objects, places, etc. and displayed.

4 is a diagram illustrating an example of a pop-up screen generated by a service interface providing unit according to an embodiment of the present invention.

Referring to FIG. 4, it is an example of a pop-up screen to which a JS+CSS package is applied. The metadata crowdsourcing system for constructing high-quality metadata according to an embodiment of the present invention requires a user interface that enables collective intelligence to be realized. Therefore, it can be configured based on web clients that can be used on various platforms such as mobile, PC, and tablet. The system of the present invention is configured in a form that can be easily applied to any web page, for example, Java Script (JS) in charge of pop-up type user interface and Cascading Style Sheets (CSS) in charge of user interface design. It can be composed of two files, through which it appears at the time the user wants to be exposed and receives the user's input. The user input is transmitted to the crowdsourcing server to measure user reliability, and the metadata accumulated from a large number of users with high reliability is used in the learning model optimization unit to continuously learn and develop metadata automatic tagging performance. have.

5 is a view for explaining a processing procedure in a crowd sourcing processing unit according to an embodiment of the present invention.

The crowdsourcing processing unit may include a question questionnaire, a correct answer confirmation unit, and a database (not shown). With reference to FIG. 8, a description will be given of the overall flow until a problem is presented to the user, data input from the user is processed, and the result is reflected in metadata.

First, when a user accesses the service and requests a problem (510), the problem questionnaire presents a problem to the user (520). In the questionnaire, the detailed question type is determined, the question is written, and the correct answer set is written. Among the automatically extracted metadata, data with low reliability are randomly extracted, and a problem type is determined based on the extracted data. When the problem type is determined, only the data appropriate to the problem type is collected from random data to create a problem to be displayed to the user, and items with high reliability among the automatically extracted metadata are included in the problem, so that the correct answer check unit will trust the user. Determine if you can.

After receiving the correct answer from the user (530), the correct answer checking unit performs the correct answer processing (540). For example, it determines how reliable an input value for a problem presented to a user is and updates the value of the database. The correct answer verification unit performs user reliability analysis, user input analysis, database management and update in detail.

The user reliability analysis is determined by how accurately the high reliability item was selected among the user input values of the question prepared and presented in the question questionnaire. For example, when a user is presented with 2 low-confidence grape pictures, 3 high-confidence grape pictures, and 4 high-confidence apple pictures and asked to select a picture containing grapes, whether to select a grape picture of low confidence. Regardless of, it is checked whether the user has selected all three pictures of grapes with high reliability, or whether the pictures of grapes are omitted and an apple picture is selected, and is added to the user's existing reliability to determine user reliability. As an example, the formula for calculating the user reliability is as follows.

User confidence = (required correct answers-incorrect answers)/(required correct answers)

User input analysis estimates how accurately an item with low reliability among the values entered by the user is selected based on user reliability. The user's input with high reliability can be estimated to be the actual correct answer with a high probability, but incorrect input may come in for various reasons, such as a lack of knowledge of the user and a situation where the problem is not accurately understood, which adversely affects the reliability of the actual metadata information. Can cause. An equation for estimating the accuracy of metadata with low reliability is as follows.

Question type accuracy = (number of correct answers-number of incorrect answers)/(total number of question types entered by the user)

Metadata accuracy = (number of existing correct answers + number of correct answers to the corresponding question)/(number of total questions) X question type accuracy X user reliability

As a result of the correct answer processing, it is determined whether or not the answer is correct (550). If it is not correct, the question is asked again to the user, and if it is correct, the user reliability score is updated and the correct answer to the question is updated (560). That is, the user reliability and metadata accuracy calculated as above are accumulated and stored in the user DB and the metadata DB, respectively. Then, metadata with high reliability is extracted and reflected in the metadata DB (570). For example, even though the current accuracy of the metadata is high, the automatically extracted metadata is not replaced unless various values of users with high reliability are accumulated. Data created by collective intelligence can be used when the state is as reliable as possible.

6 is a diagram for describing a detailed configuration and operation of a metadata learning model optimization unit according to an embodiment of the present invention.

After constructing various high-quality metadata using collective intelligence, it is necessary to regenerate the model using newly constructed data. However, building a model from scratch takes a lot of time and reduces user convenience and system utilization. Therefore, it is necessary to use an existing model without creating a model from the beginning, and to perform these processes automatically. The metadata learning model optimization unit 140 is configured to automatically perform these tasks in connection with the video metadata tagging unit 110 and the crowdsourcing processing unit 120, and specifically, the transfer learning unit 610 and active learning are performed. It consists of a part 620.

The transfer learning unit 610 extracts similar features from the object or object with the correct answer and estimates it as a specific object or object when there is no correct answer label for the object or object. Automatically tagged metadata can have very low accuracy, and even if a crowdsourcing system is used, if there are too many objects with low accuracy, there is a limit to the amount that can be processed. Accordingly, the transfer learning unit 610 extracts features of items with low accuracy among the automatically tagged metadata, and determines which objects or objects of the previously learned model have similar features. After comparing features in a pre-trained model, items with high similarity are determined as objects or objects, and the automatically extracted metadata information is updated to provide an accurate answer when a request for a similar object or thing comes in the future. .

The active learning execution unit 620 requests the correct answer from Oracle for an object or thing whose correct answer is ambiguous, so that the correct answer is used for learning data. The transfer learning unit 620 compares the previously learned data with the characteristics of groups whose correct answer is ambiguous, but if there is no object or object having similar characteristics, the correct answer is used for groups with ambiguous but similar characteristics using active learning. Ask to write a label. Here, the correct answer label is requested from Oracle. In the crowdsourcing system for constructing high-quality metadata of the present invention, the oracle becomes an entity that inputs data in the crowdsourcing system, that is, a person. Although the correct answer is ambiguous, the groups with similar characteristics update the automatically extracted metadata information when the correct answer is written by human hand, and it is used as learning data to improve the performance of automatic metadata extraction.

7 is a flowchart of a metadata crowdsourcing method according to an embodiment of the present invention.

The metadata crowdsourcing method of the present invention includes a video metadata tagging step 710, a crowdsourcing processing step 720, and a metadata learning model optimization step 730.

In the video metadata tagging step 710, an image is extracted for each scene section from video data, and metadata for each image is extracted and stored and managed. More specifically, media management including registration and deletion of video data and extraction of video sections in units of scenes, and extraction of metadata including object recognition, background recognition, face recognition, and caption generation from the extracted scene section images Perform.

The crowdsourcing processing step 720 creates a problem with the image and metadata stored in this way, receives an answer to the generated problem from the user, and calculates the user's reliability and accuracy of the problem type, thereby determining the accuracy of the metadata. It is cumulatively managed and metadata is updated accordingly. Here, the user reliability is determined based on a selection degree of metadata having a high reliability among user input values for a presented problem, and the problem type accuracy may be determined based on a selection degree of metadata having a low reliability among user input values. In addition, the metadata accuracy may be calculated by reflecting the number of previously accumulated correct answers and the number of correct answers to the corresponding question in the user reliability and problem type accuracy.

In the metadata learning model optimization step 730, the stored metadata is updated and optimized according to the calculated metadata accuracy. For example, it is possible to optimize metadata by applying transfer learning or active learning, and a specific method is as described above with reference to FIG. 6.

So far, the present invention has been looked at around the embodiments. Those of ordinary skill in the art to which the present invention pertains will be able to understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

110 Video metadata tagging unit 120 Crowdsourcing processing unit
130 Service interface provision unit 140 Metadata learning model optimization unit

Claims (12)

A video metadata tagging unit that extracts images for each scene section from video data, and extracts and stores metadata for each image;
Generates a problem with the image and metadata stored in the video metadata tagging unit, receives an answer to the generated problem from the user, calculates the user's reliability and accuracy of the problem type, and accumulates and manages the accuracy of the metadata. And a crowd sourcing processing unit for updating metadata accordingly;
A service interface providing unit for presenting the problem generated by the crowdsourcing processing unit to a plurality of unspecified users and receiving an answer to the problem; And
Metadata crowdsourcing system comprising a metadata learning model optimization unit for updating and optimizing metadata stored in the video metadata tagging unit according to the metadata accuracy. The method of claim 1, wherein the video metadata tagging unit,
Media management including registration and deletion of the video data and extraction of video sections for each scene, and metadata for extracting metadata including object recognition, background recognition, face recognition, and caption generation from the extracted scene section images Crowdsourcing system. The method of claim 1, wherein the service interface providing unit,
Metadata crowdsourcing system implemented in JavaScript to present problems and input answers through pop-up windows. The method of claim 3, wherein the service interface providing unit,
Metadata crowdsourcing system in which the design of the user interface is provided in Cascading Style Sheets. The method of claim 1,
The user reliability is determined based on the selection degree of metadata with high reliability among user input values for the presented problem, and the problem type accuracy is metadata determined based on the selection degree of metadata with low reliability among user input values. Crowdsourcing system. The method of claim 1,
The metadata accuracy is calculated by reflecting the number of previously accumulated correct answers and the number of correct answers to the corresponding question in the user reliability and the problem type accuracy. The method of claim 1, wherein the learning model optimization unit,
Metadata crowdsourcing system for optimizing the metadata by applying transfer learning or active learning. A video metadata tagging step of extracting images for each scene section from the video data, and extracting and storing metadata for each image;
Generates a problem with the stored image and metadata, receives the answer to the generated problem from the user, calculates the user's reliability and accuracy of the problem type, and accumulates and manages the accuracy of metadata through this Updating crowdsourcing processing step; And
And a metadata learning model optimization step of updating and optimizing the stored metadata according to the metadata accuracy. The method of claim 8, wherein the video metadata tagging step,
Media management including registration and deletion of the video data and extraction of video sections for each scene, and metadata for extracting metadata including object recognition, background recognition, face recognition, and caption generation from the extracted scene section images How to crowdsource. The method of claim 8,
The user reliability is determined based on a selection degree of metadata having a high reliability among user input values for the generated problem, and the problem type accuracy is determined based on a selection degree of metadata having a low reliability among user input values. Data crowdsourcing method. The method of claim 8,
The metadata accuracy is calculated by reflecting the number of previously accumulated correct answers and the number of correct answers to the corresponding question in the user reliability and the problem type accuracy. The method of claim 8, wherein the step of optimizing the learning model comprises:
Metadata crowdsourcing method for optimizing the metadata by applying transfer learning or active learning.

Images