KR102148392B1 - Video metadata tagging system and method thereof - Google Patents

Abstract

Disclosed are a semi-automatic video metadata tagging system and method for constructing integrated metadata. A video metadata tagging system according to an embodiment includes an integrated metadata storage unit storing at least one metadata for a plurality of video data, and a machine learning algorithm for image data stored in the integrated metadata storage unit. Is applied to recognize a person included in the image data, recognize the emotion of the person, and recognize the behavior of the person, perform object, background, and sound recognition within the image data, and generate sentence information, which is explanatory information about the image data. Separation of video sections to select representative images for each section by extracting the scene change information for the section where the scene is changed from the video data read from the metadata learning unit and the video content management unit, and the frame image for each section divided based on the scene change By referring to the learning result of the sub, and the metadata learning unit, and applying a predetermined machine learning algorithm, it recognizes the person, the emotion of the person, and the behavior of the person in the selected representative image, and the object, background, and It may include a metadata extracting unit that recognizes sound, generates sentence information describing the representative image, and stores it as metadata in the integrated metadata storage unit.

Description

Video metadata tagging system and method thereof

The present invention relates to a semi-automatic video metadata tagging system and method for constructing integrated metadata.

With the widespread use of mobile media, not only existing video contents such as movies, dramas, and documentaries, but also various types of video contents such as Internet broadcasting and video contents created by single creators are increasing significantly.

As video contents increase, many studies are being conducted to understand, summarize and analyze the contents of a large amount of video contents. In order to analyze such a large number of video contents more efficiently, deep learning technology has recently attracted attention. In order to effectively and successfully apply deep learning technology, it is essential to create and utilize various types of high-quality and large-capacity metadata.

As a related art, Korean Patent Publication No. 2015-0079064,'Automatic Tagging System' discloses a technique for extracting only visual and physical information and semantic information of a still image, and Korean Patent Publication No. 2011-0020158 In the'metadata tagging system, image search method, device and gesture tagging method applied thereto', a technique for extracting visual information and place information by analyzing an image is disclosed. However, this prior art is limited to tagging visual information in images, and does not guarantee the quality of metadata. In addition, it is not possible to create integrated metadata that has all of the visual information, sound information, caption information, and caption information for one image, and it is expensive and difficult to work for tagging a large amount of data.

According to an embodiment, a video metadata tagging system and method for generating integrated metadata more conveniently by using a deep learning-based recognition technique for automatic metadata extraction are proposed.

A video metadata tagging system according to an embodiment includes an integrated metadata storage unit storing at least one metadata for a plurality of video data, and a machine learning algorithm for image data stored in the integrated metadata storage unit. Is applied to recognize a person included in the image data, recognize a person's emotions, and recognize a person's behavior, perform object, background, and sound recognition in the image data, and generate sentence information, which is explanatory information about the image data. Separation of video sections to select representative images for each section by extracting the scene change information for the section where the scene is changed from the video data read from the metadata learning unit and the video content management unit, and the frame image for each section divided based on the scene change By referring to the learning result of the sub and the metadata learning unit, and applying a predetermined machine learning algorithm, the person, the emotion of the person, and the behavior of the person are recognized in the selected representative image, and the object, background, and It may include a metadata extracting unit that recognizes sound, generates sentence information describing the representative image, and stores it as metadata in the integrated metadata storage unit.

In addition, the metadata verification unit may further include a metadata verification unit that verifies and corrects whether the metadata obtained by the metadata extraction unit matches the actual metadata, and provides a user interface for verifying the metadata.

The video section separating unit divides the video data read from the video content management unit in units of frames, and compares the similarity of pixel values between the previous frame and the current frame to determine that the scene has been switched when it is greater than or equal to a predetermined threshold.

Meanwhile, in the video metadata tagging method according to another embodiment, recognition of a person included in image data constituting the video data by applying a predetermined machine learning algorithm to a plurality of video data, recognition of a person's emotion, and a person's behavior Recognizing the object, background, and sound in the image data, generating sentence information, which is explanatory information about the image data, and storing it as metadata, in the section where the scene is switched from the input video data. Selecting a representative image for each section by extracting the scene change information for each section and the frame image for each section divided based on the scene change, and applying a predetermined machine learning algorithm, for the person and the person in the selected representative image. Recognizing emotions and actions of a person, recognizing an object, a background, and a sound in the representative image, generating sentence information describing the representative image, and making it metadata.

Through the video metadata tagging system according to an embodiment, integrated metadata tagged with visual information, sound information, subtitle information, and caption information for a single image through the step of automatically extracting and verifying the metadata. It can be conveniently produced in large quantities. In addition, each metadata can be used independently and applied to various application fields, and it can satisfy the requirements of researchers and companies who want to create metadata sets directly and fields that require metadata of images specialized in the application field. There is an effect.

1 is a configuration diagram of a video metadata tagging system according to an embodiment of the present invention;
2 is a diagram showing an example of a video database managed by a video content management unit;
3A and 3B are diagrams showing an example of extracting an image from an image section separation unit and selecting a representative image;
4 is a diagram illustrating an example for explaining a process of recognizing a face in a metadata extraction unit;
5 is a diagram for explaining an example of emotion recognition using Emotion API in a metadata extraction unit;
6 is a diagram for explaining an example of object recognition using Faster R-CNN in a metadata extraction unit;
7 is a view for explaining an example of tagging and verifying time information in the metadata verification unit;
8 is a diagram for explaining an example of reproduction of a metadata tagging result reproduced in a video content management unit;
9 is a flowchart of a video metadata tagging method according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the technical field to which the present invention pertains. It is provided to completely inform the scope of the invention to those who have it, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same components throughout the specification.

In describing embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted, and terms to be described later are in the embodiments of the present invention. These terms are defined in consideration of the function of the user and may vary according to the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout this specification.

Combinations of each block of the attached block diagram and each step of the flowchart may be executed by computer program instructions (execution engine), and these computer program instructions are executed on a processor of a general purpose computer, special purpose computer or other programmable data processing equipment. As it can be mounted, its instructions executed by a processor of a computer or other programmable data processing equipment generate means for performing the functions described in each block of the block diagram or each step of the flowchart.

These computer program instructions may also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular manner, so that the computer-usable or computer-readable memory It is also possible to produce an article of manufacture containing instruction means for performing the functions described in each block of the block diagram or each step of the flow chart.

In addition, since computer program instructions can be mounted on a computer or other programmable data processing equipment, a series of operation steps are performed on a computer or other programmable data processing equipment to create a computer-executable process. It is also possible for the instructions to perform the data processing equipment to provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

In addition, each block or each step may represent a module, segment, or part of code containing one or more executable instructions for executing specified logical functions, and in some alternative embodiments mentioned in the blocks or steps. It should be noted that it is also possible for functions to occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, and the blocks or steps may be performed in the reverse order of a corresponding function as necessary.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention exemplified below may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. Embodiments of the present invention are provided to more completely describe the present invention to those of ordinary skill in the art.

1 is a block diagram of a video metadata tagging system according to an embodiment of the present invention.

The video metadata tagging system includes a video content management unit 110, an integrated metadata storage unit 120, a metadata learning unit 130, a video section separation unit 140, and a metadata extraction unit 150. A data verification unit 160 may be further included.

The integrated metadata storage unit 120 stores at least one metadata for a plurality of video data. The metadata learning unit 130 applies various machine learning algorithms to the image data stored in the integrated metadata storage unit 120 to recognize a person included in the image data, recognize the person's emotion, and recognize the person's behavior. , Performs object, background, and sound recognition in image data, and generates sentence information (caption information) that is explanatory information about image data. The video section separating unit 140 extracts the scene change information for the section in which the scene is changed from the video data read from the video content management unit 110 and the frame image for each section divided based on the scene change, and is a representative image for each section. Select The metadata extraction unit 150 refers to the learning result of the metadata learning unit 130 and applies several self-learning algorithms to recognize a person, an emotion for a person, and a person's behavior from the selected representative image. Objects, backgrounds, and sounds in the image are recognized, and sentence information describing a representative image is generated and stored in the integrated metadata storage unit 120 using this as metadata.

The metadata verification unit 160 provides a user interface for verifying and correcting whether the metadata obtained by the metadata extracting unit 150 matches the actual metadata to be verified.

2 is a diagram illustrating an example of a video database managed by a video content management unit.

The video content management unit 110 stores the video from which metadata is to be extracted in the video storage unit 170 or catalogs it, and displays it to the user by deleting or playing it according to the user's selection. And you can download the JSON file corresponding to the generated metadata.

More specifically, the video metadata tagging system according to an embodiment of the present invention manages the video through the video content management unit 110. The video content management unit performs the following functions, for example. First, the function to upload and delete the video for tagging to the system, second, the function to play through the player, the third, the function to list the uploaded videos, and the fourth, the JSON file when metadata is created It includes a function to download and finally, a tagging result playback function that plays the tagging result through the player. The image and the JSON metadata are stored and viewed in the video storage unit 170, and thus the video section separation unit 140 may extract a scene change section and a representative image for the image of the video storage unit 170.

The video section separating unit 140 divides the video data read from the video content management unit 110 into a frame unit, compares the similarity of the pixel values between the previous frame and the current frame, and determines that the scene has been switched when it exceeds a certain threshold. In this case, the similarity of pixel values can be obtained, for example, by calculating Sum of Absolute Difference (SAD). Hereinafter, the image section separation unit 140 will be described in more detail.

In order to generate metadata from an image, metadata tagging must be performed by dividing the image into each frame unit. The video section separation unit 140 of the metadata tagging system according to an embodiment of the present invention extracts the scene change section information and the frame image of the image stored in the video storage unit 170 to provide the scene information and image storage unit 180. And select a representative image among the extracted frame images.

3A and 3B are diagrams illustrating an example of extracting an image from an image section separating unit and selecting a representative image.

Various methods can be used to extract the section in which the scene is switched from the image. For example, an open source library, FFMPEG, can be used. The scene change detection method of FFMPEG extracts frames above a threshold value using SAD (Sum of Absolute Difference), which measures similarity through a pixel difference from a previous frame. SAD is close to zero when they are similar. In the present system, as an example, a threshold value of 0.25 is set to extract a scene change section, and one shot image of each section is extracted per 10 frames. For example, three representative images are selected and stored from the entire image of the extracted shot section. Depending on the type of video, the threshold value, the number of images per frame, and the criteria for selecting a representative image can be set differently. The stored representative image proceeds to image recognition through the metadata extraction unit 150.

The metadata extraction unit 150 automatically generates initial metadata first in order to facilitate metadata tagging and increase accuracy. The generated initial metadata is stored in the integrated metadata storage unit 120. The metadata extractor 150 may perform character recognition, emotion recognition, behavior recognition, object recognition, background recognition, sound recognition, and caption generation, and may apply data error detection technology and transfer learning to increase the accuracy of recognition. .

4 is a diagram illustrating an example for explaining a process of recognizing a face in a metadata extracting unit.

More specifically, the metadata extractor 150 finds a face part from the selected representative image, extracts a feature point, and then recognizes a face by applying a deep learning technique using a model based on a convolutional neural network (CNN), for example, Save it to the data storage.

In other words, the bounding box of the person's name and face coordinates in the image is automatically extracted. For face recognition, deep learning can be used to extract face parts with high accuracy. Deep learning-based face recognition can use OpenFace, an open source library, and Microsoft's Face API. A representative face recognition library finds face parts (eyes, nose, mouth) in an image, extracts facial feature points, and recognizes faces by quantifying them into vectors using a CNN-based model. A bounding box of the recognized face name and face coordinates is drawn on the representative image, which is stored in the integrated metadata storage unit 120 and the stored metadata is transmitted to the metadata verification unit 160.

5 is a diagram for explaining an example of emotion recognition using an Emotion API in a metadata extraction unit.

The metadata extracting unit 150 may recognize emotions by applying a deep learning technique to the recognized person and store them in the integrated metadata storage unit 120. Examples of deep learning-based emotion recognition include Microsoft's Emotion API and Affecitva's Emotion SDK. In emotion recognition, emotions can be classified into Anger, Disgust, Fear, Happiness, Sadness, Surprise, Neutral, and even Contempt. have. These emotions are expressed from 0 to 1, and each emotion is numerically recognized. The recognized information is passed in JSON format, and the numerical value is expressed in the face bounding box in the image. The image in which the emotion recognition has been performed is stored in the integrated metadata storage unit 120 in JSON format and transmitted to the metadata verification unit 160.

In addition, the metadata extractor 150 may recognize a behavior by applying a deep learning technique using a body part for a recognized person and a skeleton expressing connection information between the parts.

In other words, in behavioral recognition, an action means a posture or a specific motion. To recognize such an action, a recognition technique that uses a skeleton that expresses the position of a body part and the connection information between the parts is used as an example. I can. OpenPose, DensePose, etc. are methods of using deep learning to recognize a person's behavior in an image.

It is also possible to extract the face, fingers, and joints of a person in the image, recognize the action, and express it in two-dimensional coordinates, and furthermore, all pixels of the person in the RGB image can be mapped in 3D. The extracted coordinates are stored in the integrated metadata storage unit 120 in JSON format and transmitted to the metadata verification unit 160.

6 is a diagram for explaining an example of object recognition using Faster R-CNN in a metadata extraction unit.

The metadata extractor 150 may find an object in the representative image by applying a deep learning technique using a model based on a convolutional neural network (CNN). In other words, the location coordinate bounding box of the object in the image and the object name are automatically extracted. The bounding box refers to a rectangular box drawn vertically and horizontally at a position determined as an object in the image. There are various methods of using deep learning in object recognition. In particular, Faster R-CNN (Region Proposal CNN), GBD-Net (Gated bi-directional CNN) based on CNN (Convolutional Neural Network) that classifies and recognizes objects using feature maps extracted from the features of the input image. YOLO (You Only Look Once) is representative and can be used in this object recognition.

The metadata extractor 150 may find the background in the representative image by applying a deep learning technique using a CNN (Convolutional Neural Network)-based model.

More specifically, background items are classified for the background recognition of images, and are generally trained using AlexNet, GoogleNet, and VGG16 based on CNN (Convolutional Neural Network). A representative model could be MIT's Places CNN.

Background information candidate metadata with the highest probability from the representative image using deep learning-based background recognition technology is stored in the integrated metadata storage unit 120 in JSON format and transmitted to the metadata verification unit 160.

The metadata extracting unit 150 may recognize the type of sound, voice, and speaker included in the section. First, the type of sound recognition recognizes even non-verbal sounds. The type of sound can be specified differently according to the video content. For example, in the sound recognition contest DCASE (Detection and Classification of Acoustic Scenes and Events), recognition is divided into Speech, Human Laughter, Keyboard, Door knock, Coughing, etc. To proceed, this method can also be used in this embodiment.

Second, the principle of STT (Speech To Text) technology for speech recognition is to identify words or sentences from speech waveforms, extract meanings, and convert them into characters. For speech recognition, for example, voice conversation systems such as AI speakers and AI assistants, which are widely used in recent years, can be used, and representative open APIs are Google's Cloud Speech API, Naver's Clova Speech API, and Kakao's Newtone. I can.

Third, speaker recognition is generally divided into speaker verification technology and speaker identification technology. Speaker verification technology is a technology that verifies whether a speaker and a voice match, and speaker identification technology is a technology that identifies a speaker whose input voice matches the most among previously registered voices. In this embodiment, a speaker is recognized by classifying a person in an image using speaker identification technology. Speaker recognition classifies by extracting features of speech. For speaker recognition, learn and proceed with the speaker in advance for each video content. Open APIs for speaker recognition include Microsoft's Speaker Recognition and Amazon's Transcribe, and these APIs can be used.

In this way, it is possible to recognize the type of sound, voice, and speaker in the video, and automatically generate metadata for the sound. The generated sound metadata is stored in the integrated metadata storage unit 120 in JSON format, and transmitted to the metadata verification unit 160. When sound recognition is performed smoothly, subtitles (captions) in an image can be extracted in real time.

The metadata extractor 150 combines the face recognition result and the object recognition result to extract the features of the face and the object, and applies the RNN (Recurrent Neural Network) model to generate a natural language sentence, thereby describing the representative image. Can be created.

That is, by generating a caption (sentence) describing the representative image of the video, the video can be summarized, and it becomes easy to search for the desired video content.

In order to generate a caption, a model based on a recurrent neural network (RNN) among deep learning models may be used as an example. In order to create an image caption, the object recognition function and the face recognition function are previously combined. Using the object recognition function and the face recognition function, the features of the object and the face in the image are extracted, and natural language sentences are generated as input of the RNN model. You can also use the attention technique to understand the association of words in the output sentence. In the attention technique, caption generation can generate captions based on a single image or multiple images. Captions from multiple images can be extracted from features for multiple consecutive images by using 3D CNN. Based on the extracted features, a caption is generated.

The caption generation result value is stored in the integrated metadata storage unit 120 along with the image name in JSON format, and transmitted to the metadata verification unit 160.

As the accuracy of the metadata extraction unit 150 increases, the possibility of editing the metadata in the metadata verification unit 160 decreases, so that a working speed is improved and high quality metadata can be obtained more easily.

The basic metadata of the person, emotion, behavior, object, and background extracted by the metadata extraction unit 150 is transmitted to the metadata verification unit 160.

7 is a diagram for explaining an example of tagging and verifying visual information in a metadata verification unit.

The metadata verification unit 160 divides the metadata automatically extracted from the metadata extraction unit 150 and stored in the integrated metadata storage unit 120 into time, caption, sound, and caption information tagging, and Verification is performed. To this end, the metadata verification unit 160 provides a user interface for verification and is verified by interacting with a user through this.

The metadata verification unit 160 provides a user interface for visual information tagging verification, caption information tagging verification, and sound and subtitle information tagging verification, and performs verification through user manipulation through this. In the visual information tagging verification and the caption information tagging verification, the user sees the representative image in which the basic metadata is generated, and the user verifies whether the corresponding metadata is correct, thereby generating the final metadata.

8 is a diagram for explaining an example of reproduction of a metadata tagging result reproduced in a video content management unit.

In verifying the tagging result of visual information, visual information means person information (name, face coordinates, whole body coordinates, behavior, emotion), object information (object name, object position coordinates), and background information, which are visual elements in the image. do. In the case of visual information, items may be set differently according to application examples. The initial JSON result generated through the metadata extraction unit 150 and stored in the integrated metadata storage unit 120 is transmitted to the metadata verification unit 160 to generate the final metadata through a verification step.

First, among the person information, the name and face coordinates are generated through the metadata extracting unit 150 and received metadata stored in the integrated metadata storage unit 120, and the name and the location of the bounding box are corrected in the verification step of the user. When this is necessary, it can be edited using the editing function, and when a new bounding box is needed, it can be created.

The person's behavior information receives the metadata label through the metadata extraction unit 150, and in the verification step, the user determines whether the person's behavior in the image matches the label of the metadata. If there is a discrepancy, the user directly corrects it and creates the final metadata. In the case of emotion information, final metadata is generated through the same steps as behavior information.

For tagging of person information, high quality metadata can be created only when the criteria of each item are clarified. In the verification step of the person information tagging, the general person information tagging criteria are as follows, and may be set differently according to application examples.

① Tagging only the front and side faces

② Tagging only faces larger than a certain size

③ You can tag multiple times to mark emotions within one section.

④ Excluding the back of the person's body

For object information tagging verification, the object name and the bounding box position of the object information are also generated through object detection by the metadata extraction unit 150, and the metadata stored in the integrated metadata storage unit 120 is delivered, and the operator performs the verification step. Editing function is used when modification and creation are needed. In the case of objects, since metadata has the most items that can be extracted from one image, there is a difference in cost depending on the performance of the recognition unit. Since the higher the performance of object recognition, the more metadata automatically extracted, the time required for modification and creation in the verification step of object information tagging can be reduced. In addition, objects that have not passed through the verification step may be recognized as negative data, resulting in a problem of degrading the performance of object recognition.

For background information tagging verification, background information is generated through background recognition and stored in the integrated metadata storage unit 120 to receive the metadata label, and in the verification step, the operator determines whether the background of the image and the label of the metadata match. do. If there is a discrepancy, the user directly corrects it and creates the final metadata. In the verification stage, when two places appear on one screen, the item is determined by the exposure ratio.

Background information items may also vary depending on the object to be applied, but they can be used in common because the number is significantly lower than that of objects.

In the caption information tagging verification, metadata generated through caption recognition and stored in the integrated metadata storage unit 120 is transmitted, and in the verification step, the user determines whether the image status and the caption match. In case of disagreement, the user manually corrects the caption and creates the final metadata. Depending on caption recognition, the length of the caption may be generated differently. Therefore, in caption information tagging verification, additional two captions are generated. Accordingly, three captions are generated for caption information in the final metadata.

In the sound and subtitle information tagging verification, tagging is performed using an image rather than an image. Sound information is generated through sound recognition to receive the type of sound and speaker label, which are basic metadata stored in the integrated metadata storage unit 120, and the caption information receives metadata captions generated through speech recognition. In the verification step, the user determines whether the sound, speaker, and subtitle of the video match the label of the metadata. If there is a discrepancy, the user directly corrects it and creates the final metadata.

In addition, the higher the performance of sound recognition, the lower the cost required in the correction process, so that high quality data becomes important.

The finally generated metadata tagging result data will be described as follows.

The visual information result value, that is, the tagged metadata result is extracted as JSON. The extracted JSON file is stored in the integrated metadata storage unit 120 and transmitted to the video content management unit 110 through the video storage unit 170.

An example of the JSON structure of visual information is as follows.

"visual_results":
[
{
"image": "image1.jpg",
"period_num": 1,
"period_frame_num": 1,
"start_time": "03:14",
"end_time": "03:18",
"place": "home",
"persons":
{
"MIke":
{
"face_rect":
{
"x": 0,
"y": 0,
"w": 522,
"h": 383
},
"full_rect":
{
"x": 0,
"y": 0,
"w": 522,
"h": 383
},
"behavior": "stand up",
"emotion": "happy"
}

}

]

The caption information result value is an example of the JSON structure of the caption information as follows.

"caption_results":
[
{
"image": "image1.jpg",
"period_num": 1,
"start_time": "03:14",
"end_time": "04:18",
"caption_info":
[
{
"caption1": "he is happy"
},
{
"caption1": "he is nice"
},
{
"caption1": "he is healthy"
}
]
}
]

An example of a JSON structure of sound and subtitle information result values, that is, sound and subtitle information, is as follows.

"sound_script_results":
{
"start_time": "03:14",
"end_time": "03:18",
"sound_type": "speaking",
"script": "Hello Everyone"
}

The video content management unit 110 can play the metadata tagging result, and can play the generated metadata information together with the video. As the result value can be visualized, it is possible to check at a glance whether the corresponding metadata is correct.

9 is a flowchart of a video metadata tagging method according to an embodiment of the present invention.

First, pre-learned metadata is stored (210), that is, by applying a machine learning algorithm to a plurality of video data, recognition of a person included in the image data constituting the video data, recognition of a person's emotion, and recognition of a person's behavior. It performs object, background, and sound recognition in the image data, and generates text information, which is explanatory information about the image data, and stores it as metadata. The machine learning method and the detailed method of storing the metadata are as described in the video metadata tagging system.

Next, video data is input (220). Then, from the input video data, scene change information for a section in which the scene is changed and a frame image for each section divided based on the scene change are extracted, and a representative image for each section is selected (230).

By applying a machine learning algorithm, it recognizes the person, emotions, and behavior of the person in the selected representative image, recognizes the object, background, and sound in the representative image, and generates sentence information describing the representative image. Make it into metadata (240). Here, the machine learning algorithm can recognize faces, emotions, behaviors, objects, backgrounds, and sounds by applying deep learning techniques including, for example, a convolutional neural network (CNN)-based model or a recurrent neural network (RNN) model. have. Details of the selection of the representative image and generation of metadata are the same as described through the image segment separation unit 140 and the metadata extraction unit 150 described above.

It checks whether the created metadata matches the actual metadata, corrects it, and verifies it (250). The detailed method of verifying the metadata is also the same as described through the above-described metadata verification unit 160.

An image can be extracted from the image of the metadata tagging system according to an embodiment of the present invention, and integrated metadata for one image can be conveniently generated in large quantities by tagging visual information, caption information, sound information, and subtitle information. And integrated verification is also convenient.

As seen above, the work speed when generating the integrated metadata through a series of processes of this metadata tagging system will be improved, the time will be shortened, and the amount will be increased. In addition, the accuracy is improved through several verification steps after automatic extraction.

Furthermore, transfer learning technology can be applied to improve the object recognition performance of visual information. Transfer learning is a method of applying a model made in one domain to another domain, and is applied to improve performance when an existing model is used in a new domain. It is expensive to train the model again in a new domain, and it can be used to prevent the performance of the model from deteriorating in the new domain even though the training data is used a lot. In addition, it is possible to lower the cost of metadata tagging through active learning. Active learning is a method used for learning by estimating the label of unlabeled data or asking the user to input a label when unlabeled data and labeled data are mixed. If such transfer learning and active learning are used together with a metadata tagging system, the cost incurred when creating metadata can be lowered and the quality of metadata can be further improved.

Whereas existing systems were able to extract only image-based visual information metadata and use it only in a specific domain, the metadata tagging system of the present invention can analyze and summarize images by generating high-quality integrated metadata from images. It can be applied to various services using video content such as video search system, caption system, video sound analysis system, and broadcast content.

So far, the present invention has been looked at around the embodiments. Those of ordinary skill in the art to which the present invention pertains will be able to understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

110 Video content management unit 120 Integrated metadata storage unit
130 Metadata learning unit 140 Video section separation unit
150 Metadata extraction unit 160 Metadata verification unit
170 Video storage unit 180 Scene information and image storage unit

Claims (15)

An integrated metadata storage unit that stores at least one metadata for a plurality of video data;
A video section separating unit for selecting a representative image for each section by extracting scene change information for a section in which a scene is changed from the video data read from the video content management unit and a frame image for each section divided based on the scene change; And
By applying a predetermined machine learning algorithm, the selected representative image recognizes the person, the person's feelings, and the person's behavior, recognizes the object and background in the representative image, and the sound included in the video data for each section And a metadata extracting unit for recognizing and storing sentence information describing the representative image as metadata and storing it in the integrated metadata storage unit. The method of claim 1,
And a metadata verification unit providing a user interface for checking and correcting whether the metadata obtained by the metadata extracting unit matches actual metadata, and verifying the metadata. The method of claim 1, wherein the video content management unit
A video metadata tagging system, characterized in that the video from which the metadata is to be extracted is stored in a video storage unit or listed, and deleted or played according to a user's selection, and displayed to the user. The method of claim 1, wherein the video content management unit
A video metadata tagging system, characterized in that downloading a JSON file corresponding to the generated metadata. The method of claim 1, wherein the image section separation unit
A video metadata tagging system, characterized in that the video data read from the video content management unit is classified in units of frames, and the similarity of the pixel values between the previous frame and the current frame is compared to determine that the scene has been switched when it exceeds a predetermined threshold. . The method of claim 5,
A video metadata tagging system, characterized in that the similarity of the pixel values is obtained by calculating SAD (Sum of Absolute Difference). The method of claim 1, wherein the metadata extracting unit
A video meta, characterized in that after finding a face part from the selected representative image, extracting a feature point, applying a deep learning technique using a CNN (Convolutional Neural Network)-based model, recognizing a face, and storing it in an integrated metadata storage unit. Data tagging system. The method of claim 1, wherein the metadata extracting unit,
A video metadata tagging system, characterized in that, by applying a deep learning technique to the recognized person, emotions are recognized and stored in an integrated metadata storage unit. The method of claim 1, wherein the metadata extracting unit,
A video metadata tagging system, characterized in that the motion is recognized by applying a deep learning technique using a body part of the recognized person and a skeleton expressing connection information between the parts. The method of claim 1, wherein the metadata extracting unit
A video metadata tagging system, characterized in that the object and background in the representative image are found by applying a deep learning technique using a model based on a convolutional neural network (CNN). The method of claim 1, wherein the metadata extracting unit,
A video metadata tagging system, characterized in that recognizing a type of sound, a voice, and a speaker included in the section. The method of claim 1, wherein the metadata extracting unit,
A video, characterized in that sentence information describing the representative image is generated by combining the face recognition result and the object recognition result to extract the features of the face and the object, and then applying the RNN (Recurrent Neural Network) model to generate a natural language sentence. Metadata tagging system. Storing at least one metadata for a plurality of video data;
Selecting a representative image for each section by extracting scene change information for a section in which a scene is changed from the input video data and a frame image for each section divided based on the scene change; And
By applying a predetermined machine learning algorithm, the selected representative image recognizes the person, the person's feelings, and the person's behavior, recognizes the object and background in the representative image, and the sound included in the video data for each section Recognizing and generating sentence information describing the representative image and making it into metadata. The method of claim 13,
And verifying whether the generated metadata matches actual metadata, and verifying the video metadata tagging method. The method of claim 13, wherein the step of making metadata
A video metadata tagging method comprising recognizing faces, emotions, behaviors, objects, backgrounds, and sounds by applying a deep learning technique including a convolutional neural network (CNN)-based model or a recurrent neural network (RNN) model.

Images