KR20210051473A - Apparatus and method for recognizing video contents - Google Patents

Abstract

Disclosed is a video content identification method, which comprises the steps of: extracting a frame from an input video; detecting a face region from the extracted frame; detecting a clothing region from a frame in which the face region is detected; comparing the detected clothing region with one or more candidate images stored in a database using information on a face recognized based on the detected face region; and identifying the input video according to the comparison result. According to the present invention, it is possible to identify video content with the improved facial recognition accuracy.

Description

Video content identification device and method {APPARATUS AND METHOD FOR RECOGNIZING VIDEO CONTENTS}

The present invention relates to a video content identification apparatus and method, and more particularly, to a video content identification apparatus and method using face recognition and clothing area detection.

In the conventional method of identifying an image, the hash value of the original image or feature information of the audio or video extracted from the original signal called the feature point is stored in the DB, and the hash value or the feature point is extracted from the input image that requires identification, and the DB A method of comparing the similarity with the value stored in is widely used in the field of image identification or image filtering technology. However, in the case of images generated by editing different works images into one file or adding or deleting audio signals, it is difficult to identify them in this way.

Recently, as a face recognition method, a deep neural network is pre-trained using various face images, and then a method of finding which face of the input image is the most similar to the learned face by using the learned neural network has been widely used. In addition, there are attempts to automatically detect the face of a criminal or a lost child in a surveillance camera using such a technology.

In the face recognition technology using a deep learning network for the purpose of identifying such moving picture contents, it is possible to identify contents using information of a plurality of actors for a single work image having a sufficient length. On the other hand, in a video that is edited to a short length and only one actor appears, it is not easy to specify one of a number of videos in which the actor appeared. In particular, it is very difficult to expect an accurate identification result when a plurality of works images are edited and reproduced as a single moving image.

An object of the present invention for solving the above problems is to provide a method for identifying video content.

Another object of the present invention for solving the above problems is to provide a video content identification device using the video content identification method.

A video content identification method according to an embodiment of the present invention for achieving the above object includes: extracting a frame from an input video; Detecting a face area from the extracted frame; Detecting a clothing area in the frame in which the face area is detected; Comparing the detected clothing region with one or more candidate images stored in a database using information on a face recognized based on the detected face region; And identifying the input video according to the comparison result.

The one or more candidate images stored in the database may include: detecting a face region from an image constituting a data set; Recognizing a person using the detected face area; Detecting a clothing area from the image in which the face area is detected; And storing the detected clothing area together with meta information on the recognized person.

The step of detecting the face region from the extracted frame may be performed using a first neural network model trained through supervised learning on the face region of a person.

The step of detecting the clothing region from the image in which the face region is detected may be performed using a second neural network model of a single-step method trained through supervised learning on the clothing region of a person.

Comparing the detected clothing region with one or more candidate images stored in the database may be performed using a third neural network model trained through learning about the clothing region of a person.

Comparing the detected clothing area with one or more candidate images stored in the database includes at least one first image including the clothing area stored in the database and a second clothing area detected from the input image. Calculating a degree of similarity between images; And identifying a content corresponding to a first image having a maximum value among similarity values between the at least one first image and the second image as a content to which the input video belongs.

The third neural network model calculates a CNN feature map for the first image and the second image, and reconstructs the calculated CNN feature map, thereby calculating the similarity between the first image and the second image as a ratio value. And print it out.

The one or more candidate images stored in the database may be managed in association with a name of a person related to the candidate image, information on a clothing area in the candidate image, and a content identifier for the candidate image.

A video content identification apparatus according to another embodiment of the present invention for achieving the above object includes: a processor; And a memory storing at least one instruction executed through the processor,

The at least one command may include a command to extract a frame from an input video; An instruction to detect a face area in the extracted frame; A command for detecting a clothing area in the frame in which the face area is detected; An instruction for comparing the detected clothing region with one or more candidate images stored in a database using information on a face recognized based on the detected face region; And a command to identify the input video according to the comparison result.

The one or more candidate images stored in the database may include: detecting a face region from an image constituting a data set; Recognizing a person using the detected face area; Detecting a clothing area from the image in which the face area is detected; And storing the detected clothing area together with meta information on the recognized person.

The command to detect the face region from the extracted frame may be performed using a first neural network model trained through supervised learning on the face region of a person.

The command for detecting the clothing region from the image in which the face region is detected may be performed using a second neural network model of a single step method trained through supervised learning on the clothing region of a person.

The command for comparing the detected clothing region with one or more candidate images stored in the database may be performed using a third neural network model trained through learning about the clothing region of a person.

The command for comparing the detected clothing region with one or more candidate images stored in the database includes at least one first image including the clothing region stored in the database and a second clothing region detected from the input image. An instruction to calculate the similarity between the images; And a command for identifying a content corresponding to a first image having a maximum value among similarity values between the at least one first image and the second image as a content to which the input video belongs.

The third neural network model calculates a CNN feature map for the first image and the second image, and reconfigures the calculated CNN feature map to calculate the similarity between the first image and the second image as a ratio value. And print it out.

The one or more candidate images stored in the database may be managed in association with a name of a person related to the candidate image, information on a clothing area in the candidate image, and a content identifier for the candidate image.

According to the embodiments of the present invention as described above, it is possible to identify video content with improved face recognition accuracy.

In particular, compared to the conventional method, it is possible to expect more accurate identification performance for work images edited in the form of short clips and moving image contents in which a plurality of work images are edited into a single file.

Through this, it can be effectively used in the field of automatic identification and monitoring of short content and editing videos of multiple works in areas where copyright infringement occurs frequently, such as movies and TV dramas.

1 is a conceptual diagram of a video content identification device according to an embodiment of the present invention.
2 is a flow diagram of a method for identifying video content according to an embodiment of the present invention.
3 is a flowchart of a method of generating a face recognition model according to an embodiment of the present invention.
4 is a flowchart illustrating a process of identifying a face using a CNN model according to an embodiment of the present invention.
5 is a flowchart illustrating a process of detecting and storing a clothing image according to an embodiment of the present invention.
6 is a conceptual diagram of object detection used for detecting a clothing area according to the present invention.
7 shows examples of learning images including a clothing region used for learning to detect a clothing region according to the present invention.
8 is a conceptual diagram illustrating a management system of image information related to a person according to an embodiment of the present invention.
9 is a block diagram of a deep learning system for determining similarity between two images based on a clothing area according to an embodiment of the present invention.
10 and 11 show examples of feature maps reconstructed according to the present invention.
12 is a table showing the size of an image input/output in each step of a deep learning system for determining similarity according to the present invention.
13 illustrates a concept of final identification of an input image using a clothing area image registered in a database according to an embodiment of the present invention.
14 is a block diagram of a video content identification device according to another embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to a specific embodiment, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals have been used for similar elements.

Terms such as first, second, A, and B may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. The term "and/or" includes a combination of a plurality of related described items or any of a plurality of related described items.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof does not preclude in advance.

Unless otherwise defined, all terms used herein including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present application. Does not.

In order to solve the above-described problems of the prior art, the present invention detects an actor's face recognition and a clothing area using a deep learning-based object detection technology for arbitrary moving image contents edited with a short length of one or more works images, and the result is determined. It proposes a method to identify which work is used.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a video content identification device according to an embodiment of the present invention.

As shown in FIG. 1, the video content identification apparatus 100 according to an embodiment of the present invention recognizes an actor's face from an input movie or TV video content. The video content identification apparatus 100 detects a costume area of an actor whose face is recognized based on a result of actor recognition using a face. The video content identification apparatus 100 may identify an unknown video input through a re-identification process by comparing the detected image of the plurality of clothing regions with the clothing images of the same actor registered in the DB.

2 is a flow diagram of a method for identifying video content according to an embodiment of the present invention.

The overall flow of the method for identifying video content according to the present invention is as shown in FIG. 2.

According to the video content identification method, frames are extracted from the input video (S210 and S220). When a face area is detected within the frame (S230), a costume area is detected together with actor face recognition (S240) (S250). At this time, the actor face recognition and the clothing region detection may be performed using a deep learning network previously learned using the actor face and clothing region images. For the costume area image detected from the input video, a similarity comparison with the costume images of the actor registered in the DB is performed (S260), and a final identification result may be obtained through this (S270).

3 is a flowchart of a method of generating a face recognition model according to an embodiment of the present invention.

The generation of the face recognition model shown in FIG. 3 is a procedure performed as a preliminary task for the video identification method shown in FIG. 2. That is, the video identification method shown in FIG. 2 may be performed using the face recognition model generated according to the method shown in FIG. 3.

In order to create a face recognition model, a number of actors' face images are used. More specifically, a face region is detected for each input actor image (S310), and image alignment is performed on the detected face region (S320). In this case, facial feature points are used for image alignment, and facial landmarks such as eyes, nose, and mouth may be the feature points. By arranging images based on such landmarks and inputting them to the CNN model to train the CNN model (S330), a face recognition model with improved face recognition accuracy may be generated (S340).

4 is a flowchart illustrating a process of identifying a face using a CNN model according to an embodiment of the present invention.

The face identification process of FIG. 4 shows in detail the face supervised learning (S330) examined through FIG. 3. The CNN (Convolutional Neural Network) model used in the embodiment of FIG. 4 is an example, and the neural network model used for face identification in the present invention is not limited to the CNN.

Here, the CNN (convolution) neural network is one of artificial neural networks, and is used to more easily analyze data by using the convolution of input data and filters.

Artificial Neural Network (ANN) is a statistical learning algorithm inspired by biological neural networks (especially the brain in the central nervous system of animals) in machine learning and cognitive science. The artificial neural network refers to the overall model with problem-solving ability by changing the strength of synaptic bonding through learning by artificial neurons (nodes) that form a network through synaptic bonding.

Artificial neural networks include supervised learning that is optimized for a problem by inputting a guidance signal (correct answer) and unsupervised learning that does not require a guidance signal. Supervised learning is usually used when there is a clear answer and unsupervised learning is used for data clustering. Artificial neural networks rely on many inputs and are generally used to guess and approximate a function wrapped in a veil. In general, it is expressed as an interconnection of neuronal systems that calculate values from inputs, and has adaptability, so machine learning such as pattern recognition can be performed.

Referring to FIG. 4, the CNN model undergoes several (for example, three) convolution and activation functions (ReLU; Rectified Linear Unit), and Max Pooling, and additionally, a plurality of complete connections. Face identification results can be obtained through a Softmax function that normalizes the output value of the layer (Fully Connected Layer) to a value between 0 and 1.

In convolution, a feature map can be extracted through filters, and an activation function is applied to the feature map. ReLU can be used as the activation function. Features extracted through the convolutional layer undergo a pooling (or subsampling) process that reduces their size as necessary, and one of them is the max pooling technique. Max pooling is a method of cropping the activation map to the size of MxN and then extracting the largest value from it. Max pooling is based on the concept that a value with a large feature value represents other features.

If features are extracted through this process, classification is performed by putting the extracted feature values into a fully connected layer (general neural network). After the fully connected layer, a softmax function is applied. The softmax function is a kind of activation function, which can have several classifications. Therefore, it can be seen that the result derived in FIG. 4 is classified into several identifiers.

When the CNN model is sufficiently learned through the process of repetitively supervising the network shown in FIG. 4 using a large amount of face images, it can be used as a face recognition model according to the present invention. As described above, supervised learning is a method of learning by telling the correct answer. For example, this picture is actor A and this picture is actor B. This is a method of training a neural network while giving the correct answer.

The face identification result using such a face recognition model can be obtained as a probability value as shown in FIG. 4. In the example of FIG. 4, the probability that the face included in the input image is face_ID_003 is 90%, and the input image may be estimated to be face_ID_003.

5 is a flowchart illustrating a process of detecting and storing a clothing image according to an embodiment of the present invention.

According to an embodiment of the present invention, costume images for each actor for identifying a work image may be processed according to the order shown in FIG. 5 and stored in a database.

In more detail, first, frames are extracted from the input video (S510, S520). When a face area is detected within the frame (S530), a costume area is detected together with actor face recognition (S540) (S550). At this time, the actor face recognition and the clothing region detection may be performed using a deep learning network previously learned using the actor face and clothing region images. The detected clothing area is stored together with meta information related to the corresponding area and converted into a database (S560). In this case, the meta information may include the title of the video, the name of the actor, and the like. For example, in the case of a movie, a frame is extracted from a main story or a trailer, and a costume area is detected from a frame where a face is detected, and the costume image is stored in a database along with meta information such as the movie title and actor name.

6 is a conceptual diagram of object detection used for detecting a clothing area according to the present invention.

In the present invention, a deep learning network of a single stage method as shown in FIG. 6 may be used as an object detector for detecting a garment region.

That is, object detection according to the present invention is performed through feature extraction and detection radar based on CNN, and the object type and location are output as a result value. Here, the type of the object is the clothing area, and the location of the object may be the location of the clothing area.

As a method of discriminating and detecting the type of object by a single process, a method using a single shot multibox detector (SSD) or You Only Look Once (YOLO) may be used. These are deep learning networks constructed for the purpose of detecting the type and location of an object from an input image without searching for an object candidate area.

When training the deep learning network for detecting the clothing area as shown in FIG. 6, the training data is prepared in consideration of the characteristics of a work image, such as a movie or TV, for the purpose of the present invention. In the case of movies and TV, it is often filmed on a close-up screen for reasons such as acting on the actor's face, and only the upper body is visible. In this environment, in order to detect the clothing region and increase the accuracy of re-identification with the clothing images registered in the database, the correct answer region from the image data preparation stage used for learning the clothing region detector is selected from the upper body oriented images. To supervise the deep learning network.

7 shows examples of learning images including a clothing region used for learning to detect a clothing region according to the present invention.

It can be seen that in each image of FIG. 7, a clothing area is separately displayed among the entire images. These images can be used for learning of the garment region detector according to the present invention.

8 is a conceptual diagram illustrating a management system of image information related to a person according to an embodiment of the present invention.

Referring to FIG. 8, for an actor whose face is recognized, a plurality of fashion images of the actor are associated and stored with the actor's name as a vertex. In addition, content_ID is also stored and managed for each clothing area image.

Through this storage system, the image of the costume in the actor's appearance work, whose face is recognized, is compared with the content to be input later to identify the input content.

Such an image information management system may be constructed through a storage step (S560) of the costume images of actors whose faces are recognized as viewed through FIG. 5, in a DB. In addition, the database constructed in this way helps to identify unknown video contents through comparison with actor names and costume images recognized by the same process in the video identification process.

9 is a block diagram of a deep learning system for determining similarity between two images based on a clothing area according to an embodiment of the present invention.

The two images to be compared in the embodiment of FIG. 9 are images including the identified clothing area, and may be a first image (input_1) stored in a database and a second image (input_2) including a clothing area detected from the input image. .

At this time, the input image may use only the clothing area (dotted line box), or the image of the area including the face (the solid line box surrounding the entire image in FIG. 9) may be used as an input to the network.

Referring to FIG. 9, convolution and max pooling 910 (920) are repeatedly applied to an input image pair, that is, a first image and a second image, respectively, in order to determine the similarity between two images.

The convolution and max pooling 910; 920 include Convolution_1, MaxPooling_1, and MaxPooling_2, and further include Convolution_2, MaxPooling_3, and MaxPooling_4 having the same structure.

,

) 의 차이를 계산하여 두 이미지 사이의 차이에 대한 특징맵(

,

)을 새롭게 구성한다. 이를 다시 컨볼루션, 풀링, 완전연결층으로 이어지는 일련의 신경망(920)을 통과시키고 최종적으로 소프트맥스를 통해 두 이미지의 동일성을 판단한다. Then, the extracted two CNN feature maps ((

,

) By calculating the difference in the feature map (

,

) Is newly constructed. This is again passed through a series of neural networks 920 leading to convolution, pooling, and fully connected layers, and finally, the identity of the two images is determined through softmax.

Here, the series of neural networks 920 may include Convolution_3, Convolution_4, Convolution_5, Convolution_6, MaxPooling_5, and MaxPooling_6, and may include a fully connected layer at the end of the neural network.

Additionally, although not shown in FIG. 10, it may be more effective to configure a deep learning network by adding an activation function such as ReLU (Rectified Linear Unit) after convolution.

,

) 사이의 관계를 계산하는 방법을 나타내며, 입력 이미지쌍에 대해 재구성된 특징맵

와

를 도출하는 식이다.In FIG. 9, the process of calculating the difference between input image pairs can be expressed by expressing Equation 1 below. Equation 1 is the two feature maps (

,

) Represents the method of calculating the relationship between, and the reconstructed feature map for the input image pair.

Wow

Is an equation that derives.

는 첫 번째 입력 이미지가 컨볼루션 레이어(910)를 통과하여 도출된 특징맵이고,

는 두 번째 입력 이미지가 컨볼루션 레이어(910)를 통과하여 도출된 특징맵을 의미한다. 또한,

는 1로 구성된 5×5 행렬을 뜻하고,

는

행렬에서

가 중심인 5×5 행렬을 의미한다.

Is a feature map derived by passing the first input image through the convolutional layer 910,

Denotes a feature map derived by passing the second input image through the convolutional layer 910. Also,

Means a 5x5 matrix of 1s,

Is

In the procession

Denotes a 5×5 matrix with the center of.

10 and 11 show examples of feature maps reconstructed according to the present invention.

와

를 재구성한다. That is, FIGS. 10 and 11 show examples of results of applying Equation 1 to an actual feature map. In the present invention, a feature map for an input image pair in this way

Wow

Reconstruct.

12 is a table showing the size of an image input/output in each step of a deep learning system for determining similarity according to the present invention.

The table shown in FIG. 12 represents the size of the input image and the size of the output image of each layer of the deep learning system described through the embodiment of FIG. 9.

More specifically, the table of FIG. 12 shows the sizes of the input and output images of Convolution_1, MaxPooling_1, and MaxPooling_2 included in the neural network 910 configured to derive the feature vector, and the input and output of Convolution_2, MaxPooling_3 and MaxPooling_4 included in the neural network 920 Indicate the size of the image.

In addition, it shows the size of the input and output images of Convolution_3, Convolution_4, Convolution_5, Convolution_6, MaxPooling_5, and MaxPooling_6 included in the series of neural networks 930 that process the reconstructed feature map.

13 illustrates a concept of final identification of an input image using a clothing area image registered in a database according to an embodiment of the present invention.

13 shows the similarity between the actor's clothing region detected from an unknown image and the actor's costume region images registered in the DB using the learned neural network system to re-identify the costume region, and to identify an unknown input image. It is a conceptual diagram. That is, the costume area of the actor identified in the input image is compared with the costume area in the plurality of corresponding actor candidate images (movieID_0001, movieID_0002, movieID_0003, movieID_0004) in the database using a sufficiently learned neural network system. The similarity between each candidate image and the input image may be derived through comparison using the learned neural network system. Among the derived similarity values, it is possible to identify the content by checking meta information of the clothing image having the highest similarity.

In the example of FIG. 13, since the similarity between movieID_0001 and the input image is 0.05, the similarity between movieID_0002 and the input image is 0.02, the similarity between movieID_0003 and the input image is 0.9, and the similarity between movieID_0004 and the input image is 0.03, the image with the highest similarity is movieID_0003. It can be confirmed that the content has an identifier of. That is, in the example of FIG. 13, it can be confirmed that the input image belongs to the video content movieID_0003.

14 is a block diagram of a video content identification device according to another embodiment of the present invention.

A video content identification device according to an embodiment of the present invention includes at least one processor 110, a memory 120 for storing at least one command executed through the processor, and a transmission/reception device connected to a network to perform communication. It may include (130).

The video content identification device 100 may further include an input interface device 140, an output interface device 150, and a storage device 160. Each of the components included in the video content identification apparatus 100 may be connected by a bus 170 to communicate with each other.

The processor 110 may execute a program command stored in at least one of the memory 120 and the storage device 160. The processor 110 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor in which methods according to embodiments of the present invention are performed.

Each of the memory 120 and the storage device 160 may be configured with at least one of a volatile storage medium and a nonvolatile storage medium. For example, the memory 120 may be composed of at least one of read only memory (ROM) and random access memory (RAM). In addition, the storage device 160 may include the database described through the above embodiments, and such a database may include one or more candidate images.

Here, the at least one command may include: a command for causing the processor to extract a frame from an input video; An instruction to detect a face area in the extracted frame; A command for detecting a clothing area in the frame in which the face area is detected; An instruction for comparing the detected clothing region with one or more candidate images stored in a database using information on a face recognized based on the detected face region; And a command to identify the input video according to the comparison result.

The one or more candidate images stored in the database may include: detecting a face region from an image constituting a data set; Recognizing a person using the detected face area; Detecting a clothing area from the image in which the face area is detected; And storing the detected clothing area together with meta information on the recognized person.

The command to detect the face region from the extracted frame may be performed using a first neural network model trained through supervised learning on the face region of a person.

The command for detecting the clothing region from the image in which the face region is detected may be performed using a second neural network model of a single-step method trained through supervised learning on the clothing region of a person.

The command to compare the detected clothing region with one or more candidate images stored in the database may be performed using a third neural network model trained through learning about the clothing region of a person.

The command for comparing the detected clothing region with one or more candidate images stored in the database includes at least one first image including the clothing region stored in the database and a second clothing region detected from the input image. An instruction to calculate the similarity between the images; And a command for identifying a content corresponding to a first image having a maximum value among similarity values between the at least one first image and the second image as a content to which the input video belongs.

The third neural network model calculates a CNN feature map for the first image and the second image, and reconfigures the calculated CNN feature map to calculate the similarity between the first image and the second image as a ratio value. And print it out.

The one or more candidate images stored in the database may be managed in association with a name of a person related to the candidate image, information on a clothing area in the candidate image, and a content identifier for the candidate image.

As can be seen from the above embodiments, the video content identification method according to the present invention simultaneously considers the actor's face and the costume image, and edits a short edited video and a plurality of copyrighted images as a single video compared to the existing identification methods. It is a method that can be accurately identified even when regenerated.

The operation of the method according to the embodiment of the present invention can be implemented as a computer-readable program or code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. In addition, a computer-readable recording medium may be distributed over a network-connected computer system to store and execute a computer-readable program or code in a distributed manner.

Further, the computer-readable recording medium may include a hardware device specially configured to store and execute program commands, such as ROM, RAM, and flash memory. The program instructions may include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

While some aspects of the invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, where a block or apparatus corresponds to a method step or characteristic of a method step. Similarly, aspects described in the context of a method can also be represented by a corresponding block or item or a feature of a corresponding device. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, a programmable computer or electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In embodiments, the field programmable gate array may work with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by some hardware device.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will be able to variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the following claims. You will understand that you can.

Claims (20)

Extracting a frame from the input video;
Detecting a face area from the extracted frame;
Detecting a clothing area in the frame in which the face area is detected;
Comparing the detected clothing region with one or more candidate images stored in a database using information on a face recognized based on the detected face region; And
And identifying an input video according to the comparison result. The method according to claim 1,
One or more candidate images stored in the database,
Detecting a face area from an image constituting a data set;
Recognizing a person using the detected face area;
Detecting a clothing area from the image in which the face area is detected; And
A video content identification method generated through the step of storing the detected clothing area together with meta information on the recognized person. The method according to claim 1,
The step of detecting a face area from the extracted frame,
A method of identifying video contents performed using a first neural network model trained through supervised learning on a person's face region. The method according to claim 1,
The step of detecting the clothing area in the image in which the face area is detected,
A method of identifying video contents performed using a second neural network model of a single-step method trained through supervised learning on a person's face region. The method according to claim 1,
Comparing the detected clothing area with one or more candidate images stored in the database,
A method of identifying video content, performed using a third neural network model trained through learning about a person's clothing area. The method according to claim 1,
Comparing the detected clothing area with one or more candidate images stored in the database,
Calculating a similarity between at least one first image including a clothing area stored in the database and a second image including a clothing area detected from an input image; And
And identifying a content corresponding to a first image having a maximum value among similarity values between the at least one first image and the second image as a content to which the input video belongs. The method of claim 5,
The third neural network model,
A video content identification method for calculating and outputting a similarity between the first image and the second image as a ratio value by calculating a CNN feature map for the first image and the second image and reconstructing the calculated CNN feature map. The method according to claim 2,
One or more candidate images stored in the database,
A video content identification method that is managed in association with a name of a person related to the candidate image, information on a clothing area in the candidate image, and a content identifier for the candidate image. The method of claim 3,
The first neural network model is configured to include at least one of convolution, activation function, max pooling, fully connected layer, and softmax. The method of claim 5,
The third neural network model,
Convolution, activation function, Max Pooling (Max Pooling), fully connected layer (Fully connected layer), and consisting of one or more of the SoftMax (SoftMax), video content identification method. Processor; And
Includes a memory for storing at least one instruction executed through the processor,
The at least one command,
A command to extract a frame from an input video;
An instruction to detect a face area in the extracted frame;
A command for detecting a clothing area in the frame in which the face area is detected;
An instruction for comparing the detected clothing region with one or more candidate images stored in a database using information on a face recognized based on the detected face region; And
A video content identification device comprising a command to identify an input video according to the comparison result. The method of claim 11,
One or more candidate images stored in the database,
Detecting a face area from an image constituting a data set;
Recognizing a person using the detected face area;
Detecting a clothing area from the image in which the face area is detected; And
A video content identification device generated through the step of storing the detected clothing area together with meta information on the recognized person. The method of claim 11,
The command to detect the face area in the extracted frame,
A video content identification device performed using a first neural network model trained through supervised learning on a person's face region. The method of claim 11,
The command to detect the clothing area in the frame in which the face area is detected,
A video content identification device that is performed using a second neural network model of a single-step method trained through supervised learning of a person's clothing area. The method of claim 11,
The command to compare the detected clothing area with one or more candidate images stored in the database,
A video content identification device performed using a third neural network model trained through learning about a person's clothing area. The method of claim 11,
The command to compare the detected clothing area with one or more candidate images stored in the database,
A command for calculating a similarity between at least one first image including a clothing area stored in the database and a second image including a clothing area detected from an input image; And
A video content identification apparatus comprising a command for identifying a content corresponding to a first image having a maximum value among similarity values between the at least one first image and the second image as a content to which the input video belongs. The method of claim 15,
The third neural network model,
A video content identification apparatus for calculating and outputting a similarity between the first image and the second image as a ratio value by calculating a CNN feature map for the first image and the second image and reconstructing the calculated CNN feature map. The method of claim 11,
One or more candidate images stored in the database,
A video content identification device that is managed in association with a name of a person related to the candidate image, information on a clothing area within the candidate image, and a content identifier for the candidate image. The method of claim 13,
The first neural network model is configured to include at least one of convolution, activation function, max pooling, fully connected layer, and softmax. The method of claim 15,
The third neural network model,
Convolution, activation function, Max Pooling (Max Pooling), a fully connected layer (Fully connected layer), and consisting of one or more of SoftMax (SoftMax), video content identification device.

Images