KR20210064597A - Apparatus and method for analyzing video - Google Patents

Abstract

The present invention relates to an apparatus for analyzing a video. The apparatus separates the voice data of audio content in the video for each speaker, recognizes the face of a person appearing in the video content, matches the separated voice for each speaker with the recognized face, and identifies characters in every frame of the video to provide character information for each frame. It can create bookmarking information about the appearance of characters for each scene by using this information, and perform scene search and clip image generation for each character. The apparatus includes an audio analysis part, a video analysis part and a control part.

Description

Apparatus and method for video analysis {APPARATUS AND METHOD FOR ANALYZING VIDEO}

Disclosed is an apparatus for analyzing a moving picture, and more particularly, a technology for analyzing the appearance of a character in a moving picture and providing appearance information for each frame.

Editing a video is an important task that greatly affects the quality of a video, but it is done manually and takes a lot of time and money. In order to edit a video, the editor must know the overall information of the video, and it is necessary to bookmark the executive section for each scene so that it can be easily referenced during editing. In general, video editing is done manually by the editor according to a subjective concept of bookmarking.

Characters occupying an important part of the video are objective information, and if only scenes in which the characters appear can be automatically bookmarked, manual editing can be convenient. When a video is provided as an input, it is necessary to develop a device capable of analyzing the video, bookmarking only the appearance of the performer, and generating additional information using the bookmarking information.

An object of the present invention is to provide an apparatus and method for automatically bookmarking frames in which a character appears in a moving picture by analyzing a moving picture.

In addition, it is another object of the present invention to provide an apparatus and method that can analyze a moving picture and search only a scene in which a specific character appears in the moving picture.

In addition, it is another object of the present invention to provide an apparatus and method capable of generating a clip image including only a scene in which a specific character appears in the moving image by analyzing the moving image.

A video analysis apparatus according to an aspect of the present invention includes an audio analysis unit, a video analysis unit, and a control unit.

The audio analyzer separates the voices for each speaker from the audio content of the moving picture, and generates voice frame information for the voices for each speaker.

The video analysis unit recognizes the faces of characters appearing in the video content of the moving picture, classifies the characters, and generates face recognition frame information for the characters.

The control unit matches the recognized face and voice by combining the voice frame information and the face recognition frame information, and generates a character frame vector indicating whether a character appears or not with respect to the entire frame.

A video analysis method according to an embodiment of the present invention is implemented at least in part by program commands executed in a video analysis apparatus that receives and processes a video, and includes the steps of separating a speech for each speaker from audio content of the video, and for each speaker The steps of generating voice frame information for the voice, the steps of recognizing the faces of people appearing in the video content of the moving picture to classify the characters, the steps of generating face recognition frame information about the characters, and the voice frame information and matching the recognized face and voice by combining face recognition frame information, and generating a character frame vector indicating whether a character appears in the entire frame.

According to the present invention, by analyzing a moving picture, it is possible to automatically bookmark frames in which a character appears in the moving picture.

Additionally, according to the present invention, only scenes in which a specific character appears in the video can be searched by analyzing the video.

In addition, according to the present invention, it is possible to automatically generate a clip image including only a scene in which a specific character appears in the moving picture by analyzing the moving picture.

1 is a conceptual diagram illustrating a process of analyzing a video by the video analysis apparatus of the present invention.
2 is a block diagram of a video analysis apparatus of the present invention.
3 schematically illustrates a concept in which the voice separation model separates voices for each speaker into audio content.
FIG. 4 shows an example of matching each speaker's voice with a recognized face using a co-occurrence matrix.
5 illustrates a concept of correcting a character frame vector using a sliding window.
6 is a flowchart illustrating an exemplary procedure of the video analysis method of the present invention.

The foregoing and additional aspects are embodied through embodiments described with reference to the accompanying drawings. It is understood that various combinations are possible within the embodiments as long as there is no contradiction between the elements or other mentions. Each block in the block diagram may in some cases represent a physical part, but in other cases may be a part of the function of one physical part or a logical representation of a function across a plurality of physical parts. Sometimes a block or part of an entity may be a set of program instructions. All or part of these blocks may be implemented by hardware, software, or a combination thereof.

1 is a conceptual diagram illustrating a process of analyzing a video by the video analysis apparatus of the present invention. When explaining the process of analyzing a video with reference to FIG. 1, a voice separation model, which is a deep learning model trained to separate voices for each speaker, separates voices for each speaker from audio content included in a video (eg, a movie), and , generates voice frame information, which is information about frames in which the voices of characters appear in the video, for each speaker’s voice, and the face recognition model, a deep learning model trained to recognize faces in the video A face group is formed for the same person by recognizing the face of a person appearing in the image, and face recognition frame information, which is information about frames in which the character of the recognized face appears in the video, is generated. In this case, the audio processing and the image processing may be sequentially processed or processed simultaneously regardless of the order. Then, the voice frame information and the face recognition frame information are combined, and the speaker's voice and the face recognition group are automatically matched using the frequency at which the voice and face appear at the same time. Using the matching result, voice frame information, and face recognition frame information, a character frame vector indicating whether all the characters appear in the entire frame is generated. At this time, depending on the setting, if both a voice and a face appear, it may be recognized as an appearance, or it may be recognized as an appearance if either a voice or a face appears. The generated character frame vector contains parts that appear and disappear only for a very short time. A correction is made to delete short appearances for the entire character frame vector so that it is recognized as an appearance only when it appears in consecutive frames over a set threshold. Various meta information is generated using the corrected character frame vector, voice frame information, and face recognition frame information.

2 is a block diagram of a video analysis apparatus of the present invention. The video analysis apparatus 10 may be implemented as a computing device including a microprocessor, a memory, a display, and the like, or may be implemented as a plurality of computing devices connected through a network.

According to an aspect of the present invention, the video analysis apparatus 10 includes an audio analysis unit 100 , a video analysis unit 110 , and a control unit 120 .

The video analysis apparatus 10 may further include a meta-information generating unit 130 according to an additional aspect, and according to another additional aspect, a meta-information generating unit 130, a user interface 160, and a frame search unit ( 140), and may further include a meta information generating unit 130, a user interface 160, and a clip image editing unit 150 according to another additional aspect.

In addition, as shown in FIG. 2 , the video analysis apparatus 10 includes an audio analysis unit 100 , a video analysis unit 110 , a control unit 120 , a meta information generation unit 130 , and a user interface ( 160 , a frame search unit 140 , and a clip image editing unit 150 may be included.

The audio analysis unit 100, the video analysis unit 110, the control unit 120, the meta information generation unit 130, the user interface 160, the frame search unit 140, and the clip image editing unit ( 150) may be implemented as program instructions, at least a part of which is executed in the video analysis device 10, all of which are executed in one computing device, or some or all of them in another computing device.

The audio analysis unit 100 includes a voice separation model, which is a trained neural network model such as deep learning, and separates voices for each speaker from audio content of a video through the voice separation model. The voice separation model is a model trained by deep learning, etc. to separate the voices for each speaker even when multiple speakers speak simultaneously. The specific algorithm of the speech separation model is a well-known technique that has been published through various papers, and a detailed description thereof will be omitted.

3 schematically illustrates a concept in which the voice separation model separates voices for each speaker into audio content. The example of FIG. 3 conceptually illustrates an example of separating sound sources that are simultaneously uttered and mixed for each speaker.

In addition, the audio analyzer 100 generates voice frame information with respect to the separated voice for each speaker. The voice frame information is information in which a voice for each speaker is mapped to frame information of an image. For example, a video section (start time and end time) in which the separated speaker voice appears is mapped to frame information of the video by time code, elapsed playback time, or frame number.

The video analysis unit 110 includes a face recognition model, which is a trained neural network model such as deep learning, and recognizes the faces of people appearing in the video content of the moving picture through the face recognition model to classify the characters. In general, a face recognition model is a model trained by deep learning, etc. to differentiate a person by vectorizing a number of key point location information such as the location of the eyes and the location of the nose in a human face in an image. The specific algorithm of the face recognition model is a well-known technology that has been published through various papers, and a detailed description thereof will be omitted.

The face recognized by the video analysis unit 110 in the video content is a face recognized in the corresponding scene, and even if it is the same person, the face may be recognized in various ways depending on the scene. Accordingly, since a plurality of faces recognized for the same person may exist, the present invention manages the recognized faces of the same person as a face group.

In addition, the video analysis unit 110 generates face recognition frame information for the face-recognized character. The face recognition frame information is information in which a scene in which a face-recognized person appears is mapped to frame information of an image. For example, a video section (start time and end time) in which a face-recognized person appears is mapped to frame information of the video as a time code, elapsed playback time, or frame number.

The controller 120 matches the recognized face and voice by combining the voice frame information and the face recognition frame information. The voice and the recognized face, that is, a face group, are automatically matched by using the frequency of simultaneous appearance from frame information in which the voice of each speaker appears in the voice frame information and the frame information in which a face recognized in the face recognition frame information appears at the same time. According to an aspect of the present invention, matching may be performed automatically using a co-occurrence matrix. The co-occurrence matrix is a matrix used when analyzing using the frequency in which two words appear simultaneously in the same sentence in the data mining field or natural language processing field. In the present invention, the frequency of the frames in which the voice and the recognized face appear at the same time is used. Match each speaker's voice with the recognized face.

FIG. 4 shows an example of matching each speaker's voice with a recognized face using a co-occurrence matrix. In the matrix of FIG. 4 , a row represents a voice for each speaker, a column represents a recognized face (face group), and a value crossing a row and a column means the frequency of consecutive frames appearing at the same time. In each matrix, the column with the highest frequency is matched for each row. In the example of FIG. 4 , audio1 matches face group5, and audio2 matches face group1.

In addition, the control unit 120 generates a character frame vector indicating whether a character appears with respect to the entire frame. Before generating the character frame vector, a vector in which a frame in which a voice and a face appears for each character is marked as 1 and a frame that does not appear as 0 is first generated for the entire frame. An example of this vector is the audio of each face group in the character frame vector generation part of FIG. 1, 0 and 1 indicated in the face row. The controller 120 generates a character frame vector for each face group by performing an AND or OR operation on the appearance of the face and audio of each face group according to the setting.

Also, when the character appears in consecutive frames less than or equal to the threshold, the controller 120 may correct the character frame vector so that it does not appear in the corresponding frame. Since voice frame information and face recognition frame information include frame information when a character's voice or face appears briefly and then disappears in a very short time, the character frame vector created using these information contains only a very short moment. In the case of an appearance, the corresponding person is displayed as appearing. Accordingly, the control unit 120 corrects the character frame vector as not appearing when the voice and/or face of the character appears in continuous frames below the preset threshold, and at this time, the character frame is corrected. The vector can be moved and corrected using a sliding window. The size of the window is set to a specific value.

5 illustrates a concept of correcting a character frame vector using a sliding window. In general, the sliding window technique is used to correct time series information, and is corrected by averaging noise or large variability parts appearing in a short moment using surrounding values. In the case of the character frame vector, not continuous values are corrected, but discontinuous values of appearance (1)/non-appearance (0) are corrected in consideration of the majority vote of surrounding values in the window, the number of consecutive ones, etc. . In the example of FIG. 5 , since there is one appearance (1) in the window (size 5), it is illustrated that it is corrected to not appear (0). Although not shown, the opposite case is also possible. If there is one non-appearance (0) in the window and the remaining ones appear (1), the non-appearance (0) can be corrected to the appearance (1).

The user interface 160 may allow the image analysis apparatus to accept a user's input and display the processed result on the screen. Also, the user interface 160 may play and display a video.

In addition, the user interface 160 may present thumbnail images of face-recognized characters in a list form for frame search so that the user can select the characters, and in this case, the user may select a plurality of characters. The frame search result according to the user input may be presented to the user through the user interface 160 , and only the frame of the corresponding result may be reproduced and displayed through the user interface 160 .

In addition, the user interface 160 may present a thumbnail image of a face-recognized character in a list form for generating a clip image so that the user can select the character, and in this case, the user may select a plurality of characters. A clip image generated according to a user input may be reproduced and displayed through the user interface 160 .

The meta information generating unit 130 generates bookmarking information for frames in which the character appears by using the character frame vector. Using the bookmarking information, it is possible to quickly search between scenes in which a specific character appears through the user interface 160 provided by the video analysis apparatus 10 .

In addition, the meta information generating unit 130 may calculate the appearance time and number of appearances for each character using the character frame vector, and may generate appearance information for each person including the total appearance time and number of appearances of the characters. have. Using the appearance information for each person, a graph that compares and presents information on the appearance of each person may be output through the user interface 160 provided by the video analysis apparatus 10 .

In addition, the meta-information generating unit 130 defines a graph using the characters as nodes, connecting characters appearing in the same scene to edges, and defining a graph using the frequency of simultaneous appearance as an edge weight to analyze the main characters, Character relationship diagram information including a relationship diagram between characters may be generated. The meta-information generating unit 130 may express the relationship of the characters as a graph data structure, select the main characters using graph analysis techniques, and create a relationship diagram between the characters. The graph analysis technique used can be analyzed using indicators such as Degree Centrality, Closeness Centrality, Betweeness Centrality, and Eigenvector Centrality. For example, using the Degree Centrality indicator, it can be seen that the more connected the nodes are, the more important the characters are.

The frame search unit 140 may search for frames in which the character selected through the user interface 160 appears by using the bookmarking information. In this case, the search results may be provided in the form of a list.

The clip image editing unit 150 may automatically create a clip image by editing frames in which a character selected through the user interface 160 appears using the bookmarking information. The clip image editing unit 150 first searches for frames in which the selected character appears, and then combines the searched frames to generate one clip image. The generated clip image may be reproduced through the user interface 160 .

Also, the clip image editing unit 150 may automatically generate an edited image by deleting frames in which a character selected through the user interface 160 appears by using the bookmarking information.

By using various meta information including bookmarking information generated by the present invention, it can be used to edit and delete the appearance amount of a specific performer in a broadcast program, and can be easily utilized when searching for reference images in other broadcasting programs, can be automatically edited and provided, or the amount of appearances for each actor in a movie can be calculated and provided. In addition, it is possible to provide summary information including information such as main characters, playing time, and file size for the image.

According to an embodiment of the present invention, the video analysis method is implemented at least in part by program commands executed by the video analysis apparatus 10 that receives and processes a video, and includes the steps of separating voices for each speaker and generating voice frame information. and a character identification step, a face recognition frame information generation step, a face and voice matching step, and a character frame vector generation step.

The step of separating the voices for each speaker is a step of separating the voices for each speaker from the audio content of the video. The video analysis apparatus 10 separates speech for each speaker from the audio content of the video through a speech separation model, which is a learned neural network model such as deep learning. The speech separation model is a model trained with the aforementioned deep learning.

The voice frame information generating step is a step of generating voice frame information for the separated voices for each speaker. The voice frame information is information in which a voice for each speaker is mapped to frame information of an image.

The character classification step is a step of classifying the characters by recognizing the faces of the characters appearing in the video content of the moving picture. The video analysis device 10 classifies the characters by recognizing the faces of people appearing in the video content of the video through a face recognition model, which is a trained neural network model such as deep learning. The face recognition model is a model trained with the aforementioned deep learning. The face recognized in the video content by the video analysis apparatus 10 is a face recognized in the corresponding scene, and even if it is the same person, the face may be recognized in various ways depending on the scene. Accordingly, since a plurality of faces recognized for the same person may exist, the present invention manages the recognized faces of the same person as a face group.

The step of generating face recognition frame information is a step of generating face recognition frame information about the character. The image analysis apparatus generates face recognition frame information for the face-recognized character. The face recognition frame information is information in which a scene in which a face-recognized person appears is mapped to frame information of an image.

The face and voice matching step is a step of matching the recognized face and voice by combining voice frame information and face recognition frame information. The image analysis apparatus automatically identifies the voice and the recognized face, that is, a face group, using the frequency of simultaneous appearance from frame information in which the voice of each speaker appears in the voice frame information and the frame information in which a face recognized in the face recognition frame information appears. match

The character frame vector generation step is a step of generating a character frame vector indicating whether a character appears in the entire frame. Before generating a character frame vector, the image analysis device first generates a vector in which frames in which voices and faces appear for each character are marked as 1 and frames that do not appear are marked as 0 for the entire frame, and each A character frame vector is generated for each face group by performing AND or OR operation on the audio of the face group and the appearance of the face.

According to another embodiment of the present invention, the video analysis method may further include a character frame vector correction step, wherein the character frame vector correction step includes a corresponding frame when a character appears in consecutive frames below a threshold value. This is the step of correcting the character frame vector as it does not appear in .

According to another embodiment of the present invention, the video analysis method may further include the step of generating bookmarking information, and the step of generating bookmarking information is bookmarking the frames in which the character appears by using the character frame vector. This is the information generation step.

According to another embodiment of the present invention, the video analysis method may further include an appearance information generation step, wherein the appearance information generation step calculates an appearance time and number of appearances for each character using a character frame vector, and This is a step of generating appearance information for each person including time and number of appearances.

According to another embodiment of the present invention, the video analysis method may further include the step of generating the person relationship diagram information, and the person relationship diagram information generation step includes the main characters and the relationship diagram between the characters through graph analysis. The person relationship is also a step in generating information. At this time, the graph is defined by using characters as nodes, connecting characters appearing in the same scene to edges, and using the frequency of simultaneous appearances as edge weights. The image analysis apparatus can express the relationship of the characters as a graph data structure, select the main characters using graph analysis techniques, and create a relationship diagram between the characters. The graph analysis technique used can be analyzed using indicators such as Degree Centrality, Closeness Centrality, Betweeness Centrality, and Eigenvector Centrality.

According to another embodiment of the present invention, the video analysis method may further include a frame searching step, wherein the frame searching step searches for frames in which a character selected through the user interface 160 appears using bookmarking information. is a step to At least one person may be selected through the user interface 160 for selecting a character to be searched by presenting the face-recognized character of the video analysis apparatus 10 to select the character to be searched.

According to another embodiment of the present invention, the video analysis method may further include a clip image generating step, wherein the clip image generating step is a frame in which a character selected through the user interface 160 using bookmarking information appears. It is a step to create a clip image by editing them. At least one person may be selected through the user interface 160 for selecting a character to be searched by presenting the face-recognized character of the video analysis apparatus 10 to select the character to be searched.

6 is a flowchart illustrating an exemplary procedure of the video analysis method of the present invention. The procedure will be described with reference to FIG. 6 , the video analysis apparatus 10 separates the speech for each speaker from the audio content of the video through the speech separation model that is the learned neural network model ( S1000 ). The video analyzing apparatus 10 generates voice frame information, which is information in which a speaker-specific voice for a separated speaker-specific voice is mapped to frame information of an image (S1010).

The video analysis apparatus 10 recognizes the faces of persons appearing in the video content of the video through a face recognition model, which is a learned neural network model, and classifies the characters ( S1030 ). In this case, the faces recognized by the video analysis apparatus 10 are managed as a face group. The video analysis apparatus 10 generates face recognition frame information, which is information in which a scene in which a person whose face is recognized for the character appears is mapped to frame information of an image (S1050).

The video analysis apparatus 10 matches the recognized face and voice by combining the voice frame information and the face recognition frame information (S1070). In this case, the image analysis apparatus divides the voice and the recognized face, that is, a face group, by using the frequency of simultaneous appearance from frame information in which the voice of each speaker appears in the voice frame information and frame information in which a face recognized in the face recognition frame information appears at the same time. automatically match.

The video analysis device 10 generates a character frame vector indicating whether or not the character appears for the entire frame (S1090), and when the character appears in consecutive frames below the threshold value, it is assumed that the character does not appear in the frame. Correct the character frame vector.

The video analysis device 10 generates bookmarking information for frames in which the characters appear using the character frame vector (S1110), and calculates the appearance time and number of appearances for each character using the character frame vector and generates appearance information for each person including appearance time and number of appearances (S1130), and generates person relationship diagram information including main characters and relationship diagrams between characters through graph analysis (S1150). These processes are not essential steps and may be generated in an order different from the example, and all or some steps may be omitted according to a user's selection.

When the user wants to search for a specific character or a scene in which the characters appear, the video analysis apparatus 10 searches for frames in which the selected character appears through the user interface 160 using the bookmarking information (S1170). ). In this case, the selection of the character to be searched may be at least one selected through the user interface 160 for presenting the face-recognized character of the video analysis apparatus 10 to select the character to be searched.

When a user wants to create a clip image composed of a specific character or a scene in which the characters appear, the video analysis apparatus 10 uses bookmarking information to view frames in which the character selected through the user interface 160 appears. A clip image is created by editing (S1190). In this case, the selection of the character to be searched may be at least one selected through the user interface 160 for presenting the face-recognized character of the video analysis apparatus 10 to select the character to be searched.

Although the present invention has been described above with reference to the accompanying drawings, the present invention is not limited thereto, and it should be construed to encompass various modifications that can be apparent from those skilled in the art. The claims are intended to cover such variations.

10: video analysis device
100: audio analysis unit
110: video analysis unit
120: control unit
130: meta information generating unit
140: frame search unit
150: clip video editing unit
160: user interface

Claims (14)

an audio analysis unit that separates voices for each speaker from the audio content of the video and generates voice frame information on the voices of each speaker;
a video analysis unit for recognizing the faces of people appearing in the video content of the moving picture, classifying the characters, and generating face recognition frame information for the characters; and
a controller for matching the recognized face and voice by combining the voice frame information and the face recognition frame information, and generating a character frame vector indicating whether a character appears in the entire frame;
A video analysis device comprising a.
The method of claim 1,
The control unit is a moving picture analysis device for correcting the character frame vector as not appearing in the frame when the character appears in consecutive frames below the threshold value.
3. The device of claim 2, wherein the device
a meta information generating unit generating bookmarking information for frames in which characters appear;
Video analysis device further comprising a.
4. The method of claim 3,
The meta-information generator calculates the appearance time and number of appearances for each character, and generates appearance information for each person including the appearance time and number of appearances.
4. The method of claim 3,
The meta-information generator defines a graph that uses characters as nodes, connects characters appearing in the same scene to edges, and uses the frequency of simultaneous appearances as edge weights. A video analysis device that generates information about the person relationship that includes it.
4. The method of claim 3, wherein the device
a user interface for presenting a face-recognized character, selecting at least one character to be searched, and playing and displaying the retrieved frames; and
a frame search unit that searches for frames in which the selected character appears by using the bookmarking information;
Video analysis device further comprising a.
4. The method of claim 3, wherein the device
a user interface for presenting a face-recognized character, selecting at least one character to be searched, and playing and displaying the retrieved frames; and
a clip image editing unit for generating a clip image by editing frames in which the selected character appears by using the bookmarking information;
Video analysis device further comprising a.
In the video analysis method in which at least some are implemented as program commands executed in a video analysis device that receives and processes a video,
Separating each speaker's voice from the audio content of the video;
generating voice frame information for each speaker's voice;
distinguishing the characters by recognizing the faces of the people appearing in the video content of the moving picture;
generating face recognition frame information for the character;
matching the recognized face and voice by combining voice frame information and face recognition frame information; and
generating a character frame vector indicating whether or not a character appears with respect to the entire frame;
A video analysis method comprising a.
9. The method of claim 8, wherein the method
correcting the character frame vector as not appearing in the frame when the character appears in consecutive frames below the threshold value;
Video analysis method further comprising.
10. The method of claim 9, wherein the method
generating bookmarking information for frames in which characters appear;
Video analysis method further comprising.
11. The method of claim 10, wherein the method
calculating an appearance time and number of appearances for each character, and generating appearance information for each person including the appearance time and number of appearances;
Video analysis method further comprising.
11. The method of claim 10, wherein the method
generating character relationship diagram information including main characters and relationship diagrams between characters through graph analysis;
further comprising,
A graph is a video analysis method defined by using characters as nodes, connecting characters appearing in the same scene to edges, and using the frequency of simultaneous appearances as edge weights.
11. The method of claim 10, wherein the method
retrieving frames in which the selected character appears by using the bookmarking information;
further comprising,
The selection of the character to be searched is a video analysis method in which at least one person is selected through a user interface for selecting a character to be searched by presenting the face-recognized character of the video analysis device.
11. The method of claim 10, wherein the method
generating a clip image by editing frames in which the selected character appears using the bookmarking information;
further comprising,
The selection of the character to be searched is a video analysis method in which at least one person is selected through a user interface for selecting a character to be searched by presenting the face-recognized character of the video analysis device.

Images