KR100502429B1 - Apparatus and method for detecting user specified actor by using audio and visual features in drama genre video contents - Google Patents

Abstract

The present invention relates to a retrieval apparatus and method using audio and visual information for detecting a specific person's appearance section in a video content of a drama genre, wherein motion information, color histogram, and silence interval are calculated in a video content image, and the obtained values are obtained. A scene section search unit searching for and providing a scene section in the video content by using the search section; A video analysis unit analyzing and providing audio feature information and video feature information with respect to the scene section provided from the scene section search unit; Find the index corresponding to the section including the frame number selected by the user, search the feature information of the section related to the found index, compare the searched feature information with the feature information of the other section, and then A search unit providing an index; A candidate section display unit configured to display a first scene corresponding to an index of a section of high similarity provided from the search unit; A video content player driven by a user and providing a specific scene desired by the user to the search unit through the user interface unit is provided. Therefore, the video content editing system can be used as a function of dividing the video content into meaningful scene sections.

Description

Searching apparatus and method using audio and visual information for detecting specific characters' appearance section in drama genre video contents {APPARATUS AND METHOD FOR DETECTING USER SPECIFIED ACTOR BY USING AUDIO AND VISUAL FEATURES IN DRAMA GENRE VIDEO CONTENTS}

The present invention relates to a retrieval device and method using audio and visual information for detecting a specific person appearance section in a video content of a drama genre, and more particularly, to an apparatus for retrieving a scene having a specific content from a video content of a drama genre; It is about a method.

Typically, the video content retrieval system is a technician of MPEG-7 established by the Video Experts Group MPEG, a working group of the Committee SC29 under the International Standard Organization / International Electrotechnical Committee (ISO / IEC) JTC1, a multimedia-related international standardization organization. This search system is being studied by the researchers.

In other words, a method of searching for a specific content in a video content retrieval system is a method of extracting signal and statistical characteristics of an input video image, providing the extracted characteristics to a video, and comparing the characteristics provided from the video by itself. In the case of searching using audio information, it is mainly used to classify video content genres.

However, when searching while giving meaning to a scene, such as a method of searching for a scene in which a specific person desired by a user who uses video content appears, it is impossible to extract only by comparing input video or audio information.

For example, when the video content is news, a face of a person is detected in the video content of a scene in which a specific person appears, and a scene of the same face is detected in the news or a video scene is detected using the same. A method of extracting a scene in which a person's name appears through a voice recognition system is currently being studied. However, in the case of drama genre, not news, there is a problem that it is not accurate to detect a person's face.

Accordingly, the present invention has been made to solve the above-described problems, the object of which is to provide a search method that can use not only visual feature information but also audio information at the same time that a specific person desired by the user while watching TV The present invention provides a retrieval apparatus and method using audio and visual information for detecting a specific person's appearance section in a video content of a drama genre that enables searching for a scene having a high likelihood of the same person appearing in a video content.

In the present invention for achieving the above object, a retrieval apparatus using audio and visual information for detecting a specific person's appearance section in the video content of the drama genre obtains the motion information, the color histogram, and the silence interval in the video content image. A scene section search unit searching for and providing a scene section in the video content using a value; A video analysis unit analyzing and providing audio feature information and video feature information with respect to the scene section provided from the scene section search unit; Find the index corresponding to the section including the frame number selected by the user, search the feature information of the section related to the found index, compare the searched feature information with the feature information of the other section, and then A search unit providing an index; A candidate section display unit configured to display a first scene corresponding to an index of a section of high similarity provided from the search unit; It is driven by the user, characterized in that it comprises a video content player for providing a specific scene to the search unit through the user interface to find.

In addition, in the present invention for achieving the above object, a search method using audio and visual information for detecting a specific person appearance section in the video content of the drama genre is the current frame and the previous frame in the specific scene of the video content in the scene section search unit. Obtaining a motion prediction reliability value by comparing the results, and then calculating the average of the motion information of each block for the entire image, and if the motion prediction reliability value is greater than or equal to the threshold, the scene transition frame is a frame in which scene transition may occur. Obtaining the reported color histogram and distance from the previous frame color histogram, if the distance between the color histogram obtained is more than the threshold, determining that the scene transition has occurred, and calculating the silence interval frequency, and the silence calculated by the scene interval search unit If the interval frequency is above the threshold, the first frame of the new interval Setting the indexes, indexing in order of time the sections that can be used for searching among the scene sections in the video analysis unit, calculating the LPCC of the audio in the scene sections, and performing vector quantization on the calculated LPCCs. And extracting and providing a codebook as feature information, providing a specific scene that the user wants to find in the video content player through the video content player, and a frame number selected by the user in the search unit. The user can select and view an index of a section having a high similarity by searching for an index corresponding to a section, searching for feature information of a section related to the found index, and comparing the searched feature information with feature information of another section. And displaying the data so as to display it.

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram for searching for a specific character appearance section from the video content of the drama genre according to the present invention, a scene section search unit 10, a video analysis unit 20, a search unit 30, The candidate section display unit 40, a user interface unit 50, and a video content player 60 are included.

The scene section search unit 10 is a block for analyzing video content. The scene section search unit 10 obtains motion information, a color histogram, and a silent section frequency in an image, and uses the video analyzer to determine a scene section that detects a scene change in the video content. 20) to provide.

Here, when the scene selected by the user does not include audio, the scene section search unit 10 records the section which cannot be used for searching when the silence interval is low and the movement is too fast.

The video analysis unit 20 may include audio feature information and video feature information, that is, motion information, color histogram, silence interval frequency, and linear prediction coefficient Cepstrum, for the scene section provided from the scene section search unit 10. The codebook obtained from the LPCC coefficients is extracted as the feature information and provided to the search unit 30.

Here, the video analyzer 20 divides the video content to be searched into several sections through the scene section search section 10, and extracts and stores audio visual feature information for each section.

The search unit 30 finds a section to which the user-selected frame belongs by using a frame number (or play time), sets it as the input video section of the search, and extracts the LPCC coefficients from the input video section and other sections in the video content. Using the similarity of the obtained codebook, N sections are selected, and for the selected N sections, M sections are selected in order of high probability using the codebook and color histogram information, and candidates for the first scene of the M sections selected by the user are selected. The section display unit 40 is provided.

The candidate section display unit 40 displays the first scene of the M sections provided from the searcher 40 so that the user can select and watch them.

The user interface 50 is a block that interfaces between the searcher 30 and the video content player 60, and provides the searcher 30 with a temporal position of a scene provided from the video content player 60.

The video content player 60 is driven by the user, and when a user comes to watch while watching a TV, an input device (not shown) such as a remote controller or a keyboard linked to the video content player 60 appears. ) Provides the temporal position of the scene selected by the user in the content, that is, the frame number or time, to the searcher 30 through the user interface 50.

With reference to the flowchart of FIG. 2, a search method using audio and visual information for detecting a specific person appearance section in video content of a drama genre according to the present invention will be described in detail with reference to the above-described configuration.

First, the scene section search unit 10 divides the video content into meaningful scene sections and is performed as a preprocessing operation. The scene section search unit 10 compares the current frame with the previous frame in the scene to obtain motion information (step 203). ).

Here, when the motion information is estimated from the current frame to the previous frame, the ratio of the number of blocks when the reliability of the motion prediction result is smaller than the threshold and the total number of blocks in the image is higher than the threshold and the difference in the color histogram is determined. If it is found and smaller than the threshold, it is regarded as a continuous scene.

Subsequently, the scene section search unit 10 obtains a silent section frequency for a section regarded as a continuous scene and records this as an unsearchable section when the value is larger than a predetermined threshold or the value of motion information is smaller than a predetermined threshold. The color histogram stored as the information is obtained not in the whole frame but in the region where the motion information exists.

That is, the motion information used in the scene section search unit 10 divides an image into blocks having a size of 16 × 16, and obtains a motion prediction reliability value for a block divided by a mean square error (MSE) method from a previous image. Then, the average of the motion information of each block is calculated for the entire image (step 204).

At this time, the formula for obtaining the motion information is expressed as in Equation (1).

Here, it is determined whether the motion prediction reliability value obtained for the block divided by the MSE method is lower than the threshold (step 205).

In the determination step 205, if the motion prediction reliability value is lower than the threshold, the motion prediction is regarded as an unreliable block, and the motion prediction reliability of the frame is obtained by dividing the number of unreliable blocks by the number of blocks in one image frame. Is obtained (step 206).

If the motion prediction reliability is greater than or equal to the threshold in the determination step 205, the color histogram is calculated as a frame in which scene transition may occur, and the distance of the color histogram of the previous frame is obtained (step 207). Here, the motion prediction reliability of the image frame is expressed as in Equation 2.

Next, when the distance between the color histograms is calculated, it is checked whether the calculated distance between the color histograms is lower than the threshold (step 208).

If the distance between the color histograms is lower than the threshold in the check step 208, the next frame is input (step 206).

If the distance between the color histograms is greater than or equal to the threshold in the check step 208, it is determined that the scene change has occurred, and the silence interval frequency is calculated (step 209).

After the silence interval value is calculated, it is determined whether the calculated silence interval is lower than the threshold (step 210).

If the silence interval is lower than the threshold in the determination step 210, the next frame is input (step 206).

If the silence interval is greater than or equal to the threshold in the determination step 210, it is determined that a scene change has occurred and is set as the first frame of the new interval (step 211).

Here, the formula representing the silence interval is expressed as Equation 3, and the silence interval is divided into speech frames of 20 msec, and then the number of speech frames and the total speech frames whose magnitudes of the speech signals are lower than the threshold. Shown as the ratio of numbers.

After the scene section search is completed, the video content is divided into sections, and the video analyzer 20 enters the section by section.

In addition, the video analyzer 20 measures the color histogram for each image frame by applying the formula of Equation 4 to the input section, and the distance between the previous frame and the color histogram of the current frame is expressed by the formula of Equation 5. Measure while applying (step 212).

The video analysis unit 20 indexes the sections of the scene sections detected by the scene section search section 10 that can be used for searching in chronological order, and converts the LPCC of the audio in units of 20 msec audio frames within the scene section. The calculated LPCCs are subjected to vector quantization, and 32 codebooks are extracted as feature information and provided to the search unit 30 (step 213).

At this time, the vector quantization uses the LBG algorithm, and the motion information calculated by the scene section retrieval unit 10 and the VQ codebook of the quiescent histogram, the silent section frequency, and the LPCC coefficient store the feature information of the scene section along with the section index. do.

Here, the LPCC coefficient of the audio may be obtained by applying the formula of Equation 6.

Meanwhile, when a user (eg, a specific person) comes to watch while watching TV using the video content player 60, the user selects the content from the user through an input device (not shown) such as a remote controller or a keyboard. The temporal position of the scene, that is, the frame number or time, is provided to the search unit 30 through the user interface unit 50 (step 214).

The search unit 30 finds an index corresponding to a section to which the frame selected by the user belongs, that is, a section including a frame number (or play time) (step 215), and uses the feature to identify an index related to the section. After retrieving the information (step 216), the index of the section having high similarity is output to the candidate section display unit 40 by comparing it with the feature information of the other section (step 217).

That is, the search unit 30 compares the codebook of the section including the input frame number with the codebook of the other section and arranges N pieces in the order of high similarity, and the similarity between the codebook sets is determined by applying the formula of Equation 7 below. do.

Thereafter, the search unit 30 measures similarity with the input section using the color histogram information on the selected N sections. In this case, the similarity of the color histogram is measured for the sections except when the similarity between sets of codebooks exceeds the threshold, and M sections are selected in order of high similarity, and the first scene of the M sections selected by the user is displayed as a candidate section display unit ( 40). Where M is an integer smaller than N and greater than zero.

The candidate section display unit 40 displays the first scene of the M sections provided from the searcher 40 so that the user can select and watch them (step 218).

As described above, the present invention provides a search method that can use not only visual feature information but also audio information at the same time, when a user designates a scene in which a specific person wants to appear while watching a TV, the same person in video content. By searching for scenes that are likely to come out, the video editing system can be used as a function of dividing video contents into meaningful scene sections.

1 is a block diagram for searching for a specific character appearance section from the video content of the drama genre according to the present invention,

2 is a detailed flowchart illustrating a retrieval method using audio and visual information for detecting a specific person appearance section in video content of a drama genre according to the present invention.

<Description of the symbols for the main parts of the drawings>

10: scene section search unit 20: video analysis unit

30: search unit 40: candidate section display unit

50: user interface unit 60: video content player

Claims (19)

In a search apparatus using audio and visual information in the video content of the drama genre, A scene section search unit which obtains motion information, a color histogram, and a silent section frequency in the video content image, and searches for and provides a scene section in the video content using the obtained value; A video analysis unit analyzing and providing audio feature information and video feature information with respect to the scene section provided from the scene section search unit; Find an index corresponding to a section including a frame number selected by the user, search for feature information of a section related to the found index, and compare the searched feature information with feature information of another section to have a high similarity. A search unit providing an index of a section; A candidate section display unit configured to display a first scene corresponding to an index of a section having high similarity provided from the search unit; And a video content player driven by the user and providing a specific scene desired by the user to the search unit through a user interface unit. Search device using visual information. The method of claim 1, If the scene selected by the scene section search unit does not include audio, the video content of the drama genre is recorded in a section in which the silence interval is low and the movement is too fast and cannot be used for searching. Retrieval device using audio and visual information for detecting a specific character appearance section in. The method of claim 1, The feature information is extracted from the video content of the drama genre, which is extracted by motion information in a scene section, a codebook obtained from a color histogram, a silent section frequency, and a linear prediction coefficient coefficient (LPCC) coefficient. Search apparatus using audio and visual information for section detection. The method of claim 1, The video analysis unit divides the video content to be searched into several sections through the scene section search unit, and extracts and stores audio visual feature information for each section. Retrieval device using audio and visual information for the appearance section detection. The method of claim 1, And a user searches for a video by selecting a scene in which a specific person appears and searches for a video in a drama genre of video content. The method of claim 4, wherein Retrieval apparatus using audio and visual information for detecting a specific person's appearance section in the video content of the drama genre, characterized by selecting the detectable scene section using the motion information and the silent section frequency in the scene section divided into several sections . In the retrieval method using a scene section search unit, a video analysis unit, a video content player, a search unit, and audio and visual information in the video content of the drama genre, Obtaining a motion prediction reliability value by comparing a current frame in a specific scene of the video content with a previous frame in the scene section searcher, and then obtaining an average of motion information of each block for the entire image; When the motion prediction reliability value is greater than or equal to the threshold value, the scene section searcher calculates a color histogram as a frame capable of scene change and obtains a distance from the color histogram of the previous frame, and if the distance between the color histograms is greater than or equal to the threshold value, After determining that a transition has occurred, calculating a frequency of silence intervals, Setting the first frame of a new section when the calculated silence interval is greater than or equal to a threshold by the scene section search unit; The video analysis unit indexes the sections that can be used for searching among the scene sections in time order, calculates the LPCC of the audio within the scene sections, and performs vector quantization on the calculated LPCCs. (codebook) extracting and providing the feature information, Providing a specific scene that the user wants to find in the video content player through the video content player; The searcher searches for an index corresponding to a section including the frame number selected by the user, searches for feature information of a section related to the found index, and then searches between the searched feature information and feature information of another section. Displaying the indexes of the similarity sections by comparison so that the user can select and view them. Search method using audio and visual information for detecting a specific character appearance section in the video content of the drama genre comprising a. The method of claim 7, wherein When the motion information is estimated from the current frame to the previous frame, the ratio of the number of blocks when the reliability of the motion prediction result is smaller than the threshold and the total number of blocks in the image is higher than the threshold, and the difference between the color histogram is obtained. A search method using audio and visual information for detecting a specific character appearance section in a video content of a drama genre, characterized in that it is regarded as a continuous scene when smaller than a threshold. The method of claim 7, wherein The motion information is equation 1 A search method using audio and visual information for detecting a specific character appearance section in a video content of a drama genre, characterized in that it is expressed and calculated. The method of claim 7, wherein The motion prediction reliability of the image frame is Equation 2 A search method using audio and visual information for detecting a specific person appearance section in a video content of a drama genre, characterized in that it is expressed and calculated. The method of claim 7, wherein The formula representing the silence interval is Equation 3 The silence interval frequency is divided into 20 msec speech frames, and the silence interval frequency is expressed as a ratio of the number of speech frames with a lower speech threshold and the total number of speech frames. Retrieval method using audio and visual information for detecting a specific character appearance section in video content. The method of claim 7, wherein The color histogram is Equation 4 Each frame is measured by applying the formula of. The distance between the color histogram of the previous frame and the current frame is Equation 5 A search method using audio and visual information for detecting a specific character appearance section in a video content of a drama genre characterized by measuring while applying the formula of. The method of claim 7, wherein The vector quantization uses an LBG algorithm, and the VQ codebook of the motion information, the chelator histogram, the silence interval, and the LPCC coefficients calculated by the scene interval retrieval unit stores the feature information of the scene interval along with the interval index. A search method using audio and visual information for detecting a specific person appearance section in video content of a drama genre. The method of claim 13, The LPCC coefficient is Equation 6 Searching method using audio and visual information for detecting a specific character appearance section in the video content of the drama genre, characterized in that can be obtained by applying the formula of. The method of claim 7, wherein The codebook for extracting the feature information and the codebooks of different sections are compared with each other, and an arbitrary number is sorted in ascending order of similarity. Equation 7 A search method using audio and visual information for detecting a specific person appearance section in a video content of a drama genre, characterized by determining by applying a formula of. The method of claim 7, wherein If the motion prediction reliability value is lower than the threshold, the motion prediction is regarded as an unreliable block, and the motion prediction reliability of the frame is obtained by dividing the number of the unreliable blocks by the number of blocks in one image frame. Retrieval method using audio and visual information for detecting a specific character appearance section in the video content of the drama genre characterized in that it comprises. The method of claim 7, wherein And if the distance between the color histograms is lower than a threshold value, inputting a next frame. The method of claim 1, further comprising: inputting a next frame. The method of claim 7, wherein And if the silence interval is lower than a threshold, inputting a next frame. The method of claim 1, further comprising: inputting a next frame in the video content of the drama genre. 19. A recording medium on which a program implementing the method of any one of claims 7 to 18 is recorded.