KR100828166B1 - Method of extracting metadata from result of speech recognition and character recognition in video, method of searching video using metadta and record medium thereof - Google Patents

Abstract

A method for extracting metadata through speech recognition and caption recognition of a video, a video searching method using metadata, and a recording medium recording the same are provided to extract metadata information as a speech recognition result and an opened caption recognition result, reduce archiving work time by manual work, and enable broadcasting manufacturers to manufacture contents of good quality by automatically executing contents management and index work for mass broadcasting data. A method for extracting metadata through speech recognition and caption recognition of a video comprises the following steps of: inputting the video including the metadata, and extracting a start frame and a screen conversion frame of the inputted video(110); displaying the extracted start frame and screen conversion frame as a thumbnail image, and storing the displayed thumbnail image and time information of the thumbnail image(120); recognizing a voice of a speaker according to a phoneme of a voice included in the inputted video, and converting the recognized voice data into text data before extracting a keyword from the converted text data(130); extracting a caption through caption recognition from the inputted video(140); extracting metadata and title from the keyword and caption included in thumbnail images of a start shot, an end shot and between the start shot and the end shot if the start shot and the end shot are designated by a user in the start frame and the screen conversion frame of the displayed video(150); and displaying the extracted metadata, time information of the start shot, time information of the end shot and the title(160).

Description

Method of extracting metadata from result of speech recognition and character recognition in video, method of searching video using metadta and record medium kind}

1 is a flowchart of a metadata extraction method through voice recognition and subtitle recognition of a video according to an embodiment of the present invention.

2 illustrates a continuous speech recognition space searched to be applied to the present invention.

3 is a detailed flowchart of an operation (130) of extracting a keyword from the audio of the image of FIG.

FIG. 4 is a detailed flowchart of a step (step 140) of extracting captions through the caption recognition of FIG. 1.

FIG. 5 is a detailed flowchart of the start frame and screen change frame extraction step 110 of the video of FIG. 1.

FIG. 6 is a detailed flowchart of the step 150 of extracting the metadata and the title of FIG. 1.

7 is a flowchart illustrating a video searching method using metadata according to another embodiment of the present invention.

8 illustrates a graphical user interface environment of moving picture data stored in an XML document.

9 illustrates a search window of moving image data stored as an XML document.

The present invention relates to data extraction, and more particularly, to a method of extracting metadata through speech recognition and caption recognition of a video from a predetermined section or a frame of each subject, and to searching a video using metadata. It relates to a method and a recording medium recording the same.

In the flood of modern information, text-oriented materials with high readability and speed are changing to multimedia data. Accordingly, a lot of time and effort is required for processing a large amount of multimedia data.

In addition, as the number of users who are burdened with processing multimedia information that requires both eyes and ears has increased, the necessity of appropriate summary information has emerged, the necessity of summary information has emerged, and summary information providing techniques have been proposed.

For example, digital video content can be viewed and played in its entirety, but in some cases summary video highlights are automatically generated by the program provider or by the user's system itself so that the content can be understood without having to watch the entire video. It may also be provided in the form of creation.

The highlight video plays only a part of the stored video, and has a characteristic of representing the corresponding video stream. The highlight video is for storing or playing along a specific section of the video stream. When the user wants to select one of several videos stored in the digital video recorder for a limited time, the user highlights the highlight video of each video stream. By playing back only, you save time searching for the video content you want. In addition to the summary information of the stored video stream, the highlight may provide a preview function that can be used in a program guide device for selecting a video to be recorded by the user in the digital video storage device.

However, since it is necessary to extract a portion that is meaningful to represent the content of the video to the user, setting the section to generate highlights is a very difficult task.

According to Korean Patent Publication No. 0404322, which is a method of dividing a news video provided with a closed caption into news article units and compressing the news video by the divided news article units, a speech recognition method based on closed caption data is provided. Use The closed captions based on the closed captions used in the patent invention were used as a way to prevent the recognition rate of the voice recognizer from being lowered. However, since the closed captions need to be prepared first, the closed captions are generally recorded or closed captions are generated or stored. There is a problem in that it is not immediately applied to broadcast video that is not.

In addition, according to Korean Patent Publication No. 0540735, this is a method for retrieving an image frame by indexing the recognition result of each frame subtitle through the frame extraction and recognition of each frame subtitle. It connects the subtitle and the video frame containing the subtitle to find the position of the video containing the subtitle.It detects and recognizes the subtitle in the video, but the object is limited to the indexing of one video frame. Even if the subtitles are irrelevant, the subtitles may be included in the search target if the subtitles match.

In addition, according to a 1999 IEEE academic paper, "Automated generation of News content hierarchy by integrating audio, video and text information, 3025-3028p, Qian Huang et al., 4", the paper described in this paper, closed caption detection, facial recognition, anchor A method of providing search information about news through shot detection and speech recognition is proposed, but this is limited to closed captions.

As described above, in the conventional method of providing summary information, since a portion having a meaning that is representative of the content of the video must be extracted separately, the caption is detected and recognized in the set image of the section in which the highlight is to be generated, but the target of the image frame is detected. Even if the problem is limited to indexing, and subtitles that are not related to the overall article content are included, matching subtitles may be included in the search target, providing incorrect summary information, and providing summary information through closed captions. Since the fast recording for caption broadcasting should be prepared first, there is a problem in that the method is not directly applied to broadcast video in which recording, caption broadcasting is not generated or stored in general.

Accordingly, the first technical problem to be achieved by the present invention is to recognize a voice of a video which can extract a key word with voice recognition by recognizing the subtitle from the video information of the video regardless of the result of the voice recognition as well as the closed caption. It is to provide a metadata extraction method through caption recognition.

In addition, a second technical problem to be achieved by the present invention is to provide a video search method using metadata applying the metadata extraction method through the voice recognition and subtitle recognition of the video.

The present invention to achieve the first technical problem,

Inputting a video including metadata, extracting a start frame and a screen switching frame of the input video, displaying the extracted start frame and the screen switching frame as a nail image, and displaying the displayed nail image and the nail Storing time information of an image, recognizing a speaker's voice according to a phoneme of a voice included in the input video, converting the recognized voice data into text data, and extracting a keyword from the converted text data Detecting a caption from the input video and extracting the caption from the detected caption through caption recognition; when a user designates a start shot and an end shot among the displayed frame and the transition frame of the displayed video, Specified start shot, end shot and hand between the start shot and end shot Extracting metadata and a title from keywords and subtitles included in the video, and displaying the extracted metadata, time information of the start shot, time information of the end shot, and the title. It provides metadata extraction method through recognition and subtitle recognition.

The present invention to achieve the second technical problem,

Inputting a video, extracting a start frame and a screen switching frame of the input video to display the start frame and the screen switching frame as a nail image, and storing time information of the displayed nail image and the nail image; Recognizing the speaker's voice according to the phoneme of the voice included in the input video, converting the recognized voice data into text data, and extracting a keyword from the converted text data, subtitles from the input video. Extracting a caption from the detected caption through caption recognition; designating a start shot and an end shot among the start frame and the screen change frame of the displayed video to designate a designated video between the designated start shot and the end shot; Metadata and titles from keywords and subtitles contained within Extracting and displaying video data including the extracted metadata, time information of the start shot, time information of the end shot, and the title, storing the displayed video data as an XML document, and When a specific user inputs a search word to search, the present invention provides a video searching method using metadata, including outputting a storage time, a title, a playback length, and metadata of a video stored in the XML document.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

However, embodiments of the present invention illustrated below may be modified in various other forms, and the scope of the present invention is not limited to the embodiments described below. Embodiments of the invention are provided to more fully illustrate the invention to those skilled in the art.

The present invention automatically extracts metadata that can be used for search using video and voice recognition technology for videos with a large amount of various characters and audiovisual information.

This extracts each keyword by recognizing the voice and subtitle included in the video by playing the video, and then extracts the metadata through the fusion of the extracted key words.

As such, the extracted metadata may be stored as a file in XML format along with time information of the video and used as information on searching for articles in the video.

In this regard, the core technical field required for the present invention is to extract and fuse keywords from a result obtained through shot detection technology, speaker independent large vocabulary speech recognition technology, image caption detection and character recognition technology, and image and speech recognition. Technology.

The present invention begins with the playback of a moving picture. When the video is played, the start scene of the video and the main scenes to which the screen is switched are presented using the shot detection technology as the nail image on the screen along with the time information of the shot.

Basically, the shot detection technique takes an image frame once every 30 frames and compares a previously obtained image frame with a histogram distribution variation. When the variation value is larger than a predetermined value, it is determined as a screen switching frame. The screen transition frame acts as a criterion for each area of the audiovisual material including various videos, and clicks the nail image included in the screen to designate the start and end shots of the area to designate the video section.

For example, when the article content starts in the news, the start and last shots of the news article after the recognition of the voice and subtitles are completed using the news feature that displays the title caption for the article in the upper right corner of the first scene. In the case of designation, the time information corresponding to the starting shot was taken to find the subtitle recognition result of the news title subtitle included in the starting shot and displayed on the screen as the title of the article content. Through this, the main shots and titles can be expressed on the screen to understand the contents of the article at a glance.

The core areas of speech recognition technology required are speaker independent speech recognition technology and continuous keyword spotting technology. In order to recognize an unspecified number of sentences among audio information used in broadcasting, 'speaker independent speech recognition technology' is required, and 'continuous keyword recognition technology' is needed to extract key words for use as metadata from sentences. In the present invention, in order to develop a more reliable key word recognizer, in order to use a method of detecting key words through post-processing of transcription data obtained through large vocabulary continuous speech recognition, Apply a speech recognizer.

The present invention uses a speaker independent speech recognition technology capable of recognizing a plurality of unspecified voices in order to obtain reliable speech recognition performance from moving image data such as various news and entertainment programs. To this end, an acoustic model is generated from voice data pronounced by 1000 people of various birthplaces and designed so that recognition by any speaker is not a problem. In addition, the basic unit is a tri-phone, which is a context-dependent model that includes both front and rear phonemes of the current phoneme to reflect the modulation effects according to the context in order to improve the voice recognition performance. We use the continuous density hidden Markov model (continuous density HMM), which is a high performance speech recognition algorithm with no loss of compression, suitable for large vocabulary recognition systems.

In addition, in order to extract metadata from video data, a large vocabulary continuous word speech recognition technology is developed to convert speech data in sentence units into transcription data which is text data.

1 is a flowchart of a metadata extraction method through voice recognition and subtitle recognition of a video according to an embodiment of the present invention.

Referring to FIG. 1, first, a video including metadata is input, and a start frame and a screen change frame of the input video are extracted (step 110).

The video applied to the present invention may be a video including voice information and text information. When the video is input, a start frame of the video is first extracted. The image frame is stored for each image frame unit preset by the user.

The preset video frame unit may be set to 30 frames, which may be set and changed according to the quality of the video, the playback environment, and the user's embodiment.

Then, the image frame is stored in units of preset image frames, and if the variation value of the histogram distribution exceeds a threshold set by the user by comparing the histogram distribution of the previously stored image frame with the stored image frame. Extract the saved video frame as the screen change frame.

For example, in a video including a variety of news, the histogram distribution of the image is different between the anchor comment and the image information according to the anchor comment. If the contents of the video are different, the histogram distribution is different.

Therefore, after loading image frames for each image frame unit and extracting a screen change frame by comparing histogram distribution with previously stored image frames, individual information can be extracted according to the characteristics of the image.

The preset threshold may compare a screen switching time and a histogram change value of a past video such as various news, entertainment programs, dramas, and set a threshold as an average value of the histogram change value according to the screen change time.

Next, the extracted start frame and the screen change frame are displayed as a nail image, and time information of the displayed nail image and the nail image is stored (step 120).

That is, the screen switching frame is displayed as a nail image according to the start frame of the video and the change in the histogram.

The displayed nail image is stored by setting an end time of the nail image to be equal to a start time of the nail image and a start time of the next nail image.

In this way, by storing the time information in the nail image it is possible to separate each video.

Next, the speaker's voice is recognized according to the phoneme of the voice included in the input video, the recognized voice data is converted into text data, and a keyword is extracted from the converted text data (step 130).

Indeed, the language that appears in videos requires a large vocabulary that begins with a general conversation and includes technical terms. The size of the space to be searched and the search time required for the recognition are determined according to the unit of speech recognition. In the case of continuous word recognition of a large vocabulary such as broadcast news, a search space and a search time largely depend on which recognition unit is used.

For independent speech recognition of speakers, it is necessary to estimate the probability distribution of speech features of various speakers and use them in the recognition process. In general, the most commonly used algorithm for probability-based acoustic modeling is the Hidden Markov Model (HMM) algorithm.

There are various types of HMMs (DHMM, SCHMM, GMM, etc.), but the algorithm that maximizes the recognition performance while the environment of the implemented system is not influenced by the memory or CPU performance like the embedded system is continuous. The Continuous Density Hidden Markov Model (CDHMM) can be used as a distribution. Since it uses a continuous Gaussian distribution to represent the probability distribution in the state, the probability can be calculated for any feature value. In particular, when a feature is input, a continuous Gaussian mixture can be used to represent the probability distribution, so that the probability can be calculated for any feature value.

In the present invention, the acoustic model is generated through the CHMM algorithm in order to express the speaker's statistical acoustic model in more detail without quantization error. In addition, in order to model statistical characteristics independent of the speaker, an acoustic model may be generated in a tri-phone unit of phonemes considering contextual articulation effects from a voice database generated by a plurality of speakers.

In addition, the speaker's voice may be recognized in units of sentences by using a time gap between sentences.

For example, referring to FIG. 2, when a unit sentence of 'this pesticide is more than six hundred times higher than the reference value' is recognized in the Chinese herb, each word is composed of a phoneme string which is a subword unit. The phoneme, which is a subword, is composed of a tri-phone based subword node by substituting the CDHMM parameter of an already trained acoustic model. In addition, a word-to-word network is constructed by applying the word-to-word connection and the bi-gram value from the bi-gram which is the probability of each word pair from the input voice.

In this way, each word of the word network is extended to a tri-phone unit, which is a subword model, by referring to a pronunciation dictionary and an acoustic model to generate a whole search space.

Meanwhile, in order to extract keywords according to speech recognition, a database including key word candidate words, which is a method of determining the importance of an existing word, is constructed, and the word frequency (TF) and reverse literature for each word are constructed. The keyword extraction method can be used according to the inverse document frequency (IDF).

To this end, for example, a database of 30,000 words of key word candidate words is constructed by using voice transcription data that is converted from text to text in the past, and a section designated by a user using the configured key word candidate word database. The value multiplied by the word frequency calculated from the speech recognition result of can be used as a measure of the importance of the word.

Next, a caption is detected from the input video, and the caption is extracted from the detected caption through caption recognition (step 140).

In fact, the subtitles used in the video often contain a summary and a subject of the corresponding frame, so that most of the subtitles are likely to be used as metadata.

The caption recognition and caption recognition are performed through caption recognition on the restricted region, thereby obtaining caption recognition results.

Next, when a user designates a start shot and an end shot among the start frame and the screen transition frame of the displayed video, the specified start shot, the end shot, and keywords and subtitles included in the nail image between the start shot and the end shot. The metadata and the title are extracted (step 150).

When the user designates a start shot and an end shot of a desired section among the start frame and the screen transition frame extracted from the displayed video, the keyword is extracted from the specified start shot, the end shot, and the nail image between the start shot and the end shot. Extract metadata and titles from captions.

In detail, when there is a word that matches the keyword and the subtitle, the importance as metadata may be high, so that the matching word may be first extracted as metadata.

When there is a word that matches the keyword and the subtitle, the word may be weighted, the weighted word may be first extracted as metadata, and a plurality of words with the same weight may exist. In this case, the words of the metadata may be extracted in the order of 'in alphabetical order'.

The title may be set to a subtitle included in the designated start shot.

This is because the subtitle included in the start shot often includes key words in the actual video, and the title is set to the subtitle included in the designated start shot by reflecting this. It may be possible.

Finally, the extracted metadata, the time information of the start shot, the time information of the end shot, and the title are displayed to provide the user with video information on a specific section of the video (step 160).

3 is a detailed flowchart of an operation (130) of extracting a keyword from the audio of the image of FIG.

First, the speaker's voice is recognized from the input video (step 331).

The speaker independent speech recognition method and the continuous keyword recognition method may be applied to speech recognition applied to the present invention.

In order to recognize an unspecified number of sentences among audio information included in a video, a speaker independent speech recognition method is required, and a continuous keyword recognition method is required to extract keywords for use as metadata from sentences.

In the present invention, in order to extract more reliable metadata, a keyword is first detected from description data obtained through a large vocabulary continuous speech recognition method.

In order to generate a continuous recognition search space according to the continuous speech recognition method, a bi-gram based search space is generated, which generates a bigram by extracting a textualized vocabulary from an existing video.

And, to minimize the unnecessary hardware access time by converting the use of the memory created and allocated at run-time of the recognition system to static allocation for large vocabulary recognition, and minimizing unnecessary hardware access time, the table of exponential and log functions essential for probability calculation Through table-look ahead, complex mathematical operations can be precomputed and approximated to reduce computation speed.

Next, a voice recognition result section is set according to the extracted screen change frame (step 332).

As described above, when the screen switching frame is extracted, the extracted screen switching frame stores the image information until the next screen switching frame is extracted, so that the interval between the extracted screen switching frames can be set as the voice recognition result section. have.

As such, the keyword is extracted from the voice information of the voice recognition result section based on the extracted screen change frame.

Finally, a keyword is extracted from the speech information of the set speech recognition result section (step 333).

In the present invention, a method of extracting a keyword from speech information is based on the frequency of each statistical word. If a word is a word that frequently appears in the set speech recognition result section, it can be determined that the word is generally important. However, if a keyword is extracted using frequency only in words that are likely to appear in all speech recognition result areas, such as 'today' or 'to journalist.' There is a problem.

Therefore, considering this at the same time can be represented by Equation 1 below as TF-IDF (Term Frequency and Inverted Document Frequency).

In Equation 1, N denotes the number of all words existing in the configured database, n denotes the frequency of a specific word existing in the configured database, and p denotes the total words existing in the current video section to extract keywords. The number of and t refers to the frequency of the current video section of the specific word.

That is, according to Equation 1, the inverse of the frequency in the database composed of the word corresponding to the bibliographic frequency is taken, multiplied by the frequency in the video section, the frequency is low in the other video, the word having a high frequency in the video section Is extracted as a keyword according to TF-IDF.

FIG. 4 is a detailed flowchart of a step (step 140) of extracting captions through the caption recognition of FIG. 1.

Referring to FIG. 4, first, a caption candidate region is detected by performing Gaussian filtering and filtering in a color domain by preprocessing an input image (step 441).

That is, in order to detect the caption frame including the caption, the caption candidate region is detected in the sampled frame in consideration of the characteristic of forming a strong contrast ratio between the high frequency component of the caption and the background.

Here, a gradient operator of each frame is used to extract a strong high frequency component, and red, green, and blue having color information in each frame to prevent loss of color information. Operator for the RGB) component.

By using the color gradient operator, the high frequency component extraction and strong contrast ratio of the subtitles can be maintained.

This is because the color image can be used while preserving information on the color image having a three-dimensional domain, rather than using a differential operator in one-dimensional component after gray-level image conversion. Can show toughness.

From the derivative operation values obtained in each of three-dimensional (R, G, B) domains, components with strong contrast ratio can be extracted through vector operations, and subtitle components can be extracted while maximizing the background components through differential operations and vector operations. A feature image can be obtained.

Since the background having the feature of the subtitle may exist as described above, after differential operation is performed on the feature image again, the horizontal direction and the vertical direction are considered in consideration of an edge pair for the change of the feature image and the distance component of the subtitle. You can get the coordinate value of.

Next, the subtitle candidate region is verified (step 442).

This is because a subtitle candidate region may have a background having properties similar to those of characters or subtitles present in the background, and thus a verification process for the subtitle candidate region is required. The verification of the subtitle candidate area is to classify only the normal area (actually artificial subtitle area) and to extract only the subtitle candidate area accurately by separating the area that can be extracted as the actual subtitle and the text or other area existing in the background. As a result, the detection rate of the subtitle portion can be improved.

To verify the subtitle candidate area, it is not classified as a two-class subtitle area and a non-subtitle area, but a one-class problem that constitutes a class for the data domain using only the actual subtitle area. Analyze

That is, a subtitle candidate region for a normal region is formed through training, and an area within an image frame existing outside the region in which the subtitle candidate region is configured is considered an outlier to verify the subtitle candidate region. do.

Such verification of the caption candidate region can distinguish between an advantage of significantly reducing the number of training data and a character region including characters existing in a background very similar to an artificial caption.

Finally, the caption is recognized through the binarization of the detected caption area, character unit separation in the caption area, Korean consonant type classification and phoneme separation, and phoneme unit recognition steps (step 443).

Since the performance of character recognition depends on the performance of binarization, sophisticated binarization can be performed using a method of extracting a luminance component of a caption region and analyzing a histogram. For histogram analysis of luminance components, we apply smoothing with dynamic variance and dynamic filter size, and use the first and second peaks and valleys related to the background and subtitles to determine the first-order dynamic threshold. Obtain

In order to accurately binarize the boundary of the caption, a dynamic mask is generated in consideration of the caption height, and the secondary dynamic threshold is generated by analyzing the caption and background components in the mask.

Then, separation is performed according to the ratio of letter units in the binarized subtitle candidate area, which analyzes the height-to-width of the letter, and analyzes the projection value in the vertical direction near the letter boundary to separate the letter of the subtitle component. Perform

When consonants and vowels are separated from one letter, and the vertical projection is performed, the height-to-width is analyzed in case of splitting into two letters.

After separating the character units, classify the Hangul type by extracting the maximum, minimum, average, etc. of the projected values of the vertical and horizontal components as feature vectors, and perform the phoneme separation process with the information on the vowel and final components obtained after the Jamo collection type classification. Perform. The phoneme separation is performed by analyzing the maximum, minimum, and rate of change of the projection value, taking into account the vowel components of each type.

The center point is calculated from the separated consonants and vowels and projected to a normalized size to recognize each separated phoneme through a neural network using a distribution of a constant distance component and a direction component from the center point as a feature vector, and the recognized phoneme recognition. By combining the results, the captions are recognized in letter units, and the recognized captions are extracted.

FIG. 5 is a detailed flowchart of the start frame and screen change frame extraction step 110 of the input video of FIG. 1.

Referring to FIG. 5, first, an input video is played (step 511).

The input video may be a real time video and a recorded video. Preferably, the input video may include a video according to the MPEG-2 standard.

The video based on the MPEG-2 standard is a video standard with a high quality standard for a wide range of applications such as television broadcasting, telecommunications, audio and video equipment, etc., and 4 to 100 Mbps high quality standard considering the image quality of high definition television (HDTV). Because of the high utilization in the development of digital storage media such as image transmission, digital video disk (DVD), and software production, it can be easily incorporated into the present invention.

Next, the screen of the input frame of the input video is displayed as a nail image (step 512), and the image frame is stored for each frame unit set by the user (step 513).

Meanwhile, the frame unit set by the user may be arbitrarily set according to the dynamic change degree and the use state of the screen change of the image according to the image. Preferably, the frame unit set by the user is set to 30 frame units that can describe the movement of the image in more detail. Can be.

Next, the histogram distribution information of the stored image frame is obtained (step 514). The histogram is a graphical description of the division of brightness values in the image, with the horizontal axis representing the brightness values (0 to 255), with dark pixels to the left and bright pixels to the right. The vertical axis represents the ratio of pixels at each level of brightness. Therefore, by checking the histogram of the image frame, the user can measure the overall exposure of the image. Therefore, if the histogram is shifted to the left, the image is mostly composed of dark pixels and the image is dark. Looking at it, we can know the characteristics of the image.

Next, the histogram distribution of the stored image frame is compared with the histogram distribution of the immediately stored image frame to determine whether the change value of the histogram has a value greater than a threshold set by the user (step 515). If the change in the histogram distribution is greater than the threshold value, the stored image frame is displayed as a nail image (step 516). If the change in the histogram distribution is smaller than the threshold value, the image frame is loaded and stored for each frame unit set by the user. (Step 513).

As described above, in the video including the various news, the distribution of the histogram of the image is different between the anchor comment and the image information according to the anchor comment, and when the contents of the video are different, the distribution of each histogram is different.

Therefore, after loading image frames for each image frame unit and extracting a screen change frame by comparing histogram distribution with previously stored image frames, individual information can be extracted according to the characteristics of the image.

The preset threshold may compare a screen switching time and a histogram change value of a past video such as various news, entertainment programs, dramas, and set a threshold as an average value of the histogram change value according to the screen change time.

FIG. 6 is a detailed flowchart of the step 150 of extracting the metadata and the title of FIG. 1.

Referring to FIG. 6, first, a start shot and an end shot are designated among a start frame and a screen change frame of a video displayed as a nail image by a user (step 651).

 That is, the user designates a start shot and an end shot for extracting metadata among the screen change frame extracted as the change value of the histogram and the start frame extracted from the displayed video when the video is played.

 Next, if a start shot and an end shot of a desired section of the screen change frame are designated, time information of the designated start shot and end shot is obtained (step 652).

By acquiring the time information, data of the screen switching frame included between the designated start shot and the end shot are loaded together.

Next, a caption recognition result of the designated start shot is obtained (step 653). The caption recognition result may be based on the caption extraction and recognition process described above.

Finally, the caption recognition result of the obtained start shot is set as a title, and metadata of the designated start shot and end shot is extracted (step 654).

As described above, since the subtitle included in the start shot often includes key words in the actual video, the title is set to the subtitle included in the designated start shot by reflecting this. The title may be modified.

On the other hand, the extraction of the metadata is generated by fusing a subtitle and a keyword, if the subtitle is extracted in a sentence form can be extracted a subtitle word of the subtitle as a subtitle.

In general, nouns in sentences convey meaning by describing the behavior or state of an object. In other words, the main meaning to convey in the sentence is concentrated on the expression of nouns. Thus, it can be seen that the noun to be expressed in the sentence includes most of the key words of the sentence. In order to extract such subtitles, the subtitles extracted from the sentence are extracted through the morphological analysis.

In addition, while extracting nouns, unnecessary words may be removed with respect to components extracted as subtitles.

In the present invention, the subtitles are for the Korean subtitles, and may not be output as a key word for a word that may follow a number or a special symbol.

In other words, a 'unit expression' that has a part-of-speech noun that can follow a number but is not considered to be a key word in meaning may be excluded from the subtitle component.

7 is a flowchart illustrating a video searching method using metadata according to another embodiment of the present invention.

First, a video is input, the start frame and the screen change frame of the input video are extracted, the start frame and the screen change frame are displayed as a nail image, and the displayed nail image and time information of the nail image are stored. (710 courses).

Next, the speaker's voice is recognized according to the phoneme of the voice included in the input video, the recognized voice data is converted into text data, and a keyword is extracted from the converted text data (step 720).

Next, a caption is detected from the input video, and the caption is extracted from the detected caption through caption recognition (step 730).

Next, a start shot and an end shot are designated among the start frame and the screen transition frame of the displayed video to extract metadata and a title from keywords and subtitles included in the designated video between the designated start shot and the end shot. The extracted metadata, time information of the start shot, time information of the end shot, and moving image data including the title are displayed (step 740).

Processes 710 to 740 are the same as described above, and thus redundant description will be omitted.

In operation 750, the displayed moving image data is stored as an XML document.

Displaying the extracted metadata, the time information of the start shot, the time information of the end shot, and the moving picture data including the title, as an XML document by the user on a graphical user interface (GUI). Can be.

In addition, the XML document may include an XML document conforming to the standard of MPEG-7.

Referring to FIG. 8, the extracted metadata, the time information of the start shot, the time information of the end shot, and the moving picture data including the title may be confirmed on a graphic user interface (GUI). This makes it easy to import into an XML document.

Finally, when a specific user inputs a search word to search for, a storage time, a title, a playback length, and metadata of a video stored as the XML document are output (step 760).

Referring to FIG. 9, as shown in FIG. 9, when a specific user inputs a search word to search for, a web browser outputs information of a video having the same word as the search word as metadata, and through this, a specific user May know in advance information of a video corresponding to the search word, and may actually play a video corresponding to the output video information.

Each of the above steps according to the present invention can be implemented in a variety of software or hardware using a general programming technique will be readily appreciated by those skilled in the art.

And some steps of the invention may also be embodied as computer readable code on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, CD-RW, magnetic tape, floppy disks, HDDs, optical disks, magneto-optical storage devices, and carrier wave (eg, Internet It also includes the implementation in the form of). The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. Examples included in the above description are introduced for the understanding of the present invention, and these examples do not limit the spirit and scope of the present invention. It will be apparent to those skilled in the art that various embodiments in accordance with the present invention in addition to the above examples are possible. The scope of the present invention is shown not in the above description but in the claims, and all differences within the scope will be construed as being included in the present invention.

As described above, according to the present invention, the metadata information included in the broadcast content that is the result of the speech recognition and the open caption recognition of the video itself is automatically extracted, and the archiving time by manual operation of the new broadcast materials is performed. It is possible to shorten and automatically execute content management and indexing of huge amounts of broadcast data in the past so that broadcast producers can produce high-quality contents. Not only can it save money, but it also works with internet search sites that are widely applied. It also enables the searching of multimedia materials in real time and facilitates reading of past materials. To make it easier for users to use your content. It works.

Claims (16)

Inputting a video including metadata and extracting a start frame and a screen switching frame of the input video; Displaying the extracted start frame and the screen switching frame as a nail image and storing time information of the displayed nail image and the nail image; Recognizing a speaker's voice according to a phoneme of a voice included in the input video, converting the recognized voice data into text data, and extracting a keyword from the converted text data; Extracting a caption from the input video through caption recognition; When the user designates a start shot and an end shot among the displayed start frame and the screen transition frame of the displayed video, metadata from the keywords and subtitles included in the designated start shot, the end shot, and the nail image between the start shot and the end shot is displayed. Extracting a title; And And displaying the extracted metadata, the time information of the start shot, the time information of the end shot, and the title. The method of claim 1, The speaker's voice is, A method of extracting metadata through speech recognition and caption recognition of a video, characterized in that each sentence unit is recognized. The method of claim 1, Extracting the metadata and title, And if the words coincide with the subtitles, weighting the matching words, and first extracting the weighted words as metadata. Metadata extraction method through. The method of claim 1, The title is The metadata extraction method through the voice recognition and subtitle recognition of the video, characterized in that the caption of the start shot is set. The method of claim 1, Extracting the caption through the caption recognition, Detecting a caption candidate region from a frame of the input video and verifying the detected caption candidate region; And And performing caption recognition from the verified caption candidate region. The method of claim 5, wherein The subtitle candidate area is And a method for extracting metadata through speech recognition and subtitle recognition of the video, characterized in that it is detected by using a derivative operation value according to a color differential operator in a color domain after Gaussian filling to the input video. The method of claim 5, wherein The performing of caption recognition may include: Binarizing the caption candidate region using a first order dynamic threshold value and a second order dynamic threshold value; And Character-separated and phoneme-separated subtitles existing in the binarized subtitle candidate region, and then recognizing the phoneme-separated phonemes, and recognizing the subtitles by letter units according to the recognized phonemes. Metadata extraction method using speech recognition and subtitle recognition of video. The method of claim 1, Extracting the metadata and title, Extracting metadata by fusing the subtitles and keywords, and extracting a word that is a noun among the subtitles as a subtitle when the subtitle is extracted in a sentence form. How to extract metadata. The method of claim 1, Extracting a keyword from the converted character data, Constructing a database including key word candidate words for determining importance for words; And Extracting the keyword according to the frequency number of the converted character data in the video and the frequency number of the converted text data in the configured database. How to extract metadata. The method of claim 1, The video is a metadata extraction method through speech recognition and subtitle recognition of the video, characterized in that it comprises a video according to the MPEG-2 standard. Inputting a video, extracting the start frame and the screen switching frame of the input video to display the start frame and the screen switching frame as a nail image, and storing time information of the displayed nail image and the nail image; ; Recognizing a speaker's voice according to a phoneme of a voice included in the input video, converting the recognized voice data into text data, and extracting a keyword from the converted text data; Detecting a caption from the input video and extracting the caption from the detected caption through caption recognition; Specifying a start shot and an end shot among the displayed start frame and the screen transition frame to extract metadata and a title from keywords and subtitles included in the designated video between the designated start shot and the end shot, and extract the metadata. Displaying moving image data including data, time information of the start shot, time information of the end shot, and the title; Storing the displayed moving image data as an XML document; And Outputting a storage time, a title, a playback length, and metadata of the video stored as the XML document when a specific user inputs a search word to search for. The method of claim 11, Extracting the caption through the caption recognition, Setting a specific region of a frame of the video as a caption candidate region; Detecting characters included in an image frame with respect to the set caption candidate region; And Extracting a subtitle from the detected character, and extracting a word which is a noun from the extracted subtitle. The method of claim 11, Extracting a keyword from the converted character data, Constructing a database including key word candidate words for determining importance for words; And Extracting the keyword according to the frequency of the converted text data in the video and the frequency of the converted text data in the configured database. The method of claim 11, The XML document is a video search method using metadata, characterized in that it comprises a document according to the standard of MPEG-7. The method of claim 11, The title is The video search method using metadata, characterized in that the caption of the start shot is set. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 1 to 15.

Images