JPH0668168A - Video retrieval method and device by acoustic keyword - Google Patents

Abstract

PURPOSE:To provide the video retrieval method and device by means of an acoustic and voice keyword from which a required video image is retrieved with the keyword as a retrieval key. CONSTITUTION:An acoustic information analysis section analyzing a characteristic of acoustic information and converting the result into an acoustic characteristic parameter time series, and a keyword characteristic analysis section 8 analyzing the characteristic of the keyword sound being the video retrieval key and converting the result into a characteristic parameter time series of the keyword sound are provided to database sets 2, 3 storing video information and acoustic information corresponding to the video information and the sound included in the acoustic information is used for the keyword to retrieve the video information by comparing the acoustic characteristic parameter time series in the database with the characteristic parameter time series of the video retrieval key.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for retrieving an image by using an audio keyword, and more particularly, regarding a database having audio information and image information among audio information, an image using a word included in the audio information as a keyword. The present invention relates to a video search method and apparatus using audio keywords for searching information.

[0002]

2. Description of the Related Art Computers that have traditionally handled only numerical and character data as data due to advances in audio processing technology, video processing technology and other signal processing technologies,
In computers such as workstations, it has become possible to perform multimedia processing in which audio information and video information are also handled as data (the following description will be made by focusing on audio as a representative of audio). As the data compression technology has advanced and the capacity of storage devices has increased, the amount of data to be handled has increased. When handling and accumulating a large amount of various kinds of data, it is demanded that necessary information be retrieved quickly and easily.

As a method for retrieving information, generally, a method is used in which an index is added to information in advance and the indexed portion is taken out. For text data that handles character strings, matching can be performed by matching between specified keywords and data, as represented by a character string search, so that matching points can be indexed, making retrieval easy. Is.
However, in the case of information that is continuous and whose contents change with time, such as video information, it is very difficult to find a certain shape and add an index to the video portion. For example, even if the image information is different from each other, it is necessary depending on the method of searching by paying attention to the commonality of the image parts if the information content has common meaning or related information. I can't find the perfect image.

Regarding the information retrieval of video, the following measures have hitherto been adopted. 1. Sequential search This is a method of searching for a required video by continuously playing back audio and video that are continuous information simultaneously. 2. Utilize correspondence between time code First, the audio and video information is reproduced from the beginning, and at that time, the time code is associated. After that, it is a method of taking out the necessary image by using the time code as a clue.

3. Image INDEX Dividing the moving image into several sections by paying attention to the change point of the camera position (change of scene) and the change of a specific object included in the moving image from the moving image which is continuous data. After that, the part with the least change in each section, the part with the largest change, or the first video of the section is used as an index of the section's representative video, and this is used as a clue to extract the necessary video. .

[0006]

However, the above method has the following problems. 1'sequential search Basically, the search method is to reproduce the original data as it is, and no indexing or coding is performed.
It takes time to retrieve a large amount of information.

Utilizing correspondence with 2'time code Originally, the time code is a symbol that is not directly related to the audio and video information, but the user is always a meaningless symbol.・ We must be aware of the relationship between the code and the corresponding video. 3'image INDEX Depending on the type of video, there are few scene changes or little movement changes, and it is difficult to find the change point. Therefore, there is a problem that unnecessary fine indexes are added or conversely, coarse indexing is performed.

The present invention solves the above-described problems of the conventional video search method and apparatus, and makes it possible to add an index to a video, which has been difficult in the past, and to easily perform a video search. The present invention provides a video search method and device using audio keywords.

[0009]

SOLUTION: To perform a feature analysis of voice information in a database having video information and voice information corresponding to this video information, convert this into a voice feature parameter time series, and use it as a video search key. It is included in the audio information by performing a feature analysis of the keyword voice, converting this into a keyword voice feature parameter time series, and comparing the voice feature parameter time series in the database with the video search key feature parameter time series. A video search method using voice keywords for searching video information using a word as a keyword is configured, and the voice information is feature-analyzed for databases 2 and 3 having the video information and the voice information corresponding to the video information. The audio information analysis unit 6 for converting the characteristic parameters of the A keyword feature analysis unit 8 for analyzing the feature of the word voice and converting it into the feature parameter time series of the keyword voice is provided, and compares the voice feature parameter time series in the database with the feature parameter time series of the video search key. By doing so, a keyword segment extraction unit 1 for extracting a portion corresponding to a keyword from voice information
0 is provided, an index assigning unit 12 that stores an address of video information corresponding to the extracted section is provided, and a search video display unit 14 that displays the provided index and an image corresponding to the section is provided. An audio / video output unit 15 that simultaneously reproduces the video and audio of the search result is provided, and an audio keyword-based video search device that searches the video information using the words included in the audio information as keywords is also configured.

[0010]

[Example] Regarding a database having audio information and video information, a word included in the audio information is used as a keyword,
Video information is searched by adding an index to the video information. The audio information is continuous information like the video information and is information that changes with time, but in a normal conversation, a keyword word appears many times. Therefore, it is possible to use the keyword included in the audio information to extract the video of the portion in which the keyword appears. According to this method, even if the screen has different contents as the video information, the search can be performed even when the audio information contents are related. The present invention uses a voice of a keyword to extract a utterance of a keyword present in the voice information in order to search a predetermined keyword in a database having voice information and video information, and extract the extracted keyword. Is used as an index, and the video corresponding to the index is displayed on the display device to search the video information by voice.

An embodiment of the present invention will be described first with reference to FIG. FIG. 1 is a flow chart showing a configuration for inputting audio / video information, associating audio and video, and analyzing audio information. The present invention has two steps of inputting audio / video information and associating them with each other, analyzing audio information and converting it into audio characteristic parameters, and performing video search. It can be roughly divided.

The audio / video input unit 1 inputs audio information and video information from the outside, and the input audio information and video information are stored in the audio data storage unit 2 and the video data storage unit 3, respectively. Information is generally input via two media such as a microphone and a camera, in which audio information and video information are independent, but like a video tape recorder in which audio information and video information are recorded together. It is also possible to input from an external storage medium.

The voice information is converted into digital information and stored in the voice data storage unit 2 so as to be processed by a computer. Considering voice quality, the sampling frequency for digitization is preferably 8 KHz or higher.
In the following description, the audio sampling frequency is 12
It shall be KHz. The voice data is stored in the voice data storage unit 2 with a sampling number.

In contrast, video information does not require computer processing after it is input. However, since it is necessary to retrieve the information later, it is stored in the video data storage unit 3 in association with time information so that random access is possible. As a concrete method, there are a method of digitizing and fetching it in a computer and storing an address as an index, a method of storing it together with a time code in a video tape recorder, and the like. Hereinafter, a case where the frame number is used for searching will be described.

When considering the input of audio information and video information, the audio information is 8.3 in the case of 12 KHz sampling.
It should be noted that there is one sampling per microsecond and the image is usually composed of 30 screens per second. Since the input audio information and video information are stored independently, it is necessary to associate the two in order to add the index to the video using the keyword of the audio information. There are a method of storing correspondence as a calculation expression, a method of creating a correspondence table, and other methods for obtaining correspondence. Less than,
The case of creating the correspondence table will be described as an example.

The audio / video correspondence unit 4 creates an audio / video correspondence table 5 showing the correspondence between the audio and the video. Here, if the data stored in the audio data storage unit 2 and the video data storage unit 3 are both accessible by using the time, the audio / video correspondence table is unnecessary in this case.
Here, the video information is composed of 30 frames (screen) per second and can be accessed using the frame number as an index, and the audio information stores the number of audio sampling data (sampling number) as an index. This will be explained as an example. In this case, the number of audio samples is 400 for one frame of video information.

FIG. 2 is an example of an audio / video correspondence table showing the correspondence between the video frame number and the audio sampling number. In this example, the start and end values of the audio sampling number corresponding to each video frame number are audio and
It is stored in the video correspondence table. Speech information analysis unit 6
Converts the voice digital data stored in the voice data storage unit 2 into a form in which a portion matching the keyword can be extracted later. The form of conversion depends on the algorithm that detects the part that matches the keyword, but the method of performing spectrum analysis is often adopted. Fast Fourier Transform and LP for spectral analysis
The technique of C analysis can be used. Further, when performing the spectrum analysis, the digital audio data having a predetermined frame length is repeatedly processed while being shifted by a predetermined frame shift length. As a result, the time series of the voice spectrum is stored in the voice feature parameter storage unit 7.

FIG. 3 is a flow chart showing a structure for searching a video by a keyword according to the present invention. The keyword feature analysis unit 8 analyzes the voice of a keyword that should be a search key and converts it into a feature parameter time series. The analysis method in this case is the same as the analysis method of the voice information in the voice analysis unit 6. The result analyzed by the keyword feature analysis unit 8 is stored in the keyword feature parameter storage unit 9. The keyword is a word included in voice information or a word such as conversation, but there are two possible ways to input the keyword. The first method is a method of extracting a keyword from the input voice information and using it, and the second method is a method of newly inputting a keyword. In the case of extracting from the input voice which is the first method and using it as a keyword, it is necessary to realize an interface for instructing the corresponding portion of the input voice. This can be done, for example, by displaying the audio power or by displaying the spectral sequence. Further, in the case of newly inputting in the second method, it is desirable to input the voice of the same person as the speaker of the voice information in order to accurately extract the keyword section from the voice information.

The keyword section extraction unit 10 extracts the keyword section in the voice information by comparing the characteristic parameter 7 of the voice information included in the data with the characteristic parameter 9 of the keyword. Here, the word spotting technique, which is generally an applied technique of speech recognition, can be used. Although various methods have been proposed as word spotting methods, basically, the method is to search the matching section of the voice information while sequentially shifting the characteristic parameters of the keyword. In this case, in order to correct the temporal expansion / contraction of the voice, it is common to employ a DP matching method or another matching method that enables nonlinear expansion / contraction. In addition, HMM (Hidden Markov model: Hidden)
It is possible to use various speech recognition and word spotting methods as well as a method of searching the corresponding section by Markov Model).

When the keyword section extraction unit 10 extracts the correspondence between the voice information and the keyword, the keyword sampling storage unit 11 stores the voice sampling number of the start point of the voice information of the matched section. The comparison between the voice information and the keyword is continuously performed for all the voice information, and the voice sampling numbers for all the extracted keyword sections are stored in the keyword section storage unit 11.

The index assigning unit 12 obtains the frame number of the video information by referring to the audio / video correspondence table 5 with the audio sampling number stored in the keyword section storage unit 11. This frame number is stored as the index position data 13. FIG. 4 shows an example of storing index position data. Here, a reference example of the audio / video table 5 will be described with reference to FIG. In this example, the keyword section storage unit 11 has a voice sampling number 10
25 is stored. The audio sampling number 1025 is the audio sampling number 801 at the start of the frame and the audio sampling number 1200 at the end of the frame of the video frame number 3 on the audio / video correspondence table 5.
It is a number located between. Therefore, it can be seen that the audio sampling number 1025 is included in the video frame 3.

In this way, the video information frame when the keyword of the audio information appears can be designated, and this frame number is stored as the index position data 13. In the example of FIG. 5, frame number 3 is stored as index position data. Search video display section 1
4 displays a representative screen as a search result based on the index position data 13. The selection method of the representative screen is to use the first screen of the video corresponding to the keyword sound, consider the keyword and the section of the keyword as one segment, and extract the video part with the least movement from that segment. Using this as the representative screen, extracting the part with the largest change in the segment and using this as the representative screen, consider that one section of speech containing the keyword is one block and move the most from this block. There are various methods such as a method of extracting a portion having a small number of pixels and using this as a representative screen. Which of these methods is selected is determined according to the type of the target video. For example, in the case of sports, the part that changes the most,
In the case of landscape, selecting the part with the smallest change enables effective display of search results. Whichever method is adopted, one representative video frame is selected and displayed on the screen based on the video frame to which the index is added. When the first screen of the video corresponding to the keyword sound is selected, the video frame number of the index position data 13 is used as it is to retrieve the video from the video data storage unit 3. In the case of extracting a video part having the least motion in the segment using the keywords as segments, the difference in video change from a plurality of video frames sandwiched by two frame numbers of consecutive index position data 13 is small, or Choose a big one in some cases. In the case of a method of extracting a representative screen from a section that considers one section of speech including a keyword as one block
Refocusing on the audio information, the continuous audio part including the keyword is regarded as an audio block, and a plurality of video frames corresponding to the audio block are selected so as to have a small difference in video change.

As the method of displaying the selected representative screen, one representative image is displayed on the entire display and the representative images are successively displayed by the keyboard operation of the operator, a method of displaying a plurality of representative images side by side on the display, etc. There is a method. In any case, the search video display unit 14 is configured to display the search result so that the video desired by the operator can be selected from the plurality of video information to which the indexes are added.

The video / audio output unit 15 is the search video display unit 1.
Based on the representative screen displayed in 4, the video information and the audio information are output and reproduced at the same time. The output information is the search video display unit
The method of allowing the operator to select the video information to be reproduced from the plurality of representative videos displayed in 4, and the video information displayed in the search video display section 14 is repeatedly reproduced and stopped by the operator's keyboard operation as a trigger. There is a method of comprehensively playing the representative video of. The video / audio output unit 15 obtains the video frame number and the audio sampling number corresponding to the frame by referring to the audio / video correspondence table based on the index position data derived from the representative screen of the search video display unit 14. , The target audio information and video information are the audio data storage unit 2 and the video data storage unit 3.
Both are output and reproduced at the same time. FIG. 6 shows an example in which the value of the audio sampling number 801 at the start of the frame is obtained by referring to the frame number 3 as the video information and the audio / video correspondence table as the audio information, and the video / audio output unit reproduces.

Generally, when considering information retrieval in a database having video information and audio information, a person may remember a certain information portion and try to retrieve the information. There are three types of this storage: video information, audio information, and both stored. The present invention is extremely effective in the case of retrieving video information based on the memory by voice among these three ways.

[0026]

As described above, it is possible to search for a video that requires a keyword included in audio information as a search key from a multimedia database having video information and audio information. This makes it possible to efficiently search and access video information, which is continuous information.

[Brief description of drawings]

FIG. 1 is a flowchart showing a configuration in which audio / video information is input, audio and video are associated with each other, and audio information is analyzed.

FIG. 2 is an audio / video correspondence table showing correspondence between video frame numbers and audio sampling numbers.

FIG. 3 is a flowchart showing a configuration for searching a video based on a keyword.

FIG. 4 is a table of index position data.

FIG. 5 is an example of referring to an audio / video correspondence table when assigning an index.

FIG. 6 is an example of referring to an audio / video correspondence table when outputting an audio / video.

[Explanation of symbols]

 2 audio data storage unit 3 video data storage unit 6 audio information analysis unit 8 keyword feature analysis unit 10 keyword section extraction unit 12 index assignment unit 14 search video display unit 15 audio / video output unit

Claims (2)

[Claims] 1. A feature analysis of audio information on a database having video information and audio information corresponding to the video information, converting this into a time series of characteristic parameters of the audio, and characterizing a keyword audio that should be a video search key. By analyzing and converting this into a keyword audio feature parameter time series, by comparing the audio feature parameter time series in the database with the video search key feature parameter time series,
A video search method using audio keywords, wherein video information is searched using audio included in audio information as a keyword. 2. An audio information analysis unit for performing a feature analysis of the audio information on a database having image information and audio information corresponding to the image information and converting this into an audio characteristic parameter time series. It is equipped with a keyword feature analysis unit for performing a feature analysis of the keyword sound to be used and converting it into a feature parameter time series of the keyword sound, and the feature parameter time series of the sound in the database and the feature parameter time series of the video search key. A keyword section extraction unit that extracts a portion corresponding to a keyword from the audio information by comparison is provided, and an index addition unit that stores the address of the video information corresponding to the extracted section is provided. And a search video display unit that displays the video corresponding to that section, and the video and audio of the search results are To play comprises a sound and video output unit, a video search apparatus according to an acoustic keywords, characterized in that retrieving the image information as a keyword audio included in the audio information.