KR101069534B1 - Method and apparatus for searching voice data from audio and video data under the circumstances including unregistered words - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for searching audio data of audio and video in an unlimited word environment. The present invention is largely comprised of an index unit and a search unit. An index unit comprising: a voice data extractor for extracting voice data from audio and video; A voice data divider for dividing the voice data into an appropriate amount estimated in sentence units to be executed in the speech recognizer and storing the voice data in the form of a wave file; A voice recognizer that receives a voice wave file as an input and outputs the voice wave file as a text data file; It includes an indexer for generating an index table in various units using the grid information resulting from speech recognition. The search unit may include: a query expander that expands a user's query into various possible queries to use an index table in consideration of non-registered words in speech recognition; A searcher for performing a search using an index table from the expanded query; It includes a result writer that effectively displays the retrieved results to the user.

Voice search, video search, unlimited words, unregistered words

Description

Method and apparatus for searching voice data from audio and video data under the circumstances including unregistered words}

The present invention relates to a method and apparatus for retrieving voice data of audio and video in an environment including non-registered words. In particular, the present invention converts voice data into text information using voice recognition technology and generates an index table from an indexer. The present invention relates to a method and apparatus for retrieving speech contents in consideration of unregistered words in.

In recent years, with the rapid development of the Internet, materials in the form of audio and video have increased rapidly. In order to manage and use such data more efficiently, retrieval technology is essential.

However, it is impossible to precisely search only by brief information on the title or content provided through manual operation, and the user often needs to examine and judge the audio and video. Content-based retrieval using voice data, which plays an important role in audio and video, can minimize user inconvenience.

In "A system and method for generating indexing information of a multimedia data file using voice data and a system and method for retrieving the indexing information of a multimedia data file" published on August 18, 2008, the publication number 10-2008-0075266 A voice data retrieval system and method are described, and in the publication No. 10-2008-0068844 published on July 24, 2008, "Method of indexing and retrieving a voice document with text metadata, a computer readable medium" The metadata that comes with the document refers to the use of existing searches.

However, this conventional technique does not consider unregistered words. That is, a search word that is not included in the dictionary defined by the speech recognizer has a limitation in that it does not show any result even if such a content is included in the actual speech document.

The present invention is to solve the problem according to the unregistered words of the speech recognizer, and to consider the lower level with the word form in indexing the speech recognition results, and by expanding to match the index table when the unregistered words appear in the search An object of the present invention is to provide a method and apparatus for retrieving audio and video voice data in an environment including non-registered words.

The object of the present invention described above is to extract speech data from audio and video, and to divide it into an appropriate amount estimated in sentence units, store it in the form of a wave file, and then convert the data into a text data file through a speech recognizer. This can be achieved through the process of displaying the result effectively to the user by extending the search to utilize the index tables for a given user query by referring to the index table created by the.

More specifically, according to one aspect of the present invention, in a voice data retrieval apparatus including an index unit and a retrieval unit,

The index unit extracts the voice data 220 from the audio and video voice data extractor 210; A voice data divider 230 for dividing the voice data 220 into an appropriate amount estimated in sentence units for performing in the voice recognizer 250 and storing the voice data 220 in the form of a wave file 240; A voice recognizer 250 for receiving a voice wave file as an input and outputting the voice wave file in a text data file form 260; An indexer 270 for generating the index table 280 in various units using the grid-shaped information resulting from the speech recognition 260,

The search unit includes a query expander 110 which expands a user's query into various possible queries to use an index table in consideration of non-registered words in speech recognition; A searcher 120 for performing a search using an index table from the extended query 21; The present invention provides an audio and video voice data retrieval apparatus in an environment including an unregistered word including a result outputter 130 that effectively displays the retrieved result 30 to a user.

Advantageously, the speech data retrieval device uses a compressed representation that combines as many as possible overlapping portions of the grid based on time information for a system that is robust to errors in the speech recognizer, to generate a word, Create an index table that contains information such as time information and probability values.

Preferably, the voice data retrieval apparatus generates an index table by converting the lattice into syllables and phonemes, which are lower units than the combined morphemes, in order to solve the problem of unregistered words of the speech recognizer.

Preferably, the voice data retrieval apparatus utilizes various index tables by extending the query in the form of a combination of units when it is determined that the keyword to be searched by the user is a non-registered word in the voice recognizer.

Preferably, the voice data retrieval apparatus proceeds bidirectionally from the first appearing portion of the combined morpheme unit for faster retrieval, and checks whether the distance between adjacent units is within a predetermined time.

According to another aspect of the present invention, there is provided a voice data retrieval method comprising an indexing step of an indexing unit and a searching step of a searching unit.

The indexing step may include: extracting voice data from audio and video; A voice data dividing step of dividing the voice data into an appropriate amount estimated in units of sentences to be performed in a voice recognition step and storing the voice data in the form of a wave file; A voice recognition step of receiving a voice wave file as an input and outputting the voice wave file as a text data file; An indexing step for generating an index table in various units using the grid-shaped information resulting from speech recognition,

The searching step may include: a query expansion step of expanding a user's query into various possible queries to use an index table in consideration of an unregistered word in speech recognition; A search performing step of performing a search using an index table from the expanded query; The present invention provides a method for retrieving audio and video voice data in an environment including an unregistered word including a result output step of effectively displaying a retrieved result to a user.

Preferably, the speech data retrieval method uses a compressed representation that combines the overlapping portions of the lattice based on time information to the maximum for a system that is robust against errors in the speech recognition step, and uses a word for the corresponding point in the speech document. Create an index table that contains information such as time information and probability values.

Preferably, the voice data retrieval method generates an index table by converting the lattice into syllables and phonemes that are lower units than the combined morphemes in order to solve the problem of the unregistered words in the speech recognition step.

Preferably, the voice data retrieval method utilizes various index tables by extending the query in the form of a combination of units when the keyword to be searched is determined to be a non-registered word in the voice recognition step.

Preferably, the voice data retrieval method proceeds bidirectionally from the first appearing portion of the combined morpheme unit for faster retrieval, and checks whether the distance between adjacent units is within a predetermined time.

As described above, the present invention can perform audio and video voice data retrieval in an environment including unregistered words through an index and search process considering unregistered words in constructing a voice data retrieval system.

DETAILED DESCRIPTION Hereinafter, a method and apparatus for retrieving audio data of audio and video in an environment including an unregistered word according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a general overview block diagram for explaining the configuration of a search system.

In FIG. 1, in general, a voice data retrieval system is a system that effectively provides a point where keywords appear in a number of video and audio 60 through a search for a keyword 20 that the user 10 wants to find.

Voice data retrieval system according to an embodiment of the present invention is largely composed of the index unit 200 and the search unit 100.

The indexer 200 includes a voice data extractor 210 for extracting voice data 220 from audio and video; A voice data divider 230 for dividing the voice data 220 into an appropriate amount estimated in sentence units for performing in the voice recognizer 250 and storing the voice data 220 in the form of a wave file 240; A voice recognizer 250 for receiving a voice wave file as an input and outputting the voice wave file in a text data file form 260; An indexer 270 is used to generate the index table 280 in various units using the grid-shaped information resulting from the speech recognition 260.

The search unit 100 may include a query expander 110 that expands a user's query into various possible queries to use an index table in consideration of non-registered words in speech recognition; A searcher 120 for performing a search using an index table from the extended query 21; And a result outputter 130 that effectively displays the searched result 30 to the user.

The first thing to consider when building a voice data retrieval system is speech recognition. Speech recognition requires various preprocessing processes for speech audio. First of all, it is necessary to extract voice data from video and audio. This task is performed from the voice data extractor 210. The voice data extractor 210 determines whether the audio data is included in the video and the audio and stores the corresponding content.

In order to effectively perform speech recognition, speech data of appropriate length is required. If the length is too long, the speech recognition performance is lowered and the speed is also a great loss. Therefore, it is necessary to divide the speech data from the speech data extractor 210 into an appropriate amount estimated in sentence units and store them in the form of a wave file, which is performed by the speech data divider 230. The voice data divider 230 generates several short wave file types from the long voice data. This process is performed by analyzing the voice data to determine if there is silence over a certain length.

In the above, the preprocessing for the part corresponding to the input of the speech recognizer 250 has been completed. In general, how to define a unit in Korean speech recognition has a big impact on performance. When a word is recognized as a unit, many words need to be registered in a dictionary, and unregistered words frequently occur, which results in poor performance. When the morpheme is recognized as a unit, many short registered words with one or two syllables are included in the dictionary, which adversely affects the recognition performance. Therefore, it is common to use a combined morpheme that combines short and frequently occurring morphemes into one as a recognition unit.

Two models are needed to use the speech recognizer 250. One is language model 40 and the other is acoustic model 50. Speech recognition performance is poor when using each model trained in the general domain. Therefore, it is necessary to adapt each model to voice data.

In the case of the language model 40, the information provided in the audio and video is utilized to the maximum, and the relevant contents are collected and trained and appropriately mixed with those trained in the general domain, and finally used as the language model. The acoustic model 50 uses a technique for adapting a conventionally trained acoustic model to the nature of the voice data currently being used. Acoustic model adaptation techniques can be used to describe commonly used maximum a-posteriori (MAP) or maximum likelihood linear regression (MLLR) methods in the HTK Toolkit [The HTK Book, Young, S. et al., Http://htk.eng.cam.ac .uk / docs / docs.shtml].

As a result, the voice recognizer 250 receives the voice wave file as an input and outputs the voice wave file in the form of a text data file. Representative contents that can be included in the text data file format are 1-best, n-best, and lattice. In general, speech recognizers cannot be free from recognition errors. In particular, when indexing considering only the 1-best generated by the speech recognizer, the entire system becomes insensitive to speech recognition errors.

In order to construct a system more robust to speech recognition errors, a large number of candidate groups must be used from the speech recognizer. However, when the grid is used as it is, the index table size increases greatly. Therefore, we use a compressed representation that combines the overlapping parts as much as possible based on time information in the grid. This is similar to Time-based Merging for Indexing (TMI) [Word-lattice based spoken-document indexing with standard text indexers, Peng Yu, K. Thambiratnam, SLT 2008] and is slightly modified. An entry in the index table consists of a word, document ID, start point, end point, and probability value.

Through the above process, the index table 280 is generated based on the speech recognition result of the combined morpheme unit. In the case where recognition is performed in a combined morphological unit using the combined morpheme index table 283, the problem of unregistered words cannot be avoided. In the speech recognizer 250, words that are not included in the list of dictionaries cannot be recognized and are usually replaced with the most likely similarity. Even though there are keywords to be searched in the actual voice document, the voice recognizer 250 may not display any results because it is a non-registered word.

To solve this problem, as shown in FIG. 2, the index tables 281 and 282 of syllables or phonemes, which are lower units than words or combined morphemes, are generated. For this purpose, a syllable or phoneme-based speech recognizer is also used, but because of poor performance, a method of converting a grid recognized as a combined morpheme unit into a syllable or phoneme unit is used. The conversion method is simply to divide the links of the grid by the number of syllables or phonemes. The probability value remains the same and the time information is estimated by dividing by the number. Based on the syllable or phoneme grids, the index table in the lower unit is created through the same indexing process.

Thus, when it is determined that the keyword 20 to be searched by the user 10 is a non-registered word in the speech recognizer 250, the user may convert to a lower unit and use a syllable or phoneme index table to compensate for the existing problem. do.

The Korean input system basically uses word spacing, which does not match the combined morpheme used in speech recognition and indexing. In the case of Korean, there are a lot of combined words, and the spacing is not constant. Therefore, a process of converting a user input query into a combined morpheme unit is required. It is also possible to use a converter using a morpheme analyzer, but a non-registered query is not likely to be well morphologically analyzed.

The query expander 110 of the search unit 100 considers all possible combinational morphemes and syllable combinations including at least one combined morpheme when a given query is first recognized as a non-registered word. In the absence of any such combination, only combinations of syllables are considered. For each combination, if the syllable is a syllable unregistered word, the syllable is converted to a phoneme unit. In this case, queries are generated that include a total of three levels: combined morphemes, syllables, and phonemes. In some cases, only combination morphemes and phonemes can be considered. Each will use the corresponding index table during the search.

For example, as shown in FIG. 2, when the user query is' Grand Canal '20, the combined stemming index table includes' Canals' and' Business', and the syllable index table includes' Lun 'and' Ha. When ',' and 'up' are included, and 'large' corresponds to a syllable unregistered word and can be converted into a phoneme string of 'T EH', the finally expanded query 21 is (T , EH, canal, company, up), (T, EH, canal, summer, business), (T, EH, canal, business).

The above method is basically a method of making the most of the combined morpheme, and the reason for using such a method is that the combined morpheme index table 283 shows the most stable performance. Basically, it is also a unit of speech recognition, and it takes less estimation than other units to index the grid. Another reason is that in the case of combined stemming, the result list appears less in search of index tables than in other units. This means that you can search faster. If it is assumed that only the phoneme index table 281 is used, not only does the number of table searches increase, but also a large number of result lists for each of them.

The user's query is expanded into a plurality of queries 21 through the query expansion unit 110, and the searcher 120 receives the expanded query 21 as an input and performs a search using various index tables. Extract a list from the corresponding index table for each unit included in the query. The combination of these lists is included in the final result for the user to indicate that all units appear in order, and is performed by checking whether the distance between adjacent units is within a certain time.

Instead of starting from the left side of the query to the right side of the search, it is effective to proceed bidirectionally from the first occurrence of the combined morpheme unit. In the case of the combined morpheme unit, the number of lists extracted from the index table is relatively smaller than that of other units, which greatly reduces the possible combination from the previous step in the search process.

In the search, the score is calculated as follows.

, ... ,

은 확장된 사용자 질의를 의미한다. 따라서 N은 확장된 사용자 질의의 개수가 되며, I는 특정 구간을 의미한다. HScore는 해당 질의가 특정 구간에서 어떤 확률로 나타났는지를 의미하며, 이는 색인 테이블의 확률 값으로부터 구한다. Where Q means user query

, ...,

Means extended user query. Therefore, N is the number of extended user queries, and I is a specific interval. HScore means what probability the query appeared in a particular interval, which is obtained from the probability value of the index table.

After the search process, the search result 30 should be finally displayed to the user through the output unit 130. The output unit 130 sequentially displays the results according to scores, and provides an interface for quickly moving to the point where the query word appears instead of simply providing corresponding audio and video.

1 is a general overview block diagram for explaining the configuration of a search system.

FIG. 2 is a block diagram illustrating in more detail an index table and an extended query of the index unit of FIG. 1.

Claims (10)

In the voice data retrieval apparatus including an index unit and a search unit, The index unit extracts the voice data 220 from the audio and video voice data extractor 210; A voice data divider 230 for dividing the voice data 220 into an estimated amount of sentences in order to be performed by the voice recognizer 250 and storing the voice data 220 in a wave file form 240; A voice recognizer 250 for receiving a voice wave file as an input and outputting the voice wave file in a text data file form 260; An indexer 270 for generating the index table 280 in various units using the grid-shaped information resulting from the speech recognition 260, The search unit includes a query expander 110 which expands a user's query into various possible queries to use an index table in consideration of non-registered words in speech recognition; A searcher 120 for performing a search using an index table from the extended query 21; A result outputter 130 for effectively displaying the searched result 30 to the user, The indexer 270 generates an index table by converting the lattice into syllables and phonemes that are lower units than the combined morphemes in order to solve the problem of the unregistered words of the speech recognizer. And video voice data retrieval device. The method according to claim 1, wherein a word, time information, probability value, etc. for a corresponding point of the voice document is used by using a compressed expression that combines the overlapping parts of the lattice based on the time information as possible for a system that is robust against error of the speech recognizer. Voice data retrieval device to create an index table containing information of. delete The apparatus of claim 1, wherein when the keyword to be searched by the user is determined to be a non-registered word in the speech recognizer, the query is extended by using a combination of units to utilize various index tables. The voice data retrieval apparatus according to claim 1, wherein the retrieval unit performs a search by checking whether a distance between adjacent units is within a predetermined time from the first appearing portion of the combined morpheme unit for faster retrieval. In the voice data retrieval method comprising the indexing step of the index unit and the search step of the search unit, The indexing step may include: extracting voice data from audio and video; A voice data dividing step of dividing the voice data into an estimated amount in sentence units to be performed in a voice recognition step and storing the voice data in a wave file form; A voice recognition step of receiving a voice wave file as an input and outputting the voice wave file as a text data file; An indexing step for generating an index table in various units using the grid-shaped information resulting from speech recognition, The searching step may include: a query expansion step of expanding a user's query into various possible queries to use an index table in consideration of an unregistered word in speech recognition; A search performing step of performing a search using an index table from the expanded query; It includes a result output step that effectively displays the search results to the user, In order to solve the problem of unregistered words in the speech recognition step, the indexing step generates an index table by converting the lattice into syllables and phonemes, which are lower units than the combined morphemes. How to retrieve video voice data. The word, temporal information, and probability value for the corresponding point of the speech document, using a compressed representation that combines as many as possible overlaps based on temporal information in the grid for the error-tolerant system in the speech recognition step. Voice data retrieval method for generating an index table containing information such as. delete The voice data retrieval method according to claim 6, wherein when the keyword to be searched by the user is determined to be a non-registered word in the speech recognition step, the query is extended by using a combination of units to utilize various index tables. The voice data retrieval method according to claim 6, wherein the retrieval unit performs a search by checking whether a distance between adjacent units is within a predetermined time from the first appearing unit of the morpheme unit for faster retrieval.

Images