JP2013125144A - Speech recognition device and program thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize speech recognition processing with high precision while regulating the increase of memory usage of dictionary data in the case where a genre or a domain of speech content transits.SOLUTION: A speech recognition device comprises: an integrated dictionary storing section for storing phoneme string data that represents a phoneme string being a string of phonemes associated with an acoustic model, word notation data that represents a word notation associated with the phoneme string data, and word appearance probability value data in which a word appearance probability value, associated with the word notation data and representing an appearance probability of a word notation, is held for each of a plurality of linguistic models; and a correct word searching section for calculating an acoustic score of each phoneme string on the basis of the relationship between an acoustic feature amount output from an acoustic analyzing section and acoustic model, for acquiring a word appearance probability value corresponding to a predetermined linguistic model with respect to word notation data corresponding to a phoneme string from the integrated dictionary storing section to calculate a linguistic score for each word notation, and for searching for a correct word notation on the basis of the calculated acoustic score and linguistic score.

Description

  The present invention relates to a speech recognition apparatus that performs processing for recognizing input speech, and a program thereof.

  In general, in speech recognition processing, a probability value (word N-gram (engram)) of occurrence of a chain of words is learned in advance as a language model, and a correct word string optimal for input speech is relied on using this language model. Explore. For speech recognition processing using such a statistical language model, it is common to prepare a large amount of text of the same genre or domain as the speech recognition target and learn according to use. For example, genre-specific uses in broadcast programs are for cooking, gardening, health and the like.

  If the genre or domain of the utterance content of the input speech that is the target of speech recognition changes over time, such as a long-time information program in broadcasting, the language model learned from the text of a general topic, Sufficient recognition performance cannot be obtained.

  Therefore, Non-Patent Document 1 describes a method of performing speech recognition after selecting an optimal one language model from among a plurality of language models prepared for each use or domain, according to some criteria before recognition. ing.

  Japanese Patent Application Laid-Open No. 2004-228561 describes a technique in which a plurality of speech recognitions are performed in parallel using a plurality of language models, and an optimum single recognition result is selected according to some criteria.

  Patent Document 2 proposes a method of generating a language model in which a plurality of language models are integrated into one, and selecting an optimal language model from a speech recognition result using the integrated language model. ing.

  Non-Patent Document 2 proposes a method of developing a plurality of grammars as a single global grammar.

JP 2004-198831 A JP 2010-224029 A

Shinichi Honma, Akio Kobayashi, Takahiro Oku, Satoshi Imai, “Examination of Topic Switching Language Model for Information Programs”, Proceedings of the Acoustical Society of Japan, 3-P-9, 2011 Lee Sung-nobu and Shikahiro Shikano, “Recognition Engine Julius / Julian-3.3 for Simultaneous Recognition and Dynamic Switching of Multiple Grammars”, Proc. Of the Acoustical Society of Japan, 3-9-12, pp. 153-154, 2002 Year

However, the conventional technology has the following problems.
For example, in the technique described in Patent Document 1, since a plurality of voice recognition processes are operated in parallel, there is a problem that the amount of calculation and memory consumption increase in proportion to the degree of parallelism, and the device scale increases. .
Further, in the technique described in Patent Document 2, when a plurality of language models are integrated into one, even when words having the same notation are registered as vocabulary in each language model, each language model is supported. Those word notations are identified as different ones, and another word N-gram is given to each word. Therefore, there is a problem that the same word is expanded at a plurality of locations in the tree-structured dictionary data to be searched for the correct word, which is redundant, resulting in an increase in use memory and an increase in speech recognition processing time.
In the technology described in Non-Patent Document 1, a speech recognition process is executed after selecting one language model from a plurality of language models before recognition. For this reason, the language model cannot be dynamically switched or adjusted.
In the technique described in Non-Patent Document 2, a tree-structured dictionary for each grammar is independently expanded in a memory, and the same word occupies the memory at a plurality of locations. For this reason, there is a problem that the use memory increases and the speech recognition processing time increases.

  The present invention has been made based on the above problem recognition. For example, when the genre or domain of the utterance content changes with time in one program as in an information program, a plurality of languages are used. The present invention provides a speech recognition apparatus that can perform recognition processing while performing adjustments such as switching models and can suppress an increase in the amount of memory used for dictionary data corresponding to a plurality of language models.

  In addition, the present invention interrupts speech recognition processing that is operating online when selecting and using one of a plurality of language models or using a plurality of language models with weighted linear combination. It is an object of the present invention to provide a speech recognition apparatus that can perform recognition processing without delay.

  [1] In order to solve the above-described problem, a speech recognition apparatus according to an aspect of the present invention includes phoneme sequence data representing a phoneme sequence that is a sequence of phonemes associated with an acoustic model that is a statistic relating to an acoustic feature. A word notation data representing a word notation associated with the phoneme string data and a word appearance probability value associated with the word notation data and representing an appearance probability of the word notation corresponding to each of a plurality of language models An integrated dictionary storage unit that stores appearance probability value data, an acoustic analysis unit that analyzes input speech and outputs an acoustic feature amount, and a relationship between the acoustic feature amount output from the acoustic analysis unit and the acoustic model To calculate an acoustic score for each phoneme string, and to calculate the word appearance probability value corresponding to a predetermined language model for the word notation data corresponding to the phoneme string Acquired from engagement dictionary storage unit calculates the language score of each of the word notation, the calculated the acoustic score and the language score comprises a correct word searching unit and outputting the searched word notation correct.

  Here, the integrated dictionary storage unit stores phoneme string data and word notation data in an area common to a plurality of language models. In addition, the word notation data corresponds to the notation mixed with kanji characters in Japanese. The word appearance probability value data is a probability value based on a word chain such as a unigram, bigram, or trigram. When the number of word chains is 2 or more, a conditional probability based on the immediately preceding word (sequence) is used. In addition, word appearance probability value data hold | maintains a word appearance probability value for every language model.

  [2] Further, according to one aspect of the present invention, in the speech recognition apparatus, the word appearance probability value data stored in the integrated dictionary storage unit is language model selection control data indicating which language model is selected. And includes an integrated dictionary management unit that updates the language model selection control data in accordance with settings, and indicates that the correct word search unit is selected by the language model selection control data. The word appearance probability value corresponding to the predetermined language model is acquired from the integrated dictionary storage unit, and a language score for each word notation is calculated.

  [3] Further, according to an aspect of the present invention, the speech recognition apparatus includes a weight value storage unit that stores a weight value for each language model, and the integrated dictionary management unit includes the word appearance probability value data A new word appearance probability value obtained by synthesizing the plurality of language models is obtained by weighted averaging the word appearance probability values corresponding to the plurality of language models with the weight values acquired from the weight value storage unit. The word appearance probability value data is updated with the calculated word appearance probability value.

  With this configuration, a plurality of language models can be mixed at a desired ratio to synthesize a new language model and used for recognition processing. Further, when the mixture ratio (weight value) is changed, the synthesized language model can be automatically updated.

  [4] Further, according to one aspect of the present invention, in the speech recognition apparatus, the integrated dictionary management unit corresponds to each of the plurality of language models, the word notation data, the word appearance probability value, and the Pronunciation data corresponding to each of the word notation data is acquired, the phonetic data is expanded into the phoneme string data for each word notation data, written to the integrated dictionary storage unit, and associated with the expanded phoneme string data And writing the word notation data to the integrated dictionary storage unit, writing the word appearance probability value associated with the word notation data to the word appearance probability value data of the integrated dictionary storage unit in association with the language model, When there is a word having a common word notation among language models, the word appearance probability value for each language model is associated with a single word notation data. The word appearance probability value data is written in the word appearance probability value data, and the word appearance probability value for each language model is associated with a single phoneme string data when there is a word with common pronunciation data between language models. Write data.

  With this configuration, a plurality of language models are integrated, and regarding phoneme string data and word notation data depending on each language model, statically integrated tree structure data common between the language models is expanded on the memory, The word appearance probability value for each language model can be associated from the word notation data.

  [5] Further, according to one aspect of the present invention, in the speech recognition apparatus, the integrated dictionary storage unit corresponds to a partial phoneme sequence from a word start to a mid-word phoneme in the phoneme sequence data, for each language model. Further, intra-word probability value data for holding the maximum value of the word appearance probability values associated with all the word notation data sharing the partial phoneme string as an intra-word probability value corresponding to the partial phoneme string. The correct word search unit is configured to exclude word notation data sharing the corresponding partial phoneme sequence from search targets when the intra-word probability value is lower than a predetermined threshold.

  [6] Further, according to one aspect of the present invention, phoneme string data representing a phoneme string that is a phoneme string associated with an acoustic model that is a statistic relating to an acoustic feature value, and a word notation associated with the phoneme string data And a word appearance probability value data for storing a word appearance probability value associated with the word notation data and representing a word appearance probability value corresponding to each of a plurality of language models. A storage unit, an acoustic analysis unit that analyzes input speech and outputs statistical data of acoustic feature values, and an acoustic feature value output from the acoustic analysis unit for each phoneme sequence according to a relationship with the acoustic model A score is calculated, and the word appearance probability value data corresponding to a predetermined language model for the word notation data corresponding to the phoneme string is stored in the integrated dictionary storage unit A computer as a speech recognition apparatus comprising: a correct word search unit that acquires and calculates a language score for each word notation, and searches for and outputs a correct word notation based on the calculated acoustic score and the language score. It is a program to make it function.

According to the present invention, highly accurate speech recognition can be realized by switching a plurality of language models, and the memory usage of dictionary data corresponding to the plurality of language models can be suppressed.
Further, according to the present invention, the recognition process is not delayed or interrupted when the word appearance probability value table is updated.

It is a block diagram which shows the function structure of the speech recognition apparatus by one Embodiment of this invention. It is the schematic which shows the structural example of the data for phoneme sequence search contained in the dictionary data which the integrated dictionary memory | storage part by the same embodiment memorize | stores. It is another schematic diagram which shows the structural example of the data for phoneme string search contained in the dictionary data which the integrated dictionary memory | storage part by the same embodiment memorize | stores. It is the schematic which shows the structure of the data which the weight vector memory | storage part by the same embodiment memorize | stores. It is the schematic which shows the structural example of the word termination | terminus N-gram table memorize | stored in the integrated dictionary memory | storage part by the embodiment. It is the schematic which shows the structural example of the N-gram table in a word memorize | stored in the integrated dictionary memory | storage part by the embodiment. It is the flowchart which showed the procedure of the speech recognition process (especially update of an N-gram table and search of a correct word) by the embodiment.

Next, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a functional configuration of the speech recognition apparatus according to the embodiment. As illustrated, the speech recognition apparatus 1 includes an acoustic analysis unit 2, a correct word search unit 3, an integrated dictionary storage unit 4, and language model storage units 6-1, 6-2, ..., 6-M. , 7-M, pronunciation dictionary storage units 7-1, 7-2,..., 7-M, integrated dictionary management unit 8, acoustic model storage unit 9, weight vector storage unit 10 (weight value storage unit), A manual setting unit 11 and an automatic setting unit 12 are included.

  The language model storage unit 6-1 and the pronunciation dictionary storage unit 7-1 are used as a pair. The language model storage unit 6-2 and the pronunciation dictionary storage unit 7-2 are used as a pair, and so on.

  As an example, the speech recognition apparatus 1 is used for the purpose of producing subtitle text for broadcasting from the sound of a broadcast program. In this case, the input sound is the sound of the broadcast program itself, or the sound of the broadcast program that is restated by the announcer dedicated to caption production (so-called “risk peak sound”).

  The acoustic analysis unit 2 reads input speech and extracts acoustic feature quantities such as power and frequency characteristics. In addition, the acoustic analysis unit 2 extracts the acoustic feature amount and simultaneously detects the voice utterance start point and the utterance end point in the input voice. Detection of the utterance start point and utterance end point itself can be performed by existing techniques. For example, the acoustic analysis unit 2 sequentially calculates the ratio between the maximum cumulative likelihood for speech and the cumulative likelihood for non-speech while referring to an acoustic model for speech and a non-speech acoustic model for each speaker cluster. Then, by comparing the calculated ratio value with a preset threshold value, the utterance start point and utterance end point are detected.

  The correct word search unit 3 searches the correct word using the dictionary data read from the integrated dictionary storage unit 4 based on the acoustic feature amount extracted by the acoustic analysis unit 2, and recognizes the text obtained as a result. Output as result text. Specifically, the correct word search unit 3 calculates an acoustic score for each phoneme sequence based on the relationship between the acoustic feature amount output from the acoustic analysis unit 2 and the acoustic model stored in the acoustic model storage unit 9. To do. In addition, the correct word search unit 3 acquires a word appearance probability value corresponding to the selected language model from the integrated dictionary storage unit for the word notation data corresponding to the phoneme string, and obtains a language score for each word notation. calculate. Then, the correct word search unit 3 searches for and outputs the correct word notation based on the calculated acoustic score and language score. The correct word searching unit 3 acquires a word appearance probability value corresponding to the language model indicated by the control flag from the integrated dictionary storage unit, and uses it for language score calculation.

  The integrated dictionary storage unit 4 stores integrated dictionary data obtained by integrating information of a plurality of language models and a plurality of pronunciation dictionaries. Specifically, the integrated dictionary storage unit 4 stores data related to the word start node and the word end node, phoneme string data, word notation data, a word end N-gram table, and an intra-word N-gram table. To do. In addition, the integrated dictionary storage unit 4 stores information on links for associating those data. Details of these data will be described later.

  The language model storage units 6-1, 6-2,..., 6-M store language model data, and the language model 1, language model 2,. It corresponds. Each language model data holds word notation data and word appearance probability value data in association with each other. In this embodiment, the word appearance probability value is a bigram probability value, but may be a probability value based on an arbitrary number of word chains.

  The pronunciation dictionary storage units 7-1, 7-2,..., 7 -M store pronunciation dictionary data, and the language model 1, language model 2,. It corresponds. Each pronunciation dictionary data holds word notation data and pronunciation data in association with each other. The phonetic data is equivalent to a phoneme string.

  The integrated dictionary management unit 8 includes data of a plurality of language models, that is, language model storage units 6-1, 6-2,..., 6-M and pronunciation dictionary storage units 7-1, 7-2,. The data read from 7-M is integrated, and the phoneme string data and the word notation data common to the language models are expanded on the memory, and the word end N-gram table for holding the probability value for each language model and A value is written in the intra-word N-gram table. Details of the language model integration and expansion processing will be described later.

  The integrated dictionary management unit 8 also integrates the integrated dictionary storage unit when necessary in the utterance pause between the utterance end and the next utterance start based on the timing of the utterance start and end of the utterance detected by the acoustic analysis unit 2. 4 is updated. Specifically, when the weight vector of the weight vector storage unit 10 has been changed, the integrated dictionary management unit 8 uses the changed weight vector to change the word end N-gram table and the intra-word N-gram. Recalculate table values and update these tables. Details of this table update processing will be described later.

  Further, the integrated dictionary management unit 8 updates a control flag for controlling selection of a language model (case) by manual setting or automatic setting. By updating this flag, the language model used by the correct word searching unit 3 can be instantaneously switched. Even when a language model (“case X” described later) that is a weighted synthesis result of a plurality of language models is selected, the integrated dictionary management unit 8 similarly updates the control flag. .

The acoustic model storage unit 9 stores statistics related to acoustic features in a format associated with phonemes. The statistic of the acoustic feature quantity in the acoustic model storage unit 9 is, for example, an average value and a variance value (a variance covariance matrix when expressed as a vector) such as phoneme power and frequency characteristics.
As the acoustic feature amount, for example, a hidden Markov model (HMM) is used.

  The weight vector storage unit 10 stores a vector of weight values representing the contribution of each language model for synthesizing a plurality of language models. Details of the weight vector will be described later.

  The manual setting unit 11 sets the value of the weight vector by manual setting from the user. The manual setting unit 11 may directly input the value of each weighting factor for the language model, or a combination of several weighting factors in advance (for example, case 1, case 2,. X) may be set so that one of them can be selected.

  The automatic setting unit 12 estimates the topic by analyzing the text of the speech recognition result, and performs setting to switch the language model according to the identified topic. Estimating the topic itself can be performed using existing technology. For example, the frequency of appearance for each word in the recognition result text is counted, and the topic can be estimated using the frequency of appearance of all words as a feature vector and the similarity between vectors based on a cosine scale or the like. The automatic setting unit 12 estimates a topic at any time and selects a weight vector from candidates prepared in advance or newly generates a weight vector according to the estimation result.

  FIG. 2 is a schematic diagram illustrating a configuration example of phoneme string search data included in the dictionary data stored in the integrated dictionary storage unit 4. As shown in the figure, the dictionary data stored in the integrated dictionary storage unit 4 includes data representing nodes and directed links. Each node is represented as structure data, for example. The directed link is represented by using a pointer to a storage area at the end point of the link. In the figure, reference numeral 20 denotes a word start node corresponding to the word start. Reference numerals 31 to 40 are phoneme nodes corresponding to phonemes. Reference numerals 51 to 57 are word notation nodes corresponding to word notations represented by Japanese kanji kana mixing. Reference numeral 70 denotes a word end node corresponding to the word end. Note that the phoneme nodes 31 to 40 and the link between the phoneme nodes represent a phoneme string that is a search target space by the correct word searching unit 3, and this is referred to as phoneme string data.

  The word notation data stored in the integrated dictionary storage unit 4 corresponds to the word notation node and represents the word notation. Further, the word notation data is associated by a link from the terminal phoneme node in the phoneme string data.

  For example, since the word “morning” corresponding to the word notation node 51 is pronounced by the phoneme string / asa /, the phoneme node 31 of / a /, the phoneme node 32 of / s /, and the phoneme node of / a / 34 is associated with a serial network. Further, for example, the word “Tomorrow” corresponding to the word notation node 53 is pronounced with the phoneme string / asu /, so the phoneme node 31 of / a / and the phoneme node 32 of / s / Share with node 51 (word “morning”). Then, the word notation node 53 (the word “tomorrow”) is associated with the network branching from the phoneme node 32 of / s / to the phoneme node 35 of / u / (a network in series with the phoneme nodes 31, 32, and 35). It is done. In addition, not only the word notation node 51 (word “morning”) but also the word notation node 52 (word “hemp”) is associated with the phoneme string / asa /.

  In addition, each of the phoneme nodes 31 to 40 is associated with the statistic of the acoustic feature amount in the acoustic model storage unit 9. This association is realized by each phoneme node holding link information (not shown in FIG. 2).

  Each of the phoneme nodes 31 to 40 holds link information (not shown in FIG. 2) for associating with an entry in the intra-word N-gram table. This association is performed by the value of the pointer or phoneme node identification information. The intra-word N-gram table will be described later.

  Each of the word notation nodes 51 to 57 holds link information (not shown in FIG. 2) to the entry in the word end N-gram table. The word end N-gram table will be described later.

  FIG. 3 is another schematic diagram showing a configuration example of phoneme string search data included in the dictionary data stored in the integrated dictionary storage unit 4. The data shown in FIG. 3 and the data described in FIG. 2 are the same in conceptual configuration. In FIG. 3, reference numeral 21 denotes a word start node. 41 to 43 are phoneme nodes. 58 to 61 are word notation nodes. Reference numeral 71 denotes a word end node. Note that the phoneme nodes 41 to 43 and the link between the phoneme nodes represent a phoneme string that is a search target space by the correct word searching unit 3, and this is referred to as phoneme string data.

In this example, the word notation w ₁ and the word notation w ₂ are both associated with the phoneme string / n ₁ n ₂ /. In other words, _{w 1} and _{w 2} is a homonym. The word notation w ₃ and the word notation w ₄ are both associated with the phoneme string / n ₁ n ₃ /. In other words, _{w 3} and _{w 4} is a homonym. In the description of the intra-word N-gram table and the end-of-word N-gram table later, reference will be made to the data in FIG.

  FIG. 4 is a schematic diagram illustrating a configuration of data stored in the weight vector storage unit 10. As illustrated, the weight vector storage unit 10 stores the weight value of each language model for each case. In the illustrated example, the vector of case 1 is (1.0, 0.0,..., 0.0). That is, in case 1, the weight of the language model 1 is 1.0, and the weights of all other language models are 0.0. In case 2, the weight of the language model 2 is set to 1.0, and the weights of all other language models are set to 0.0. In case X, the weights of the language models 1 and 2 are set to 0.5 each, and the weights of the other language models are all set to 0.0. That is, in case 1 or case 2, it means that one language model is used independently from among a plurality of language models. Further, as in case X, using a weight vector, a statistic obtained by arbitrarily weighting linear combination of a plurality of language models can be used as a language model.

The meaning of the weight vector for the language model is as follows. That is, the weighting coefficient for an arbitrary language model m∈ {1, 2,..., M} is λ _m, and the (n−1) th word w _n−1 to the nth word w included in the input speech. _{If the} appearance probability based on the language model m of the word chain (bigram) to _n is P _m (w _n | w _n−1 ), a language model (case) composed of weighted linear combinations from these M language models. The appearance probability of the word chain of X) can be calculated as the following equation (1) by adding the respective weighting factors to the appearance probability of each language model.

  Here, it is assumed that the condition expressed by the following equation (2) is satisfied.

That is, the sum of the weight values for a certain case is 1, and each weight value is non-negative.
In addition, although the appearance probability of the two-word chain from the immediately preceding word to the next word, that is, the bigram probability has been described here, the appearance probability of the three-word chain including the previous immediately preceding word as a condition, That is, the trigram probability can be similarly calculated using the weight.

  The manual setting unit 11 or the automatic setting unit 12 selects a weight vector from the case 1, the case 2,..., The case X, or changes the vector value of the case X as appropriate. decide. Further, when only the weight of a specific language model in the selected weight vector is 1.0 as in case 1 and case 2, the same effect as that of simply selecting the original language model is obtained. It is done.

  FIG. 5 is a schematic diagram illustrating a configuration example of a word end N-gram table (word appearance probability value data) stored in the integrated dictionary storage unit 4. The example shown here is data when using bigram probabilities. Note that as described above, the word notation nodes described in FIG. 2 and FIG. 3 are associated by links from the word end N-gram table entries.

As illustrated, the word end N-gram table holds the probability value (word appearance probability value) of the word N-gram for each case of the language model. For example, in Case 1, as the probability value when connecting to each next word _{w 2} and _{w 4} with respect to previous word _{w 1,} respectively, and holds 0.8 and 0.2. At this time, the total probability of connection to the next word with respect to the immediately preceding word w ₁ is 1.0. That is, immediately before the word _{w 1} - a connection probability 0.8 follows the word _{w 2,} immediately before the word _{w 1} - sum of the connection probability 0.2 follows the word _{w 4} 1.0. Similarly, in Case 1, the sum of the connection probability of previous word w ₂ to the next word is also 1.0. Further, this word end N-gram table similarly holds probability values based on N-grams (here, bigrams) of words in case 2, case X, and the like.

Case X will be described in detail. As shown in FIG. 4, when the weight value in case X is 0.5 for each of language model 1 and language model 2, the connection between immediately preceding word w ₁ and next word w ₂ in case X in FIG. The probability is the sum of the value obtained by multiplying the probability 0.8 in Case 1 by the weight 0.5 and the value obtained by multiplying the probability 0.0 in Case 2 by the weight 0.5, that is, 0.4. Similarly, the connection probability of the immediately preceding word w ₁ -next word w ₃ in case X is a value obtained by multiplying probability 0.0 in case 1 by weight 0.5 and probability 0.4 in case 2 with weight 0.5. The sum of the value and the value obtained by multiplying by 0.2, that is, 0.2. Similarly, the connection probability of the immediately preceding word w ₁ -next word w ₄ in case X is a value obtained by multiplying probability 0.2 in case 1 by weight 0.5 and probability 0.6 in case 2 with weight 0.5. The sum of the value and the value obtained by multiplying by 0.4, that is, 0.4. The same applies to the other immediately preceding words.

  That is, the word end N-gram table is associated with the word notation data, and the word appearance probability value representing the conditional appearance probability of the word notation on the premise of the immediately preceding word or the immediately preceding word chain is represented by a plurality of language models. Hold corresponding to each.

  The word end N-gram table has a control flag (language model selection control data) for controlling selection of a language model (case). In the illustrated example, since the case 1 is selected, the value of the control flag corresponding to the case 1 is “◯”. Further, the value of the control flag corresponding to case 2 or case X not selected is “×”. While the speech recognition process is being executed, multiple tree structures of phoneme string data are expanded in the memory for each language model by referring to the data of the language model indicated by this control flag “○”. It is possible to perform processing equivalent to switching the language model without switching the entire tree structure for each language model. Moreover, the language model can be changed quickly only by updating the control flag.

  FIG. 6 is a schematic diagram illustrating a configuration example of an intra-word N-gram table (intra-word probability value data) stored in the integrated dictionary storage unit 4. As illustrated, the intra-word N-gram table holds a probability value (intra-word probability value) corresponding to a phoneme node for each immediately preceding word. As described above, the association from each phoneme node to the entry in the intra-word N-gram table is performed. In addition, the intra-word N-gram table has a control flag (language model selection control data) for controlling the selection of a language model (case), similarly to the above-described word end N-gram table. In the selected case, the value of the control flag is “◯”. In the case where the control flag is not selected, the value of the control flag is “×”.

  That is, the integrated dictionary storage unit 4 corresponds to the partial phoneme sequence from the beginning of the word to the mid-word phoneme in the phoneme sequence data (the mid-word phoneme is a phoneme that does not have a direct link to the word notation node). For each, an intra-word N-gram table is stored that holds the maximum value of word appearance probability values associated with all word notation data sharing a partial phoneme string as an intra-word probability value corresponding to the partial phoneme string. The meaning of word notation data that shares partial phoneme strings (phoneme nodes) will be described next.

The data example shown in FIG. 6 corresponds to the tree structure of the phoneme string described in FIG. As shown in FIG. 3, the phoneme node n ₁ is the words w ₁ , w ₂ , w _3, and w ₄ corresponding to the word notation nodes 58 to 61. That is, the phoneme node n ₁ shares the words w ₁ , w ₂ , w _3, and w ₄ . In general, the set of shared words (more words containing the phoneme nodes) in the phoneme node p and _{s p, s 1 = {w} 1, w 2, w 3, w 4}, s 2 = {w 1, w ₂ }, s ₃ = {w ₃ , w ₄ }. In other words, the set of all words notation nodes sharing the phoneme node p is the s _p. The probability value held in the intra-word N-gram table is the maximum value of connection probabilities when a word included in a shared word set shared by a certain phoneme node is set as the next word on the premise of a certain previous word.

That is, the maximum value of the chain the probabilities of occurrence of previous word w _n-1 to the shared word set s _p in the subsequent phoneme node p _(a set of a plurality of words containing phonemes node p) is represented by the following formula (3) .

In Expression (3), P _m (w _n | w _n−1 ) is a connection probability from the word w _{n −1} to the word w _n in the case m (any single language model or weighted linear sum). And the above-mentioned word end N-gram table holds the probability value.

  In this example, the appearance probability of the two-word chain from one previous word to the next word, that is, the maximum value of the bigram probability has been described. However, the appearance of the three-word chain including the previous previous word as a condition is also described. The probability, that is, the maximum value of the trigram probability, the appearance probability of a single word independently of the immediately preceding word, that is, the maximum value of the unigram probability may be used. Even when these trigram probabilities, unigram probabilities, and the like are used, the method for calculating the value held in the intra-word N-gram table is the same.

For example, in case 1 (only language model 1 is used in the example shown in FIG. 4), when the immediately preceding word is w ₁ , the set of shared words of phoneme node n ₁ is s ₁ = {w ₁ , w ₂ , w _3, since _{w 4} is a}, previous word _{w 1} - a next word _{w 1} of the connection probability 0.0, previous word _{w 1} - a connection probability 0.8 follows the word _{w 2,} immediately before the word _{w 1} - next word and w ₃ connection probability 0.0, previous word _{w 1} - maximum value of the connection probability 0.2 follows the word _{w 4} is 0.8. Therefore, the probability value of the immediately preceding word w ₁ -phoneme node n ₁ is 0.8. The same applies to other immediately preceding words and other phoneme nodes.

  By using such an intra-word N-gram table, when performing speech recognition processing, instead of delaying the application of the language model (addition of logarithmic language score) to the end of the word, at each phoneme node during the search, Based on the maximum value of the chain appearance probability for the words that share the phoneme, it is possible to terminate the search for the word with low possibility (that is, pruning in the search space). In other words, when the maximum value of the chain appearance probability for a word that shares the phoneme is lower than a predetermined threshold, the possibility that any of the shared word sets is a correct word is lower than the threshold. The search space can be safely pruned.

  Although it is possible to write probability values for all possible phoneme nodes in the intra-word N-gram table in advance, in order to avoid an enormous amount of data, the following measures are taken. May be. The first contrivance is that the probability value does not change between phoneme nodes after the phoneme node where no further branching occurs in the tree of phoneme strings (that is, when there is only one type of word pronunciation common to the phoneme node). Omit registration of probability values. The second idea is to pre-calculate a probability value only for a phoneme node close to the beginning of a word having a high calculation frequency and register it in the intra-word N-gram table. Then, for a phoneme node with a low calculation frequency and close to the word end, a probability value is calculated only when it is necessary for recognition.

Next, a procedure of processing by the voice recognition device 1 will be described.
FIG. 7 is a flowchart showing an operation procedure performed by the speech recognition apparatus 1.

  First, in step S1, the integrated dictionary management unit 8 reads the data of each language model from the language model storage units 6-1 to 6-M, and the pronunciation dictionary data from the pronunciation dictionary storage units 7-1 to 7-M. read out. The integrated dictionary management unit 8 reads out acoustic model data from the acoustic model storage unit 9. Then, the integrated dictionary management unit 8 creates integrated dictionary data by processing such as expanding the tree structure of phoneme nodes, and writes the data in the integrated dictionary storage unit 4. Data generated at this time includes the tree structure of the phoneme node shown in FIGS. 2 and 3 and the data of the word notation node associated therewith, the word end N-gram table shown in FIG. 5, and FIG. In-word N-gram table indicated by. Details of the integrated dictionary creation processing by the integrated dictionary management unit 8 will be described later.

  Next, in step S <b> 2, the automatic setting unit 12 sets the case 1 vector to be used as the initial value of the weight vector. That is, the weighting factor for language model 1 is set to 1.0, and the weighting factor for other language models is set to 0.0. At this time, the control flags of the word end N-gram table and the intra-word N-gram table are set so as to select a specific case.

  Next, in step S3, the integrated dictionary management unit 8 checks whether or not the weight vector is manually set to a value other than the initial value. If there is a manual setting of the weight vector (step S3: YES), the process proceeds to the next step S4, and if there is no manual setting (step S3: NO), the process jumps to step S5.

  Next, when it progresses to step S4, the integrated dictionary management part 8 recalculates the probability value which a word end N-gram table and an intra-word N-gram table hold | maintain according to the value of the weight vector after a change. ,Update.

  Next, in step S5, the acoustic analysis unit 2 analyzes the input speech and extracts acoustic feature values. Specifically, the acoustic analysis unit 2 obtains a discrete time series of acoustic feature values based on an input voice having a predetermined time length, corresponding to the time length. At this time, the acoustic analysis unit 2 also simultaneously detects the utterance start point and utterance end point by the method described above.

  Next, in step S6, the acoustic analysis unit 2 determines whether or not the current input speech is an inter-speech pause (silence, non-speech) according to the timing of the speech start end and speech end detected in step S5. Determine. If it is a pause between utterances (step S6: YES), the process proceeds to the next step S7, and if it is not a pause between utterances (step S6: NO), the process jumps to step S11.

  Next, in step S7, the integrated dictionary management unit 8 checks whether or not the weight vector is automatically set by the automatic setting unit 12 after the previous time. If it is automatically set (step S7: YES), the process proceeds to step S8. If it is not automatically set (step S7: NO), the process proceeds to step S9.

  Next, when the process proceeds to step S8, the setting of the weight vector to be used is updated so that the changed weight vector is used. After the processing in this step, the process proceeds to step S10.

  Next, in step S9, the integrated dictionary management unit 8 checks whether or not the weight vector has been set (changed) by the manual setting unit 11 since the previous time. If it has been changed (step S9: YES), the process proceeds to step S10. If it has not been changed (step S9: NO), the process jumps to step S11.

  Next, when it progresses to step S10, the integrated dictionary management part 8 recalculates the probability value which a word end N-gram table and an intra-word N-gram table hold | maintain according to the value of the weight vector after a change. ,Update.

  In step S11, the correct word search unit 3 uses the integrated dictionary data (including the N-gram table) stored in the integrated dictionary storage unit 4 based on the acoustic feature amount obtained by the acoustic analysis unit 2. The correct word is searched for, and the score (likelihood) of each correct candidate is calculated. The correct word searching unit 3 calculates, for example, the sum of the logarithm of the probability value of the word unit in the time series direction as a score.

  Then, in step S12, when an appropriate timing is reached, such as when the end of the word is reached, when the end of the sentence is reached, or when the end of the utterance is reached, the correct word search unit 3 determines the maximum likelihood. A word string is sequentially output as a recognition result text.

  Next, in step S <b> 13, the acoustic analysis unit 2 determines whether or not the input voice is terminated. When the input is completed (step S13: YES), the process of the entire flowchart is terminated. If the input is not yet finished (step S13: NO), the process returns to step S5 to analyze the next input voice. That is, the voice recognition device 1 repeats the voice recognition process until the input voice is finished.

<Details of correct word search processing>
Here, the detail of the process of the correct word search part 3 is demonstrated. The correct word search unit 3 searches the integrated dictionary in the integrated dictionary storage unit 4 in the above-described procedure, and when the search reaches a word notation node (reference numerals 51 to 60), the probability value corresponding to the word is determined as the word end. The logarithm is read from the N-gram table, the logarithm is taken, and the logarithmic value is added to the acoustic score of the sequence (correct answer candidate) including the word. Needless to say, the addition of logarithmic values of probability values in the series direction has the same meaning as the score obtained by multiplying the probability values. Further, as described in the explanation of the word end N-gram table, not only a method of searching for a correct word by the maximum value of the bigram probability, but also a method of using the maximum value of the trigram probability or the maximum value of the unigram probability. It may be adopted. Further, an N-gram having a chain number of 4 or more may be used.

  Regarding the case indicated by “◯” in the control flag (the selected single language model or the weighted linear sum thereof), the next word not described in the word end N-gram table is as follows. To handle the chain appearance probability. That is, when the next word is not included in the vocabulary in the case, the connection probability to the next word is set to 0.0. If the next word is included in the vocabulary in this case, the N-gram with the lower chain number (that is, the lower bigram when using a trigram. When the bigram is used. The probability value of the back-off language model using the lower unigram) is used.

  In addition, the correct word search unit 3 uses the acoustic model associated with the phoneme node to evaluate the hypothesis candidate word at each phoneme node in order to evaluate which phoneme the input speech is close to when searching in the above-described procedure. The acoustic score for is calculated. The correct word searching unit 3 calculates the acoustic score P (O | W) by the following equation (4).

Here, W is a word string in the input speech O. _Wn is the nth word in the input speech O. Also, O _n is the vector sequence of acoustic features of the input speech corresponding to the n-th word (time series).

  In addition, the correct word search unit 3 traces a transition from the word end node of each word to the word start node of the next word in order to recognize a phrase or sentence composed of a plurality of words, not only in word units. As a result, the correct word search unit 3 can search for consecutive words one after another. For example, when recognizing the utterance of the sentence “Autumn in the morning”, the correct word searching unit 3 performs acoustic analysis using the acoustic model associated with each of the phoneme nodes 31 to 40 from the word start node 20 of FIG. The score calculation is performed, and only phoneme nodes having high possibility (that is, acoustic score higher than a predetermined threshold) are stored as correct answer candidates, and phonemes having low possibility (that is, acoustic score is lower than the predetermined threshold) are stored. Abandon the search after the node. That is, the phoneme node is pruned. Then, the correct word search unit 3 adopts the word sequence of “autumn” → “no” → “morning” → “ne”, which maximizes the sum of the acoustic score and the language score, as a speech recognition process. To complete. Here, the language score is a value obtained by taking the logarithm of the word chain appearance probability (word N-gram).

  The correct word search unit 3 can perform pruning of the search space as described above by referring to the intra-word N-gram table that holds the maximum value of the chain appearance probability for words that share phonemes. In other words, when the intra-word probability value is lower than the predetermined threshold, the correct word search unit 3 excludes the word notation data sharing the corresponding partial phoneme string from the search target.

  In other words, instead of delaying the application of the language model (adding the logarithmic language score) to the end of the word, the possibility is based on the maximum value of the chain appearance probability for each word sharing the phoneme at each phoneme node in the search. It is possible to abort the search for low words. In other words, the correct word search unit 3 does not delay the application of the language model (addition of the logarithmic language score) until the end of the word, but the chain appearance probability of the word sharing the phoneme at each phoneme node in the search. Temporarily add the maximum value (pre-read language score).

<Details of integrated dictionary management processing>
Next, details of the processing of the integrated dictionary management unit 8 will be described. In the initial stage (step S1 in FIG. 7), the integrated dictionary management unit 8 sequentially reads M language models and M pronunciation dictionaries corresponding to them and expands them into a tree structure, as shown in FIG. 2 and FIG. Generate an integrated dictionary. A word end N-gram table and an intra-word N-gram table are generated corresponding to these M language models and further corresponding to the case of the weighted linear sum of these M language models.

  At this time, the integrated dictionary management unit 8 overlaps the M language models when a word with a common word notation or a word with a common phoneme string (that is, the same pronunciation) is included. Thus, the tree structure data can be shared among the common phoneme strings without expanding into the tree structure of the phoneme strings. As an example of a specific procedure, the integrated dictionary management unit 8, for example, finishes expanding all the words in the vocabulary of the language model 1 into a tree structure, and then uses the same word notation and the same pronunciation ( Phoneme strings) are not expanded again as separate nodes. That is, when the same word notation exists in both the language model 1 and the language model 2, the word end N-gram table associated with the word notation node created for the existing language model 1 is used. Corresponding to the entry, the probability value for the language model 2 (case 2) is written. If a new word notation with the same pronunciation is present in the language model 2, the phoneme string network created for the language model 1 is used as it is, and only the word notation node associated therewith is newly added. create. The integrated dictionary management unit 8 creates integrated dictionary data for the language model 3 and later in the same manner.

  In other words, the integrated dictionary management unit 8 acquires word notation data and word appearance probability values from the language model storage units 6-1 to 6-M corresponding to each of the plurality of language models, and also stores the pronunciation dictionary. The pronunciation data corresponding to each of the word notation data is acquired from the sections 7-1 to 7-M. Then, the integrated dictionary management unit 8 expands the pronunciation data for each word notation data into a tree structure of phoneme string data and writes it in the integrated dictionary storage unit 4. Then, the integrated dictionary management unit 8 writes the word notation data in the integrated dictionary storage unit 4 in association with the expanded phoneme string data of the tree structure. Then, the integrated dictionary management unit 8 writes the word appearance probability value associated with the word notation data in the word end N-gram table of the integrated dictionary storage unit 4 in association with the language model. However, the integrated dictionary management unit 8 is configured to associate each word model with a single word expression data (that is, occupy only a single memory area) when a word having a common word expression exists between the language models. Is written in the word end N-gram table, and there is a single phoneme string data (i.e., a single memory) when there is a word with common pronunciation data (i.e., phoneme string) between language models. The word appearance probability value for each language model is written in the word end N-gram table so as to be related to the chain of phoneme nodes occupying only the region.

  As a result, the integrated dictionary management unit 8 does not need to expand the tree structure for each language model on the memory, and can statically integrate a plurality of language models into a dictionary having a tree structure that eliminates redundancy. It becomes possible. In other words, it is possible to significantly reduce the amount of memory consumed while switching between a plurality of language models, and to reduce the hardware scale of the apparatus.

  If the weight vector stored in the weight vector storage unit 10 has been changed by the manual setting unit 11 or the automatic setting unit 12, the integrated dictionary management unit 8 may change the word end N-gram table and the word The probability value stored in the N-gram table is recalculated and updated. The processing in steps S4 and S10 in FIG. 7 corresponds to this. Since the weight vectors of case 1 and case 2 shown in FIG. 4 are unchanged, it is necessary to update the word end N-gram table and the intra-word N-gram table by changing the vector value itself. Absent. Cases can be switched simply by switching the selection with the control flags in the word end N-gram table and the intra-word N-gram table. On the other hand, since the value of the weight vector can be changed in case X, when it is changed, the integrated dictionary management unit 8 uses the changed vector value to change the word end N-gram table and the intra-word N-gram. Recalculate the probability values in the table.

  In other words, the integrated dictionary management unit 8 uses the weight values obtained from the weight vector storage unit 10 for word appearance probability values corresponding to a plurality of language models in the word end N-gram table and the intra-word N-gram table. By calculating the weighted average, a word appearance probability value of a new language model (corresponding to case X) obtained by combining a plurality of language models is calculated, and the word end N-gram table is calculated using the calculated word appearance probability value. The intra-word N-gram table is updated. This table update is triggered when the weight vector is initially set or when the weight vector is changed during the utterance pause.

  Step S4 in FIG. 7 is a process before starting the reading of the input voice, and therefore there is no special time constraint regarding the table update. On the other hand, since the process of step S10 is a process during pause between utterances, it is necessary to finish the table update process before the start of the next utterance. Therefore, if the recalculation of the probability value is not completed within the time between pauses, the integrated dictionary management unit 8 does not update the word end N-gram table and the intra-word N-gram table at that time, Continue recalculating probability values in the process. In this case, in the next utterance, the correct word search unit 3 searches for the correct word using the values of the word end N-gram table and the intra-word N-gram table before update as they are. In the next utterance pause, the integrated dictionary management unit 8 updates these tables using the recalculated probability values. Such control can prevent the speech recognition processing from being interrupted or delayed.

The following effects can be obtained by the configuration of the speech recognition apparatus 1 described above.
Since it is possible to switch between a plurality of language models and to synthesize them by taking a linear sum of weight vectors, the recognition accuracy is improved even for input speech over a plurality of topics.
When integrating a plurality of language models, information about phoneme strings and words is developed on the memory without duplication, so that memory consumption can be suppressed. As a result, the apparatus scale can be reduced.
Moreover, since the language model used for recognition can be switched only by switching the control flag, it is possible to quickly follow changes in the topic online during the recognition process.
Moreover, when searching using the intra-word N-gram table, candidates with low possibility can be pruned, so that the amount of calculation for searching for correct words can be reduced.
In this way, highly accurate speech recognition can be realized in real time while reducing the amount of memory used.

  Note that the functions of the speech recognition apparatus in the above-described embodiment may be realized by a computer. In that case, a program for realizing each function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, a “computer-readable recording medium” dynamically holds a program for a short time, like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included, and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

Although the embodiment has been described above, the present invention can also be implemented in the following modified example.
For example, in the case of using a language model based on weighted linear combination, when the weight vector for the word chain appearance probability is changed, the integrated dictionary management unit 8 causes the tree structure to be searched in the correct word search for the input speech. The word end N-gram table and the intra-word N-gram table are recalculated and updated only for the intra-word phonemes and the end words in the generalized word dictionary, and the other intra-word phonemes and end words are updated. Do not do it. Thereby, it is possible to quickly respond to the change of the weight vector on-line and to prevent the recognition process from being delayed or interrupted.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The present invention can be used for voice recognition processing in general. For example, it can be used to create caption data for a broadcast program. In particular, it can be used for speech recognition processing in which the topic changes with time.

DESCRIPTION OF SYMBOLS 1 Speech recognition apparatus 2 Acoustic analysis part 3 Correct word search part 4 Integrated dictionary memory | storage parts 6-1, 6-2, ..., 6-M language model memory | storage parts 7-1, 7-2, ..., 7 -M Pronunciation dictionary storage unit 8 Integrated dictionary management unit 9 Acoustic model storage unit 10 Weight vector storage unit (weight value storage unit)
11 Manual setting unit 12 Automatic setting unit 20, 21 Word start node 31-43 Phoneme node 51-61 Word notation node 70, 71 Word end node

Claims (6)

Phoneme string data representing a phoneme string that is a string of phonemes associated with an acoustic model that is a statistic relating to an acoustic feature, word notation data representing a word notation associated with the phoneme string data, and the word notation data An integrated dictionary storage unit that stores word appearance probability value data that holds word appearance probability values that are associated and represent word appearance probability values corresponding to each of a plurality of language models;
An acoustic analyzer that analyzes input speech and outputs acoustic features;
The acoustic score for each phoneme string is calculated based on the relationship between the acoustic feature value output from the acoustic analysis unit and the acoustic model, and the word notation data corresponding to the phoneme string corresponds to a predetermined language model. Correct word search for obtaining a word appearance probability value from the integrated dictionary storage unit, calculating a language score for each word notation, and searching for and outputting a correct word notation based on the calculated acoustic score and the language score And
A speech recognition apparatus comprising: The speech recognition device according to claim 1,
The word appearance probability value data stored in the integrated dictionary storage unit includes language model selection control data indicating which language model is selected,
An integrated dictionary management unit that updates the language model selection control data according to settings,
The correct word search unit obtains the word appearance probability value corresponding to the predetermined language model indicated to be selected by the language model selection control data from the integrated dictionary storage unit, and stores each word notation Calculate the language score of
A speech recognition apparatus characterized by that. The speech recognition device according to claim 2,
A weight value storage unit for storing a weight value for each language model;
The integrated dictionary management unit performs weighted averaging of the word appearance probability values corresponding to a plurality of language models in the word appearance probability value data with the weight values acquired from the weight value storage unit, thereby the plurality of languages Calculating a new word appearance probability value obtained by synthesizing the model, and updating the word appearance probability value data with the calculated word appearance probability value;
A speech recognition apparatus characterized by that. The speech recognition device according to any one of claims 2 and 3,
The integrated dictionary management unit
Corresponding to each of the plurality of language models, the word notation data, the word appearance probability value, and pronunciation data corresponding to each of the word notation data,
For each word notation data, the pronunciation data is expanded into the phoneme string data and written to the integrated dictionary storage unit,
Write the word notation data in the integrated dictionary storage unit in association with the expanded phoneme string data,
The word appearance probability value associated with the word notation data is written in the word appearance probability value data of the integrated dictionary storage unit in association with the language model,
When there is a word having a common word notation among language models, the word appearance probability value for each language model is written in the word appearance probability value data so as to be associated with a single word notation data, When there is a common word in pronunciation data, the word appearance probability value for each language model is written to the word appearance probability value data so as to be associated with a single phoneme string data.
A speech recognition apparatus characterized by that. The speech recognition device according to any one of claims 1 to 4,
The integrated dictionary storage unit is associated with all the word notation data sharing the partial phoneme sequence for each language model corresponding to the partial phoneme sequence from the word start to the mid-word phoneme in the phoneme sequence data. In-word probability value data that holds the maximum value of the word appearance probability value as an in-word probability value corresponding to the partial phoneme sequence, is also stored.
The correct word search unit excludes word notation data sharing the corresponding partial phoneme sequence from search targets when the intra-word probability value is lower than a predetermined threshold,
A speech recognition apparatus characterized by that. Phoneme string data representing a phoneme string that is a string of phonemes associated with an acoustic model that is a statistic relating to an acoustic feature, word notation data representing a word notation associated with the phoneme string data, and the word notation data An integrated dictionary storage unit that stores word appearance probability value data that holds word appearance probability values that are associated and represent word appearance probability values corresponding to each of a plurality of language models;
An acoustic analyzer that analyzes input speech and outputs acoustic features;
The acoustic feature amount output from the acoustic analysis unit calculates an acoustic score for each phoneme string according to the relationship with the acoustic model, and corresponds to a predetermined language model for the word notation data corresponding to the phoneme string The word appearance probability value data to be obtained is obtained from the integrated dictionary storage unit, a language score is calculated for each word notation, and a correct word notation is searched and output based on the calculated acoustic score and the language score. A correct word search unit;
A program for causing a computer to function as a speech recognition apparatus.

Images