JP2019015950A - Voice recognition method, program, voice recognition device, and robot - Google Patents

Abstract

To improve the recognition accuracy in the case of an infant speaker, or even under the circumstance in which an input utterance is greatly influenced by the noise.SOLUTION: The speech recognition method includes extracting a first utterance from sound picked up by a microphone corresponding to a speech processing device, calculating the reliability of a first utterance recognition result and the first utterance, exerting an utterance of repeat asking on the basis of the calculated reliability of the first utterance, extracting a second utterance obtained by the repeat asking by the microphone, calculating the reliability of a second utterance recognition result and the second utterance, and generating a recognition result from the first utterance recognition result and the second utterance recognition result on the basis of the calculated reliability of the second utterance.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to a speech recognition technique.

  In recent years, various speech recognition methods have been proposed for estimating a word string indicating utterance content from uttered speech data.

  For example, Patent Document 1 discloses the following speech recognition method. That is, the uttered speech data is divided into a plurality of phoneme sections x, a phoneme model is assigned to each phoneme section x, and the likelihood Psn of the phoneme model p assigned to the nth phoneme section x and the phoneme model p other than The maximum likelihood value Pmaxn of the phoneme section x and the differential likelihood Pdn of the likelihood Psn are obtained. Then, the likelihood Psn and the difference likelihood Pdn are respectively input to the correct phoneme segment likelihood model and the incorrect solution phoneme likelihood model, and the likelihood Lnc of the correct phoneme segment likelihood model and the incorrect phoneme likelihood model The likelihood Lni is obtained. Then, the difference likelihood cn between the likelihood Lnc and the likelihood Lni is obtained, the addition value of all phoneme sections of the difference likelihood cn is obtained as the word reliability WC, and if the word reliability WC is equal to or greater than the threshold, the speech The phoneme sequence assigned to the data is output, and if the word reliability WC is less than the threshold, the phoneme sequence is rejected.

  However, since Patent Document 1 considers only phoneme likelihood and does not consider language likelihood, there is a problem that a phoneme string having naturalness as a language cannot be reproduced.

  Therefore, Non-Patent Document 1 discloses a method for estimating a word string by recognizing speech by using acoustic likelihood and language likelihood. Specifically, in Non-Patent Document 1, the word string W that maximizes the product of the probabilities shown on the right side of Equation (1) is selected as the recognition result. Here, w is an arbitrary word string, and P (O | w) is a probability (acoustic likelihood) that the phoneme string of the word string w is O, and is calculated by an acoustic model. P (w) is a probability (language likelihood) indicating the likelihood of w as a language, and is calculated by a language model based on appearance frequency information of consecutive words such as n-gram.

  In this method, the product of the acoustic likelihood and the language likelihood is output together with the recognition result as the likelihood of the recognition result (text reliability).

JP-A-11-249688

Takaaki Hori and Gen Tsukada, Journal of Information Processing Society of Japan, Vol. 45, No. 10, PP.1020-1026. Speech recognition with weighted finite state transducers, October 2004

  However, in Non-Patent Document 1, if the input utterance is ambiguous or the input utterance is strongly influenced by noise or the like, the reliability of the sentence is low. And when the reliability of a sentence is low, there exists a subject that possibility that an error will be included in a recognition result becomes high.

  This indication is made in order to solve such a subject.

A speech recognition method according to an aspect of the present disclosure includes:
A first utterance spoken by a speaker intended for a word is received via a microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
For each of the N phonemes constituting the first utterance, the appearance probabilities of all types of phonemes are calculated,
A phoneme string in which phonemes having the maximum appearance probability corresponding to each of the N phonemes from the first phoneme constituting the first utterance to the Nth phoneme are arranged in order; a first phoneme string corresponding to the first utterance; Recognized,
A first value is calculated by multiplying appearance probabilities of N phonemes constituting the first phoneme string;
If the first value is less than a first threshold, let the speaker output a voice prompting the speaker to speak the one word again,
A second utterance re-spoken by the speaker with the intention of the one word is received via the microphone, and the second utterance is composed of M (M is a natural number of 2 or more) phonemes,
For each of the M phonemes constituting the second utterance, the appearance probability is calculated for all types of phonemes,
A phoneme string in which phonemes having the maximum appearance probability corresponding to each of the M phonemes from the first phoneme constituting the second utterance to the Mth phoneme are arranged in order; a second phoneme string corresponding to the second utterance; Recognized,
A second value is calculated by multiplying the appearance probabilities of M phonemes constituting the second phoneme sequence;
When the second value is less than the first threshold, phonemes having an appearance probability equal to or higher than the second threshold in the first phoneme string and phonemes having an appearance probability equal to or higher than the second threshold in the second phoneme string Extract and
Extracting words including the extracted phonemes from a dictionary stored in a memory, the dictionary associates each word with a phoneme string corresponding to each word,
If the extracted word is one, the extracted word is recognized as corresponding to the one word.

  The present disclosure can improve recognition accuracy even when the speaker is an infant or in an environment where the input utterance is greatly affected by noise.

It is a figure which shows an example of the whole structure of the speech dialogue system in Embodiment 1. FIG. It is a figure which shows an example of the appearance probability calculated for every phoneme in the utterance which consists of two phonemes. FIG. 3 is a diagram summarizing products of appearance probabilities for combinations of phonemes of the first phoneme and phonemes of the second phoneme in FIG. 2. 3 is a flowchart illustrating an example of recognition processing in the first embodiment. 5 is a diagram illustrating an example of a conversation in Embodiment 1. FIG. It is a figure which shows an example of the 1st recognition result and the 2nd recognition result with respect to the example of a dialog of FIG. It is a figure which shows an example of the data structure of a word dictionary. It is a figure which shows an example of the recognition candidate word extracted from the 1st recognition result. In Embodiment 1, it is a figure which shows another example of the process which narrows down a recognition candidate word from a 1st recognition result and a 2nd recognition result. It is a figure which shows an example of the whole structure of the voice interactive system in Embodiment 2. FIG. It is a figure which shows an example of the audio | voice signal divided | segmented into the some flame | frame. 10 is a flowchart illustrating an example of recognition processing in the second embodiment. FIG. 11 is a diagram illustrating an example of a search space when a 1-gram language model is employed in a specific example of the second embodiment. FIG. 10 is a diagram showing an example of a word dictionary when a 2-gram language model is adopted in a specific example of the second embodiment. FIG. 10 is a diagram illustrating an example of a search space when a 2-gram language model is employed in a specific example of the second embodiment. It is a figure which shows the search space when the appearance probability of each phoneme of the 1st recognition result and each phoneme of the 2nd recognition result in the specific example of Embodiment 2 is synthesize | combined. FIG. 10 is a diagram showing an example of the overall configuration of a voice interaction system in a third embodiment. 10 is a flowchart for explaining an example of recognition processing in the third embodiment. FIG. 10 is a diagram illustrating an example of 5-best of the first recognition result in the third embodiment. FIG. 20 is a diagram illustrating an example of 5-best of the second recognition result in the third embodiment. It is an external view of the robot by which the speech recognition apparatus which concerns on Embodiment 1-3 is mounted.

(Knowledge that became the basis of this disclosure)
Voice that analyzes utterance content from the voice uttered by the user and returns a natural response based on the analysis result to realize a natural dialogue with the user, or provide services such as device control or information provision Technologies related to dialogue systems are being studied.

  In a general voice recognition system for adults, the recognition accuracy exceeds 90%, and even if it cannot be recognized, the recognition result with low reliability is discarded and the speech is slowly spoken by listening back, A recognition result with high reliability can be acquired sufficiently by having the user speak clearly.

  However, in a general speech recognition system, the recognition accuracy is low in an environment where the speech of an infant in the language acquisition stage or the input speech is greatly affected by noise, so even if it is listened to again, a highly reliable recognition result There is a problem that cannot be obtained.

  In Non-Patent Document 1, although a word string having language-likeness can be output, the above problem cannot be solved because there is no disclosure about listening back when a recognition result with low reliability is obtained.

  In Patent Document 1, when a recognition result with low reliability is obtained, it is only disclosed that the recognition result is discarded, and there is no disclosure about replaying. Can't solve the problem.

  Therefore, the present inventor does not discard the recognition result with low reliability as it is, but considers the recognition result and the recognition result obtained by listening back, when the speaker is an infant or the input utterance is The present disclosure has been conceived by obtaining knowledge that recognition accuracy can be improved even in an environment that is greatly affected by noise.

A speech recognition method according to an aspect of the present disclosure includes:
A first utterance spoken by a speaker intended for a word is received via a microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
For each of the N phonemes constituting the first utterance, the appearance probabilities of all types of phonemes are calculated,
A phoneme string in which phonemes having the maximum appearance probability corresponding to each of the N phonemes from the first phoneme constituting the first utterance to the Nth phoneme are arranged in order; a first phoneme string corresponding to the first utterance; Recognized,
A first value is calculated by multiplying appearance probabilities of N phonemes constituting the first phoneme string;
If the first value is less than a first threshold, let the speaker output a voice prompting the speaker to speak the one word again,
A second utterance re-spoken by the speaker with the intention of the one word is received via the microphone, and the second utterance is composed of M (M is a natural number of 2 or more) phonemes,
For each of the M phonemes constituting the second utterance, the appearance probability is calculated for all types of phonemes,
A phoneme string in which phonemes having the maximum appearance probability corresponding to each of the M phonemes from the first phoneme constituting the second utterance to the Mth phoneme are arranged in order; a second phoneme string corresponding to the second utterance; Recognized,
A second value is calculated by multiplying the appearance probabilities of M phonemes constituting the second phoneme sequence;
When the second value is less than the first threshold, phonemes having an appearance probability equal to or higher than the second threshold in the first phoneme string and phonemes having an appearance probability equal to or higher than the second threshold in the second phoneme string Extract and
Extracting words including the extracted phonemes from a dictionary stored in a memory, the dictionary associates each word with a phoneme string corresponding to each word,
If the extracted word is one, the extracted word is recognized as corresponding to the one word.

  According to this configuration, the first value of the first phoneme string obtained by recognizing the first utterance intended for one word is lower than the first threshold value, and the reliability of the first phoneme string is low. However, the first phoneme string is not discarded. Then, when the second value of the second utterance intended for one word obtained by listening is lower than the first threshold and the reliability of the second phoneme sequence is low, each of the first phoneme sequence and the second phoneme sequence Therefore, a phoneme with high reliability is extracted, and a word corresponding to one word is extracted by comparing with a dictionary.

  Thus, even if a recognition result with low reliability is obtained for the first utterance, this configuration does not discard the recognition result and recognizes the recognition result with low reliability for the second utterance. Use when results are obtained. Therefore, even if a highly reliable recognition result is not obtained by listening back, one word is recognized using a reliable phoneme from the first phoneme sequence and the second phoneme sequence that are both recognition results. Therefore, the recognition accuracy of one word can be improved.

  Furthermore, in this configuration, since words including highly reliable phonemes from the first phoneme string and the second phoneme string are extracted from the dictionary, it is possible to prevent a linguistically unnatural recognition result from being obtained. .

  As described above, this configuration can improve the recognition accuracy even when the speaker is an infant or in an environment where the input utterance is greatly affected by noise.

In the above configuration, when there are a plurality of extracted words, a voice asking the speaker whether the extracted words are spoken or not is output through the speaker.
Receiving a positive or negative answer from the speaker via the microphone;
The word corresponding to the positive answer may be recognized as corresponding to the one word.

  According to this configuration, when a plurality of words including a reliable phoneme are extracted from the dictionary from the first phoneme string and the second phoneme string, the speaker is directly confirmed as to which word is spoken. Therefore, recognition accuracy can be improved.

ｔは、前記第１発話を構成するフレームを指定する番号を示し、
Ｔは、前記第１発話を構成するフレームの総数を示し、
Ｐ_Ａ１（ｏ_ｔ，ｓ_ｔ｜ｓ_ｔ−１）は、前記第１発話の１番フレームからｔ−１番フレームまでの状態ｓ_ｔ−１に対応する音素列の次に、ｔ番フレームで任意の音素が出現し、状態ｓ_ｔに対応する音素列に遷移する確率を示し、
ｏ_ｔは前記第１発話から得られ、前記任意の音素を推定するための物理量を示し、
前記任意の音素は全種類の音素を示し、
Ｐ_Ｌ１（ｓ_ｔ，ｓ_ｔ−１）は、前記第１発話において前記状態ｓ_ｔ−１に対応する単語列の次に、ｔ番フレームで任意の単語が出現し、前記状態ｓ_ｔに対応する単語列に遷移する確率を示し、
前記信頼度Ｘ１が閾値以上であるか判定し、
前記信頼度Ｘ１が前記閾値未満である場合は、前記発話者へ前記一の単語列を再度発話するように促す音声をスピーカを通して出力させ、
前記一の単語列を意図して前記発話者によって再度発話された第２発話を、前記マイクを介して受信し、
前記第２発話の信頼度Ｘ１が前記閾値未満である場合は、前記第１発話と前記第２発話から推定される全ての単語列に対して合成信頼度Ｘを算出し、

ｔは、前記第１発話および前記第２発話を構成するフレームを指定する番号を示し、
Ｔは、前記第１発話および前記第２発話を構成するフレームの総数を示し、
Ｐ_Ａ１（ｏ_ｔ，ｓ_ｔ｜ｓ_ｔ−１）は、前記第１発話の１番フレームからｔ−１番フレームまでの状態ｓ_ｔ−１に対応する音素列の次に、ｔ番フレームで任意の音素が出現し、状態ｓ_ｔに対応する音素列に遷移する確率を示し、
ｏ_ｔは、前記第１発話から得られ、前記任意の音素を推定するための物理量を示し、
前記任意の音素は、全種類の音素を示し、
Ｐ_Ａ２（ｑ_ｔ，ｓ_ｔ｜ｓ_ｔ−１）は、前記第２発話の１番フレームからｔ−１番フレームまでの状態ｓ_ｔ−１に対応する音素列の次に、ｔ番フレームで任意の音素が出現し、状態ｓ_ｔに対応する音素列に遷移する確率を示し、
ｑ_ｔは前記第２発話から得られ、前記任意の音素を推定するための物理量を示し、
Ｐ_Ｌ（ｓ_ｔ，ｓ_ｔ−１）は、前記第１発話において前記状態ｓ_ｔ−１に対応する単語列の次に、ｔ番フレームで任意の単語が出現し、前記状態ｓ_ｔに対応する単語列に遷移する確率を示し、
前記合成信頼度Ｘのうち最大値を与える前記状態ｓ_ｔに対応する単語列を、前記一の単語列として認識する。 A voice recognition method according to another aspect of the present disclosure includes:
A first utterance uttered by a speaker with the intention of a single word string is received via a microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
Calculating the reliability X1 of the word string estimated for the first utterance;

t indicates a number for specifying a frame constituting the first utterance;
T indicates the total number of frames constituting the first utterance,
P _A1 (o _t , s _t | s _t-1 ) is the t-th frame after the phoneme string corresponding to the state s _t-1 from the first frame to the t-1 frame of the first utterance. any phoneme appeared, indicates the probability of transition to the phoneme string corresponding to the state s _t,
o _t is obtained from the first utterance and indicates a physical quantity for estimating the arbitrary phoneme;
The arbitrary phonemes represent all types of phonemes,
_{P L1 (s t, s t} -1) is the next word string corresponding to the state _{s t-1} in the first utterance, any word is found at t-th frame, corresponding to the state _{s t} The probability of transition to a word string
Determining whether the reliability X1 is greater than or equal to a threshold;
If the reliability X1 is less than the threshold, a voice prompting the speaker to speak the one word string again is output through a speaker;
Receiving a second utterance re-uttered by the speaker with the intention of the one word string, via the microphone;
When the reliability X1 of the second utterance is less than the threshold value, the composite reliability X is calculated for all word strings estimated from the first utterance and the second utterance,

t indicates a number that designates a frame constituting the first utterance and the second utterance;
T represents the total number of frames constituting the first utterance and the second utterance,
P _A1 (o _t , s _t | s _t-1 ) is the t-th frame after the phoneme string corresponding to the state s _t-1 from the first frame to the t-1 frame of the first utterance. any phoneme appeared, indicates the probability of transition to the phoneme string corresponding to the state s _t,
o _t is a physical quantity obtained from the first utterance and used to estimate the arbitrary phoneme;
The arbitrary phonemes represent all types of phonemes,
P _A2 (q _t , s _t | s _t−1 ) is the t th frame after the phoneme string corresponding to the state s _t−1 from the first frame to the t−1 frame of the second utterance. any phoneme appeared, indicates the probability of transition to the phoneme string corresponding to the state s _t,
q _t is obtained from the second utterance and indicates a physical quantity for estimating the arbitrary phoneme;
_{P L (s t, s t} -1) is the next word string corresponding to the state _{s t-1} in the first utterance, any word is found at t-th frame, corresponding to the state _{s t} The probability of transition to a word string
A word string corresponding to the state s _t giving the maximum value of the combined confidence X, recognized as the one word sequence.

According to this configuration, the first utterance intended for one word sequence is divided into the T frame, when a transition to a state s _t from state s _t-1 to t-1 th frame to the t-th frame A word sequence that maximizes the product of the probability P _A1 (o _t , s _t | s _t-1 ) of the phoneme sequence and the probability P _L1 (s _t , s _t-1 ) of the word sequence Recognized as a column.

And even if the reliability X1 of the word sequence of the first utterance is lower than the threshold value and the reliability of the word sequence of the first utterance is low, the word sequence of the first utterance is not discarded. Then, lower than the reliability X1 is the threshold of the word sequence of the second speech intended for one word string obtained by reflective listening, if the reliability of the word sequence of the second utterance also low, the first utterance in the state s _t The sum of the phoneme sequence probability P _A1 (o _t , s _t | s _t-1 ) and the phoneme sequence probability P _A2 (q _t , s _t | s _t-1 ) of the second utterance, and the state s _t The product of the word string probabilities P _L (s _t , s _t−1 ) is calculated as the combined reliability X, and the word string that maximizes the combined reliability X is recognized as one word.

  Thus, even if a recognition result with low reliability is obtained for the first utterance, this configuration does not discard the recognition result and recognizes the recognition result with low reliability for the second utterance. Use when results are obtained. For this reason, even if a highly reliable recognition result cannot be obtained by listening back, since one word string is recognized by combining both recognition results, the recognition accuracy of one word string can be improved. .

  Furthermore, in this configuration, not only the probability of the phoneme string but also the probability of the word string is taken into consideration, so that a linguistically unnatural recognition result can be prevented.

  As described above, this configuration can improve the recognition accuracy even when the speaker is an infant or in an environment where the input utterance is greatly affected by noise.

ｔ１は、前記第１発話を構成するフレームを指定する番号を示し、
Ｔ１は、前記第１発話を構成するフレームの総数を示し、
Ｐ_Ａ１（ｏ_ｔ１，ｓ_ｔ１｜ｓ_ｔ１−１）は、前記第１発話の１番フレームからｔ１−１番フレームまでの状態ｓ_ｔ１−１に対応する音素列の次に、ｔ１番フレームで任意の音素が出現し、状態ｓ_ｔ１に対応する音素列に遷移する確率を示し、
ｏ_ｔ１は前記第１発話から得られ、前記任意の音素を推定するための物理量を示し、
前記任意の音素は全種類の音素を示し、Ｐ_Ｌ１（ｓ_ｔ１，ｓ_ｔ１−１）は、前記第１発話において前記状態ｓ_ｔ１−１に対応する単語列の次に、ｔ１番フレームで任意の単語が出現し、前記状態ｓ_ｔ１に対応する単語列に遷移する確率を示し、
前記信頼度Ｘ１の最大値ＭａｘＸ１が閾値以上であるか判定し、
前記最大値ＭａｘＸ１が前記閾値未満である場合は、
前記信頼度Ｘ１の上位Ｍ個（Ｍは２以上の自然数）を与える前記第１発話に対して推定される第１単語列を抽出し、
前記発話者へ前記一の単語列を再度発話するように促す音声をスピーカを通して出力させ、
前記一の単語列を意図して前記発話者によって再度発話された第２発話を、マイクを介して受信し、
前記第２発話に対して推定される全ての単語列の信頼度Ｘ２を算出し、

ｔ２は、前記第２発話を構成するフレームを指定する番号を示し、
Ｔ２は、前記第２発話を構成するフレームの総数を示し、
Ｐ_Ａ２（ｏ_ｔ２，ｓ_ｔ２｜ｓ_ｔ２−１）は、前記第２発話の１番フレームからｔ２−１番フレームまでの状態ｓ_ｔ２−１に対応する音素列の次に、ｔ２番フレームで任意の音素が出現し、状態ｓ_ｔ２に対応する音素列に遷移する確率を示し、
ｏ_ｔ２は前記第２発話から得られ、前記任意の音素を推定するための物理量を示し、
Ｐ_Ｌ２（ｓ_ｔ２，ｓ_ｔ２−１）は、前記第２発話において前記状態ｓ_ｔ２−１に対応する単語列の次に、ｔ２番フレームで任意の単語が出現し、前記状態ｓ_ｔ２に対応する単語列に遷移する確率を示し、
前記信頼度Ｘ２の最大値ＭａｘＸ２が閾値以上であるか判定し、
前記最大値ＭａｘＸ２が前記閾値未満である場合は、前記信頼度Ｘ２の前記上位Ｍ個を与える前記第２発話に対して推定される第２単語列を抽出し、
前記第１単語列と前記第２単語列とに共通する単語列がある場合は、前記共通する単語列を前記一の単語列として認識する。 A speech recognition method according to still another aspect of the present disclosure is provided.
A first utterance uttered by a speaker with the intention of a single word string is received via a microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
Calculating reliability X1 of all word strings estimated for the first utterance;

t1 indicates a number for specifying a frame constituting the first utterance;
T1 indicates the total number of frames constituting the first utterance,
P _A1 (o _t1 , s _t1 | s _t1-1 ) is the t1 frame after the phoneme sequence corresponding to the state s _t1-1 from the first frame to the t1-1 frame of the first utterance. Indicates the probability that an arbitrary phoneme appears and transitions to a phoneme string corresponding to the state s _t1 ,
o _t1 is obtained from the first utterance and indicates a physical quantity for estimating the arbitrary phoneme;
The arbitrary phonemes indicate all types of phonemes, and P _L1 (s _t1 , s _t1-1 ) is arbitrary in the frame t1 next to the word string corresponding to the state s _t1-1 in the first utterance. And the probability of transition to a word string corresponding to the state s _t1 ,
Determining whether the maximum value MaxX1 of the reliability X1 is equal to or greater than a threshold;
When the maximum value MaxX1 is less than the threshold value,
Extracting a first word string estimated for the first utterance giving the top M pieces of reliability X1 (M is a natural number of 2 or more);
Outputting a voice prompting the speaker to speak the one word string again through a speaker;
Receiving a second utterance re-spoken by the speaker with the intention of the one word string, via a microphone;
Calculating reliability X2 of all word strings estimated for the second utterance;

t2 indicates a number that designates a frame constituting the second utterance;
T2 indicates the total number of frames constituting the second utterance,
_{P A2 (o t2, s t2} | s t2-1) is the next phoneme string corresponding to the state _{s t2-1} to t2-1 numbered frame from 1 numbered frame of the second speech, at t2 numbered frame Indicates the probability that an arbitrary phoneme appears and transitions to a phoneme string corresponding to the state s _t2 ,
o _t2 is obtained from the second utterance and indicates a physical quantity for estimating the arbitrary phoneme;
P _L2 (s _t2 , s _t2-1 ) corresponds to the state s _t2 when an arbitrary word appears in the t2 frame after the word string corresponding to the state s _t2-1 in the second utterance. The probability of transition to a word string
It is determined whether the maximum value MaxX2 of the reliability X2 is greater than or equal to a threshold value,
If the maximum value MaxX2 is less than the threshold, extract a second word string estimated for the second utterance that gives the top M pieces of the reliability X2,
When there is a word string common to the first word string and the second word string, the common word string is recognized as the one word string.

According to this configuration, the first utterance intended for one word sequence is divided into the T frame, when a transition to a state s _t from state s _t-1 to t-1 th frame to the t-th frame Of the phoneme string P _A1 (o _t , s _t | s _t-1 ) and the word string probability P _L1 (s _t , s _t-1 ) are calculated as the reliability X1.

  When the maximum value MaxX1 of the reliability X1 is lower than the threshold value and the reliability of the word string recognized from the first utterance is low, the first word string having the top M reliability X1 is extracted, Two utterances are obtained.

  If the maximum value MaxX2 of the reliability X2 of the word sequence of the second utterance is lower than the threshold value and the reliability of the word sequence of the second utterance is also low, the second word sequence having the top M reliability X2 is extracted. When there is a common word string in the first word string and the second word string, the common word string is recognized as one word string.

  Thus, even if a recognition result with low reliability is obtained for the first utterance, this configuration does not discard the recognition result and recognizes the recognition result with low reliability for the second utterance. Use when results are obtained. For this reason, even if a highly reliable recognition result is not obtained by listening back, the word sequence recognized in both the first utterance and the second utterance is recognized as one word sequence, so one word Column recognition accuracy can be increased.

  Furthermore, in this configuration, not only the probability of the phoneme string but also the probability of the word string is taken into consideration, so that a linguistically unnatural recognition result can be prevented.

  As described above, this configuration can improve the recognition accuracy even when the speaker is an infant or in an environment where the input utterance is greatly affected by noise.

  The above speech recognition method may be applied to a robot.

  In addition, the present disclosure can be realized not only as a speech recognition method that performs the characteristic processing as described above, but also includes a speech recognition unit that includes a processing unit for executing the characteristic steps included in the speech recognition method. It can also be realized as a device. Moreover, it can also be realized as a computer program that causes a computer to execute the characteristic steps included in such a speech recognition method. Needless to say, such a computer program can be distributed via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

  Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. Numerical values, shapes, components, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements. In all the embodiments, the contents can be combined.

(Embodiment 1)
FIG. 1 is a diagram illustrating an example of the overall configuration of the voice interaction system according to the first embodiment. The voice interaction system shown in FIG. 1 includes a voice recognition device 100, a microphone 400, a speaker 410, a service application server 420, and a control device 430.

  The speech recognition apparatus includes 100, a CPU (Central Processing Unit) 20 as a processor, and a memory 30. The CPU 20 includes a speech recognition unit 200, a word reliability determination unit 210, an intention interpretation unit 220, an action selection unit 230, a response generation unit 240, a speech synthesis unit 250, and an utterance extraction unit 260. The memory 30 includes a word dictionary 301 and a recognition result storage unit 302. The speech recognition unit 200 includes a phoneme estimation unit 201, a word estimation unit 202, and a phoneme appearance probability determination unit 203.

  The word dictionary 301 stores combinations of words and phoneme strings that can be recognized by the speech recognition apparatus 100. FIG. 7 is a diagram illustrating an example of a data configuration of the word dictionary. In the word dictionary, words such as “mango” and “brick” and a phoneme string of each word such as “mango:” and “renga” are stored in association with each other.

  Returning to FIG. A program that causes a computer to function as the speech recognition apparatus 100 is stored in a memory 30 incorporated in a robot or a terminal in which the speech recognition apparatus 100 is mounted, and is executed by a processor such as the CPU 20. All elements constituting the speech recognition apparatus 100 may be mounted on the same terminal, or on another terminal or server connected via an arbitrary network such as an optical fiber, a radio, or a public telephone line. It may be implemented individually, and the voice interaction processing may be realized by the voice recognition device 100 and another terminal or server communicating with each other.

  The microphone 400 is constituted by, for example, a directional microphone, and is incorporated in a terminal or a robot on which the speech recognition apparatus 100 is mounted. Moreover, the microphone 400 may be configured by an arbitrary sound collection device such as a hand microphone, a pin microphone, or a table microphone. In this case, the microphone 400 is connected to a terminal on which the speech recognition apparatus 100 is mounted via wired or wireless. Moreover, the microphone 400 may be comprised with the microphone mounted in the device with sound collection and communication functions, such as a smart phone or a tablet terminal.

  The speaker 410 may be incorporated in a terminal or robot on which the voice recognition device 100 is mounted, or may be connected to a terminal or robot on which the voice recognition device 100 is mounted via a wired or wireless connection. The speaker 410 may be configured by a speaker mounted on a device having sound collection and communication functions such as a smartphone or a tablet terminal.

  The service application server 420 is a cloud server that provides users with a plurality of services such as weather, storytelling, news, and games via a network. For example, the service application server 420 acquires a speech recognition result by the speech recognition apparatus 100 and determines a service to be executed according to the recognition result. The service provided by the service application server 420 may be realized by a program having a function of acquiring an execution result in the service application server 420 via a network, or implemented by the service application server 420 and the speech recognition apparatus 100. It may be realized by a robot stored in the memory or a program stored in a memory on the terminal.

  The control device 430 is configured by a device such as a television or an air conditioner connected to the speech recognition device 100 by wire or wireless, and is a device that is controlled by receiving a speech recognition result from the speech recognition device 100.

  The speech extraction unit 260 extracts a speech signal being spoken from the speech signal output from the microphone 400 and outputs the speech signal to the phoneme estimation unit 201. Here, the utterance extraction unit 260 detects that the utterance has been started, for example, when a sound of a predetermined volume or higher continues for a certain period of time, and outputs an audio signal input from the microphone 400 to the phoneme estimation unit 201. Start. In addition, when the speech extraction unit 260 detects that the voice having a volume lower than the predetermined volume has continued for a predetermined period or longer, the speech extraction unit 260 stops outputting the voice signal to the phoneme estimation unit 201. In the present embodiment, it is assumed that the utterance extraction unit 260 extracts a voice signal of a voice uttered by a speaker with the intention of one word. The speaker is an infant in the language acquisition stage.

  The phoneme estimation unit 201 divides the speech signal input by the utterance extraction unit 260 into a plurality of speech sections configured in a predetermined time unit, and calculates the appearance probabilities of all types of phonemes in each phoneme section. A phoneme refers to a minimum unit of speech in a language, and is represented by symbols such as “a” and “i”, for example. All types of phonemes refer to all phonemes used for speech. All types of phonemes are modeled by acoustic models. As an acoustic model, HMM (Hidden Markov Model: Hidden Markov Model) is mentioned, for example.

  The number of phoneme types varies depending on the language, but is about 40 for Japanese. Here, the phoneme estimation unit 201 may estimate a phoneme string by collecting continuous common phoneme segments as one phoneme segment using an HMM. Then, the phoneme estimation unit 201 estimates a phoneme combination that maximizes the product of appearance probabilities in all phoneme sections as a phoneme string uttered by the speaker.

  The word estimation unit 202 extracts from the word dictionary 301 the word that most closely matches the phoneme string estimated by the phoneme estimation unit 201, and estimates the extracted word as a word spoken by the speaker.

  FIG. 2 is a diagram illustrating an example of the appearance probability calculated for each phoneme in an utterance composed of two phonemes. FIG. 3 is a diagram summarizing the products of appearance probabilities for combinations of phonemes of the first phoneme and phonemes of the second phoneme in FIG. 2.

  For example, it is assumed that a word composed of two phonemes is uttered and the appearance probability of phonemes shown in FIG. 2 is obtained. In FIG. 2, the appearance probabilities of phonemes “a” and “u” are calculated as “0.4” and “0.5” for the first phoneme, respectively, and phonemes “i” and “i” for the second phoneme. The appearance probabilities of “e” are calculated as “0.3” and “0.6”, respectively.

  In this case, “ai”, “ae”, “ui”, and “ue” are obtained as phoneme combinations of the first phoneme and the second phoneme, and the product of the appearance probabilities of each combination is “0.12”. ”,“ 0.24 ”,“ 0.15 ”, and“ 0.30 ”.

  Therefore, the combination having the maximum product appearance probability of the first phoneme and the second phoneme is “ue” whose appearance probability is “0.30”. At this time, the word dictionary 301 is searched for the phoneme string “ue”, and a word that matches the phoneme string “ue” is output as a recognition result. At this time, the product of the appearance probabilities of each phoneme, that is, “ue” = “0.30” is the reliability of the recognized word.

  The word reliability determination unit 210 compares the word reliability recognized by the word estimation unit 202 (an example of the first value and the second value) with a predetermined threshold value TH1 (an example of the first threshold value) to determine the word When the reliability of the first utterance is less than the threshold TH1, a recognition result including the phoneme string of the word recognized by the word estimation unit 202 and the appearance probability of each phoneme is stored in the recognition result storage unit 302 as the first recognition for the first utterance. Remember as a result. In this case, the word reliability determination unit 210 outputs, to the response generation unit 240, a voice generation instruction that prompts re-speaking in order for the speaker to speak one word again.

  The word reliability determination unit 210 performs the second utterance intended for one word to the speaker by re-utterance, and when the word recognition unit 202 obtains the second recognition result, the reliability of the second recognition result is It is determined whether or not the threshold value is less than TH1.

  The phoneme appearance probability determination unit 203 recognizes the word recognized by the first recognition result and the second recognition result when the word reliability determination unit 210 determines that the reliability of the second recognition result is less than the threshold TH1. For each of the words that have been generated, a phoneme having a phoneme appearance probability greater than or equal to the threshold TH2 is extracted.

  The word estimation unit 202 extracts a word including the phoneme string extracted by the phoneme appearance probability determination unit 203 from the word dictionary 301, and determines a word as a final recognition result based on the extraction result.

  Moreover, the word reliability determination part 210 outputs a recognition result to the intention interpretation part 220, when the word reliability is more than threshold TH1.

  The intention interpretation unit 220 estimates the type of response (for example, a match or a question answer) and the type of action (such as shiritori, hide and seek, or television control) from the recognition result. Then, the intention interpretation unit 220 outputs the response type estimation result to the response generation unit 240 and also outputs the behavior type estimation result to the behavior selection unit 230.

  The action selection unit 230 determines the service to be executed or the control device 430 to be controlled from the estimation result of the intention interpretation unit 220. If the action selection unit 230 determines to execute the service, the action selection unit 230 transmits a request to provide the service to be executed to the service application server 420. In addition, when the action selection unit 230 determines to control the control device 430, the behavior selection unit 230 outputs a control instruction to the control device 430 to be controlled.

  When the response generation unit 240 acquires an estimation result of the response type from the intention interpretation unit 220, the response generation unit 240 generates a response sentence corresponding to the estimation result. On the other hand, when the response generation unit 240 obtains a voice generation instruction for prompting re-speech from the word reliability determination unit 210, the response generation unit 240 generates a response sentence to hear back one word from the speaker.

  The voice synthesis unit 250 converts the response sentence generated by the response generation unit 240 into a voice signal and outputs the voice signal to the speaker 410. The speaker 410 converts the voice signal output from the voice synthesizer 250 into voice and outputs the voice to the outside.

  FIG. 4 is a flowchart illustrating an example of recognition processing in the first embodiment. First, the utterance extraction unit 260 determines whether or not there is a voice input in the microphone 400 (step S100). When it is determined that there is no voice input (NO in step S100), the process of step S100 is repeated until there is a voice input.

  On the other hand, when it is determined that there is a voice input (YES in step S100), the utterance extraction unit 260 extracts a voice signal being uttered from the voice signal output from the microphone 400 (step S101).

  Next, the speech recognition unit 200 performs speech recognition processing (step S102). Specifically, the phoneme estimation unit 201 divides the speech signal extracted by the utterance extraction unit 260 into a plurality of speech sections, generates a feature amount of the speech signal in each speech section, and uses the generated feature amount as an acoustic model. By collating, the phonemes of each speech segment are estimated. At this time, the phoneme estimation unit 201 calculates the appearance probability of phonemes for each speech section, and combines consecutive speech sections of the same phoneme by using the HMM. For example, assuming that the first phoneme, the second phoneme, and the third phoneme that make up the uttered speech are configured, the phoneme estimation unit 201 performs each of the first phoneme, the second phoneme, and the third phoneme. Appearance probability of all types of phonemes is calculated.

  For example, the first phoneme has all types such that the probability of the phoneme “a” is “0.4”, the probability of the phoneme “i” is “0.1”, and the probability of the phoneme “u” is “0.2”. For each of the phonemes, the appearance probability of the first phoneme is calculated. For the second phoneme and the third phoneme, the appearance probabilities of all types of phonemes are calculated in the same manner as the first phoneme.

  Then, the phoneme estimation unit 201 estimates a combination of three phonemes that maximizes the product of the first phoneme appearance probability, the second phoneme appearance probability, and the third phoneme appearance probability as a phoneme sequence of the utterance speech.

  Next, the word estimation unit 202 refers to the word dictionary 301 stored in the memory 30 and selects a word that matches the phoneme string estimated by the phoneme estimation unit 201. If there is no matching word in the word dictionary 301, the word estimation unit 202 causes the phoneme estimation unit 201 to estimate a phoneme string of a word having the next highest product of the appearance probabilities of each phoneme. Then, the word estimation unit 202 searches the word dictionary 301 for a word that matches the estimated phoneme string. When a word that matches the word dictionary 301 is obtained in this way, the word estimation unit 202 adopts the product of the appearance probabilities of the phoneme string of the matched word as the reliability of the word, The phoneme string and the appearance probability of each phoneme constituting the phoneme string are output to the word reliability determination unit 210 as a recognition result.

  Next, the word reliability determination unit 210 determines whether or not the reliability of the recognized word is greater than or equal to the threshold value TH1 (step S103). When the word reliability is equal to or higher than the threshold TH1 (YES in step S103), the word reliability determination unit 210 determines whether or not the first recognition result is stored in the recognition result storage unit 302 (step S104). ). Here, the first recognition result is a recognition result of speech uttered before the speech obtained in step S101 and indicates a recognition result stored in the recognition result storage unit 302.

  That is, when the reliability of the word recognized by the previous utterance is less than the threshold TH1, and the recognition result of the utterance is stored in the recognition result storage unit 302, the recognition result becomes the first recognition result.

  When the first recognition result is stored (YES in step S104), the word reliability determination unit 210 deletes the first recognition result stored in the recognition result storage unit 302 (step S105), and the recognition result is displayed. The result is output to the intention interpretation unit 220. Next, the intention interpretation unit 220 performs intention understanding processing based on the recognition result (step S106).

  On the other hand, when the first recognition result is not stored in the recognition result storage unit 302 (NO in step S104), the process proceeds to step S106. In step S106, the intention interpretation unit 220 estimates a response type and an action type from the recognition result. In step S107, the response generation unit 240 generates a response sentence corresponding to the estimated response type. In step S107, the action selection unit 230 determines the service to be executed or the control device 430 to be controlled according to the estimated action type, and if the service is determined, the service application server 420 sends a service provision request. When the control device 430 is determined, a control instruction is output to the control device 430 to be controlled.

  On the other hand, when the reliability of the recognized word is less than the threshold TH1 (NO in step S103), the word reliability determination unit 210 refers to the recognition result storage unit 302 to determine whether the first recognition result is stored. Is determined (step S110). When the first recognition result is not stored (NO in step S110), the word reliability determination unit 210 uses the first utterance as the phoneme string of the word estimated by the word estimation unit 202 and the appearance probability of each phoneme. The recognition result (first recognition result) is stored in the recognition result storage unit 302 (step S109), and a voice generation instruction for prompting re-speech is output to the response generation unit 240.

  Next, the response generation unit 240 generates a response sentence such as “Please say it again slowly?”, Causes the voice recognition unit 200 to generate a voice signal of the generated response sentence, and generates a response of the generated voice signal. Audio is output from the speaker 410 (step S108). When the voice of the reply response sentence is output in step S108, the speech recognition apparatus 100 enters a standby state for a re-utterance intended for one word by the speaker, and the process returns to S100.

  As a result, the second utterance is made by the speaker, and the second recognition result for the second utterance is obtained in the same manner as the first utterance by the processing in steps S100 to S102. And if the reliability of a 2nd recognition result is less than threshold value TH1, it will determine with NO by step S103, and a process will progress to S110.

  On the other hand, if the reliability of the second recognition result is equal to or higher than the threshold value TH1 (YES in step S103), the second recognition result is determined as one word intended by the speaker, and the processing from step S105 to step S107 is executed. The

  When the first recognition result is stored in the recognition result storage unit 302 in step S110 (YES in step S110), the phoneme appearance probability determination unit 203 stores the first recognition result stored in the recognition result storage unit 302. Then, phonemes equal to or greater than a predetermined threshold TH2 (an example of the second threshold) are extracted from the second recognition result for the recurrent speech by the speaker obtained in step S102 (step S111).

  Next, the word estimation unit 202 refers to the word dictionary 301 and extracts a word including a phoneme having a threshold value TH2 or more as a recognition candidate word in the phoneme string of the first recognition result (step S112). Next, the word estimation unit 202 narrows down the recognition candidate words from the list of recognition candidate words extracted in step S112 by words including phonemes having a threshold value TH2 or more in the phoneme string of the second recognition result (step S113).

  FIG. 5 is a diagram illustrating an example of the dialogue in the first embodiment. In FIG. 5, the robot refers to the robot on which the speech recognition apparatus 100 is mounted, and the numbers attached after the robot indicate the utterance order of the robot. The infant refers to an infant who interacts with the robot, and the numbers attached after the infant indicate the utterance order.

  First, the robot speaks “What kind of fruit do you like?” (Robot 1) to the infant, and the infant speaks “Apple” (Infant 1). However, since the reliability of the word recognized for the utterance of “apple” (toddler 1) is low here, the robot performs a replay in step S108.

  By this rehearsal, the toddler was re-speaking to “apple” (toddler 2), but the reliability of this re-speech was also low. The processing of the speech recognition apparatus 100 in this case will be described below with reference to FIGS. 6, 7, and 8.

  FIG. 6 is a diagram illustrating an example of the first recognition result and the second recognition result with respect to the dialogue example of FIG. As shown in FIG. 6, in the first recognition result, the word “mango” is recognized for the utterance “infringe” of the infant, and the reliability of this word is less than the threshold value TH1. Therefore, the first recognition result is stored in the recognition result storage unit 302. As shown in FIG. 6, the breakdown of the first recognition result is that the recognition word is “mango”, the recognition phoneme string is “m”,..., “O:”, and the phoneme appearance probability “ 0.4 ”,...,“ 0.6 ”.

  Because the reliability of the first recognition result was low, the infant spoke again with an “apple” when asked by the robot, “Please say slowly again?”, But the second recognition result was recognized to recognize “gentian”. In the second recognition result, the reliability is equal to or less than the threshold value TH1. As shown in FIG. 6, the breakdown of the second knowledge result is that the recognized word is “Lindou”, the recognized phoneme string is “r”,..., “O:”, and the phoneme appearance probability is “0”. .9 ”,...,“ 0.5 ”.

  Here, the threshold TH2 of the phoneme appearance probability is set to 0.7. In this case, the phoneme appearance probability determination unit 203 extracts a phoneme “n” and a phoneme “g” having a phoneme appearance probability of 0.7 or more from the first recognition result. Moreover, the phoneme appearance probability determination unit 203 extracts a phoneme “r”, a phoneme “i”, and a phoneme “n” having a phoneme appearance probability of 0.7 or more from the second recognition result.

  Next, the word estimation unit 202 refers to the word dictionary 301 and extracts, as recognition candidate words, words including continuous “n” and “g” phoneme sequences extracted from the first recognition result. Among the words illustrated in FIG. 7, words including the continuous phoneme string “ng” are “mango”, “brick”, “apple”, and “apple juice”.

  Therefore, the word estimation unit 202 extracts “mango”, “brick”, “apple”, and “apple juice” as recognition candidate words as shown in FIG. FIG. 8 is a diagram illustrating an example of recognition candidate words extracted from the first recognition result.

  Furthermore, the word estimation unit 202 narrows down the recognition candidate words by extracting words including the continuous phoneme string “rin” extracted from the second recognition result from the extracted recognition candidate words. Among the recognition candidate words illustrated in FIG. 8, words including the continuous phoneme string “rin” are “apple” and “apple juice”.

  Therefore, the word estimation unit 202 finally narrows down “apple” and “apple juice” as recognition candidate words in step S113.

  If it is assumed that the threshold value TH3 is 3 in step S115 in FIG. 4, the word estimation unit 202 determines YES in step S115 because the number of recognition candidate words finally narrowed down is two. In step S116, the word estimation unit 202 generates a speech signal of confirmation utterances for confirming recognition candidate words one by one, such as “Is it an apple?” Or “Apple juice?” And output from the speaker 410.

  The speaker performs, for example, an affirmative utterance (for example, “Yes”) or a negative utterance (for example, “No”) for the confirmation utterance. When the word estimation unit 202 recognizes an utterance affirmed with respect to the confirmation utterance, the word estimation unit 202 recognizes a word corresponding to the confirmation utterance as an utterance intended for one word. On the other hand, when recognizing a negative utterance with respect to the confirmation utterance, the word estimation unit 202 confirms the next recognition candidate word.

  FIG. 9 is a diagram illustrating another example of processing for narrowing recognition candidate words from the first recognition result and the second recognition result in the first embodiment. In the example of FIG. 9, the narrowing-down method in the case where phonemes having a threshold value TH2 or more are not continuous in the first recognition result and the second recognition result is shown.

  In FIG. 9, the example of a dialog is the same as FIG. In the example of FIG. 9, the first recognition result in which the word “rumba” is recognized for the utterance “apple”, and the second recognition result in which the word “yellow powder” is recognized for the recurrent utterance “apple”. Is obtained. In the example of FIG. 9, since the reliability is less than the threshold value TH1 = 0.7 for both the first recognition result and the second recognition result, the recognition candidate word is determined using the word “rumba” and the word “yellow powder”. Perform the process of narrowing down.

  As shown in FIG. 9, in the first recognition result, phonemes having a threshold TH2 = 0.7 or more are “r” and “n”, and the order of both phonemes is “r” before “n”. . In the second recognition result, phonemes with the threshold TH2 = 0.7 or more are “i” and “o”, and the order of both phonemes is “i” before “o”.

  Therefore, in the example of FIG. 9, the word estimation unit 202 sets “r” → “n” from the word dictionary 301 regardless of whether a phoneme exists between “r” and “n”. A word in which phonemes are arranged in order is extracted as a recognition candidate word. Next, the phoneme appearance probability determination unit 203 selects “i” → “o” from the extracted recognition candidate words regardless of whether a phoneme exists between “i” and “o”. The words arranged in this order are extracted, and the recognition candidate words are further narrowed down.

  Returning to FIG. In step S114, when the number of recognition candidate words is narrowed down to one (YES in step S114), the word estimation unit 202 determines the narrowed-down word as a recognition result, causes the process to transition to step S105, and after step S105 The process is executed.

  On the other hand, when the number of recognition candidate words cannot be narrowed down to one (NO in step S114), the phoneme appearance probability determination unit 203 determines whether or not the number of recognition candidate words is narrowed down to two or more and a threshold value TH3 or less ( Step S115). If the number of narrowed recognition candidate words is two or more and the threshold value TH3 or less (YES in step S115), the word estimation unit 202 performs a confirmation utterance for confirming the narrowed recognition candidate words to the speaker one by one. The voice synthesizer 250 is instructed to perform it (step S116). As the confirmation utterance, for example, if an apple is included in one of the narrowed recognition candidate words, an utterance such as “Did you say an apple?” Is mentioned.

  When an utterance that means affirmation such as “Yes” or “Yes” is performed from the speaker with respect to the confirmation utterance, the word estimation unit 202 determines the recognized recognition candidate word as a recognition result. If the recognition result is confirmed in step S117 (YES in step S117), the process proceeds to S105, and the processes after S105 are executed.

  On the other hand, when two or more recognition candidate words cannot be narrowed down to the threshold value TH3 or less (NO in step S115), the process proceeds to step S109, and the word estimation unit 202 uses the second recognition result as the recognition result in the memory 30. The data is stored in the storage unit 302. At this time, if the same recognition result exists in the past, the recognition result is overwritten on the past recognition result. At this time, the word estimation unit 202 may include all the narrowed-down recognition candidate words in the second recognition result and store them in the recognition result storage unit 302.

  On the other hand, if a positive utterance is not performed for all recognition candidate words in step S116 and the recognition result is not confirmed (NO in S117), the phoneme appearance probability determination unit 203 gives up recognition. The process ends.

  Thus, according to the speech recognition apparatus 100 of the first embodiment, even if a recognition result with low reliability is obtained for the first utterance, the recognition result is not discarded and the recognition result is This is used when a recognition result with low reliability is obtained for the utterance. For this reason, even if a highly reliable recognition result is not obtained by listening back, the phoneme string included in the first recognition result and the phoneme string included in the second recognition result may be used by using a reliable phoneme. The words are recognized. As a result, the recognition accuracy of one word can be improved.

  When the recognition result cannot be narrowed down uniquely by the first recognition result and the second recognition result, that is, when it is determined NO in Step S115 and the second recognition result is stored in the recognition result storage unit 302 ( In step S109), the speech recognition apparatus 100 may acquire the third recognition result by further listening. When the reliability is less than the threshold TH1 in the third recognition result, the phoneme appearance probability determination unit 203 may perform narrowing using the first, second, and third recognition results. In this case, the phoneme appearance probability determination unit 203 selects the recognition candidate words narrowed down by the first recognition result and the second recognition result as phonemes having an appearance probability of the threshold TH2 or more from the phoneme string recognized by the third recognition result. You can narrow down by the words you include. Even in this case, if the number of recognition candidate words is not less than or equal to the threshold TH3, the phoneme appearance probability determination unit 203 may perform further replay and repeat the replay until the number of recognition candidate words is less than or equal to the threshold TH3.

(Embodiment 2)
FIG. 10 is a diagram illustrating an example of the overall configuration of the voice interaction system according to the second embodiment. 10 differs from FIG. 1 in that a word estimation unit 202, a phoneme appearance probability determination unit 203, and a word reliability determination unit 210 are respectively a sentence estimation unit 1202, a phoneme appearance probability synthesis unit 1203, and a sentence trust. The degree determination unit 1210 is replaced.

  The speech recognition unit 200 according to the first embodiment is configured to recognize only one word as speech, whereas the configuration of the speech recognition unit 200 according to the second embodiment is configured from arbitrary words. The sentence (word string) is recognized.

  The phoneme estimation unit 201 estimates a phoneme string using a hidden Markov model (HMM), and the sentence estimation unit 1202 estimates a sentence (word string) using a finite state grammar or n-gram. .

  By combining the HMM and the finite state grammar or n-gram, a search space composed of a directed graph in which a plurality of phonemes are connected in a network is configured. Therefore, the speech recognition process is reduced to a search problem of the network route. That is, the voice recognition process is a process of finding a network path that best matches the input voice signal and using a word string corresponding to the path as a recognition result. Specifically, the speech recognition process is a process for obtaining a word string W (S) that maximizes the product of the phoneme and the word appearance probability in the following equation (2).

FIG. 11 is a diagram illustrating an example of an audio signal divided into a plurality of frames. As shown in FIG. 11, a frame refers to an input audio signal divided at a constant time interval such as 25 msec. o _t indicates a feature vector in the t-th frame. A feature vector is an example of a physical quantity used for estimating a phoneme, and is obtained from the volume of an audio signal. T represents the length of the input audio signal in the number of frames. As the feature vector, for example, a Mel-Frequency Cepstrum Coefficients can be adopted. s _t represents a state where the process has reached the t-th frame.

11, an arrow 1101 of right represents the state _{s t.} With respect to the phoneme string, in the state _{s t,} "kykyo: o: nno" or "kyo: no" sequence of phonemes are estimated. Note that “kykyo: o: nno” and “kyo: no” depend on the difference in the acoustic model. If the same phoneme successive phoneme estimator 201 acoustic models of binding is been not use the estimation result of the state s _t is the latter. For simplicity, the following description will be made using an acoustic model of one frame and one phoneme.

In addition, with regard to the word string, in the state s _t, word string of "Today" it has been estimated. Therefore, P _A (o _t , s _t | s _t−1 ) represents the probability of transition from the phoneme string corresponding to the state s _t−1 to the phoneme string corresponding to the state s _t (appearance probability of the phoneme string). . P _L (s _t , s _t-1 ) represents the probability of the language model (word string appearance probability) for transition from the word string corresponding to the state s _t-1 to the word string corresponding to the state s _t . Note that the word string appearance probability P _L (s _t , s _t-1 ) is applied when the state s _t-1 and the state s _t are word boundaries, and is 1 except for the word boundary. W (S) is the state transition process S, i.e., represent the word sequence corresponding to the state _{s t.}

  The word string finally estimated for the speech signal of the input utterance corresponds to the phoneme string from the first frame to the Tth frame. The phoneme sequence is estimated in order from the first, such as the first frame → second frame →... → Tth frame. When there is any utterance, first, the phoneme estimation unit 201 estimates the number of phoneme strings that can be estimated with respect to the speech signal of the utterance. The phoneme sequence that can be estimated is the first frame, the second frame from the first frame, the third frame from the first frame, etc., in addition to the phoneme sequence for the entire utterance. A phoneme string up to the middle of a continuous utterance is also included.

  Next, the sentence estimation unit 1202 assigns as many assignable words as possible to the estimated phoneme string. Then, the sentence estimation unit 1202 multiplies the estimated appearance probability of the phoneme string by the appearance probability of the assigned word, and finally estimates the combination of the phoneme string and the word that obtains the maximum value as the word string. Here, the product of the appearance probability of the estimated phoneme string and the appearance probability of the assigned word indicates the reliability of the word string composed of the estimated phoneme string and the words assigned thereto. This will be specifically described below.

When “Today's weather” is uttered, the phoneme estimation unit 201 starts from the state s ₁ , that is, from the first frame phoneme sequence (in this case, the phoneme), in order from the entire utterance (here, from the first frame). estimating a phoneme sequence of states s ₉ of T = to the 9th frame), we calculate the probability for each estimated phoneme string.

When the phoneme string in the state s ₁ is estimated as “ky”, the phoneme string up to the state s ₂ , that is, the second frame is estimated as “kyo:”, for example. In this case, the appearance probability P _A (o ₂ , s ₂ | s ₁ ) of the phoneme string up to the second frame represents the probability that the phoneme “o:” appears after the phoneme “ky”.

The number of phoneme strings in state s ₂ is not limited to “kyo:”, but there are all types of phoneme numbers, but the appearance probability of the phoneme string varies depending on the characteristics of the speech when it is actually spoken. In this case, because it is spoken as "today's weather", the phoneme string of state s _2, the phoneme string "kyo:" appearance it is a sequence of phonemes probability P _A of "kyu:" than the occurrence probability P _A of Get higher. Similarly, the phoneme string of state _{s 9,} the phoneme string "kyo: notenki" probability of occurrence _{P A} of it is a phoneme string: higher than the occurrence probability _{P A} of "kyo nodenchi".

The sentence estimation unit 1202 first assigns words to the phoneme string estimated by the phoneme estimation unit 201. For example, when the phoneme string in the state s ₉ is estimated as “kyo: notenki”, a word such as “Today's weather” or “Kyoto's weather” is assigned. Next, the sentence estimation unit 1202 uses the word appearance probability P _L (s _t , s _t-1 ) for each assigned word, using the word appearance probability based on a language model such as n-gram. Calculate For example, when the sentence estimation unit 1202 uses a 2-gram language model, the word appearance probability P _L (s _t , s _t−1 ) for “today” is “to” next to “today”. ”Represents the probability of appearance, and the word appearance probability P _L (s _t , s _t−1 ) for“ Kyo ”represents the probability of“ no ”appearing after“ Kyo ”.

The appearance probabilities of these words are stored in the word dictionary 301. If the appearance probability of the word “today” is greater than the appearance probability of the word “Kyoto” for the phoneme string “kyo: notenki” in the state s ₉ , “today ’s weather” word of the occurrence probability _{P L (s t, s t} -1) of the people is greater than the probability appearance of the word for "Today's weather" _{P L (s t, s t} -1) for. Here, an example of 2-gram has been described, but the calculation of the word appearance probability is the same regardless of which n-gram (n is a natural number) is used.

The sentence reliability determination unit 1210 includes each of the phoneme string appearance probability P _A (o _t , s _t | s _t−1 ) estimated by the phoneme estimation unit 201 and all the phoneme strings estimated by the sentence estimation unit 1202. Are multiplied by the appearance probabilities P _L (s _t , s _t−1 ) of the plurality of word strings assigned to, and the reliability of the plurality of word strings is calculated. Then, the sentence reliability determination unit 1210 recognizes a word string having the maximum reliability among the plurality of reliability as a final word string. That is, the sentence estimation unit 1202 recognizes W (s) in Expression (2) as a final word string.

  The phoneme appearance probability synthesis unit 1203 synthesizes the appearance probability of each phoneme by taking the sum of the appearance probability of each phoneme in the first utterance and the appearance probability of each phoneme in the second utterance. When the appearance probabilities of each phoneme are combined, the sentence estimation unit 1202 uses the combined appearance probabilities of each phoneme to generate a plurality of words using the same method as the method obtained for the first utterance. The reliability of the sequence is calculated, and the word sequence having the maximum reliability is used as the final recognition result. That is, the sentence estimation unit 1202 uses the word string W (s) in Expression (3) as the final recognition result.

  Here, the first utterance is not a response utterance to a reply, but a response to an inquiry from the speech recognition apparatus 100 or an utterance by a conversation from the user to the speech recognition apparatus 100. The second utterance refers to a response utterance in response to a reply, and refers to an utterance by a speaker who intends the first utterance.

In Expression (3), P _A1 represents the appearance probability of the phoneme string of the first utterance, and P _A2 represents the appearance probability of the phoneme string of the second utterance. At this time, the sum of the appearance probabilities of each phoneme in the first utterance and the second utterance may be a value obtained by weighted addition according to the reliability of the first utterance and the reliability of the second utterance. For example, if the reliability of the first utterance is α and the reliability of the second utterance is β, the sum of the appearance probabilities is obtained by multiplying the appearance probability of each phoneme of the first utterance by the weight value α / α + β. An addition value obtained by multiplying the appearance probability of each phoneme of the second utterance by the weight value β / α + β may be employed.

  The sentence reliability determination unit 1210 determines whether or not the reliability (product of the phoneme string appearance probability and the word string appearance probability) with respect to the recognition result of the first utterance estimated by the sentence estimation unit 1202 is greater than or equal to a threshold value TH1. To do. Then, when the reliability is less than the threshold TH1, the sentence reliability determination unit 1210 stores the recognition result for the first utterance in the recognition result storage unit 302 as the first recognition result, and performs a replay. Here, the first recognition result includes information necessary for estimating the word string. For example, the recognized word string, the phoneme string corresponding to the word string, and each of the phoneme strings constituting the phoneme string Contains the phoneme appearance probability.

  FIG. 12 is a flowchart illustrating an example of recognition processing according to the second embodiment. The processing in step S200 and step S201 is the same as the processing in step S100 and step S101 shown in FIG.

  The voice recognition unit 200 performs a voice recognition process (step S202). Specifically, the phoneme estimation unit 201 estimates the phonemes of each speech section using an acoustic model, as in the first embodiment. The sentence estimation unit 1202 assigns the word string registered in the word dictionary 301 to the phoneme string estimated by the phoneme estimation unit 201. At this time, the sentence estimation unit 1202 assigns an assignable word string to each of all phoneme strings estimated by the phoneme estimation part 201, and one or more word strings for each estimated phoneme string Get the result of the assignment. Then, the sentence estimation unit 1202 outputs a word string that maximizes the product of the appearance probability of the phoneme string and the appearance probability of the assigned word string as the recognition result, and the word obtained as the recognition result using the maximum product value The column reliability is output to the sentence reliability determination unit 1210.

  Next, the sentence reliability determination unit 1210 determines whether or not the reliability of the word string recognized by the sentence estimation unit 1202 is greater than or equal to the threshold value TH1 (step S203). If the text reliability is equal to or higher than the threshold TH1 (YES in step S203), the process proceeds to step S204. Steps S204 to S207 are the same as steps S104 to S107 shown in FIG.

On the other hand, when the reliability of the word string recognized by the sentence estimation unit 1202 is less than the threshold TH1 (NO in step S203), the sentence reliability determination unit 1210 refers to the recognition result storage unit 302 and performs the first recognition result. Is stored (step S210). When the first recognition result is not stored (NO in step S210), the sentence reliability determination unit 1210, the word string recognized by the sentence estimation unit 1202, the phoneme string corresponding to the word string, the formula (2) ) P _A (o _t , s _t | s _t-1 ), and the appearance probability of each phoneme is stored in the recognition result storage unit 302 as a recognition result (first recognition result) of the first utterance (step) S209). In step S208, the speech recognition apparatus 100 performs listening back as in step S108 shown in FIG. The second utterance is performed by the speaker by this replay, and the second recognition result for the second utterance is obtained by the processing of step S200 to step S202 as with the first utterance. If the reliability of the second recognition result is less than the threshold TH1, NO is determined in step S203, and the process proceeds to S210.

  On the other hand, if the reliability of the second recognition result is equal to or higher than the threshold TH1 (YES in step S203), the second recognition result is determined as one word string intended by the speaker, and the processing in steps S205 to S207 is executed. Is done.

  On the other hand, when the first recognition result is stored in the recognition result storage unit 302 (YES in step S210), the phoneme appearance probability synthesis unit 1203 is included in the first recognition result stored in the recognition result storage unit 302. The sum of the appearance probability of each phoneme in the phoneme string and the appearance probability of each phoneme in the phoneme string of the second utterance obtained in step S202 is calculated (step S211).

  Next, the sentence estimation unit 1202 calculates a composite appearance probability to be described later by multiplying the sum of the appearance probabilities of each phoneme of the first utterance and the second utterance, and multiplies the composite appearance probability by the word appearance probability. Thus, the reliability of each word string is calculated, and the word string giving the maximum reliability is recognized as one word string uttered by the speaker (step S212). When the process of step S212 ends, the process transitions to step S203.

(Specific example of Embodiment 2)
Next, a specific example of the second embodiment will be described. In this specific example, for the sake of simplicity, the speech recognition apparatus 100 that recognizes a sentence using a model that can estimate only two word strings (sentences) of “I am an apple” and “Is a mango” will be described.

  It is assumed that the phoneme estimation unit 201 estimates “ringodesu” and “mango: desu” as phoneme sequences for an utterance. In this case, the appearance probability of each phoneme string is calculated as a product of the appearance probabilities of the phonemes constituting each phoneme string.

  FIG. 13 is a diagram illustrating an example of a search space when a 1-gram language model is adopted in the specific example of the second embodiment.

  In the search space of FIG. 13, the first phoneme “sil” is an abbreviation of “silent” and indicates a silent section. Moreover, in FIG. 13, each alphabet shows a phoneme, and the numerical value described under each alphabet is the appearance probability of each phoneme. In this search space, an element “sil” is arranged at the beginning and the end, respectively, and includes a phoneme string “ringodesu” and a phoneme string “mango: desu”. Specifically, the search space branches from the leading element “sil” into two phoneme strings “ringo” and “mango:”, merges again with the phoneme string “desu”, and the final element “sil”. Has led to

  In this case, the appearance probability of the phoneme sequence “ringodesu” is calculated as 0.7 × 0.5 × 0.5 ×... × 0.9 × 0.9, and the appearance probability of the phoneme sequence “mango: desu”. Is calculated as 0.2 × 0.3 × 0.4 ×... × 0.9 × 0.9.

  Here, it is assumed that three words “apple”, “mango”, and “is” and the appearance probability of each word are registered in the word dictionary 301. In this case, the sentence estimation unit 1202 obtains the search space shown in FIG. 13 by assigning these three words to each phoneme string. The numerical value shown to the right of each word indicates the appearance probability of the word.

  Generally, n-gram is used as the word appearance probability. In n-gram, it is assumed that the word appearance probability depends on the immediately preceding word. In the example of FIG. 13, 1-gram is used. Since 1-gram does not depend on the immediately preceding word, the appearance probability of a single word is used. At this time, the probability that “apple” is uttered as the first word is 0.6, and the probability that “mango” is uttered as the first word is 0.4. Further, the probability that “I” is uttered after “mango” and “apple” is 1.

  The sentence estimation unit 1202 extracts all the paths connecting the leading element “sil” to the final “sil” as phoneme strings, and can be assigned to each phoneme string among the words registered in the word dictionary 301. Assign multiple word strings. In the example of FIG. 13, the word “apple” is assigned to the phoneme string “ringo”, the word “mango” is assigned to the phoneme string “mango:”, and the word “is” is assigned to the phoneme string “desu”. Therefore, in the example of FIG. 13, the word strings “I am an apple” and “I am a mango” are obtained.

  Then, the appearance of the word “apple” in the multiplication value “0.7 × 0.5 ×... 0.9” of the appearance probabilities of each phoneme of the phoneme sequence “rigodesu” + “sil” of the word sequence “I am apple” The probability “0.6” and the appearance probability “1” of “is” are multiplied, and the reliability of the word string “is apple” is obtained. Similarly, the reliability of the word string “is mango” is obtained.

  Of the word strings “I am apple” and “I am mango”, the word string having the maximum reliability is estimated as the recognition result. In the example of FIG. 13, since the reliability of the word string “I am an apple” is greater than the reliability of the word string “I am a mango”, the recognition result is the word string “I am an apple”.

  In the case of 2-gram, it is assumed that the word appearance probability depends only on the immediately preceding word. That is, a 2-gram dictionary consisting of only three words “apple”, “mango”, and “is” is as shown in FIG. FIG. 14 is a diagram illustrating an example of the word dictionary 301 when the 2-gram language model is adopted in the specific example of the second embodiment. The combinations of 2-grams obtained from the three words “apple”, “mango”, and “is” including “sil” are as follows. That is, the 2-gram combination includes three sets of “apple”, “mango”, and “is” for “sil”, and “is”, “mango”, and “sil” for “apple”. 3 pairs of “I”, “apple” and “sil” for “mango”, and 3 pairs of “apple”, “mango” and “sil” for “do” And a total of 3 × 4 = 12 combinations are possible. Therefore, in the word dictionary 301 shown in FIG. 14, these 12 sets of 2-gram word strings are registered.

  A 2-gram search space using the word dictionary 301 shown in FIG. 14 is expressed as shown in FIG. FIG. 15 is a diagram illustrating an example of a search space when a 2-gram language model is employed in the specific example of the second embodiment. In FIG. 15, the phoneme string and the appearance probability of each phoneme are the same as those in FIG.

  At this time, when the word dictionary 301 as shown in FIG. 14 is stored, the probability that “apple” appears in the first word, that is, the probability that “apple” appears after the element “sil” is 0. .3. The probability that “mango” appears in the first word, that is, the probability that “mango” appears after the element “sil” is 0.2.

  The probability that “Is” will appear after “Apple” is 0.5, and the probability that “Is” will appear after “Mango” is 0.4. Further, the probability that the element “sil” appears after “is” is 0.6. In this case, the word string that maximizes the product of the appearance probability of the phoneme string of each path shown in the graph of FIG. 15 and the appearance probability of the 2-gram word string is adopted as the recognition result. That is, the appearance probability of each phoneme in the phoneme string “ringodesu” and the occurrence probabilities of “sil-apple”, “apple-is”, and “is-sil” (= 0.3, 0.5, and 0) .6) is calculated as the reliability of the word string “I am an apple”. Similarly, the reliability of the word string “is mango” is also calculated. In this example, since the reliability of the word string “I am an apple” is higher than the reliability of the word string “I am a mango”, the recognition result is finally the word string “I am an apple”. This is the same processing even when n-gram is 3-gram or more.

  The sentence reliability determination unit 1210 determines whether or not the reliability of the word string estimated by the sentence estimation unit 1202 is greater than or equal to a threshold value TH1. When the reliability of the first recognition result for the first utterance and the reliability of the second recognition result for the second utterance are both less than the threshold TH1, the phoneme appearance probability synthesis unit 1203 has an appearance probability of each phoneme in the first utterance. Is multiplied by the sum of the appearance probabilities of each phoneme in the second utterance to calculate a composite appearance probability.

  The sentence estimation unit 1202 recognizes a word string (sentence) using the combined appearance probability calculated by the phoneme appearance probability combining unit 1203.

  FIG. 16 is a diagram illustrating a search space when the appearance probabilities of each phoneme of the first recognition result and each phoneme of the second recognition result in the specific example of the second embodiment are combined. FIG. 16 shows a directed graph of the phoneme sequence “ringodesu” and the phoneme sequence “mango: desu”, as in FIG. 15, and the appearance probability of the first utterance and the appearance probability of the second utterance are shown for each phoneme. Has been. In the example of FIG. 16, a word of 1-gram is assigned. In FIG. 16, the numerical value described immediately below each phoneme indicates the appearance probability of the first utterance, and the numerical value described immediately below the first utterance indicates the appearance probability of the second utterance.

  For example, the appearance probability of the phoneme “r” in the first utterance of the phoneme string “ringodesu” is 0.7, and the appearance probability of the phoneme “r” in the second utterance is 0.3.

  Here, the combined appearance probability of the phoneme string “ringodesu” is (0.7 + 0.3) × (0.5 + 0.4) ×... × (0.9 + 0.9). The synthetic appearance probability of the phoneme string “mango: desu” is (0.2 + 0.4) × (0.3 + 0.5) ×... × (0.9 + 0.9).

  In this case, the sentence estimation unit 1202 assigns a 1-gram word string registered in the word dictionary 301 to each of the phoneme string “ringodesu” and the phoneme string “mango: desu”.

  Then, the sentence estimation unit 1202 calculates the reliability of each word string by multiplying the combined appearance probability calculated by the phoneme appearance probability combining unit 1203 by the word appearance probability. Then, the sentence estimation unit 1202 recognizes the phoneme string having the maximum reliability as one word string intended by the speaker.

  In FIG. 16, the reliability of the word string “I am an apple” has a probability of “apple” appearing as the first word is 0.6, and the probability that “is” appears after “apple” is 1. Therefore, (0.7 + 0.3) × (0.5 + 0.4) ×... × (0.9 + 0.9) × 0.6 × 1 is calculated. Similarly, the reliability of the word string “is a mango” has a probability that “mango” appears in the first word is 0.4, and “is” next to “mango” has an appearance probability of 1. 0.2 + 0.4) × (0.3 + 0.5) ×... × (0.9 + 0.9) × 0.4 × 1.

  Here, since the word string “I am an apple” is more reliable than the word string “I am a mango”, it is recognized that the word string “I am an apple” is spoken.

  As described above, according to the speech recognition apparatus 100 of the second embodiment, even if a recognition result with low reliability is obtained for the first utterance, the recognition result is not discarded and the recognition result is This is used when a recognition result with low reliability is obtained for the utterance. For this reason, even if a highly reliable recognition result cannot be obtained by listening back, since one word string is recognized by combining both recognition results, the recognition accuracy of one word string can be improved. .

  Note that in step S209, the recognition result stored in the recognition result storage unit 302 may be not only the previous recognition result but also a plurality of past recognition results obtained by listening back. In this case, the phoneme appearance probability synthesis unit 1203, in step S211, the appearance probability of each phoneme of a plurality of phoneme strings obtained as a result of recognition in the past plural times and each phoneme of the phoneme string obtained as the latest recognition result. And the appearance probability of.

(Embodiment 3)
FIG. 17 is a diagram illustrating an example of the overall configuration of the voice interaction system according to the third embodiment. 17, the difference from FIG. 10 is that the phoneme appearance probability synthesis unit 1203 is omitted and a common candidate extraction unit 270 is added.

  In the third embodiment, the sentence estimation unit 1202 estimates a word string in the same manner as in the second embodiment, but does not use the word string with the highest reliability as the recognition result, but the top n words in the descending order of reliability. Are extracted as recognition candidates, and the top n recognition candidates (n-best) are used as recognition results. n-best refers to n recognition candidates in descending order of reliability among a plurality of recognition candidates included in the recognition result.

  The common candidate extraction unit 270 determines that the maximum reliability value in the first recognition result is less than the threshold TH1 and the maximum reliability value in the second recognition result is less than the threshold TH1 by the sentence reliability determination unit 1210. The first utterance recognition candidate (n-best) and the second utterance recognition candidate (n-best) are compared, common recognition candidates are extracted, and a word string to be finally recognized is determined based on the extraction result. To do.

  FIG. 18 is a flowchart illustrating an example of recognition processing according to the third embodiment. The processing from step S300 to step S307 is basically the same as the processing from step S200 to step S207 shown in FIG. However, in step S303, the sentence reliability determination unit 1210 determines whether or not the maximum value of the reliability of the first recognition result estimated by the sentence estimation unit 1202 is greater than or equal to the threshold value TH1.

  In step S303, when the maximum value of the reliability of the first recognition result is equal to or greater than the threshold value TH1 (YES in step S303), a recognition candidate having the maximum value is determined as one word string intended by the speaker. The processing from S305 to step S307 is executed.

  On the other hand, in step S303, when the maximum reliability value in the first recognition result is less than the threshold value TH1 (NO in step S303), the sentence reliability determination unit 1210 refers to the recognition result storage unit 302 and performs the first recognition. It is determined whether the result is stored (step S310). When the first recognition result is not stored (NO in step S310), as shown in FIG. 19, the sentence reliability determination unit 1210 includes the reliability (phoneme) among the recognition candidates included in the recognition result of the first utterance. N recognition candidates are acquired as n-best from the sentence estimation unit 1202 and stored in the recognition result storage unit 302 in descending order of the product of the appearance probability for each word and the product of the appearance probability for each word. In step S308, similar to step S208 shown in FIG. The second utterance is performed by the speaker by this replay, and the second recognition result for the second utterance is obtained by the processing of step S300 to step S302 as with the first utterance. If the maximum value of the reliability of the second recognition result is less than the threshold value TH1, NO is determined in step S303, and the process proceeds to S310.

  On the other hand, if the maximum value of the reliability of the second recognition result is equal to or greater than the threshold value TH1 (YES in step S303), a recognition candidate having the maximum value is determined as one word string intended by the speaker, and steps S305 to S305 are performed. The process of step S307 is executed.

  On the other hand, when the first recognition result is stored in the recognition result storage unit 302 (YES in step S310), the common candidate extraction unit 270 determines the n-best of the first recognition result and the n-best of the second recognition result. Are compared (step S311).

  Next, the common candidate extraction unit 270 determines whether there is a common recognition candidate as a result of the comparison (step S312). If there are common recognition candidates (YES in step S312), the common candidate extraction unit 270 determines whether or not there are a plurality of common candidates (step S313). When there are a plurality of common recognition candidates (YES in step S313), the common candidate extraction unit 270 adds the reliability of the first recognition result and the reliability of the second recognition result for each of the plurality of common recognition candidates. Is calculated. Then, the common candidate extraction unit 270 may determine a recognition candidate having the maximum reliability as a final recognition result, or select a plurality of recognition candidates as a final recognition result in descending order of the reliability. You may decide. When the process of step S313 ends, the process transitions to step S304. Further, the common candidate extraction unit 270 performs the utterance confirmation described in step S116 in FIG. 4 on the plurality of recognition candidates obtained in descending order of the reliability, and the recognition candidate whose consent is obtained by the speaker. May be determined as the final recognition result.

  FIG. 19 is a diagram illustrating an example of 5-best of the first recognition result in the third embodiment. FIG. 20 is a diagram illustrating an example of 5-best of the second recognition result in the third embodiment. In FIG. 19 and FIG. 20, common recognition candidates are “I want to eat an apple” and “I can fly a parakeet”. At this time, the sum of the reliability of the first recognition result and the second recognition result is 0.96 (= 0.54 + 0.42) for “I want to eat an apple” and 0.47 (= 0 for “I can fly a parrot”) 20 + 0.27). In this case, “I want to eat an apple” having the maximum sum of reliability is determined as the final recognition result. Alternatively, both recognition candidates may be determined as final reliability.

  On the other hand, if there is no common recognition candidate (NO in step S312), the process transitions to step S309. In step S <b> 309, the common candidate extraction unit 270 stores the second recognition result in addition to the first recognition result in the recognition result storage unit 302, and outputs an instruction to generate a response to the response to the response generation unit 240. Then, the speaker is further heard back (step S308). Thereby, the third recognition result is acquired. And if the maximum value of the reliability of a 3rd recognition result is less than threshold value TH1, a 1st, 2nd, and 3rd recognition result will be compared and a common recognition candidate will be extracted. In this case, if there is a recognition candidate common to at least two of the first, second, and third recognition results, the recognition candidate is extracted as a common recognition result.

  Thus, according to the speech recognition apparatus 100 according to Embodiment 3, even if a recognition result with low reliability is obtained for the first utterance, the recognition result is not discarded, and the recognition result is This is used when a low-reliability recognition result is obtained for two utterances. Therefore, even if a highly reliable recognition result is not obtained by replay, one word string is recognized because the word string recognized in both the first utterance and the second utterance is recognized. Column recognition accuracy can be increased.

(robot)
The speech recognition apparatus 100 may be mounted on a robot 500 as shown in FIG. FIG. 21 is an external view of a robot 500 on which the speech recognition apparatus 100 according to the first to third embodiments is mounted. The robot 500 includes a spherical main casing 501, a first spherical crown portion 502, and a second spherical crown portion 503. The main housing 501, the first spherical crown portion 502, and the second spherical crown portion 503 constitute a sphere as a whole. That is, the robot 500 has a spherical shape. In addition, the robot 500 includes a camera 504 in the second crown portion 503 and a distance sensor 505, a speaker 410, and a microphone 400 in the first crown portion 502.

  The camera 504 acquires an image of the surrounding environment of the robot 500. The distance sensor 505 acquires distance information to the surrounding environment of the robot 500. In this aspect, the robot 500 includes the camera 504 in the second crown portion 503 and the distance sensor 505, the speaker 410, and the microphone 400 in the first crown portion 502, but is not limited thereto. The camera 504, the distance sensor 505, the speaker 410, and the microphone 400 may be provided in at least one of the first spherical crown portion 502 and the second spherical crown portion 503.

  The center of the first spherical crown portion 502 and the center of the second spherical crown portion 503 are fixedly connected by a shaft (not shown) provided inside the main casing 501. The main casing 501 is rotatably attached to the shaft. A frame (not shown) and a display unit (not shown) are attached to the shaft. A first motor (not shown) for rotating the main casing 501 is attached to the frame. As the first motor (not shown) rotates, the main casing 501 rotates relative to the first and second spherical crown portions 502 and 503, and the robot 500 moves forward or backward. The first motor and the main casing 501 are an example of a moving mechanism. When the robot 500 moves forward or backward, the first spherical crown portion 502 and the second spherical crown portion 503 are in a stopped state, so that the camera 504, the distance sensor 505, the microphone 400, and the speaker 410 are positioned in front of the robot 1. Maintained facing. Further, an image showing the eyes and mouth of the robot 1 is displayed on the display unit. The display unit is attached so that the angle with respect to the shaft can be adjusted by power from a second motor (not shown). Therefore, the direction of the eyes and mouth of the robot is adjusted by adjusting the angle of the display unit with respect to the shaft. Note that since the display unit is attached to the shaft independently of the main housing 501, the angle with respect to the shaft does not change even when the main housing 501 rotates. Therefore, the robot 500 can move forward or backward with the direction of the eyes and mouth fixed.

  The present disclosure can improve the accuracy of speech recognition, and is useful, for example, in the technical field of robots that interact with an infant whose speech is ambiguous.

20 CPU
30 memory 100 speech recognition device 200 speech recognition unit 201 phoneme estimation unit 202 word estimation unit 203 phoneme appearance probability determination unit 210 word reliability determination unit 220 intention interpretation unit 230 action selection unit 240 response generation unit 250 speech synthesis unit 260 utterance extraction unit 270 Common candidate extraction unit 301 Word dictionary 302 Recognition result storage unit 400 Microphone 410 Speaker 500 Robot 1202 Sentence estimation unit 1203 Phoneme appearance probability synthesis unit 1210 Sentence reliability determination unit

Claims (11)

A speech recognition method,
A first utterance spoken by a speaker intended for a word is received via a microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
For each of the N phonemes constituting the first utterance, the appearance probabilities of all types of phonemes are calculated,
A phoneme string in which phonemes having the maximum appearance probability corresponding to each of the N phonemes from the first phoneme constituting the first utterance to the Nth phoneme are arranged in order; a first phoneme string corresponding to the first utterance; Recognized,
A first value is calculated by multiplying appearance probabilities of N phonemes constituting the first phoneme string;
If the first value is less than a first threshold, let the speaker output a voice prompting the speaker to speak the one word again,
A second utterance re-spoken by the speaker with the intention of the one word is received via the microphone, and the second utterance is composed of M (M is a natural number of 2 or more) phonemes,
For each of the M phonemes constituting the second utterance, the appearance probability is calculated for all types of phonemes,
A phoneme string in which phonemes having the maximum appearance probability corresponding to each of the M phonemes from the first phoneme constituting the second utterance to the Mth phoneme are arranged in order; a second phoneme string corresponding to the second utterance; Recognized,
A second value is calculated by multiplying the appearance probabilities of M phonemes constituting the second phoneme sequence;
When the second value is less than the first threshold, phonemes having an appearance probability equal to or higher than the second threshold in the first phoneme string and phonemes having an appearance probability equal to or higher than the second threshold in the second phoneme string Extract and
Extracting words including the extracted phonemes from a dictionary stored in a memory, the dictionary associates each word with a phoneme string corresponding to each word,
If the extracted word is one, the extracted word is recognized as corresponding to the one word;
Speech recognition method. If there are a plurality of extracted words, the voice that asks the speaker whether the extracted words are spoken is output through the speaker.
Receiving a positive or negative answer from the speaker via the microphone;
Recognizing the word corresponding to the positive answer as corresponding to the one word;
The speech recognition method according to claim 1. A speech recognition method,
A first utterance uttered by a speaker with the intention of a single word string is received via a microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
Calculating the reliability X1 of the word string estimated for the first utterance;

t indicates a number for specifying a frame constituting the first utterance;
T indicates the total number of frames constituting the first utterance,
P _A1 (o _t , s _t | s _t-1 ) is the t-th frame after the phoneme string corresponding to the state s _t-1 from the first frame to the t-1 frame of the first utterance. any phoneme appeared, indicates the probability of transition to the phoneme string corresponding to the state s _t,
o _t is obtained from the first utterance and indicates a physical quantity for estimating the arbitrary phoneme;
The arbitrary phonemes represent all types of phonemes,
_{P L1 (s t, s t} -1) is the next word string corresponding to the state _{s t-1} in the first utterance, any word is found at t-th frame, corresponding to the state _{s t} The probability of transition to a word string
Determining whether the reliability X1 is greater than or equal to a threshold;
If the reliability X1 is less than the threshold, a voice prompting the speaker to speak the one word string again is output through a speaker;
Receiving a second utterance re-uttered by the speaker with the intention of the one word string, via the microphone;
When the reliability X1 of the second utterance is less than the threshold value, the composite reliability X is calculated for all word strings estimated from the first utterance and the second utterance,

t indicates a number that designates a frame constituting the first utterance and the second utterance;
T represents the total number of frames constituting the first utterance and the second utterance,
P _A1 (o _t , s _t | s _t-1 ) is the t-th frame after the phoneme string corresponding to the state s _t-1 from the first frame to the t-1 frame of the first utterance. any phoneme appeared, indicates the probability of transition to the phoneme string corresponding to the state s _t,
o _t is a physical quantity obtained from the first utterance and used to estimate the arbitrary phoneme;
The arbitrary phonemes represent all types of phonemes,
P _A2 (q _t , s _t | s _t−1 ) is the t th frame after the phoneme string corresponding to the state s _t−1 from the first frame to the t−1 frame of the second utterance. any phoneme appeared, indicates the probability of transition to the phoneme string corresponding to the state s _t,
q _t is obtained from the second utterance and indicates a physical quantity for estimating the arbitrary phoneme;
_{P L (s t, s t} -1) is the next word string corresponding to the state _{s t-1} in the first utterance, any word is found at t-th frame, corresponding to the state _{s t} The probability of transition to a word string
A word string corresponding to the state s _t giving the maximum value of the combined confidence X, recognized as the one word sequence,
Speech recognition method. A speech recognition method,
A first utterance uttered by a speaker with the intention of a single word string is received via a microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
Calculating reliability X1 of all word strings estimated for the first utterance;

t1 indicates a number for specifying a frame constituting the first utterance;
T1 indicates the total number of frames constituting the first utterance,
P _A1 (o _t1 , s _t1 | s _t1-1 ) is the t1 frame after the phoneme sequence corresponding to the state s _t1-1 from the first frame to the t1-1 frame of the first utterance. Indicates the probability that an arbitrary phoneme appears and transitions to a phoneme string corresponding to the state s _t1 ,
o _t1 is obtained from the first utterance and indicates a physical quantity for estimating the arbitrary phoneme;
The arbitrary phonemes indicate all types of phonemes, and P _L1 (s _t1 , s _t1-1 ) is arbitrary in the frame t1 next to the word string corresponding to the state s _t1-1 in the first utterance. And the probability of transition to a word string corresponding to the state s _t1 ,
Determining whether the maximum value MaxX1 of the reliability X1 is equal to or greater than a threshold;
When the maximum value MaxX1 is less than the threshold value,
Extracting a first word string estimated for the first utterance giving the top M pieces of reliability X1 (M is a natural number of 2 or more);
Outputting a voice prompting the speaker to speak the one word string again through a speaker;
Receiving a second utterance re-spoken by the speaker with the intention of the one word string, via a microphone;
Calculating reliability X2 of all word strings estimated for the second utterance;

t2 indicates a number that designates a frame constituting the second utterance;
T2 indicates the total number of frames constituting the second utterance,
_{P A2 (o t2, s t2} | s t2-1) is the next phoneme string corresponding to the state _{s t2-1} to t2-1 numbered frame from 1 numbered frame of the second speech, at t2 numbered frame Indicates the probability that an arbitrary phoneme appears and transitions to a phoneme string corresponding to the state s _t2 ,
o _t2 is obtained from the second utterance and indicates a physical quantity for estimating the arbitrary phoneme;
P _L2 (s _t2 , s _t2-1 ) corresponds to the state s _t2 when an arbitrary word appears in the t2 frame after the word string corresponding to the state s _t2-1 in the second utterance. The probability of transition to a word string
It is determined whether the maximum value MaxX2 of the reliability X2 is greater than or equal to a threshold value,
If the maximum value MaxX2 is less than the threshold, extract a second word string estimated for the second utterance that gives the top M pieces of the reliability X2,
If there is a word string common to the first word string and the second word string, the common word string is recognized as the one word string;
Speech recognition method.   A program for causing a computer to execute the speech recognition method according to claim 1. A speech recognition device comprising a processor, a memory, a microphone, and a speaker,
The processor is
Receiving a first utterance spoken by a speaker with the intention of a word via the microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
For each of the N phonemes constituting the first utterance, the appearance probabilities of all types of phonemes are calculated,
A phoneme string in which phonemes having the maximum appearance probability corresponding to each of the N phonemes from the first phoneme constituting the first utterance to the Nth phoneme are arranged in order; a first phoneme string corresponding to the first utterance; Recognized,
A first value is calculated by multiplying appearance probabilities of N phonemes constituting the first phoneme string;
If the first value is less than a first threshold, the speaker is prompted to output a voice prompting the speaker to speak the one word again,
A second utterance re-spoken by the speaker with the intention of the one word is received via the microphone, and the second utterance is composed of M (M is a natural number of 2 or more) phonemes,
For each of the M phonemes constituting the second utterance, the appearance probability is calculated for all types of phonemes,
A phoneme string in which phonemes having the maximum appearance probability corresponding to each of the M phonemes from the first phoneme constituting the second utterance to the Mth phoneme are arranged in order; a second phoneme string corresponding to the second utterance; Recognized,
A second value is calculated by multiplying the appearance probabilities of M phonemes constituting the second phoneme sequence;
When the second value is less than the first threshold, phonemes having an appearance probability equal to or higher than the second threshold in the first phoneme string and phonemes having an appearance probability equal to or higher than the second threshold in the second phoneme string Extract and
Extracting words including the extracted phonemes from the dictionary stored in the memory, the dictionary associates each word with a phoneme string corresponding to each word,
If the extracted word is one, the extracted word is recognized as corresponding to the one word;
Voice recognition device. A voice recognition device according to claim 6;
A housing containing the voice recognition device;
A moving mechanism for moving the housing;
Robot equipped with. A speech recognition device comprising a processor, a microphone, and a speaker,
The processor is
Receiving a first utterance uttered by a speaker with the intention of one word string via the microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
Calculating the reliability X1 of the word string estimated for the first utterance;

t indicates a number for specifying a frame constituting the first utterance;
T indicates the total number of frames constituting the first utterance,
P _A1 (o _t , s _t | s _t-1 ) is the t-th frame after the phoneme string corresponding to the state s _t-1 from the first frame to the t-1 frame of the first utterance. any phoneme appeared, indicates the probability of transition to the phoneme string corresponding to the state s _t,
o _t is obtained from the first utterance and indicates a physical quantity for estimating the arbitrary phoneme;
The arbitrary phonemes represent all types of phonemes,
_{P L1 (s t, s t} -1) is the next word string corresponding to the state _{s t-1} in the first utterance, any word is found at t-th frame, corresponding to the state _{s t} The probability of transition to a word string
Determining whether the reliability X1 is greater than or equal to a threshold;
If the reliability X1 is less than the threshold, a voice prompting the speaker to speak the one word string again is output through the speaker.
Receiving a second utterance re-uttered by the speaker with the intention of the one word string, via the microphone;
When the reliability X1 of the second utterance is less than the threshold value, the composite reliability X is calculated for all word strings estimated from the first utterance and the second utterance,

t indicates a number that designates a frame constituting the first utterance and the second utterance;
T represents the total number of frames constituting the first utterance and the second utterance,
P _A1 (o _t , s _t | s _t-1 ) is the t-th frame after the phoneme string corresponding to the state s _t-1 from the first frame to the t-1 frame of the first utterance. any phoneme appeared, indicates the probability of transition to the phoneme string corresponding to the state s _t,
o _t is a physical quantity obtained from the first utterance and used to estimate the arbitrary phoneme;
The arbitrary phonemes represent all types of phonemes,
P _A2 (q _t , s _t | s _t−1 ) is the t th frame after the phoneme string corresponding to the state s _t−1 from the first frame to the t−1 frame of the second utterance. any phoneme appeared, indicates the probability of transition to the phoneme string corresponding to the state s _t,
q _t is obtained from the second utterance and indicates a physical quantity for estimating the arbitrary phoneme;
_{P L (s t, s t} -1) is the next word string corresponding to the state _{s t-1} in the first utterance, any word is found at t-th frame, corresponding to the state _{s t} The probability of transition to a word string
A word string corresponding to the state s _t giving the maximum value of the combined confidence X, recognized as the one word sequence,
Voice recognition device. A voice recognition device according to claim 8;
A housing containing the voice recognition device;
A moving mechanism for moving the housing;
Robot equipped with. A speech recognition device comprising a processor, a microphone, and a speaker,
The processor is
Receiving a first utterance uttered by a speaker with the intention of one word string via the microphone;
The first utterance is composed of N phonemes (N is a natural number of 2 or more),
Calculating reliability X1 of all word strings estimated for the first utterance;

t1 indicates a number for specifying a frame constituting the first utterance;
T1 indicates the total number of frames constituting the first utterance,
P _A1 (o _t1 , s _t1 | s _t1-1 ) is the t1 frame after the phoneme sequence corresponding to the state s _t1-1 from the first frame to the t1-1 frame of the first utterance. Indicates the probability that an arbitrary phoneme appears and transitions to a phoneme string corresponding to the state s _t1 ,
o _t1 is obtained from the first utterance and indicates a physical quantity for estimating the arbitrary phoneme;
The arbitrary phonemes indicate all types of phonemes, and P _L1 (s _t1 , s _t1-1 ) is arbitrary in the frame t1 next to the word string corresponding to the state s _t1-1 in the first utterance. And the probability of transition to a word string corresponding to the state s _t1 ,
Determining whether the maximum value MaxX1 of X1 is equal to or greater than a threshold;
When the maximum value MaxX1 is less than the threshold value,
Extracting a first word string estimated for the first utterance giving the top M pieces of reliability X1 (M is a natural number of 2 or more);
A voice prompting the speaker to speak the one word string again is output through the speaker;
Receiving a second utterance re-uttered by the speaker with the intention of the one word string, via the microphone;
Calculating reliability X2 of all word strings estimated for the second utterance;

t2 indicates a number that designates a frame constituting the second utterance;
T2 indicates the total number of frames constituting the second utterance,
_{P A2 (o t2, s t2} | s t2-1) is the next phoneme string corresponding to the state _{s t2-1} to t2-1 numbered frame from 1 numbered frame of the second speech, at t2 numbered frame Indicates the probability that an arbitrary phoneme appears and transitions to a phoneme string corresponding to the state s _t2 ,
o _t2 is obtained from the second utterance and indicates a physical quantity for estimating the arbitrary phoneme;
P _L2 (s _t2 , s _t2-1 ) corresponds to the state s _t2 when an arbitrary word appears in the t2 frame after the word string corresponding to the state s _t2-1 in the second utterance. The probability of transition to a word string
It is determined whether the maximum value MaxX2 of the reliability X2 is greater than or equal to a threshold value,
If the maximum value MaxX2 is less than the threshold, extract a second word string estimated for the second utterance that gives the top M pieces of the reliability X2,
If there is a word string common to the first word string and the second word string, the common word string is recognized as the one word string;
Voice recognition device. A speech recognition device according to claim 10;
A housing containing the voice recognition device;
A moving mechanism for moving the housing;
Robot equipped with.

Images