JP7471921B2 - Speech dialogue device, speech dialogue method, and speech dialogue program - Google Patents

Description

The present invention relates to a voice dialogue device, a voice dialogue method, and a voice dialogue program for executing voice dialogue.

Compared to dialogue systems that use text input, communication robots capable of voice dialogue with humans are more likely to accept erroneous voice input. There are various patterns of erroneous voice input, such as voice activity detection errors, voice recognition errors, recognition of irrelevant voice, and distorted speech due to hesitation or mid-utterance corrections. Therefore, when such voice input is accepted, a dialogue with a human cannot be established unless the communication robot executes a process to determine whether the input voice is normal or not and to carry out appropriate exception handling.

Examples of such voice dialogue technology include Patent Document 1 and Patent Document 2. Patent Document 1 discloses a robot that notifies a target person by voice of voice data corresponding to words for which the voice recognition accuracy did not meet a predetermined value. This robot acquires a speech from the target person, and outputs to the target voice data from the acquired speech that corresponds to words for which the voice recognition accuracy did not meet a predetermined value.

Patent Document 2 discloses a dialogue control device that controls a dialogue device to perform a confirmation action performed by a human, and reduces erroneous responses from the dialogue device. This dialogue control device includes a scenario storage unit that stores a conversation scenario that starts a dialogue by outputting a voice that triggers a dialogue from the dialogue device, a response scenario that responds to an utterance from the user, and a confirmation scenario that confirms whether or not to start a dialogue, and a scenario selection unit that receives as input a conversation start index S indicating whether or not to start a dialogue from the dialogue device by outputting a voice that triggers a dialogue, and a response start index R indicating whether or not to respond to a certain voice, where J and K are each an integer equal to or greater than 1, and selects the conversation scenario, response scenario, or confirmation scenario based on the magnitude relationship between the conversation start index S and J thresholds, and the magnitude relationship between the response start index R and K thresholds.

JP 2018-81147 A JP 2018-87847 A

However, the above-mentioned Patent Documents 1 and 2 do not take into consideration which parts of the input dialogue voice the communication robot was able to recognize and which parts it was unable to recognize. Therefore, the user who uttered the dialogue voice cannot know which parts were conveyed to the communication robot and which parts were not. As a result, the user ends up having to ask the same question over and over again, which increases the user's annoyance and may cause the dialogue with the user to break down.

The purpose of the present invention is to facilitate smooth dialogue with users.

A voice dialogue device according to one aspect of the invention disclosed in the present application is a voice dialogue device having a processor that executes a program and a storage device that stores the program, wherein the processor executes a generation process that, when a word string contains a word whose reliability is equal to or greater than a first threshold value and there are multiple words in the first and second halves of the word string whose reliability is less than the first threshold value, generates a reply sentence that asks the source of the spoken voice about words in the word string that are equal to or greater than the first threshold value .

A representative embodiment of the present invention can facilitate smooth dialogue with the user. Problems, configurations, and effects other than those described above will become clear from the explanation of the following examples.

FIG. 1 is an explanatory diagram illustrating an example of a voice dialogue between a user and a communication robot according to a first embodiment. FIG. 2 is a diagram illustrating an example of a system configuration of the voice dialogue system according to the first embodiment. FIG. 3 is a block diagram showing an example of a hardware configuration of the speech dialogue device. FIG. 4 is a block diagram of an example of a functional configuration of the voice dialogue apparatus according to the first embodiment. FIG. 5 is an explanatory diagram illustrating an example of a voice recognition result. FIG. 6 is an explanatory diagram showing an example of conversion from a word string to a part-of-speech string. FIG. 7 is an explanatory diagram 1 showing an example of a part-of-speech classification definition of a part-of-speech string. FIG. 8 is an explanatory diagram 2 showing an example of a part-of-speech classification definition of a part-of-speech string. FIG. 9 is an explanatory diagram illustrating an example of reliability calculation by the reliability acquisition unit. FIG. 10 is a flowchart of an example of a voice dialogue processing procedure performed by the voice dialogue device according to the first embodiment. FIG. 11 is a flowchart showing a detailed example of the process of generating a reflection sentence (step S1005) shown in FIG. FIG. 12 is a flowchart illustrating an example of a detailed processing procedure of the mask word inference process (step S1106) illustrated in FIG. FIG. 13 is an explanatory diagram showing an example of the mask word inference process (step S1106) shown in FIG. FIG. 14 is a flowchart illustrating an example of a detailed process procedure of the restatement utterance interpretation process (step S1007) illustrated in FIG. FIG. 15 is a flowchart showing a detailed example of the procedure of the dialogue control process (steps S1008 and S1403) shown in FIG. 10 and FIG. FIG. 16 is a flowchart showing an example of the flow of a dialogue between a user and the voice dialogue device. FIG. 17 is a block diagram of an example of a functional configuration of the voice dialogue apparatus according to the second embodiment. As shown in FIG. FIG. 18 is a flowchart of an example of a voice dialogue processing procedure performed by the voice dialogue device according to the second embodiment.

<Example of voice dialogue>
Fig. 1 is an explanatory diagram showing an example of a voice dialogue between a user and a communication robot (hereinafter, simply referred to as "robot") according to the first embodiment. Fig. 1 shows patterns P1 to P5 of requests for reflection from the robot 102 when a user 101 utters a speech 110 of "What time will the sunrise be on Mt. Fuji tomorrow?" to the robot 102. Note that at least one of the volume, speech rate, pronunciation accuracy, and surrounding environment of the speech 110 is assumed to be different for each of the following (A) to (E).

(A) shows an example of a response in the reflection pattern P1. Specifically, for example, the robot 102 recognizes the speech voice 110 as "Ox△□ of tomorrow's sunrise over Mt. Fuji" ("Ox△□" is the part that could not be recognized due to low reliability), and generates a response sentence (hereinafter, the reflection sentence) to reflect the part with a high reliability above a threshold (of tomorrow's sunrise over Mt. Fuji) as "I didn't hear it very well. What is 'tomorrow's sunrise over Mt. Fuji'?" as the reflection pattern P1, and speaks it to the user 101 who originated the speech.

(B) shows an example of a response in the repeat pattern P2. Specifically, for example, the robot 102 recognizes the speech 110 as "Ox△Mt. FujiOx△□What's the time?" (where "Ox△" and "Ox△□" are parts that could not be recognized because their reliability was below the threshold), generates a repeat sentence "'Mt. Fuji'? 'Time'? Please ask the question again," which repeats the words "Mt. Fuji" and "Time," which have a reliability above the threshold, as the repeat pattern P2, and speaks it to the user 101.

(C) shows an example of a response in the repeat pattern P3. Specifically, for example, the robot 102 recognizes the speech sound 110 as "What time will the sunrise be tomorrow at XX △ □?" (where "XX △ □" is a portion that could not be recognized because its reliability was below the threshold), generates a guess (Mt. Fuji) for the portion (XX △ □) with a low reliability below the threshold and a repeat sentence that confirms the speech sound 110 uttered by the user 101 including the guess, "Did you say 'What time will the sunrise be tomorrow at Mt. Fuji'?" as the repeat pattern P3, and repeats it to the user 101.

(D) shows an example of a response in the pattern P4. Specifically, for example, the robot 102 recognizes the speech 110 with high accuracy, and generates a response sentence, "Did you say, 'What time will the sun rise on Mt. Fuji tomorrow?'" to confirm the entire recognized question as the pattern P4, and speaks it to the user 101.

(E) shows an example of a response in the pattern P5 of asking back. Specifically, for example, the robot 102 cannot recognize the speech 110, and generates a asking back sentence, "I didn't hear it very well, so please say it again," as the pattern P5 of asking back, requesting the user 101 to ask again.

The robot 102 calculates the speech recognition result of the spoken voice 110, not only the speech recognition reliability but also the language likelihood (perplexity of the language model), and selects either a response using normal dialogue control or the generation of a review sentence using the above-mentioned review patterns P1 to P5, depending on the speech recognition reliability and language likelihood.

In this way, by informing the user 101 that some or all of the spoken voice 110 could not be recognized, the robot 102 can prompt the user 101 to repeat the necessary speech and smoothly advance the dialogue with the user 101. In addition, the user 101 can know which parts of the spoken voice 110 have been conveyed and which parts have not.

<Speech dialogue system>
2 is an explanatory diagram showing an example of the system configuration of the voice dialogue system according to Example 1. The voice dialogue system 200 is, for example, a client-server system, and includes a server 201 and an information processing device 104 such as a robot 102 and a smartphone 103. The server 201 and the information processing device 104 can communicate with each other via a network 202 such as the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

In the case of a client-server system, the voice dialogue program is installed in the server 201. Therefore, the server 201, as a voice dialogue device, executes voice recognition processing, voice recognition reliability calculation, language likelihood calculation, and response sentence generation. In this case, the information processing device 104 becomes a dialogue interface that inputs the spoken voice 110, converts the data of the inputted spoken voice 110, transmits the converted voice data to the server 201, receives the response sentence from the server 201, and speaks the response sentence.

On the other hand, in the case of a stand-alone type, the voice dialogue program is installed in the information processing device 104, and the server 201 is not required. Therefore, as a voice dialogue device, the information processing device 104 executes input of spoken voice 110, data conversion of the inputted spoken voice 110, voice recognition processing, calculation of the reliability of the voice recognition, calculation of the language likelihood, generation of a response sentence, and speaking of the response sentence.

<Example of hardware configuration of a voice dialogue device>
3 is a block diagram showing an example of a hardware configuration of the voice dialogue device. The voice dialogue device 300 has a processor 301, a storage device 302, an input device 303, an output device 304, and a communication interface (communication IF) 305. The processor 301, the storage device 302, the input device 303, the output device 304, and the communication IF 305 are connected by a bus 306. The processor 301 controls the voice dialogue device 300. The storage device 302 is a working area for the processor 301. The storage device 302 is a non-temporary or temporary recording medium that stores various programs and data. Examples of the storage device 302 include a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), and a flash memory. The input device 303 inputs data. The input device 303 may be, for example, a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, or a biosensor. The output device 304 outputs data. The output device 304 may be, for example, a display, a printer, or a speaker. The communication IF 305 is connected to the network 202 and transmits and receives data.

<Example of functional configuration of the voice dialogue device 300>
4 is a block diagram showing an example of a functional configuration of a voice dialogue device 300 according to the first embodiment. The voice dialogue device 300 is a computer in which a voice dialogue program is installed. The voice dialogue device 300 can access a voice recognition model 411, a standard language model 412, a domain-specific language model 413, a question sentence language model 414, a dialogue context language model 415, and a dialogue knowledge DB 416 (database) 416. These are stored in the storage device 302 of the voice dialogue device 300 shown in FIG. 3 or in the storage device 302 of another computer that can communicate with the voice dialogue device 300.

The speech recognition model 411 is a statistical model of the correspondence between acoustic features and phonemes, the correspondence between phoneme strings and words, and word strings. An example of the speech recognition model 411 is a weighted finite-state transducer (WFST) that combines an acoustic model and a language model into a single object. An example of the acoustic model is a hybrid acoustic model (DNN-HMM) of a Deep Neural Network (DNN) and a Hidden Markov Model (HMM). An example of the language model is N-gram.

The standard language model 412 is a statistical model of word sequences obtained from texts in various fields, including dialogue. The standard language model 412 may be, for example, a word N-gram or a recurrent neural network (RNN). The standard language model 412 is used to evaluate the likelihood of a particular language (Japanese in this example).

The domain-specific language model 413 is a statistical model of word sequences obtained from text related to the operating environment of the robot 102 (for example, if the operating environment is a tourist guide, then text related to the tourist spot guide) or a collection of anticipated questions. The domain-specific language model 413 may be, for example, a word N-gram or an RNN. The domain-specific language model 413 is used to evaluate the likelihood of the content being spoken in the operating environment. The standard language model 412 may be a language model that aggregates domain-specific language models 413 for multiple types of fields, or may be a language model related to general spoken language excluding the domain-specific language models 413.

The question language model 414 is a statistical model of word sequences at the part-of-speech level obtained from a collection of expected questions and example questions actually received from the user 101. The question language model 414 may be, for example, an N-gram part-of-speech model or an RNN. The question language model 414 is a language model used to determine whether an utterance by the user 101 is a question from the word sequence.

The dialogue context language model 415 is a statistical model that learns statistics of any word and the group of words that appear around it, as well as statistics of phrase arrangements, for texts in any field, including dialogue. An example of the dialogue context language model 415 is Bidirectional Encoder Representations from Transformers (BERT).

The dialogue knowledge DB416 is a database that stores examples of anticipated questions and past questions in association with their answers.

The voice dialogue device 300 also has a voice recognition unit 401, a language likelihood acquisition unit 402, a reliability acquisition unit 403, a reflection determination unit 404, a dialogue control unit 405, a reflection sentence generation unit 406, a dialogue history management unit 407, and a voice synthesis unit 408. Specifically, these are realized by, for example, having the processor 301 execute a voice dialogue program stored in the storage device 302 shown in FIG. 3.

The speech recognition unit 401 receives the waveform data of the spoken voice 110 and converts the waveform data into speech feature X using the speech recognition model 411. Specifically, for example, the speech recognition unit 401 cuts out the waveform data into pieces having a length of about 25 msec and converts them into a speech feature vector x. For example, 13-dimensional Mel Frequency Cepstrum Coefficients (MFCC) are used for the speech feature vector x.

The speech recognition unit 401 converts the entire section of the waveform data while shifting this speech feature vector x by about 10 [msec], and generates speech feature X = [x_1, ..., x_t, ..., x_T]. x_t is the speech feature vector converted in the t-th section of the waveform data. T is the number of sections of the waveform data. Then, the speech recognition unit 401 finds the most likely word string Wt under the conditions given the speech feature X, using the following formula (1). Note that W is a word string.

The speech recognition unit 401 outputs the word string Wt to the language likelihood acquisition unit 402. The speech recognition unit 401 also generates a word lattice and outputs it to the confidence acquisition unit 403. The word string Wt and the word lattice are the speech recognition result.

Figure 5 is an explanatory diagram showing an example of a speech recognition result. The speech recognition result 500 includes a word string Wt and a word lattice WL. The word string Wt (as well as the word string W) is composed of the spelling, reading, and part of speech of each of one or more words. If the word string Wt is "kyo", "wa", and "sunny", the word string Wt is composed of the spelling "kyo", the reading "kyo", and the part of speech "noun" of the word "kyo", the spelling "ha", the reading "wa", and the part of speech "particle" of the word "wa", and the spelling "hare", the reading "hare", and the part of speech "noun" of the word "sunny".

Note that the speech recognition model 411 and each language model (except for the dialogue context language model 415) have the same specifications for word division and part of speech system. The BERT used in the dialogue context language model 415 automatically determines word division when the dialogue context language model 415 is trained.

The word lattice WL is network structure data of candidate hypotheses for speech recognition. A numerical value is assigned to each word, which is the sum of the acoustic likelihood and linguistic likelihood. For example, the sum of the acoustic likelihood and linguistic likelihood for word A (today) is "0.7", the sum of the acoustic likelihood and linguistic likelihood for word B (is) is "0.6", and the sum of the acoustic likelihood and linguistic likelihood for word C (sunny) is "0.7". The sequence of words on the path that maximizes the sum of these numerical values is the word string Wt.

The voice recognition unit 401 may also transfer waveform data of the user 101's dialogue voice to another computer that can communicate with the voice dialogue device 300, and obtain the voice recognition result 500 generated by the other computer.

Returning to FIG. 4, the language likelihood acquisition unit 402 uses multiple language models for the word string Wt input from the speech recognition unit 401 to calculate the language likelihood (test set perplexity) for each language model. The multiple language models are a standard language model 412, a domain-specific language model 413, a question language model 414, and a dialogue context language model 415. Of these, the standard language model 412 and the question language model 414 are essential language models for the language likelihood acquisition unit 402.

The occurrence probability of word string W1n when N-gram (word N-gram, part of speech N-gram) is, for example, as shown in formula (2) below, the occurrence probability of word string W1n when RNN is used is as shown in formula (3) below, and the occurrence probability of word string W1n when BERT is used is as shown in formula (4) below. Word string W1n means the arrangement of words from word W1 to word Wn (n is the total number of words contained in word string W1n), and here word string W1n = word string Wt. The language likelihood acquisition unit 402 outputs the language likelihood calculated for each language model to the reflection determination unit 404.

When calculating the language likelihood using the part-of-speech N-gram question language model 414 according to the above formula (2), the language likelihood acquisition unit 402 converts the word string Wt into a part-of-speech string. Since each word in the word string Wt is composed of a spelling, a reading, and a part of speech, the language likelihood acquisition unit 402 deletes the spelling and reading from each word in the word string Wt and converts it into a word string consisting only of parts of speech, that is, a part-of-speech string.

Figure 6 is an explanatory diagram showing an example of conversion from a word string to a part-of-speech string. In the case of a word string Wt, it is converted into a part-of-speech string of noun → particle → noun. The linguistic likelihood acquisition unit 402 regards the part-of-speech string as a word string W1n and calculates the linguistic likelihood by applying the above formula (2). Note that the linguistic likelihood acquisition unit 402 may assign conjugated forms to parts of speech that have conjugated forms, and base forms (words) to pronouns and particles, so that more information is not lost than necessary due to the conversion to a part-of-speech string.

Figures 7 and 8 are explanatory diagrams showing an example of a part-of-speech classification definition 700 for a part-of-speech string, which conforms to the short unit definition (part-of-speech classification in a dictionary for morphological analysis) defined by the National Institute for Japanese Language and Linguistics. The part-of-speech classification definition 700 is, for example, information stored in the storage device 302. The part-of-speech classification definition 700 has a major category 701, a medium category 702, a small category 703, and supplementary information 704. The major category 701 defines the parts of speech contained in words W1 to Wn. The medium category 702 defines the items into which the major category 701 is subdivided. The small category 703 defines the items into which the medium category 702 is subdivided. The supplementary information 704 defines information to be supplemented to the major category 701 to the small category 703. The parts of speech of each word W1 to Wn in the word string Wt include information such as major category 701 to minor category 703, the conjugated form for parts of speech that have conjugated forms, and the English spelling for katakana words derived from English words.

When converting the word string Wt into a parts-of-speech string, the language likelihood acquisition unit 402 refers to the parts-of-speech classification definition 700, and for each part of speech, only the information in the parts-of-speech classification definition 700 is retained, and the base form (word) is assigned to parts of speech whose base form is described in the supplemental information 704. This prevents information loss in the parts-of-speech string.

The language likelihood acquisition unit 402 may transfer the word string Wt input from the speech recognition unit 401 to another computer that can communicate with the speech dialogue device 300, and acquire the language likelihood for each language model calculated by the other computer based on the word string Wt.

Returning to FIG. 4, the reliability acquisition unit 403 calculates the speech recognition reliability (hereinafter simply "reliability") of the word string Wt and each of the words W1 to Wn that make up the word string Wt based on the word lattice WL. Specifically, for example, the reliability is an index value that indicates how reliable the word string Wt obtained by the speech recognition unit 401 is. Here, the larger the value, the higher the reliability.

Figure 9 is an explanatory diagram showing an example of reliability calculation by the reliability acquisition unit 403. The reliability acquisition unit 403 first converts the word lattice WL into a network 600. For a word with only one state transition (e.g., word E), the reliability acquisition unit 403 inserts a state transition with an empty symbol ε, whose word reliability is 1.0, to align it with the state transition of word E.

The confidence acquisition unit 403 then normalizes the confidence of each word A to F in the network 600 to 1.0 for each section, and converts it into a word confusion network 601. The confidence acquisition unit 403 also calculates the harmonic mean of the normalized confidences of words A to F in the word confusion network 601, and sets this as the confidence of the word string Wt. The confidence acquisition unit 403 outputs the confidence of the normalized words A to F and the confidence of the word string Wt to the reflection determination unit 404.

The reliability acquisition unit 403 may transfer the word lattice WL input from the speech recognition unit 401 to another computer that can communicate with the speech dialogue device 300, and acquire the reliability calculated by the other computer based on the word lattice WL.

Returning to FIG. 4, the ask-back determination unit 404 determines whether or not to ask the user 101 back. Specifically, for example, the ask-back determination unit 404 uses the language likelihood for each language model from the language likelihood acquisition unit 402 and the reliability of the word string Wt from the reliability acquisition unit 403 to determine which process to execute: dialogue control by the dialogue control unit 405 or ask-back processing by the ask-back sentence generation unit 406.

Note that a threshold value is set for each of the language likelihood for each language model, the reliability of the word string Wt, and the reliability of each normalized word W1 to Wn. If any one of the language likelihood for each language model and the reliability of the word string Wt is less than the threshold, the reflection determination unit 404 executes the reflection sentence generation process. Note that the reliability of each normalized word W1 to Wn is used by the reflection sentence generation unit 406.

When the ask-back determination unit 404 determines that there is no need to ask the user 101 to ask again, the dialogue control unit 405 refers to the dialogue knowledge DB 416 and generates a response sentence corresponding to the speech recognition result 500 of the spoken voice 110.

When the ask-back determination unit 404 determines that the user 101 should ask the user 101 to ...

The dialogue history management unit 407 accumulates the word string Wt of the speech 110 of the user 101 and the response sentence generated by the voice dialogue device 300. The voice synthesis unit 408 converts the response sentence generated by the dialogue control unit 405 or the reflective sentence generation unit 406 into voice and outputs it. When the voice dialogue device 300 is realized by the server 201 of a client-server system, the voice dialogue device 300 does not have the voice synthesis unit 408, and transmits the response sentence generated by the dialogue control unit 405 or the reflective sentence generation unit 406 to the information processing device 104 such as the robot 102 or smartphone 103 that serves as the client. In this case, the client has the voice synthesis unit 408, and converts the response sentence received from the server 201 into voice and outputs it.

<Example of voice dialogue processing procedure>
Fig. 10 is a flowchart showing an example of a voice dialogue processing procedure by the voice dialogue device 300 according to the embodiment 1. As shown in Fig. 5, the voice dialogue device 300 inputs the dialogue voice of the user 101, executes the voice recognition process by the voice recognition unit 401, and outputs the voice recognition result 500 (step S1001).

Next, the speech dialogue device 300 acquires the reliability of the word string Wt and the reliability of the words W1 to Wn that make up the word string Wt, as shown in FIG. 9, by the reliability acquisition unit 403 (step S1002). In addition, the speech dialogue device 300 acquires the language likelihood for each language model by the language likelihood acquisition unit 402 (step S1003).

In addition, in step S1003, when the dialogue context language model 415 is used, the language likelihood acquisition unit 402 acquires the language likelihood using as input the sentence (word string) that combines the voice dialogue device 300 and the user's utterance 110 prior to the user's utterance 110.

for example,
Speech dialogue device 300: "Hello."
User 101: "Hello, what's your name?"
Speech dialogue device 300: "My name is robot."
User 101: "Oh really, that's cute."
Speech dialogue device 300: "Thank you very much."
User 101: “What can you do?” (utterance 110)
Take the following dialogue as an example.

In this case, the word strings obtained from the speech recognition unit 401 for the speech dialogue device 300's speech "Hello" before the speech dialogue device 300's speech 110 "What can you do?", the speech dialogue device 300's speech "Hello, what's your name", the speech dialogue device 300's speech "My name is Robot 102", the speech dialogue device 300's speech "Oh, really, that's cute", and the speech dialogue device 300's speech "Thank you" are concatenated to obtain the word string "Hello. Hello, what's your name. My name is Robot. Oh, really, that's cute. Thank you. What can you do.".

The voice dialogue device 300 inputs this connected word sequence "Hello. Hello, what's your name? My name is Robot. Oh really, that's cute. Thank you. What can you do?" into the dialogue context language model 415 and calculates the language likelihood. In this way, by inputting not only the spoken voice 110 of the user 101 but also the previous dialogue into the dialogue context language model 415, it is possible to improve the accuracy of the language likelihood obtained from the dialogue context language model 415.

When using the dialogue context language model 415 in step S1003, the language likelihood acquisition unit 402 may generate a sentence by concatenating each word of the word string Wt in the speech recognition result 500, and may generate a word string Wt' by morphological analysis of a morphological analyzer included in the dialogue context language model 415. As a result, the word strings Wt and Wt' may have different word divisions. Then, the language likelihood acquisition unit 402 calculates the likelihood of the word string Wt'.

Then, the voice dialogue device 300 determines whether or not to ask the user 101 to reflect using the reflection determination unit 404 (step S1004). Specifically, for example, the voice dialogue device 300 determines whether or not any one of the language likelihood for each language model and the reliability of the word string Wt is less than a threshold value. If any one of the language likelihood for each language model and the reliability of the word string Wt is less than a threshold value (step S1004: Yes), that is, if the user 101 should be asked to reflect, the voice dialogue device 300 executes a reflection process using the reflection sentence generation unit 406 (step S1005).

The voice dialogue device 300 generates a response sentence from one of the response patterns P1 to P5 through the reflective sentence generation process (step S1005), or notifies the user that no response sentence will be generated, and then proceeds to step S1009. Details of the reflective sentence generation process (step S1005) will be described later with reference to FIG. 11.

On the other hand, if neither the language likelihood for each language model nor the reliability of the word string Wt is less than the threshold (step S1005: No), i.e., if there is no need to ask the user 101 to repeat the question, the speech dialogue device 300 determines whether the user 101 is currently being asked to repeat the question using the repeat pattern P1, P3, or P4 (step S1006).

If the user is currently asking back in the reflection pattern P1, P3, or P4 (step S1006: Yes), the voice dialogue device 300 executes a restatement utterance interpretation process by the reflection sentence generation unit 406 (step S1007) and proceeds to step S1009. The restatement utterance interpretation process (step S1007) is a process of interpreting the speech restated by the user 101 and responding according to the interpretation. Details of the restatement interpretation process (step S1007) will be described later with reference to FIG. 14.

On the other hand, if the user is not asking for a response in the reflection pattern P1, P3, or P4 (step S1006: No), the voice dialogue device 300 executes a dialogue control process by the dialogue control unit 405 (step S1008) and proceeds to step S1009. The dialogue control process (step S1008) is a process for generating a response sentence in response to the voice uttered by the user 101. Details of the dialogue control process (step S1008) will be described later with reference to FIG. 15.

In step S1009, the dialogue history management unit 407 of the voice dialogue device 300 accumulates, as a dialogue history (step S1009), the word string Wt of the user's 101 speech and the response sentences generated by the ask-back process (step S1005), the rephrasing interpretation process (step S1007), and the dialogue control process (step S1008).

Then, the voice dialogue device 300 converts the response sentence generated by the dialogue control unit 405 or the reply sentence generation unit 406 into voice and outputs it by the voice synthesis unit 408 (step S1010). Note that if the voice dialogue device 300 is realized as a stand-alone type, the voice dialogue device 300 executes the processes of steps S1001 to S1010 shown in FIG. 10.

On the other hand, when the voice dialogue device 300 is realized by the server 201 of a client-server system, the voice dialogue device 300 executes steps S1001 to S1009 and transmits a response sentence to a communication device such as a communication robot 102 or a smartphone that serves as a client. Then, the client converts the response sentence generated by the dialogue control unit 405 or the replies sentence generation unit 406 into voice and outputs it using the voice synthesis unit 408 (step S1010).

<Reply Sentence Generation Process (Step S1005)>
Fig. 11 is a flowchart showing a detailed example of the process of generating a reflection sentence (step S1005) shown in Fig. 10. The reflection sentence generator 406 determines whether the reliability of the word string Wt is less than a threshold value (step S1101).

If the reliability of the word string Wt is less than the threshold (step S1102: Yes), the reflection sentence generation unit 406 determines whether the total reliability of words W1 to Wn is less than the threshold (step S1102). If the total reliability of words W1 to Wn is less than the threshold (step S1102: Yes), the reflection sentence generation unit 406 determines which part of words W1 to Wn has a reliability less than the threshold (step S1103).

If it is part A (step S1103: part A), the process proceeds to step S1104, if it is part B (step S1103: part B), the process proceeds to step S1105, and if it is part C (step S1103: part C), the process proceeds to step S1106. Note that, as an example, if the word string Wt corresponds to both part A and part B, part A is applied preferentially, and if it corresponds to both part B and part C, part B is applied preferentially.

Part A refers to multiple words that exist in the first or second half of the word string Wt and have a reliability below a threshold. If the number of words n in the words W1 to Wn is an even number, the first half refers to words W1 to W(n/2) of the words W1 to Wn, and the second half refers to words W(n/2+1) to Wn of the words W1 to Wn. If the number of words n in the words W1 to Wn is an odd number, the first half refers to words W1 to W((n+1)/2) of the words W1 to Wn, and the second half refers to words W((n+1)/2) to Wn of the words W1 to Wn.

However, the multiple words include at least one independent word. An independent word is a word of a part of speech other than an auxiliary word (verb, adjective, adjectival verb, noun, adverb, conjunction, conjunction, interjection). An auxiliary word is a word whose part of speech is an auxiliary verb or particle. In the case of part A (step S1103: part A), a group of words that were difficult for the voice dialogue device 300 to recognize exist in the first half or second half of the speech of the user 101. In other words, part A corresponds to a case where the voice dialogue device 300 was unable to hear half of the speech of the user 101.

In the case of part A (step S1103: part A), the reflection sentence generation unit 406 generates a reflection sentence that asks for the remaining part of the word string Wt excluding part, i.e., the words whose reliability is equal to or greater than the threshold, as reflection pattern P1 (see FIG. 1) (step S1104), and proceeds to step S1009.

Part B refers to multiple words (including at least one independent word) that exist discretely in the word string Wt and have a reliability below the threshold. However, in order to avoid overlap with part A, there must be at least one word whose reliability is below the threshold in each of the first and second halves of the word string Wt. In other words, part B corresponds to a case where the voice dialogue device 300 cannot hear fragments of the spoken voice 110 of the user 101.

In the case of part B (step S1103: part B), a asking-back sentence that asks back about words in the word string Wt that do not fall into part B and have a reliability equal to or greater than a threshold is generated as asking-back pattern P2 (see FIG. 1) (step S1104), and the process proceeds to step S1009.

Furthermore, part C is one independent word in the word string Wt whose reliability is less than the threshold value, or two consecutive words (including at least one independent word) in the word string Wt whose reliability is less than the threshold value. However, if there is an overlap with part A, part A takes precedence (part C may be applied with priority over part A). In other words, part C corresponds to a case where the voice dialogue device 300 cannot hear a part of the spoken voice 110 of the user 101.

In the case of part C (step S1103: part C), the reflection sentence generation unit 406 executes a mask word estimation process (step S1106). The mask word estimation process (step S1106) is a process of estimating a mask word. A mask word is a word that corresponds to part C. Details of the mask word estimation process (step S1106) will be described later with reference to FIG. 12.

After executing the mask word estimation process (step S1106), the reflection sentence generation unit 406 determines whether the estimation of the mask word is successful (step S1107). If the estimation of the mask word is successful (step S1107: Yes), the reflection sentence generation unit 406 generates the estimation result of the part C whose reliability is less than the threshold value and a reflection sentence that confirms the speech voice 110 of the user 101 including the estimation result as a reflection pattern P3 (see FIG. 1).

In the case of the reflection pattern P3 in FIG. 1, the estimated result of part C is "Mt. Fuji," and the reflection sentence for confirming the speech 110 of the user 101 including the estimated result is "What time will sunrise be tomorrow at Mt. Fuji?"

On the other hand, if the mask word estimation is not successful (step S1107: No), the reflection sentence generation unit 406 generates a reflection sentence requesting a repeat question, i.e., a repeat request for the spoken voice 110, as reflection pattern P5 (see FIG. 1) (step S1109), and proceeds to step S1009.

Also, in step S1102, if the reliability of all words W1 to Wn is less than the threshold value (step S1102: Yes), the reflection sentence generation unit 406 also executes step S1109 and proceeds to step S1009.

In addition, in step S1101, if the reliability of the word string Wt is equal to or greater than the threshold (step S1102: No), the reflection sentence generation unit 406 determines whether the language likelihood of the standard language model 412 is less than the threshold (step S1110). If the language likelihood of the standard language model 412 is less than the threshold (step S1110: Yes), step S1109 is executed and the process proceeds to step S1009.

On the other hand, if the language likelihood of the standard language model 412 is equal to or greater than the threshold (step S1110: No), the reflection sentence generation unit 406 determines whether the language likelihood of the question sentence language model 414 is less than the threshold (step S1111).

If the language likelihood of the question language model 414 is equal to or greater than the threshold (step S1111: No), the language likelihood of the domain-specific language model 413 or the dialogue context language model 415 is less than the threshold. Therefore, the reflection sentence generation unit 406 generates a reflection sentence that repeats and confirms the entire question (the speech 110 of the user 101) as a reflection pattern P4 (see FIG. 1) (step S1112), and proceeds to step S1009. Also, even if neither the domain-specific language model 413 nor the dialogue context language model 415 is used, if the language likelihood of the question language model 414 is equal to or greater than the threshold (step S1111: No), the reflection sentence generation unit 406 executes step S1112.

On the other hand, if the language likelihood of the question language model 414 is less than the threshold value (step S1111: Yes), the reflection sentence generation unit 406 does not generate a reflection sentence, notifies the non-generation of a reflection sentence (step S1113), and proceeds to step S1009. In this way, the reflection sentence generation unit 406 can generate reflection sentences of the reflection patterns P1 to P5 according to the reliability of each word and the language likelihood of each language model.

<Masked Word Estimation Process (Step S1106)>
Fig. 12 is a flowchart showing a detailed example of the processing procedure of the mask word inference process (step S1106) shown in Fig. 11. Fig. 13 is an explanatory diagram showing an example of the mask word inference process (step S1106) shown in Fig. 11. The mask word inference process (step S1106) is executed in the case of step S1103: part C. In Fig. 13, a word string Wt including "○×▽" which is part C is set as a word string 1300 of "Please tell me the height of ○×▽".

The reflection sentence generation unit 406 extracts the reading of the word "○×▽" in part C (step S1201). In FIG. 13, it is assumed that "Fusan" has been extracted as the reading 1302 of "○×▽". Next, the reflection sentence generation unit 406 masks the word "○×▽" in part C to make it "***" (step S1202). The word string 1300 after masking the word "○×▽" in part C is set as word string 1301.

Next, the reflection sentence generator 406 inputs the masked word sequence 1301 into a BERT 1310, which is an example of the dialogue context language model 415, to predict mask words (step S1203). Here, the predicted results 1303 are "Landmark Tower" (predicted mask word 1303A), "Tokyo Tower" (predicted mask word 1303B), and "Mt. Fuji" (predicted mask word 1303C).

Next, the reflection sentence generator 406 extracts the readings of the predicted mask words 1303A to 1303C (step S1204). Here, the predicted extraction results 1304 are "Landmark Tower" (predicted mask word reading 1304A), "Tokyo Tower" (predicted mask word reading 1304B), and "Fujisan" (predicted mask word reading 1304C).

Then, the reflection sentence generator 406 compares the reading 1302 of the word "○×▽" in part C, "Fusan", with each of the predicted mask word readings 1304A to 1304C, for example, using the edit distance (Levenshtein distance) (step S1206). The reflection sentence generator 406 determines whether any of the predicted mask word readings 1304A to 1304C is within a predetermined distance from the reading 1302 of the word "○×▽" in part C, "Fusan" (step S1206).

If there are no predicted mask word readings 1304A-1304C within the specified distance (step S1206: No), the reflection sentence generator 406 ends the mask word estimation process (step S1106) and proceeds to step S1107. In this case, in step S1107, the mask word estimation fails (step S1107: No).

On the other hand, if there are readings 1304A to 1304C of the predicted mask word within the specified distance (step S1206: Yes), the reflection sentence generation unit 406 selects the predicted mask word with the shortest edit distance as the predicted mask word with the closest reading (step S1207). Here, as an example, "Fujisan" (predicted mask word 1303C) with the reading "Fujisan" (predicted mask word reading 1304C) is selected. Then, the reflection sentence generation unit 406 ends the mask word estimation process (step S1106) and proceeds to step S1107. In this case, in step S1107, the mask word estimation is successful (step S1107: Yes).

<Repair utterance interpretation process (step S1007)>
Fig. 14 is a flowchart showing a detailed example of the process procedure of the repair utterance interpretation process (step S1007) shown in Fig. 10. The repair utterance interpretation process (step S1007) is executed if step S1006: Yes, that is, if the user is asking the user to reflect in the reflection pattern P1, P3 or P4.

Step S1006: If the answer is Yes, and the pattern of the current repetition is P1 (Step S1401: P1), the repetition sentence generation unit 406 combines the user 101's restated speech (the word string Wt of the currently input speech) with the recognition result of the previous utterance (Step S1402), and proceeds to the dialogue control process (Step S1403).

Here, let us say that the recognition result of the previous utterance, which is the listening pattern P1, is "Tomorrow's sunrise over Mt. Fuji." "Tomorrow's sunrise over Mt. Fuji" is a group of words whose reliability for each word is equal to or exceeds a threshold value.

If "tomorrow's sunrise on Mt. Fuji" is the first half of the word sequence of the previous utterance of user 101, the repeat sentence generation unit 406 concatenates the user's 101's restatement speech (the word sequence Wt of the currently input utterance (e.g., "tell me the time")) to the end of "tomorrow's sunrise on Mt. Fuji" to generate "tell me the time of sunrise on Mt. Fuji tomorrow."

Let us also assume that the recognition result of the previous utterance, which is the reflection pattern P1, is "Tell me the time of sunrise." "Tell me the time of sunrise" is a word group in which the reliability of each word is equal to or greater than a threshold value.

If "Tell me the time of sunrise" is the first half of the word sequence of the previous speech of the user 101, the repeat sentence generation unit 406 concatenates the user 101's restatement speech (the word sequence Wt of the currently input speech (for example, "Mt. Fuji tomorrow")) to the beginning of "Tell me the time of sunrise" to generate "Mt. Fuji tomorrow, tell me the time of sunrise."

However, the review sentence generation unit 406 deletes one of the parts that match the user's 101 restated speech (the word sequence Wt of the currently input speech) and the recognition result of the previous utterance to prevent redundancy.

For example, if "Tell me the time of sunrise" is the recognition result of the previous utterance, and the word sequence of the previous utterance of the user 101 is "The time of sunrise on Mt. Fuji tomorrow," the combined result will be "Tell me the time of sunrise on Mt. Fuji tomorrow." In this case, since "the time of sunrise" appears twice, the repeat sentence generation unit 406 deletes one "the time of sunrise" to make it "Tell me the time of sunrise on Mt. Fuji tomorrow."

If the pattern of the current asking back is P3 or P4 (step S1401: P3, P4), the asking back sentence generator 406 judges whether the answer of the user 101 (the word string Wt of the currently input speech) to the asking back pattern P3 or P4 (e.g., "Did you say, 'Tell me the height of Mt. Fuji?') is positive or negative (step S1404). If the answer is positive (step S1404: positive), the process proceeds to the dialogue control process (step S1403). The dialogue control process (step S1403) will be described later with reference to FIG. 15.

On the other hand, if the answer is no (step S1404: no), the rephrase sentence generation unit 406 generates a response sentence requesting the question to be asked again (e.g., "Please ask the question again") (step S1406), and proceeds to step S1007.

<Dialogue control process (steps S1008, S1403)>
Fig. 15 is a flowchart showing a detailed example of the processing procedure of the dialogue control process (steps S1008, S1403) shown in Fig. 10 and Fig. 14. First, the dialogue control unit 405 refers to the dialogue knowledge DB 416 to search for an assumed question sentence that is similar to the word string Wt of the speech recognition result 500 (step S1501). Specifically, for example, the dialogue control unit 405 calculates the similarity between the word string Wt and each assumed question sentence in the dialogue knowledge DB 416 and the word string Wt based on the edit distance between the word string Wt and the assumed question sentence in the dialogue knowledge DB 416. Here, as an example, the reciprocal of the edit distance is taken as the similarity. Therefore, an assumed question sentence with a larger similarity value is more similar to the word string Wt.

Next, the dialogue control unit 405 determines whether there is an expected question sentence whose similarity is equal to or greater than the threshold value (step S1502). If there is no expected question sentence whose similarity is equal to or greater than the threshold value (step S1503: No), the dialogue control unit 405 generates a response sentence to the effect that the meaning of the question is not understood, and proceeds to steps S1009 and S1007. On the other hand, if there is an expected question sentence whose similarity is equal to or greater than the threshold value (step S1502: Yes), the dialogue control unit 405 outputs an answer sentence that corresponds to the expected question sentence whose similarity is equal to or greater than the threshold value in the dialogue knowledge DB 416 as a response sentence (step S1504), and proceeds to steps S1009 and S1007.

<Example of dialogue>
16 is a flowchart showing an example of the flow of a dialogue between the user 101 and the voice dialogue device 300. It is assumed that the user 101 utters "Tell me the height of Mt. Fuji" (step S1601). When the voice dialogue device 300 recognizes "Tell me the height of Mt. Fuji" as "Tell me the height of ***" (step S1602), it infers the "***" in the masked portion and responds "Is it Mt. Fuji?" (step S1603) (request pattern P3).

If the user 101 responds with "No" to the question "Is it Mt. Fuji?" (step S1604), the voice dialogue device 300 utters "Please ask the question again" since the estimation result of the masked portion, "Mt. Fuji", has been denied (step S1605).

Furthermore, if the user 101 responds with "Yes" to the question "Is it Mt. Fuji?" in step S1603 (step S1606), the voice dialogue device 300 recognizes the utterance of the user 101 in S1601 as "Tell me the height of Mt. Fuji." Therefore, the voice dialogue device 300 regards "The height of Mt. Fuji" as an expected question sentence, searches for the corresponding answer "3,776 meters" from the dialogue knowledge DB 416 (step S1610), and speaks "The height of Mt. Fuji is 3,776 meters" (step S1611).

In addition, if the voice dialogue device 300 recognizes "*** of Mt. Fuji" in response to the speech 110 of the user 101 in step S1601, "Tell me the height of Mt. Fuji" (step S1607), it responds with "What on Mt. Fuji?" (step S1608) (request pattern P1). In response to this, if the user 101 responds with "height" (step S1609), the voice dialogue device 300 searches for the corresponding answer "3776 meters" from the dialogue knowledge DB 416, regarding "The height of Mt. Fuji" as the expected question (step S1610), and speaks "The height of Mt. Fuji is 3776 meters" (step S1611).

In this way, according to the first embodiment, it is possible to select the ask-back patterns P1 to P5 for the user's utterance 110 according to the word reliability and language likelihood. In particular, by evaluating the user's utterance 110 using the standard language model 412, the domain-specific language model 413, the question language model 414, and the dialogue context language model 415, it is possible to determine whether the utterance is likely to be Japanese, likely to be spoken on the spot, likely to be a question, and likely taking into account the dialogue context, and by asking for the utterance based on the determination result, it is possible to prevent the dialogue from breaking down.

In this way, by asking the user 101 to repeat the question only a minimum number of times based on the reliability of the voice recognition, the user 101 can be prevented from having to ask the same question multiple times, thereby reducing the inconvenience to the user 101.

Example 2 is an example of preventing a false positive, i.e., transition to step S1004: No, in the case where the language likelihood of the domain-specific language model 413 or the dialogue context-specific language model is equal to or greater than a threshold value due to a unique expression in the speech 110 of the user 101 in Example 1. This prevents the voice dialogue device 300 from requesting the user 101 to ask a question again (step S1405), from responding that the meaning of the expected question sentence is not understood by the dialogue control process (steps S1008, S1403) (step S1503), or from giving an unlikely answer corresponding to an unlikely expected question sentence (step S1504). Note that, since the contents of Example 2 will be mainly described here, the description of the parts that overlap with Example 1 will be omitted.

<Example of functional configuration of the voice dialogue device 300>
Fig. 17 is a block diagram showing an example of a functional configuration of a voice dialogue apparatus 300 according to a second embodiment. The voice dialogue apparatus 300 includes an individuality language model 1701 and an individual identification unit 1702 in addition to the configuration shown in Fig. 4 of the first embodiment. The individuality language model 1701 is stored in the storage device 302 of the voice dialogue apparatus 300 shown in Fig. 3 or in the storage device 302 of another computer capable of communicating with the voice dialogue apparatus 300. Specifically, the individual identification unit 1702 is realized by, for example, causing the processor 301 to execute a voice dialogue program stored in the storage device 302 shown in Fig. 3.

The individual language model 1701 is a language model created based on the characteristics (individuality) unique to the user 101, and specifically, for example, is created for each user 101 from the dialogue history of each user 101. Therefore, the dialogue history management unit 407 manages the dialogue history for each user 101. The individual language model 1701 is realized, for example, by a word N-gram or an RNN.

The personal identification unit 1702 identifies the user 101. Specifically, the personal identification unit 1702 manages biometric information of the user 101, such as fingerprints, palm veins, irises, facial images, and voice, and if the biometric information matches the inputted biometric information, identifies the inputter as the user 101 who has that biometric information. The personal identification unit 1702 may also manage a user ID and password, and if the inputted user ID and password match, identify the inputter as the user 101.

<Example of voice dialogue processing procedure>
18 is a flowchart showing an example of a voice dialogue processing procedure by the voice dialogue apparatus 300 according to the embodiment 2. The voice dialogue apparatus 300 executes steps S1801 to S1804 prior to steps S1001 to S1010 shown in FIG.

Specifically, for example, the voice dialogue device 300 identifies the user 101 who inputs the biometric information by the personal identification unit 1702 (step S1801). The voice dialogue device 300 determines whether the user 101 who inputs the biometric information is a registered user 101 (step S1802). Specifically, for example, the voice dialogue device 300 determines whether the registered biometric information matches the input biometric information, and if they match, determines that the user 101 who inputs the biometric information is a registered user 101.

If the user 101 is a registered user (step S1802: Yes), the voice dialogue device 300 loads the individual language model 1701 and dialogue history of the user 101 (step S1803) and proceeds to step S1001. On the other hand, if the user 101 is not a registered user (step S1802: No), the voice dialogue device 300 creates a new individual language model 1701 and dialogue history of the user 101 (hereinafter, new user 101) (step S1804). Specifically, for example, the voice dialogue device 300 interacts with the new user 101 to obtain a dialogue history, and creates the individual language model 1701 of the new user 101 based on the obtained dialogue history. Then, proceeds to step S1001.

The individual language model 1701 is used in language likelihood calculation (step S1003). Then, if the language likelihood of the question language model 414 is equal to or greater than the threshold value in step S1111 shown in FIG. 11 (step S1111: No), the language likelihood of any of the domain-specific language model 413, the dialogue context language model 415, or the individual language model 1701 is less than the threshold value. Therefore, the reflection sentence generation unit 406 generates a reflection sentence that repeats and confirms the entire question (the speech of the user 101) as reflection pattern P4 (see FIG. 1) (step S1112), and proceeds to step S1009.

Even if none of the domain-specific language model 413, the dialogue context language model 415, and the individual language model 1701 are used, if the language likelihood of the question language model 414 is equal to or greater than the threshold value (step S1111: No), the reflection sentence generation unit 406 executes step S1112.

In this way, according to the second embodiment, it is possible to suppress erroneous determination of language likelihood caused by the unique phrasing of the user 101, and facilitate smooth dialogue.

The voice dialogue device 300 according to the above-mentioned first and second embodiments can also be configured as follows (1) to (13).

(1) In a voice dialogue device 300 having a processor 301 that executes a voice dialogue program and a storage device 302 that stores the voice dialogue program, the processor 301 executes a reliability acquisition process (step S1102) that acquires the reliability of each word W1 to Wn that constitutes a word string Wt related to the spoken voice 110, and a reflection sentence generation process (step S1005) that generates a reflection sentence to ask the user 101, who is the source of the spoken voice 110, based on the reliability acquired by the reliability acquisition process (step S1102).

This allows the voice dialogue device 300 to generate an appropriate reflection sentence from, for example, the reflection patterns P1 to P3 and P5 shown in FIG. 11.

(2) In the speech dialogue device 300 described above in (1), in the process of generating a reflection sentence (step S1005), if the reliability of all words W1 to Wn constituting the word string Wt is less than the first threshold value (step S1102: Yes), the processor 301 generates a reflection sentence that requests the spoken voice 110 again (step S1109).

As a result, the voice dialogue device 300 can generate a review sentence for the review pattern P5 from among the review patterns P1 to P3 and P5 shown in FIG. 11, for example, and request the user 101 to ask the question again.

(3) In the speech dialogue device 300 described above in (1), in the process of generating a reflection sentence (step S1005), if the word string Wt contains a word whose reliability is equal to or greater than the first threshold (step S1102: No), the processor 301 generates a reflection sentence based on the position of the word in the word string Wt whose reliability is less than the first threshold.

As a result, the voice dialogue device 300 can generate a review sentence according to the position of a word whose reliability is less than the first threshold value, for example, from among the review patterns P1 to P3 and P5 as shown in FIG. 11.

(4) In the speech dialogue device 300 of (3) above, in the process of generating a reflection sentence (step S1005), if a word whose reliability is less than the first threshold value is present in the first or second half of the word string Wt (step S1103: part A), the processor 301 generates a reflection sentence to ask back about either the first or second half that does not contain a word whose reliability is less than the first threshold value (step S1104).

As a result, the voice dialogue device 300 can generate a reply sentence of, for example, reply pattern P1 from reply patterns P1 to P3 and P5 as shown in FIG. 11, and prompt the user 101 to speak the part of the spoken voice 110 that he or she was able to hear, and to repeat the part that he or she was unable to hear.

(5) In the speech dialogue device 300 of (3) above, in the process of generating a reflection sentence (step S1005), if there are multiple words with a reliability less than the first threshold value in both the first and second halves of the word string Wt (step S1103: part B), the processor 301 generates a reflection sentence that asks for words in the word string Wt that are equal to or greater than the first threshold value (step S1105).

As a result, the voice dialogue device 300 can generate a request for a repeat question from, for example, the request pattern P2 from among the request patterns P1 to P3 and P5 as shown in FIG. 11, and have the user 101 speak the part of the spoken voice 110 that the user 101 was able to hear.

(6) In the voice dialogue device 300 of (3) above, when a word string contains one word whose reliability is less than the first threshold value, or two consecutive words whose reliability is less than the first threshold value (step S1103: part C), the processor 301 executes a mask word estimation process (step S1106) to estimate what kind of word the one word or two consecutive words are based on the dialogue context language model 415 that has been trained on statistics of a word and the word group that appears around it, and statistics of phrase arrangements, and in the reflection sentence generation process (step S1005), the processor 301 generates a reflection sentence according to the estimation result obtained by the mask word estimation process (step S1106).

As a result, the voice dialogue device 300 can reduce the frequency of asking to repeat, for example, by estimating from the context a single word whose reliability is less than the first threshold value, or two consecutive words whose reliability is less than the first threshold value.

(7) In the speech dialogue device 300 of (6) above, in the reflection sentence generation process (step S1005), if the estimation by the mask word estimation process (step S1106) is successful (step S1107: Yes), the processor 301 generates a reflection sentence that includes the estimated word and confirms the spoken voice 110 (step S1108).

As a result, the voice dialogue device 300 can generate, for example, a ask-back sentence as a ask-back pattern P3 to confirm the estimated words and the entire spoken voice 110, and can inform the user 101 which parts were difficult to recognize and how the entire speech was recognized, thereby reducing the frequency of asking back.

(8) In the voice dialogue device 300 of (6) above, in the reflective sentence generation process (step S1005), if the estimation by the mask word estimation process (step S1106) fails (step S1107: No), the processor 301 generates a reflective sentence that re-requests the spoken voice 110.

This allows the voice dialogue device 300 to generate a repeat sentence of, for example, the repeat pattern P5 and ask the user 101 to ask the question again.

(9) In the speech dialogue device 300 of (1) above, in the process of generating a reflection sentence (step S1005), if the reliability of the word string Wt is less than the first threshold value (step S1101: Yes), the processor 301 generates a reflection sentence based on the reliability of each word W1 to Wn that constitutes the word string Wt.

As a result, if the reliability of the word string Wt is less than the first threshold value, it is determined that the speech 110 has not been correctly recognized, and the system can select from the re-listening patterns P1 to P3 and P5 how to ask for re-listening based on the reliability of each word W1 to Wn.

(10) In the voice dialogue device 300 of (1) above, the processor 301 executes a language likelihood acquisition process (step S1103) to acquire multiple language likelihoods obtained as a result of inputting a word string to each of the multiple language models 412 to 415, and in the reflection sentence generation process (step S1005), if the reliability of the word string Wt is equal to or greater than a first threshold (step S1101: No), the processor 301 generates a reflection sentence based on the multiple language likelihoods acquired by the language likelihood acquisition process (step S1103).

As a result, when the reliability of the recognition of the speech sound 110 is high, it is possible to determine which language model the word sequence Wt is most likely to be the result of, and select the replay pattern P4, P5, or no replay.

(11) In the speech dialogue device 300 of (10) above, in the process of generating a reflection sentence (step S1005), if the first language likelihood obtained as a result of inputting the word string Wt to the standard language model 412, which is a statistical model of word sequences obtained from texts in multiple fields among the multiple language models 412 to 415, is less than the second threshold value (step S1110: Yes), the processor 301 generates a reflection sentence that re-requests the spoken voice 110 (step S1109).

In this way, if the utterance is unlikely to be in standard language, a repeat sentence of the repeat pattern P5 can be generated to prompt the user 101 to ask the question again.

(12) In the voice dialogue device 300 of (11) above, in the process of generating a reflection sentence (step S1005), if the first language likelihood is equal to or greater than the second threshold (step S1110: Yes), the processor 301 does not generate a reflection sentence (step S1113) if the second language likelihood obtained as a result of inputting the word string Wt into the question sentence language model 414, which determines whether the spoken voice 110 is a question based on the sequence of words that constitute the question sentence, is less than the second threshold (step S1111: Yes).

This makes it possible to suppress unnecessary repetition of asking for clarification when a question is deemed plausible by the question language model 414, and to facilitate smooth dialogue.

(13) In the speech dialogue device 300 of (11) above, in the process of generating a reflection sentence (step S1005), if the first language likelihood is equal to or greater than the second threshold, the processor 301 inputs the word string Wt into the question sentence language model 414, which determines whether the spoken speech 110 is a question based on the sequence of words that make up the question sentence, and if the second language likelihood obtained as a result of inputting the word string Wt is equal to or greater than the second threshold (step S1111: No), generates a reflection sentence that asks for the spoken speech 110 again (step S1112).

As a result, even if the spoken voice 110 is recognized as a question, if the question language model 414 determines that it is not plausible as a question, a repeat question of the repeat pattern P4 can be generated, and the entire question can be asked again.

The present invention is not limited to the above-described embodiments, but includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to having all of the configurations described. Furthermore, a portion of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, other configurations may be added, deleted, or replaced with part of the configuration of each embodiment.

Furthermore, each of the configurations, functions, processing units, processing means, etc. described above may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having the processor 301 interpret and execute a program that realizes each function.

Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC (Integrated Circuit) card, an SD card, or a DVD (Digital Versatile Disc).

In addition, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines necessary for implementation. In reality, it is safe to assume that almost all components are interconnected.

W1 to Wn Word WL Word lattice Wt Word string 101 User 102 Communication robot 103 Smartphone 104 Information processing device 110 Speech 200 Speech dialogue system 201 Server 202 Network 300 Speech dialogue device 301 Processor 302 Storage device 401 Speech recognition unit 402 Language likelihood acquisition unit 403 Reliability acquisition unit 404 Reflection determination unit 405 Dialogue control unit 406 Reflection sentence generation unit 407 Dialogue history management unit 408 Speech synthesis unit 411 Speech recognition model 412 Standard language model 413 Domain-specific language model 414 Question sentence language model 415 Dialogue context language model 416 Dialogue knowledge DB

Claims (15)

A speech dialogue apparatus having a processor that executes a program and a storage device that stores the program,
The processor,
An acquisition process for acquiring the reliability of each word constituting a word string related to the spoken voice;
a generation process for generating a reply sentence for asking back to a source of the spoken voice which is the first half or the second half of the word string when the word string contains a word whose reliability is equal to or greater than a first threshold value and a word whose reliability is less than the first threshold value in the first half or the second half of the word string and which does not contain a word whose reliability is less than the first threshold value;
A voice dialogue device which executes the above. A speech dialogue apparatus having a processor that executes a program and a storage device that stores the program,
The processor,
An acquisition process for acquiring the reliability of each word constituting a word string related to the spoken voice;
a generation process for generating a reply sentence for asking the speaker of the spoken voice about the words in the word string that are equal to or greater than the first threshold value when the word string includes a word whose reliability is equal to or greater than the first threshold value and a plurality of words whose reliability is less than the first threshold value in both the first and second halves of the word string;
A voice dialogue device which executes the above. A speech dialogue apparatus having a processor that executes a program and a storage device that stores the program,
The processor,
An acquisition process for acquiring the reliability of each word constituting a word string related to the spoken voice;
an estimation process for estimating what kind of word the one word or the two consecutive words are based on a dialogue context language model that has been trained on statistics of a word and a word group that appears around the word and statistics of phrase sequences when the word string includes a word whose reliability is equal to or greater than a first threshold value and a word whose reliability is less than the first threshold value or two consecutive words whose reliability is less than the first threshold value;
a generation process of generating a reflection sentence for asking the source of the uttered voice back according to an estimation result of the estimation process;
A voice dialogue device which executes the above. 4. The speech dialogue device according to claim 3,
In the generation process, when the estimation by the estimation process is successful, the processor generates a reflection sentence that includes the estimated word and confirms the spoken voice.
1. A voice dialogue device comprising: 4. The speech dialogue device according to claim 3,
In the generation process, when the estimation process fails, the processor generates a reflection sentence for requesting the speech voice again.
1. A voice dialogue device comprising: A speech dialogue apparatus having a processor that executes a program and a storage device that stores the program,
The processor,
An acquisition process for acquiring the reliability of each word constituting a word string related to the spoken voice and the reliability of the word string;
a generation process for generating a reflection sentence for asking a speaker of the utterance of the spoken voice back based on the reliability of each word constituting the word string when the reliability of the word string is less than a first threshold value;
A voice dialogue device which executes the above. A speech dialogue apparatus having a processor that executes a program and a storage device that stores the program,
The processor,
An acquisition process for acquiring a reliability of a word string related to a speech sound and a plurality of language likelihoods obtained as a result of inputting the word string into each of a plurality of language models;
a generation process for generating a reflection sentence for asking a speaker of the spoken voice back based on a plurality of linguistic likelihoods acquired by the acquisition process when the reliability of the word string acquired by the acquisition process is equal to or greater than a first threshold value;
A voice dialogue device which executes the above. The speech dialogue device according to claim 7,
In the generation process, when a first language likelihood obtained as a result of inputting the word sequence into a first language model which is a statistical model of word sequences obtained from texts in a plurality of fields among the plurality of language models is less than a second threshold value, the processor generates a reflection sentence for requesting the spoken voice again.
1. A voice dialogue device comprising: 9. The speech dialogue device according to claim 8,
In the generation process, when the first language likelihood is equal to or greater than the second threshold value, the processor does not generate the reflection sentence if a second language likelihood obtained as a result of inputting the word sequence into a second language model that determines whether the spoken voice is a question based on a sequence of words constituting a question sentence is less than the second threshold value.
1. A voice dialogue device comprising: 9. The speech dialogue device according to claim 8,
In the generation process, when the first language likelihood is equal to or greater than the second threshold value, the processor generates a reflection sentence for asking the spoken voice to repeat the spoken voice if a second language likelihood obtained as a result of inputting the word sequence into a second language model that determines whether the spoken voice is a question based on a sequence of words constituting a question sentence is equal to or greater than the second threshold value.
1. A voice dialogue device comprising: A voice dialogue method executed by a voice dialogue device having a processor that executes a program and a storage device that stores the program, comprising:
The speech dialogue method includes:
The processor,
An acquisition process for acquiring the reliability of each word constituting a word string related to the spoken voice;
a generation process for generating a reply sentence for asking back to a source of the spoken voice which is the first half or the second half of the word string when the word string contains a word whose reliability is equal to or greater than a first threshold value and a word whose reliability is less than the first threshold value in the first half or the second half of the word string and which does not contain a word whose reliability is less than the first threshold value;
A speech dialogue method comprising the steps of: A voice dialogue method executed by a voice dialogue device having a processor that executes a program and a storage device that stores the program, comprising:
The speech dialogue method includes:
The processor,
An acquisition process for acquiring the reliability of each word constituting a word string related to the spoken voice;
a generation process for generating a reply sentence for asking the speaker of the spoken voice about the words in the word string that are equal to or greater than the first threshold value when the word string includes a word whose reliability is equal to or greater than the first threshold value and a plurality of words whose reliability is less than the first threshold value in both the first and second halves of the word string;
A speech dialogue method comprising the steps of: A voice dialogue method executed by a voice dialogue device having a processor that executes a program and a storage device that stores the program, comprising:
The speech dialogue method includes:
The processor,
An acquisition process for acquiring the reliability of each word constituting a word string related to the spoken voice;
an estimation process for estimating what kind of word the one word or the two consecutive words are based on a dialogue context language model that has been trained on statistics of a word and a word group that appears around the word and statistics of phrase sequences when the word string includes a word whose reliability is equal to or greater than a first threshold value and a word whose reliability is less than the first threshold value or two consecutive words whose reliability is less than the first threshold value;
A speech dialogue method comprising the steps of: The processor:
An acquisition process for acquiring the reliability of each word constituting a word string related to the spoken voice;
a generation process for generating a reply sentence for asking back to a source of the spoken voice which is the first half or the second half of the word string when the word string contains a word whose reliability is equal to or greater than a first threshold value and a word whose reliability is less than the first threshold value in the first half or the second half of the word string and which does not contain a word whose reliability is less than the first threshold value;
A speech dialogue program comprising: The processor:
An acquisition process for acquiring the reliability of each word constituting a word string related to the spoken voice;
a generation process for generating a reply sentence for asking the speaker of the spoken voice about the words in the word string that are equal to or greater than the first threshold value when the word string includes a word whose reliability is equal to or greater than the first threshold value and a plurality of words whose reliability is less than the first threshold value in both the first and second halves of the word string ;
A speech dialogue program comprising:

Images