JP6832503B2 - Information presentation method, information presentation program and information presentation system - Google Patents

Description

The present disclosure relates to an information presentation method, an information presentation program, and an information presentation system that translates voices related to a dialogue by a plurality of speakers and presents auxiliary information for assisting the dialogue.

Conventionally, in a dialogue support device that supports dialogue by a plurality of speakers, a technique for presenting information for supplementing knowledge according to the content of dialogue to support dialogue (see, for example, Patent Document 1) has been known. ..

Japanese Unexamined Patent Publication No. 2013-73355

However, in Patent Document 1, further improvement is required.

The information presentation method according to one aspect of the present disclosure is an information presentation method in an information presentation system, in which dialogue text is generated by voice recognition of voices related to dialogue by a plurality of speakers, and the dialogue text is translated. By generating the dialogue translation text, the dialogue translation voice is generated by synthesizing the dialogue translation text, and it is determined whether or not there is auxiliary information for assisting the dialogue based on the dialogue text. When the auxiliary information exists, at least one of the plurality of speakers indicates that the auxiliary information exists according to the usage status of the information presentation system. Present to people.

According to the above aspect, further improvement can be realized.

It is a block diagram which shows the structure of the translation terminal of the information presentation system of this embodiment. It is a block diagram which shows the structure of the translation server of the information presentation system of this embodiment. It is a figure which shows an example of the information which is stored in the auxiliary explanatory text storage part. It is a figure which shows an example of the information which is stored in the auxiliary information storage part. It is a flowchart for demonstrating the operation of the information presentation system in this Embodiment. It is a flowchart for demonstrating the operation of the customer utterance processing executed in a translation terminal and a translation server. It is a 1st flowchart for demonstrating operation of a clerk utterance process executed in a translation terminal and a translation server. It is a 2nd flowchart for demonstrating operation of a clerk utterance process executed in a translation terminal and a translation server. It is a figure which shows an example of the screen which is displayed on the translation terminal at the time of presenting the existence of auxiliary information.

(Background to the invention of one aspect of the present disclosure)
First, one aspect of the viewpoint of the present disclosure will be described.

The above-mentioned Patent Document 1 discloses a technique for displaying supplementary information based on a speech act tag indicating a speech act and a speaker tag indicating a speaker. As the speech act tag, "greeting", "response", "question" and the like are shown (see FIG. 8 of Patent Document 1). Further, as the timing of displaying the supplementary information, a method of determining the necessity of presenting the information or the timing of presenting the information is shown depending on the combination of the speaker and the utterance act (FIG. 10 of Patent Document 1).

As described above, in Patent Document 1, it is possible to control the necessity of presenting information and the timing of presenting information based on the utterance act and the speaker. However, the necessity of presenting information and the timing of presenting information are determined by the dialogue support device, not by the customer or the customer service person.

In this way, when the dialogue support device determines the necessity of presenting information and the timing of presenting information, it is not possible to respond to the customer's request to omit the presentation of supplementary information because the customer is in a hurry. Further, in the conventional dialogue support device, the supplementary information is notified even when it is not necessary to present the supplementary information because the customer service person who handles the dialogue support device has sufficient knowledge.

Further, Patent Document 1 discloses a method of sounding a sound, blinking a display, or vibrating a device as a method of notifying a user of the presentation of supplementary information when presenting supplementary information. However, these methods may interfere with the customer service business, which is mainly a dialogue, and cannot inform that there is supplementary information in a natural form in the dialogue.

In order to solve the above problems, the information presentation method according to one aspect of the present disclosure is an information presentation method in an information presentation system, and a dialogue text is produced by recognizing voices related to dialogues by a plurality of speakers. Auxiliary information for generating the dialogue translation text by generating and translating the dialogue text, generating the dialogue translation voice by synthesizing the dialogue translation text by voice synthesis, and assisting the dialogue based on the dialogue text. Is present, and if the auxiliary information is present according to the usage status of the information presentation system of at least one of the plurality of speakers, the presence of the auxiliary information is described. Present to at least one of multiple speakers.

According to this configuration, the dialogue text is generated by voice recognition of the voice related to the dialogue by a plurality of speakers. The dialogue translation text is generated by translating the dialogue text. Dialogue-translated speech is generated by synthesizing the dialogue-translated text. Based on the dialogue text, it is determined whether or not there is auxiliary information to assist the dialogue, and if there is auxiliary information, it depends on the usage status of the information presentation system of at least one of the plurality of speakers. , The presence of auxiliary information is presented to at least one of a plurality of speakers.

Therefore, since it is presented that the auxiliary information exists before the auxiliary information for assisting the dialogue is presented, the auxiliary information is not necessarily presented, and the speaker needs to present the auxiliary information. Auxiliary information can be presented only when there is, and smooth dialogue can be realized.

Further, in the above-mentioned information presenting method, the auxiliary information may be presented by receiving an instruction to present the auxiliary information from at least one of the plurality of speakers and following the received instruction to present the auxiliary information.

According to this configuration, the presentation instruction of the auxiliary information is received from at least one of the plurality of speakers, and the auxiliary information is presented according to the received presentation instruction.

Therefore, the auxiliary information can be presented to the speaker who needs to present the auxiliary information.

Further, in the above information presentation method, the dialogue translation voice is generated by synthesizing the dialogue translation text by voice waveform synthesis, the explanation text explaining the auxiliary information is specified from the dialogue text, and the explanation text is voice-analyzed. Auxiliary information voice may be generated by synthesizing.

According to this configuration, the dialogue-translated speech is generated by synthesizing the speech waveform of the dialogue-translated text. Explanatory text describing the auxiliary information is identified from the dialogue text. Auxiliary information speech is generated by voice analysis and synthesis of the explanatory text.

Therefore, since the auxiliary information voice is generated by voice analysis and synthesis of the explanatory text explaining the auxiliary information, the auxiliary information can be presented by voice.

Further, in the above information presentation method, the fundamental frequency of the auxiliary information voice may be different from the fundamental frequency of the dialogue translation voice.

According to this configuration, the fundamental frequency of the auxiliary information voice is different from the fundamental frequency of the interactive translation voice, so that the speaker can use the translated voice of the utterance and the voice of the auxiliary information presented by the information presentation system. Can be distinguished.

Further, in the above information presentation method, the first voice of the first speaker asking a question in the first language is acquired, and the question is answered in a second language different from the first language. Acquires the second voice of the second speaker, generates the first dialogue text by recognizing the first voice, and translates the first dialogue text into the second language. As a result, the first dialogue translation text is generated, the first dialogue translation text is generated by voice synthesis of the first dialogue translation text, and the second dialogue is recognized by voice recognition of the second voice. A text is generated, the second dialogue text is translated into the first language to generate a second dialogue translation text, and the second dialogue translation text is voice-synthesized to perform a second dialogue translation. A voice is generated, and it is determined whether or not the auxiliary information exists based on at least one of the first dialogue text and the second dialogue text, and if the auxiliary information exists, the second The existence of the auxiliary information may be presented to at least one of the first speaker and the second speaker according to the usage status of the information presentation system of the speaker.

According to this configuration, the first voice of the first speaker asking the question in the first language is acquired, and the question is answered in the second language different from the first language. The speaker's second voice is acquired. The first dialogue text is generated by recognizing the first voice. The first dialogue translation text is generated by translating the first dialogue text into the second language. The first dialogue translation voice is generated by voice synthesis of the first dialogue translation text. The second dialogue text is generated by recognizing the second voice. A second dialogue translation text is generated by translating the second dialogue text into the first language. The second dialogue translation voice is generated by voice synthesis of the second dialogue translation text. Whether or not auxiliary information exists is determined based on at least one of the first dialogue text and the second dialogue text. When the auxiliary information is present, the existence of the auxiliary information is presented to at least one of the first speaker and the second speaker, depending on the usage status of the information presentation system of the second speaker.

Therefore, when the first speaker asks a question in the first language and the second speaker answers in the second language, the assistance is provided only when the second speaker requires the presentation of auxiliary information. Information can be presented and smooth dialogue can be realized.

Further, in the above information presentation method, the usage status includes the number of times of translation into the first language, and if the auxiliary information is present, whether or not the number of times of translation into the first language is greater than the predetermined number of times. If it is determined that the number of times of translation into the first language is greater than the predetermined number of times, it is not necessary to present the existence of the auxiliary information to the second speaker.

According to this configuration, the usage status includes the number of translations into the first language. If the auxiliary information is present, it is determined whether or not the number of translations into the first language is greater than the predetermined number. If it is determined that the number of translations into the first language is greater than the predetermined number, the presence of auxiliary information is not presented to the second speaker.

Therefore, if the number of translations into the first language is greater than the predetermined number, it can be presumed that the auxiliary information is known to the second speaker, and it is necessary to show that the auxiliary information exists. Therefore, the process of presenting unnecessary information can be omitted.

Further, in the above information presentation method, the usage status includes the usage time from the start of the second speaker's use of the information presentation system to the present, and when the auxiliary information exists, the usage When it is determined whether or not the time is longer than the predetermined time and it is determined that the usage time is longer than the predetermined time, it is not necessary to present the existence of the auxiliary information to the second speaker.

According to this configuration, the usage status includes the usage time from the start of use of the information presentation system by the second speaker to the present. If the auxiliary information exists, it is determined whether or not the usage time is longer than the predetermined time. If it is determined that the usage time is longer than the predetermined time, the existence of the auxiliary information is not presented to the second speaker.

Therefore, when the usage time of the information presentation system by the second speaker is longer than the predetermined time, it can be estimated that the auxiliary information is known information to the second speaker, so that the auxiliary information exists. It is not necessary to present the information, and the process of presenting unnecessary information can be omitted.

Further, in the above information presenting method, the auxiliary information includes information for explaining the explanation target, the usage status includes the number of times the auxiliary information is presented corresponding to the same explanation target, and the auxiliary information exists. In that case, it is determined whether or not the number of presentations is greater than the predetermined number of times, and if it is determined that the number of presentations is greater than the predetermined number of times, it is not necessary to present the existence of the auxiliary information to the second speaker. May be good.

According to this configuration, the auxiliary information includes information for explaining the explanation target. The usage status includes the number of times the auxiliary information corresponding to the same explanation target is presented. When the auxiliary information exists, it is determined whether or not the number of presentations is more than the predetermined number. If it is determined that the number of presentations is greater than the predetermined number, the existence of auxiliary information is not presented to the second speaker.

Therefore, when the number of times the auxiliary information corresponding to the same explanation target is presented is larger than the predetermined number of times, it can be estimated that the auxiliary information is known to the second speaker, so that the auxiliary information exists. It is not necessary to present the information, and the process of presenting unnecessary information can be omitted.

The information presentation program according to another aspect of the present disclosure comprises a computer, a dialogue text generator that generates dialogue text by recognizing voices related to dialogues by a plurality of speakers, and a dialogue text generator that translates the dialogue texts. A dialogue translation text generation unit that generates a dialogue translation text, a dialogue translation voice generation unit that generates a dialogue translation voice by synthesizing the dialogue translation text, and an assistance for assisting the dialogue based on the dialogue text. An auxiliary information determination unit that determines whether or not information exists, and an information presentation system that includes the computer as a component of at least one of the plurality of speakers when it is determined that the auxiliary information exists. Depending on the usage situation, it functions as a transmission unit that transmits the auxiliary information and the interactively translated voice in order to present to at least one of the plurality of speakers that the auxiliary information exists.

According to this configuration, the dialogue text is generated by voice recognition of the voice related to the dialogue by a plurality of speakers. The dialogue translation text is generated by translating the dialogue text. Dialogue-translated speech is generated by synthesizing the dialogue-translated text. Based on the dialogue text, it is determined whether or not auxiliary information for assisting the dialogue exists, and if the auxiliary information exists, an information presentation system including a computer of at least one of a plurality of speakers as a component. Auxiliary information and interactively translated audio are transmitted to present to at least one of a plurality of speakers that the auxiliary information exists, depending on the usage status of the computer.

Therefore, since it is presented that the auxiliary information exists before the auxiliary information for assisting the dialogue is presented, the auxiliary information is not necessarily presented, and the speaker needs to present the auxiliary information. Auxiliary information can be presented only when there is, and smooth dialogue can be realized.

The information presentation system according to another aspect of the present disclosure includes a terminal and a server communicably connected to the terminal, and the terminal includes a voice acquisition unit that acquires voice related to a dialogue by a plurality of speakers. The server includes a transmission unit that transmits the acquired voice to the server, and the server generates a dialogue text by recognizing the voice related to the dialogue with the reception unit that receives the voice. The dialogue translation text generation unit that generates the dialogue translation text by translating the dialogue text, the dialogue translation voice generation unit that generates the dialogue translation voice by synthesizing the dialogue translation text, and the dialogue text. Auxiliary information determination unit that determines whether or not auxiliary information for assisting the dialogue exists based on the above, and if it is determined that the auxiliary information exists, at least one of the plurality of speakers. The terminal includes a transmitting unit that transmits the auxiliary information and the interactively translated voice to the terminal according to the usage status of the information presentation system, and the terminal is a receiving unit that receives the auxiliary information and the interactively translated voice. A voice output unit that outputs the interactively translated voice and a presentation unit that presents the existence of the auxiliary information to at least one of the plurality of speakers are provided.

According to this configuration, voices related to dialogues by a plurality of speakers are acquired at the terminal. The acquired voice is sent to the server. Audio is received at the server. Dialogue text is generated by recognizing the voice related to the dialogue. The dialogue translation text is generated by translating the dialogue text. Dialogue-translated speech is generated by synthesizing the dialogue-translated text. Based on the dialogue text, it is determined whether or not auxiliary information exists to assist the dialogue. When it is determined that the auxiliary information exists, the auxiliary information and the interactively translated voice are transmitted to the terminal according to the usage status of the information presentation system of at least one of the plurality of speakers. Auxiliary information and interactively translated voice are received at the terminal. Dialogue translation audio is output. The presence of ancillary information is presented to at least one of the speakers.

Therefore, since it is presented that the auxiliary information exists before the auxiliary information for assisting the dialogue is presented, the auxiliary information is not necessarily presented, and the speaker needs to present the auxiliary information. Auxiliary information can be presented only when there is, and smooth dialogue can be realized.

(Embodiment)
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same reference numerals are used for the same components. In addition, the following embodiments are examples that embody the present disclosure, and do not limit the technical scope of the present disclosure.

FIG. 1 is a block diagram showing a configuration of a translation terminal of the information presentation system of the present embodiment.

The information presentation system translates voices related to dialogues by a plurality of speakers and presents auxiliary information for assisting the dialogues. The information presentation system includes a translation terminal 100 and a translation server 200. The translation terminal 100 and the translation server 200 are connected to each other so as to be able to communicate with each other via a network.

The translation terminal 100 may be, for example, a tablet computer, a smartphone, a mobile phone, or a notebook PC (personal computer). Further, the translation terminal 100 may be a dedicated terminal for translating the voice related to the dialogue by a plurality of speakers.

As shown in FIG. 1, the translation terminal 100 of the information presentation system of the present embodiment has a voice input unit 101, a format conversion unit 102, a response control unit 103, a communication unit 104, a voice output unit 105, and a response holding unit 106. , Auxiliary information presentation setting unit 107, auxiliary information presence / absence determination unit 108, auxiliary information presentation determination unit 109, auxiliary information presentation unit 110, video output unit 111, and user instruction input unit 112.

The voice input unit 101 collects the voices of a customer who speaks a language other than Japanese, such as a foreigner visiting Japan, or a clerk who speaks Japanese, and generates an analog voice signal. The voice input unit 101 acquires voice signals uttered by a plurality of talking speakers.

The format conversion unit 102 converts the analog audio signal generated by the audio input unit 101 into digital data by, for example, pulse code modulation (PCM) to generate a PCM audio signal.

The response control unit 103 determines the speaker of the spoken voice input from the voice input unit 101. Further, the response control unit 103 transmits to the translation server 200 the result of translating the PCM voice signal generated by the format conversion unit 102 and the customer's utterance held by the response holding unit 106. Further, the response control unit 103 receives the translation result and the content information related to the auxiliary information from the translation server 200. Further, the response control unit 103 outputs the translation result received from the translation server 200 by voice through the voice output unit 105. In addition, the response control unit 103 acquires auxiliary information and controls the presentation of the acquired auxiliary information in accordance with the instructions of the clerk input from the user instruction input unit 112.

The communication unit 104 performs data communication with the translation server 200 via a communication line such as the Internet. The communication unit 104 transmits voices related to dialogues by a plurality of speakers to the translation server 200. The communication unit 104 receives from the translation server 200 the result of translating the customer's utterance, the result of translating the clerk's utterance, and the auxiliary information corresponding to the clerk's utterance and the customer's utterance.

The voice output unit 105 voice-outputs the result of translating the customer's utterance received from the translation server 200, the result of translating the clerk's utterance, and the auxiliary information corresponding to the clerk's utterance and the customer's utterance.

The response holding unit 106 is, for example, a non-volatile memory, and holds the result of translating the customer's utterance received from the translation server 200.

The auxiliary information presentation setting unit 107 sets whether or not to present the auxiliary information, and stores the presentation necessity information indicating whether or not to present the set auxiliary information. The clerk can set in advance whether or not to present the auxiliary information through a user interface such as the auxiliary information presentation setting screen. That is, by setting the presentation necessity information indicating that the auxiliary information presentation setting unit 107 does not present the auxiliary information, the clerk can carry out the customer service business using the translation function without interrupting the auxiliary information. ..

The auxiliary information presence / absence determination unit 108 determines whether or not auxiliary information corresponding to the utterance of the clerk and the utterance of the customer exists.

The auxiliary information presentation determination unit 109 presents auxiliary information based on the setting contents of the auxiliary information presentation setting unit 107, the determination result of the auxiliary information presence / absence determination unit 108, and the user instruction content input from the user instruction input unit 112. Judge whether or not.

The auxiliary information presenting unit 110 presents auxiliary information corresponding to the utterance of the clerk and the utterance of the customer via the audio output unit 105 and the video output unit 111.

The video output unit 111 displays auxiliary information (content information) corresponding to the utterance of the clerk and the utterance of the customer.

The user instruction input unit 112 provides a user interface for instructing the presentation of auxiliary information.

FIG. 2 is a block diagram showing a configuration of a translation server of the information presentation system of the present embodiment.

As shown in FIG. 2, the translation server 200 of the information presentation system of the present embodiment includes a communication unit 201, a control unit 202, a voice recognition unit 203, a translation unit 204, an intention understanding unit 205, and an auxiliary explanatory text storage unit 206. , Auxiliary information storage unit 207, voice waveform synthesis unit 208, and voice analysis synthesis unit 209.

The communication unit 201 performs data communication with the translation terminal 100 via a communication line such as the Internet.

The control unit 202 controls the voice recognition unit 203, the translation unit 204, the intention understanding unit 205, the voice waveform synthesis unit 208, and the voice analysis synthesis unit 209 to obtain the customer's voice or the clerk's voice received from the translation terminal 100. Translate and synthesize the translation result by voice. Further, the control unit 202 understands the intention of the customer's utterance and the clerk's utterance, and determines the auxiliary explanation and the auxiliary information according to the utterance intention.

The voice recognition unit 203 recognizes the voice of the customer or the voice of the clerk and generates the dialogue text. The voice recognition unit 203 generates a dialogue text by recognizing the voice related to the dialogue.

The translation unit 204 generates the dialogue translation text by translating the dialogue text generated by the speech recognition unit 203.

The intention understanding unit 205 determines whether or not auxiliary information exists based on the dialogue text. The intention understanding unit 205 extracts the entity to be explained from the dialogue text of the customer or the clerk, classifies the intention of the utterance from the dialogue text, and determines whether or not the entity and the auxiliary information corresponding to the intention exist. As the intention of the utterance, for example, it indicates whether the utterance is related to the way to the entity or the information of the entity. The intention understanding unit 205 identifies the explanatory text explaining the auxiliary information from the dialogue text.

The auxiliary explanatory text storage unit 206 is, for example, a non-volatile memory, and stores an explanatory text output by the translation terminal 100 by voice when presenting auxiliary information.

FIG. 3 is a diagram showing an example of information stored in the auxiliary explanatory text storage unit 206. The explanatory text ID 301 is an identifier of the explanatory text stored in the auxiliary explanatory text storage unit 206. The auxiliary information explanatory text 302 is an explanatory text indicating an explanatory text for explaining the auxiliary information. The auxiliary explanatory text storage unit 206 stores the explanatory text ID 301 and the auxiliary information explanatory text 302 in association with each other.

The auxiliary information storage unit 207 is, for example, a non-volatile memory, and stores the entity extracted from the dialogue text of the customer or the clerk, the utterance intention of the dialogue text, and the explanatory text corresponding to the entity and the utterance intention in association with each other. ..

FIG. 4 is a diagram showing an example of information stored in the auxiliary information storage unit 207. Entity 401 is an explanatory object included in the dialogue text of the customer or the clerk. The content ID 402 is an identifier of the entity 401. The intention ID 403 is an identifier of the utterance intention of the customer or the clerk. For example, the intent ID "0001" corresponds to the way to the entity, and the intent ID "0002" corresponds to the information of the entity. The descriptive text ID 404 is an identifier of the descriptive text used to describe the entity. The content information is an address indicating a storage destination of image information used for explaining an entity.

The voice waveform synthesis unit 208 performs waveform synthesis type voice synthesis. Waveform synthesis type speech synthesis is a method of synthesizing speech by concatenating pre-recorded sound fragments. Waveform-synthesized speech synthesis is generally of higher quality than analytical-synthesized speech synthesis and has characteristics similar to human voice. The voice waveform synthesis unit 208 generates the dialogue translation voice by synthesizing the dialogue translation text obtained by translating the dialogue text.

The speech analysis synthesis unit 209 performs analysis synthesis type speech synthesis. Analytical synthesis type speech synthesis is a method of synthesizing speech by adjusting parameters such as base frequency or timbre. The voice generated by the analytical synthesis type voice synthesis has the characteristic that it becomes a voice that can be heard like a robot, so that it is not mistaken for a human voice. The voice analysis synthesis unit 209 generates the auxiliary information voice by voice analysis and synthesis of the explanatory text explaining the auxiliary information. The fundamental frequency of the auxiliary information voice is different from the fundamental frequency of the interactive translation voice.

When it is determined that the auxiliary information exists, the communication unit 201 transmits the auxiliary information and the interactive translation voice to the translation terminal 100 in order to present the existence of the auxiliary information to at least one of the plurality of speakers. .. When the response control unit 103 of the translation terminal 100 receives the auxiliary information from the translation server 200, the response control unit 103 presents to at least one of the plurality of speakers that the auxiliary information exists. The user instruction input unit 112 receives an instruction to present auxiliary information from at least one of a plurality of speakers. The auxiliary information presentation unit 110 presents auxiliary information according to the received presentation instruction.

As described above, the communication unit 201 acquires the first voice of the first speaker (customer) asking a question in the first language, and responds to the question in a second language different from the first language. Acquire the second voice of the second speaker (clerk) who answers. The voice recognition unit 203 generates the first dialogue text by recognizing the first voice. The translation unit 204 generates the first dialogue translation text by translating the first dialogue text into the second language. The voice waveform synthesis unit 208 generates the first dialogue-translated voice by synthesizing the first dialogue-translated text. The voice recognition unit 203 generates the second dialogue text by recognizing the second voice. The translation unit 204 generates the second dialogue translation text by translating the second dialogue text into the first language. The voice waveform synthesis unit 208 generates a second dialogue translation voice by synthesizing the second dialogue translation text. The intention understanding unit 205 determines whether or not auxiliary information exists based on at least one of the first dialogue text and the second dialogue text. When the auxiliary information exists, the auxiliary information presentation determination unit 109 presents the existence of the auxiliary information to at least one of the first speaker (customer) and the second speaker (clerk).

FIG. 5 is a flowchart for explaining the operation of the information presentation system according to the present embodiment. In the present embodiment, an example in which a customer and a clerk interact with each other will be described. The language spoken by the customer is different from the language spoken by the clerk. The information presentation system translates the customer's utterance and then outputs the voice, and translates the clerk's utterance and then outputs the voice. The operation of the information presentation system shown in FIG. 5 is started after the customer speaks.

First, the voice input unit 101 of the translation terminal 100 acquires the spoken voice and generates a voice signal (step S1). The customer speaks to the translation terminal 100. The language spoken by the customer and the language spoken by the clerk are preset or selected by the customer or clerk prior to utterance. Further, the format conversion unit 102 of the translation terminal 100 converts the voice signal generated by the voice input unit 101 into digital data by, for example, pulse code modulation (PCM), and generates a PCM voice signal.

Next, the response control unit 103 of the translation terminal 100 determines whether or not the acquired utterance voice is the utterance voice of the customer (step S2). For example, the translation terminal 100 may include a customer voice input start button for accepting voice input by the customer and a clerk voice input start button for accepting voice input by the clerk. The user instruction input unit 112 determines which of the customer voice input start button and the clerk voice input start button is pressed. The response control unit 103 determines that the utterance voice acquired after the customer voice input start button is pressed is the utterance voice from the customer, and determines that the utterance voice acquired after the clerk voice input start button is pressed. Is determined to be a voice spoken by a clerk.

The translation terminal 100 may include a customer voice input device that accepts voice input by the customer and a clerk voice input device that accepts voice input by the clerk. Further, the response control unit 103 may determine whether the voice is spoken by the customer or the clerk by discriminating the language of the input voice. Further, any determination method may be used as the determination method for determining whether or not the acquired utterance voice is the utterance voice of the customer.

When it is determined that the uttered voice is the uttered voice of the customer (YES in step S2), the customer utterance process described later is performed (step S3). Further, when it is determined that the utterance voice is not the utterance voice of the customer, that is, when it is determined that the utterance voice is the utterance voice of the clerk (NO in step S2), the clerk speech processing described later is performed (step). S4).

FIG. 6 is a flowchart for explaining the operation of the customer utterance processing executed by the translation terminal 100 and the translation server 200. The guest utterance process of FIG. 6 is started when the response control unit 103 of the translation terminal 100 determines that the utterance voice is the customer's utterance voice.

First, the communication unit 104 of the translation terminal 100 transmits the customer utterance information including the PCM voice signal of the customer utterance, the identifier indicating the customer utterance, the language type of the customer utterance, and the language type of the clerk utterance to the translation server 200. (Step S11).

Next, the communication unit 201 of the translation server 200 receives the customer utterance information transmitted by the translation terminal 100 (step S12).

Next, the voice recognition unit 203 of the translation server 200 uses the received PCM voice signal of the customer utterance and the language type of the customer utterance to voice-recognize the PCM voice signal and generate the dialogue text of the customer utterance (step S13). ).

Next, the translation unit 204 of the translation server 200 translates the dialogue text of the customer's language into the language of the clerk by using the generated dialogue text of the customer's utterance, the language type of the customer's utterance, and the language type of the clerk's speech. A dialogue translation text of the guest utterance is generated (step S14).

Next, the voice waveform synthesis unit 208 of the translation server 200 uses the generated dialogue translation text of the customer utterance and the language type of the clerk utterance to synthesize the dialogue translation text of the customer utterance, and translates and synthesizes the customer utterance. Is generated (step S15).

Next, the communication unit 201 of the translation server 200 translates the customer utterance translation information including the dialogue translation text of the customer utterance, the translated synthetic voice of the customer utterance, the identifier indicating the customer utterance, and the language type of the clerk utterance. It is transmitted to 100 (step S16).

Next, the communication unit 104 of the translation terminal 100 receives the customer utterance translation information transmitted by the translation server 200 (step S17).

Next, the response control unit 103 of the translation terminal 100 stores the received dialogue translation text of the customer utterance and the language type of the clerk utterance in the response holding unit 106 (step S18).

Next, the voice output unit 105 of the translation terminal 100 outputs the translated synthetic voice of the received customer utterance by voice (step S19). The video output unit 111 may display the character on the display screen when the translated synthetic voice of the customer utterance is output as voice, and control the display as if the character on the display screen is speaking.

FIG. 7 is a first flowchart for explaining the operation of the clerk utterance process executed by the translation terminal 100 and the translation server 200, and FIG. 8 is a clerk utterance process executed by the translation terminal 100 and the translation server 200. It is the 2nd flowchart for demonstrating the operation of. The clerk utterance process of FIG. 7 is started when the response control unit 103 of the translation terminal 100 determines that the utterance voice is the clerk's utterance voice.

First, the response control unit 103 of the translation terminal 100 acquires the interactive translation text of the customer utterance held in the response holding unit 106 (step S21).

Next, the communication unit 104 of the translation terminal 100 uses the PCM voice signal of the clerk's utterance, the identifier indicating that the clerk's utterance is spoken, the language type of the clerk's utterance, the language type of the customer's utterance, and the customer utterance acquired from the response holding unit 106. The clerk utterance information including the dialogue translation text of the above is transmitted to the translation server 200 (step S22).

Next, the communication unit 201 of the translation server 200 receives the clerk utterance information transmitted by the translation terminal 100 (step S23).

Next, the voice recognition unit 203 of the translation server 200 uses the received PCM voice signal of the clerk utterance and the language type of the clerk utterance to voice-recognize the PCM voice signal and generate the dialogue text of the clerk utterance (step S24). ).

Next, the translation unit 204 of the translation server 200 translates the dialogue text of the clerk's language into the customer's language by using the generated dialogue text of the clerk's utterance, the language type of the clerk's utterance, and the language type of the customer's utterance. A dialogue translation text of a clerk's utterance is generated (step S25).

Next, the voice waveform synthesis unit 208 of the translation server 200 uses the generated dialogue translation text of the clerk's utterance and the language type of the customer's utterance to synthesize the dialogue translation text of the clerk's utterance, and the translation synthesis voice of the clerk's utterance. Is generated (step S26).

Next, the intention understanding unit 205 of the translation server 200 understands the intentions of the customer utterance and the clerk utterance by using the dialogue translation text of the customer utterance and the dialogue text of the clerk utterance (step S27). That is, the intention understanding unit 205 extracts an entity from the dialogue translation text of the customer utterance and the dialogue text of the clerk utterance, and the utterance asks the direction to the entity from the dialogue translation text of the customer utterance and the dialogue text of the clerk utterance. Acquires an intent ID that classifies whether the utterance is an utterance asking for information about an entity. For the extraction of entities, statistical methods such as machine learning may be used, or grammatical rules may be used. Further, as the classification method, a statistical method such as machine learning may be used, or a method of estimating from the dialogue translation text and the expressions included in the dialogue text according to a predetermined rule may be used.

Next, the intention understanding unit 205 of the translation server 200 determines whether or not the auxiliary information exists by using the acquired entity and the intention ID (step S28). The intention understanding unit 205 determines whether or not the auxiliary information exists by searching the auxiliary information storage unit 207 using the entity and the intention ID. When the auxiliary information storage unit 207 has the auxiliary information corresponding to the acquired entity and the intention ID, the intention understanding unit 205 determines that the auxiliary information exists, and the auxiliary information corresponding to the acquired entity and the intention ID is supplemented. If it does not exist in the information storage unit 207, it is determined that the auxiliary information does not exist.

Here, if it is determined that the auxiliary information does not exist (NO in step S28), the process proceeds to step S32.

On the other hand, when it is determined that the auxiliary information exists (YES in step S28), the control unit 202 of the translation server 200 acquires the explanatory text and the content information of the auxiliary information (step S29). The control unit 202 acquires the explanatory text ID corresponding to the acquired entity and the intention ID from the auxiliary information storage unit 207, and acquires the explanatory text (explanatory text) corresponding to the acquired explanatory text ID from the auxiliary explanatory text storage unit 206. To do. Further, the control unit 202 acquires the content information corresponding to the acquired entity and the intention ID from the auxiliary information storage unit 207.

The explanatory text and the content information do not always exist, and the control unit 202 may acquire only the explanatory text when there is no content information, or may acquire only the content information when there is no explanatory text. Good.

Next, the translation unit 204 of the translation server 200 translates the explanation text of the auxiliary information into the customer's language by using the explanation text of the auxiliary information, the language type of the explanation text, and the language type of the customer's utterance, and explains the auxiliary information. Generate the translated text (step S30).

Next, the speech analysis synthesis unit 209 of the translation server 200 uses the generated explanatory translation text of the auxiliary information and the language type of the guest utterance to synthesize the explanatory translation text of the auxiliary information, and translates and synthesizes the auxiliary information. Is generated (step S31).

Next, the communication unit 201 of the translation server 200 includes a dialogue translation text of the clerk's utterance, a translation of the clerk's utterance, an identifier indicating that the clerk's utterance, an explanatory translation text, a translation of auxiliary information, and a customer's utterance. The clerk utterance translation information including the language type and content information is transmitted to the translation terminal 100 (step S32). If it is determined in step S28 that the auxiliary information does not exist, the explanatory translation text, the translated synthetic voice of the auxiliary information, and the content information are not transmitted. That is, when it is determined that the auxiliary information does not exist, the communication unit 201 includes the dialogue translation text of the clerk's utterance, the translated synthetic voice of the clerk's utterance, the identifier indicating that the clerk's utterance, and the language type of the customer's utterance. The utterance translation information is transmitted to the translation terminal 100.

Next, the communication unit 104 of the translation terminal 100 receives the clerk utterance translation information transmitted by the translation server 200 (step S33).

Next, the voice output unit 105 of the translation terminal 100 outputs the translated synthetic voice of the clerk's utterance included in the clerk's utterance translation information received by the communication unit 104 by voice (step S34). The video output unit 111 may display a character on the display screen when the translated synthetic voice of the clerk's utterance is output as voice, and control the display as if the character on the display screen is speaking.

Next, the auxiliary information presence / absence determination unit 108 of the translation terminal 100 determines whether or not the clerk utterance translation information received by the communication unit 104 includes auxiliary information (explanatory translation text, translation synthesis voice of auxiliary information, and content information). (Step S35). Here, if it is determined that the auxiliary information is not included (NO in step S35), the process ends.

On the other hand, when it is determined that the auxiliary information is included (YES in step S35), whether or not the auxiliary information presentation determination unit 109 of the translation terminal 100 presents the auxiliary information from the setting information of the auxiliary information presentation setting unit 107. Is determined. Whether or not to present auxiliary information is preset in the setting information. Here, if it is determined that the auxiliary information is not presented (NO in step S36), the process ends.

On the other hand, when it is determined to present the auxiliary information (YES in step S36), the auxiliary information presentation determination unit 109 of the translation terminal 100 presents the existence of the auxiliary information to at least one of the customer and the clerk (step S37). .. By being presented that the auxiliary information is present, it is possible for the clerk or the customer to decide whether or not to present the auxiliary information. That is, when the clerk does not need to present the auxiliary information, the clerk can continue the dialogue with the customer without presenting the auxiliary information, and when the clerk needs to present the auxiliary information, the clerk needs to present the auxiliary information. The clerk can interrupt the dialogue with the customer and have him / her present the auxiliary information.

FIG. 9 is a diagram showing an example of a screen displayed on the translation terminal when presenting the existence of auxiliary information. For example, the auxiliary information presentation determination unit 109 indicates that the auxiliary information exists by displaying the mark 12 indicating that the auxiliary information exists on the screen 11 of the translation terminal 100.

The auxiliary information presentation determination unit 109 may indicate that the auxiliary information exists by outputting a voice notification sound indicating that the auxiliary information exists.

Next, the auxiliary information presentation determination unit 109 of the translation terminal 100 determines whether or not there is a user instruction for presenting auxiliary information from the user instruction input unit 112 (step S38). Here, if it is determined that there is no user instruction (NO in step S38), the process ends. The user instruction input unit 112 may accept the user instruction by touching the mark 12 indicating that the auxiliary information displayed on the screen 11 of the translation terminal 100 exists. If the mark 12 is not touched for a predetermined time after the mark 12 is displayed, the auxiliary information presentation determination unit 109 may erase the mark 12.

The auxiliary information presentation determination unit 109 may indicate that the auxiliary information exists by turning on or blinking the LED provided in the translation terminal 100. In this case, the user instruction input unit 112 may accept the user instruction by pressing the button provided on the translation terminal 100. Further, if the button is not pressed for a predetermined time after the LED is turned on or blinked, the auxiliary information presentation determination unit 109 may turn off the LED.

On the other hand, when it is determined that there is a user instruction (YES in step S38), the auxiliary information presenting unit 110 of the translation terminal 100 outputs the translated synthetic voice of the auxiliary information by voice via the voice output unit 105 (step S39). ..

Next, the auxiliary information presentation unit 110 of the translation terminal 100 displays the content information of the auxiliary information via the video output unit 111 (step S40). The auxiliary information presentation unit 110 may output only the translated / synthesized voice of the auxiliary information without displaying the content information. Further, the auxiliary information presentation unit 110 may display only the content information without outputting the translated / synthesized voice of the auxiliary information.

In the information presentation system of the present embodiment, the intention understanding process shown in step S27 and the determination process of whether or not the auxiliary information shown in step S28 exists are performed in the language spoken by the clerk, but the customer utterance. It may be done in the language of. In that case, the processing of steps S27 and S28 is performed using the dialogue text of the customer utterance instead of the dialogue translation text of the customer utterance, and the dialogue translation text of the clerk utterance is used instead of the dialogue text of the clerk utterance. Will be done.

In the information presentation system of the present embodiment, the explanation is made assuming that the customer and the clerk speak different languages, but it can also be used when the customer and the clerk speak the same language. is there. In that case, the processes of step S14, step S15, step S19 in FIG. 6, step S25 in FIG. 7, step S26, and step S34 in FIG. 8 may be omitted.

Further, the translation server 200 may include an utterance count storage unit that stores the identification information that identifies the clerk and the clerk's utterance count in association with each other. When it is determined that the auxiliary information exists, the control unit 202 of the translation server 200 may refer to the number of utterances and determine whether or not the number of clerk utterances is greater than the predetermined number of times. If it is determined that the number of clerk utterances exceeds a predetermined number, it is determined that the presentation of the auxiliary information is unnecessary, and it is not necessary to indicate that the auxiliary information exists.

Here, a specific dialogue between the customer and the clerk will be described. For example, if a customer speaks to a Japanese-speaking clerk in English, "I would like to send my luggage to the United States," the information presentation system translates this customer's utterance into Japanese and outputs it as voice. .. Then, the clerk speaks in Japanese, "There is an international courier service," and the information presentation system translates the clerk's utterance into English and outputs it by voice. At this time, the information presentation system extracts the wording "international courier service" as an entity (explanation target) from the dialogue text of the clerk, and acquires auxiliary information about the international courier service. The information presentation system presents to the clerk that auxiliary information exists. When the information presentation system receives an instruction to present auxiliary information from a clerk, it outputs auxiliary information by voice saying "The size of luggage that can be sent by international courier is up to 160 cm and the weight is up to 25 kg."

The translation server 200 may further include a translation count storage unit that stores the identification information that identifies the clerk and the number of translations into the language spoken by the customer in association with each other. Then, when it is determined that the auxiliary information exists, the control unit 202 of the translation server 200 may refer to the translation number storage unit and determine whether or not the number of translations into the language of the customer utterance is greater than the predetermined number. Good. When the control unit 202 determines that the number of times of translation into the language of the customer utterance is more than the predetermined number of times, it is determined that the presentation of the auxiliary information is unnecessary, and it is not necessary to show the clerk that the auxiliary information exists. .. For example, for a clerk who uses the information presentation system more than a predetermined number of times, the auxiliary information is likely to be known information. Therefore, when the number of times of translation into the language of the customer's utterance is more than the predetermined number of times, it is determined that the presentation of the auxiliary information is unnecessary, and it is not necessary to show the clerk that the auxiliary information exists.

Further, the translation server 200 may further include a usage time storage unit that stores the identification information that identifies the clerk and the usage time from the start of use of the information presentation system to the present. Then, when it is determined that the auxiliary information exists, the control unit 202 of the translation server 200 refers to the usage time storage unit, and the usage time from the time when the clerk starts using the information presentation system to the present is more than the predetermined time. You may decide whether it is long or not. When the control unit 202 determines that the usage time is longer than the predetermined time, it determines that it is not necessary to present the auxiliary information, and it is not necessary to indicate to the clerk that the auxiliary information exists. For example, for a clerk who uses the information presentation system for a longer time than a predetermined time, the auxiliary information is likely to be known information. Therefore, when the usage time of the information presentation system of the clerk is longer than the predetermined time, it is not necessary to judge that the presentation of the auxiliary information is unnecessary and not to show the clerk that the auxiliary information exists.

Further, the translation server 200 may further include a presentation count storage unit that stores the identification information that identifies the clerk and the presentation count of the auxiliary information corresponding to the same explanation target (entity) in association with each other. Then, when it is determined that the auxiliary information exists, the control unit 202 of the translation server 200 may determine whether or not the number of times the auxiliary information corresponding to the same explanation target is presented is greater than the predetermined number of times. When the control unit 202 determines that the number of presentations is greater than the predetermined number of times, the control unit 202 determines that the presentation of the auxiliary information is unnecessary, and does not have to show the store clerk that the auxiliary information exists. For example, when the auxiliary information corresponding to the above-mentioned "international courier service" is presented a plurality of times, the clerk can explain the international courier service without the auxiliary information. Therefore, when the auxiliary information corresponding to the specific explanation target is presented more than a predetermined number of times, it is determined that the presentation of the auxiliary information is unnecessary, and it is not necessary to show the clerk that the auxiliary information exists.

The information presentation method, the information presentation program, and the information presentation system according to the present disclosure do not necessarily present the auxiliary information, and can make the auxiliary information presented only when the speaker needs to present the auxiliary information. It is useful as an information presentation method, an information presentation program, and an information presentation system that can realize a smooth dialogue, translate voices related to a dialogue by a plurality of speakers, and present auxiliary information for assisting the dialogue. is there.

100 Translation terminal 101 Voice input unit 102 Format conversion unit 103 Response control unit 104 Communication unit 105 Voice output unit 106 Response holding unit 107 Auxiliary information presentation setting unit 108 Auxiliary information presence / absence judgment unit 109 Auxiliary information presentation judgment unit 110 Auxiliary information presentation unit 111 Video output unit 112 User instruction input unit 200 Translation server 201 Communication unit 202 Control unit 203 Voice recognition unit 204 Translation unit 205 Intention understanding unit 206 Auxiliary explanatory text storage unit 207 Auxiliary information storage unit 208 Voice waveform synthesis unit 209 Voice analysis synthesis unit

Claims (10)

It is an information presentation method in the information presentation system.
Dialogue text is generated by voice recognition of voices related to dialogues by multiple speakers.
A dialogue translation text is generated by translating the dialogue text,
A dialogue-translated voice is generated by synthesizing the dialogue-translated text.
Based on the dialogue text, it is determined whether or not auxiliary information for assisting the dialogue exists.
When the auxiliary information is present, it is determined whether or not the usage record value of the information presentation system of at least one of the plurality of speakers is larger than a predetermined value.
When it is determined that the usage record value is larger than the predetermined value, the existence of the auxiliary information is not presented to at least one of the plurality of speakers, and the usage record value is equal to or less than the predetermined value. If it is determined, the existence of the auxiliary information is presented to at least one of the plurality of speakers.
Information presentation method. Upon receiving an instruction to present the auxiliary information from at least one of the plurality of speakers,
Present the auxiliary information according to the received presentation instruction,
The information presentation method according to claim 1. The dialogue-translated voice is generated by synthesizing the speech waveform of the dialogue-translated text.
An explanatory text explaining the auxiliary information is specified from the dialogue text,
Auxiliary information voice is generated by voice analysis and synthesis of the above explanation text.
The information presentation method according to claim 1 or 2. The fundamental frequency of the auxiliary information voice is different from the fundamental frequency of the interactive translation voice.
The information presentation method according to claim 3. The second speaker of the second speaker who answers the question in a second language different from the first language while acquiring the first voice of the first speaker asking the question in the first language. Get the voice of
The first dialogue text is generated by recognizing the first voice.
By translating the first dialogue text into the second language, the first dialogue translation text is generated.
The first dialogue translation voice is generated by synthesizing the first dialogue translation text.
A second dialogue text is generated by recognizing the second voice.
A second dialogue translation text is generated by translating the second dialogue text into the first language.
A second dialogue translation voice is generated by voice synthesis of the second dialogue translation text.
Whether or not the auxiliary information exists is determined based on at least one of the first dialogue text and the second dialogue text.
When the auxiliary information is present, it is determined whether or not the usage record value of the information presentation system of the second speaker is larger than the predetermined value.
When it is determined that the usage record value is larger than the predetermined value, the existence of the auxiliary information is not presented to at least one of the first speaker and the second speaker, and the usage record value is the said. When it is determined that the value is equal to or less than a predetermined value, the existence of the auxiliary information is presented to at least one of the first speaker and the second speaker.
The information presentation method according to any one of claims 1 to 4. The usage record value includes the number of times of translation into the first language.
When the auxiliary information is present, it is determined whether or not the number of translations into the first language is greater than the predetermined number of times.
If it is determined that the number of translations into the first language is greater than the predetermined number, the presence of the auxiliary information is not presented to the second speaker.
The information presentation method according to claim 5. The usage record value includes the usage time from the start of use of the information presentation system by the second speaker to the present.
If the auxiliary information is present, it is determined whether or not the usage time is longer than the predetermined time, and the usage time is determined.
If it is determined that the usage time is longer than the predetermined time, the existence of the auxiliary information is not presented to the second speaker.
The information presentation method according to claim 5. The auxiliary information includes information for explaining the object to be explained.
The usage record value includes the number of times the auxiliary information is presented corresponding to the same explanation target.
If the auxiliary information is present, it is determined whether or not the number of presentations is greater than the predetermined number.
If it is determined that the number of presentations is greater than the predetermined number, the existence of the auxiliary information is not presented to the second speaker.
The information presentation method according to claim 5. Computer,
A dialogue text generator that generates dialogue text by recognizing voices related to dialogues by multiple speakers,
A dialogue translation text generator that generates a dialogue translation text by translating the dialogue text,
A dialogue translation voice generation unit that generates a dialogue translation voice by synthesizing the dialogue translation text,
An auxiliary information determination unit that determines whether or not auxiliary information for assisting the dialogue exists based on the dialogue text,
When it is determined that the auxiliary information exists, the usage record of determining whether or not the usage record value of the information presentation system having the computer as a component of at least one of the plurality of speakers is larger than a predetermined value is determined. Value judgment department and
When it is determined that the usage record value is larger than the predetermined value, the auxiliary information and the interactive translation voice are transmitted so as not to present the existence of the auxiliary information to at least one of the plurality of speakers. Instead, when it is determined that the usage record value is equal to or less than the predetermined value, the auxiliary information and the dialogue-translated voice are presented in order to present to at least one of the plurality of speakers that the auxiliary information exists. To function as a transmitter to send
Information presentation program. With the terminal
It is equipped with a server that is communicatively connected to the terminal.
The terminal
A voice acquisition unit that acquires voice related to dialogues by multiple speakers,
A transmitter that transmits the acquired voice to the server, and
With
The server
A receiver that receives the audio and
A dialogue text generation unit that generates dialogue text by recognizing the voice related to the dialogue,
A dialogue translation text generator that generates a dialogue translation text by translating the dialogue text,
A dialogue translation voice generation unit that generates a dialogue translation voice by synthesizing the dialogue translation text,
An auxiliary information determination unit that determines whether or not auxiliary information for assisting the dialogue exists based on the dialogue text,
When it is determined that the auxiliary information exists, a usage record value determination unit that determines whether or not the usage record value of the information presentation system of at least one of the plurality of speakers is larger than a predetermined value, and
When it is determined that the usage record value is larger than the predetermined value, the auxiliary information and the dialogue translation voice are not transmitted, and when it is determined that the usage record value is equal to or less than the predetermined value, the auxiliary information and the dialogue translation voice are not transmitted. A transmitter that transmits the interactive translation voice to the terminal,
With
The terminal
A receiving unit that receives the auxiliary information and the interactively translated voice, and
An audio output unit that outputs the interactive translation audio,
A presenting unit that presents to at least one of the plurality of speakers that the auxiliary information exists, and
Information presentation system equipped with.

Images