JP6697270B2 - Communication support system, communication support method, and program - Google Patents

Description

  The present invention relates to a communication support system that supports communication between users who use different languages, a communication support method, and a program that causes a computer to function as the communication support system.

  Along with the globalization of economy and culture, a technique for translating the utterance content uttered by a user who uses a certain language into a language different from the certain language has been developed.

  For example, in Patent Document 1, the most probable speech recognition result is selected using the speech recognition scores received from two or more speech recognition devices, and the selected speech recognition results received from two or more translation devices are selected. A control device for selecting the most likely translation result using a translation score is described. The control device selects a voice synthesis result by using the voice synthesis score received from each of the two or more voice synthesis devices, and transmits the selected voice synthesis result to the second terminal device which outputs the voice.

  Further, in Patent Document 2, an input natural language speech signal is analyzed, and the analyzed speech is analyzed using a natural language acoustic model of a plurality of languages and a natural language pronunciation dictionary of the natural language language model. An automatic language identification device for searching in parallel is described. The automatic language identification device identifies the input natural language by comparing the likelihoods of the searched results.

JP, 2011-90100, A (published on May 6, 2011) JP-A-2004-347732 (published on December 9, 2004)

  However, in the related art as described above, when the device recognizes that the content uttered by the user is in a language different from the language used by the user, the content uttered by the user is correctly translated. Can not do it. Therefore, there is a problem that a user who uses a certain language and a user who uses a language different from the certain language cannot communicate smoothly.

  The present invention has been made in view of the above problems, and an object thereof is to provide a technique for facilitating communication between users who use different languages.

  In order to solve the above problems, a communication support system according to an aspect of the present invention provides a display unit having a first area for a first user and a second area for a second user, and a voice input. And a control unit, wherein the control unit obtains first voice information indicating a voice of the first user via the voice input unit, and the first voice A recognition process for recognizing the first voice content indicated by the information as each of the plurality of languages is performed, and the first recognition of the display target is performed from the first recognition content indicating the recognition content recognized as each of the plurality of languages. A selection process for selecting the content is performed, and the first recognition content of the display target is displayed in the first area of the display unit.

  In order to solve the above problems, a communication support method according to an aspect of the present invention includes an acquisition step of acquiring first voice information indicating a voice of a first user, and the first voice information. The first recognition content to be displayed is selected from the recognition step of recognizing the first voice content shown as each of the plurality of languages and the first recognition content showing the recognition content recognized as each of the plurality of languages. And a display step of displaying the first recognition content of the display target in the first area for the first user.

  In order to solve the above problems, a program according to one aspect of the present invention includes a display unit having a first area for a first user and a second area for a second user, and a voice input. A program for causing a computer to function as a communication support system comprising a control unit and a control unit, wherein the control unit receives first voice information indicating a voice of a first user via the voice input unit. From the acquisition process for acquiring, the recognition process for recognizing the first voice content indicated by the first voice information as each of the plurality of languages, and the first recognition content indicating the recognition content recognized as each of the plurality of languages Selection processing for selecting the first recognition content to be displayed, and display processing for displaying the first recognition content to be displayed in the first area of the display unit.

  According to one aspect of the present invention, it is possible to facilitate communication between users who use different languages.

It is a block diagram showing the composition of the important section of the communication support system in Embodiment 1 of the present invention. It is a figure which shows typically the communication support system in Embodiment 1 of this invention. FIG. 6 is a sequence diagram showing an example of a flow of processing when a service provider utters a communication in the communication support system in the first exemplary embodiment of the present invention. FIG. 3 is a diagram showing an example of an image displayed on a display unit in the first embodiment of the present invention. In the communication support system in Embodiment 1 of this invention, it is a sequence diagram which shows the other example of the flow of a process when a service provider speaks. FIG. 6 is a sequence diagram showing a flow of processing when a service user utters a communication in the communication support system according to the first exemplary embodiment of the present invention. FIG. 6 is a diagram showing another example of an image displayed on the display unit in the first embodiment of the present invention. FIG. 9 is a diagram showing still another example of the image displayed on the display unit in the first embodiment of the present invention. It is an example of the database stored in the terminal storage unit in the second embodiment of the present invention. It is a flowchart which shows the flow of a process of the client terminal in Embodiment 2 of this invention. FIG. 9 is a diagram showing an example of an image displayed on a display unit in Embodiment 2 of the present invention. It is a figure which shows an example of the image displayed on a display part in Embodiment 3 of this invention. It is a figure which shows an example of the image displayed on the display part in Embodiment 4 of this invention. It is a figure which shows an example of the image displayed on a display part in Embodiment 5 of this invention. It is an example of the database stored in the terminal storage unit in the sixth embodiment of the present invention. It is a flowchart which shows the flow of a process of the client terminal in Embodiment 6 of this invention. It is a figure which shows an example of the image displayed on the display part in Embodiment 6 of this invention. It is a flowchart which shows the flow of a process of the client terminal in Embodiment 7 of this invention. It is a figure which shows an example of the image displayed on the display part in Embodiment 7 of this invention. It is a figure which shows an example of the image displayed on the display part in Embodiment 8 of this invention. It is a block diagram which illustrated the hardware constitutions of the computer which can be used as a recognition server, a translation server, a support server, and a client terminal.

  Hereinafter, embodiments of the present invention will be described in detail. For convenience of description, members having the same functions as the members described in the above embodiment will be designated by the same reference numerals, and the description thereof will be omitted.

[Embodiment 1]
(Communication support system 1)
FIG. 2 is a diagram schematically showing the communication support system 1 according to the first embodiment of the present invention. As shown in FIG. 2, the communication support system 1 includes a recognition server 10, a translation server 20, a support server 30, and a client terminal 40. In this specification, when the client terminal 40 and the support server 30 are installed locally, and the recognition server 10 and the translation server 20 are installed on the network 2 (the support server 30 recognizes the recognition server 10 via the network 2). , And a configuration capable of communicating with the translation server 20), but the locations where the support server 30, the recognition server 10, and the translation server 20 are installed are not particularly limited.

  In the communication support system 1, according to voices uttered by a plurality of users (first user and second user) who are trying to communicate using the communication support system (1), the content of the voice is set as a predetermined language. At least one of the recognized recognition content and (2) translation content obtained by translating the recognition content into a predetermined language is displayed on the display unit 46 of the client terminal 40. As described above, the communication support system 1 is a system that supports communication between the first user and the second user who use different languages.

  In the present specification, a case where the first user is a service user (for example, a customer who uses a convenience store) and the second user is a service provider (for example, a clerk of the convenience store) will be described as an example. However, the present invention is not limited to this. For example, the first user may be the service provider and the second user may be the service user. Another example of the second user is a device installed in a convenience store, which is operated by a customer to display product information and order products.

  The configuration of each device constituting the communication support system 1 will be described with reference to FIG. FIG. 1 is a block diagram showing a main configuration of a communication support system 1 according to the first embodiment of the present invention.

(Recognition server 10)
As shown in FIG. 1, the recognition server 10 includes a recognition server communication unit 12 and a recognition server control unit 14.

  The recognition server communication unit 12 is a communication interface for communicating with an external device.

  The recognition server control unit 14 is an arithmetic device that integrally controls each configuration of the recognition server 10. More specifically, the recognition server control unit 14 causes the recognition server communication unit 12 to (1) the service provider voice information indicating the voice uttered by the service provider, or (2) the service user uttered. Acquire service user voice information indicating voice.

  Then, the recognition server control unit 14 performs a recognition process of recognizing the voice content indicated by the acquired voice information as a plurality of preset languages. In the following, "recognizing A voice content indicated by A voice information" may be referred to as "recognizing A voice information". The recognition server control unit 14 may be configured to recognize voice content as a designated language among a plurality of preset languages by designating the recognized language, or depending on the obtained voice information. The language to be recognized may be changed.

  For example, when the recognition server control unit 14 is designated to recognize the acquired voice information in English, the recognition server control unit 14 performs a recognition process of recognizing the voice content indicated by the acquired voice information in English. When the recognition server control unit 14 is set to recognize in Japanese when the service provider voice information is acquired, when the service provider voice information is acquired, the recognition server control unit 14 acquires the voice content indicated by the service provider voice information. Recognition processing that recognizes in Japanese is performed.

  In the present specification, when the recognition server control unit 14 acquires the service provider voice information, the recognition server control unit 14 recognizes the voice content indicated by the service provider voice information as a preset language used by the service provider. Perform processing. In addition, when the service user voice information is acquired, the recognition server control unit 14 performs a recognition process of recognizing in a designated language among a plurality of preset languages.

  Then, the recognition server control unit 14 recognizes the recognition content indicating the recognized content (provider recognition content indicating the recognition content that recognizes the voice content indicated by the service provider voice information or the voice content indicated by the service user voice information. The recognition result including the user recognition content indicating the recognized recognition content) and the recognition accuracy indicating the certainty of the recognition processing is output via the recognition server communication unit 12. Note that “recognizing the voice content A indicated by the voice information as the language B” includes the meaning of interpreting A as the language B.

(Translation server 20)
The translation server 20, as shown in FIG. 1, includes a translation server communication unit 22 and a translation server control unit 24.

  The translation server communication unit 22 is a communication interface for communicating with an external device.

  The translation server control unit 24 is an arithmetic unit that integrally controls each configuration of the translation server 20. More specifically, the translation server control unit 24 acquires the provider recognition content or the user recognition content via the translation server communication unit 22.

  Then, the translation server control unit 24 performs a translation process of translating the acquired recognition content into a plurality of preset languages. The translation server control unit 24 may be configured to translate as a designated language among a plurality of preset languages by designating a language to translate, or translate according to the acquired recognition content. It may be configured to change the language.

  For example, when the translation server control unit 24 is designated to translate the acquired recognition content into English, the translation server control unit 24 performs a translation process of translating the acquired recognition content into English. Further, when the translation server control unit 24 is set to translate into Japanese when the user recognition content is acquired, when the user recognition content is acquired, the translation server control unit 24 translates the user recognition content into Japanese. Perform processing.

  In the present specification, the translation server control unit 24, when acquiring the provider recognition content, performs translation processing to translate into a designated language among a plurality of preset languages. In addition, the translation server control unit 24, when acquiring the user recognition content, translates it into a preset language used by the service provider.

  Then, the translation server control unit 24 outputs the translation content indicating the translated content via the translation server communication unit 22.

(Support server 30)
As shown in FIG. 1, the support server 30 includes a support server communication unit 32 and a support server control unit 34.

  The support server communication unit 32 is a communication interface for communicating with an external device.

  The support server control unit 34 is an arithmetic device that integrally controls each configuration of the support server 30. Details of the support server control unit 34 will be described below.

(Support server control unit 34)
As shown in FIG. 1, the support server control unit 34 also functions as an information management unit 342, a selection unit 344, a display mode determination unit 346, and a display information output unit 348. Detailed processing of each unit will be described later with reference to the drawings.

  The information management unit 342 manages the information acquired via the support server communication unit 32.

  The selection unit 344 performs a selection process of selecting the recognition content to be displayed from the acquired recognition content.

  The display mode determination unit 346 determines the display mode for displaying the recognition content or the translation content on the client terminal 40.

  The display information output unit 348 outputs display information, which is information for displaying the recognition content or the translation content on the client terminal 40, via the support server communication unit 32.

(Client terminal 40)
As shown in FIG. 1, the client terminal 40 includes a client terminal communication unit 42, a client terminal control unit 44, a display unit 46, a voice input unit 48, an operation unit (operator) 50, and a terminal storage unit 52. ..

  The client terminal communication unit 42 is a communication interface for communicating with an external device.

  The display unit 46 is a display device that displays an image represented by the acquired image signal. The display unit 46 also has a first area for service users and a second area for service providers.

  The voice input unit 48 acquires voice around the client terminal 40 and outputs voice information indicating the acquired voice.

  The operation unit 50 is a device that receives a user operation and outputs an operation signal indicating the received operation.

  The terminal storage unit 52 is a storage device that stores a database and the like.

  The client terminal control unit 44 is an arithmetic unit that integrally controls each configuration of the client terminal 40. Details of the client terminal control unit 44 will be described below.

(Client terminal control unit 44)
As shown in FIG. 1, the client terminal control unit 44 also functions as a voice information acquisition unit 442, a voice recognition unit 444, an operation signal acquisition unit 446, a display information acquisition unit 448, and a display control unit 450. Detailed processing of each unit will be described later with reference to the drawings.

  The voice information acquisition unit 442 acquires voice information via the voice input unit 48.

  The voice recognition unit 444 executes the same processing as that of the recognition server control unit 14 described above.

  The operation signal acquisition unit 446 acquires the operation signal via the operation unit 50.

  The display information acquisition unit 448 acquires display information via the client terminal communication unit 42.

  The display control unit 450 outputs an image signal indicating an image to be displayed on the display unit 46.

(1 when there is an utterance from the service provider)
In the communication support system 1, a process when a service provider makes a speech will be described with reference to FIG. FIG. 3 is a sequence diagram showing an example of a flow of processing in the communication support system 1 according to the first embodiment of the present invention when a service provider makes a speech. In the following description, unless otherwise specified, the language used by the service provider is Japanese, and the languages for translating the content recognized by the provider are English, Chinese, and Korean, as an example. explain. In the description with reference to FIG. 3, the case where the recognition process is executed by the recognition server 10 will be described.

(Step S2)
The voice information acquisition unit 442 of the client terminal 40 acquires the provider voice information indicating the voice uttered by the service provider via the voice input unit 48. Specifically, the voice information acquisition unit 442 acquires, via the voice input unit 48, the provider voice information that is issued by the service provider and indicates “Are you looking for something?”.

  As an example of a method for the client terminal 40 to determine whether or not the acquired voice information is voice information indicating the voice uttered by the provider, the service user side voice input unit attached to the service user side. 48a and a service provider side voice input unit 48b attached to the service provider side, and the voice information acquired via the service provider side voice input unit 48b is determined to be the provider voice information. Be done.

(Step S4)
The voice information acquisition unit 442 outputs the provider voice information to the support server 30 via the client terminal communication unit 42.

(Step S6)
The information management unit 342 of the support server 30 acquires the provider voice information via the support server communication unit 32.

(Step S8)
When the information managing unit 342 acquires the provider voice information, the information managing unit 342 acquires the provider voice information via the support server communication unit 32 in order to acquire the recognition result of recognizing the provider voice content indicated by the provider voice information. Output to the recognition server 10.

(Step S10)
The recognition server control unit 14 of the recognition server 10 acquires the provider voice information via the recognition server communication unit 12.

(Step S12)
The recognition server control unit 14 recognizes the voice content indicated by the acquired provider voice information as the language used by the service provider. Specifically, the recognition server control unit 14 recognizes the provider voice content indicated by the provider voice information indicating "Are you looking for something?" As Japanese.

(Step S14)
The recognition server control unit 14 outputs the recognition result including the provider recognition content indicating the recognized content and the recognition accuracy indicating the certainty of the recognition processing in step S12 to the support server 30 via the recognition server communication unit 12. To do. Specifically, since the recognition server control unit 14 recognizes “What are you looking for?” In Japanese, the recognition accuracy is high and the provider recognition content is also “What are you looking for?”.

(Step S16)
The information management unit 342 of the support server 30 acquires the recognition result via the support server communication unit 32. The information management unit 342 outputs the acquired recognition result to the display mode determination unit 346.

(Step S18)
Further, the information management unit 342 translates the provider recognition content included in the acquired recognition result, so that the provider recognition content is set in each of a plurality of preset languages, English, Chinese, and Korean. The provider recognition content is output to the translation server 20 together with the translation instruction.

(Step S20)
The translation server control unit 24 of the translation server 20 acquires the provider recognition content via the translation server communication unit 22.

(Step S22)
The translation server control unit 24 translates the acquired provider recognition content into English, Chinese, and Korean designated by the support server 30. Specifically, the translation server control unit 24 translates "What are you looking for?" Into English, Chinese, and Korean.

(Step S24)
The translation server control unit 24 outputs the provider translation content translated in the translation process to the support server 30 via the translation server communication unit 22.

(Step S26)
The information management unit 342 of the support server 30 acquires the provider translation content via the support server communication unit 32. The information management unit 342 outputs the acquired provider translation content to the display mode determination unit 346.

(Step S28)
The display mode determination unit 346 determines the display mode to be displayed on the display unit 46 of the client terminal 40 when the provider translation content is acquired. Specifically, the display mode determination unit 346 causes the provider recognition content included in the recognition result acquired in step S16 to be displayed in the second area for the service provider, and the provider translation content for the service user. The display mode is determined so that it is displayed in the first area. Then, the display mode determination unit 346 outputs the display information including the determined display mode, the provider recognition content, and the provider translation content to the display information output unit 348.

(Step S30)
The display information output unit 348 outputs the acquired display information to the client terminal 40 via the support server communication unit 32.

(Step S32)
The display information acquisition unit 448 of the client terminal 40 acquires the display information via the client terminal communication unit 42. The display information acquisition unit 448 outputs the acquired display information to the display control unit 450.

(Step S34)
Upon obtaining the display information, the display control unit 450 refers to the information included in the display information and causes the display unit 46 to display the image. Specifically, the display control unit 450 refers to the display mode included in the display information, displays the provider recognition content included in the display information in the second area for the service provider, and includes the content in the display information. The provider translation content is displayed in the first area for service users. An example of the image displayed on the display unit 46 at this time is shown in FIG. FIG. 4 is a diagram showing an example of an image displayed on the display unit 46 in the first embodiment of the present invention, (a) is an image displayed on the first area 46a, and (b) is It is an image displayed in the second area 46b.

  As shown in (a) of FIG. 4, in the image displayed in the first area 46a, the contents of the provider recognition are (1) the text 600 including the translated contents translated into English, and (2) the Chinese characters. A text 602 including the translated content translated and (3) a text 604 including the translated content translated into Korean are included. Further, the image displayed in the first area 46a is for displaying the translated content of the other language when translated into other languages other than English, Chinese, and Korean in the communication support system. A button 606 for accepting an operation may be included.

  Further, as shown in FIG. 4B, the image displayed in the second area 46b includes the text 700 including the contents of the provider recognition.

(When there is a utterance from the service provider 2)
Next, another process in the communication support system 1 when a service provider speaks will be described with reference to FIG. FIG. 5 is a sequence diagram showing another example of the flow of processing in the communication support system 1 according to the first embodiment of the present invention when a service provider makes a speech. In the description using FIG. 5, the case where the recognition process is executed in the client terminal 40 will be described.

(Step S2)
The voice information acquisition unit 442 of the client terminal 40 acquires the provider voice information indicating the voice uttered by the service provider via the voice input unit 48. The voice information acquisition unit 442 outputs the acquired provider voice information to the voice recognition unit 444.

(Step S40)
The voice recognition unit 444 recognizes the voice content indicated by the acquired provider voice information as the language used by the service provider.

(Step S42)
The voice recognition unit 444 outputs the recognition result including the provider recognition content indicating the recognized content and the recognition accuracy indicating the accuracy of the recognition processing in step S40 to the support server 30 via the client terminal communication unit 42. ..

(Step S44)
The information management unit 342 of the support server 30 acquires the recognition result via the support server communication unit 32. The information management unit 342 outputs the acquired recognition result to the display mode determination unit 346.

(Step S18 to Step S34)
Since the process is the same as the process described above, the description thereof will be omitted.

  As described above, the recognition process of recognizing the voice content included in the voice information may be executed by the client terminal 40 or the recognition server 10. When the recognition process is executed in the client terminal 40, there is an effect that the communication amount between the client terminal 40, the support server 30, and the recognition server 10 can be reduced. On the other hand, when the recognition processing is executed by the recognition server 10, there is an effect that the load on the client terminal 40 can be reduced. Alternatively, the recognition process may be executed in the client terminal 40 first, and when the recognition accuracy is lower than a predetermined value, the recognition server 10 may execute the recognition process. Therefore, unless otherwise specified, the device that performs the recognition process is not limited.

(When the service user speaks)
In the communication support system 1, a process (communication support method) when a service user speaks will be described with reference to FIG. FIG. 6 is a sequence diagram showing a flow of processing in the communication support system 1 according to the first embodiment of the present invention when a service user makes a speech. In the following explanation, unless otherwise specified, the user recognition content is recognized as English, Chinese, and Korean, and the language for translating the user recognition content is Japanese, as an example. To do.

(Step S52: acquisition step)
The voice information acquisition unit 442 of the client terminal 40 performs an acquisition process of acquiring user voice information indicating the voice uttered by the service user via the voice input unit 48. Specifically, the voice information acquisition unit 442, via the voice input unit 48, is issued by the service user to the above-mentioned service provider “are you looking for something?”.

The user voice information indicating is acquired.

(Step S54)
The voice information acquisition unit 442 outputs the user voice information to the support server 30 via the client terminal communication unit 42.

(Step S56)
The information management unit 342 of the support server 30 acquires the user voice information via the support server communication unit 32.

(Step S58)
When the information management unit 342 acquires the user voice information, the information management unit 342 acquires the user voice information via the support server communication unit 32 in order to acquire the recognition result of recognizing the user voice content indicated by the user voice information. Output to the recognition server 10.

(Step S60)
The recognition server control unit 14 of the recognition server 10 acquires the user voice information via the recognition server communication unit 12.

(Step S62: Recognition step)
The recognition server control unit 14 recognizes the voice content indicated by the acquired user voice information as English, Chinese, and Korean. Specifically, the recognition server control unit 14 recognizes the pronunciation “Woshang Yao Coffee” as English, Chinese, and Korean.

(Step S64)
The recognition server control unit 14 outputs the recognition result including the user recognition content indicating the recognized content and the recognition accuracy indicating the certainty of the recognition processing in step S62 to the support server 30 via the recognition server communication unit 12. To do. Specifically, the recognition server control unit 14 recognizes “Woshang Yao Coffee” in Chinese as a recognition result with high recognition accuracy, and the user recognition content is also high.

become. On the other hand, the recognition server control unit 14 recognizes "Woshan Yao coffee" in English, and the recognition accuracy is low, and the user recognition content is "What are y'all coffee".

(Step S66)
The information management unit 342 of the support server 30 acquires the recognition result via the support server communication unit 32. The information management unit 342 outputs the acquired recognition result to the selection unit 344.

(Step S68: selection step)
The selection unit 344 refers to the recognition result and selects the user recognition content to be displayed. Specifically, the selection unit 344 refers to the recognition certainty included in the recognition result, and selects the user recognition content recognized with the recognition certainty whose recognition certainty is higher than a predetermined threshold. More specifically, when the recognition accuracy of recognizing “Woshan Yao Coffee” as English and Chinese is higher than a predetermined threshold and the recognition accuracy of recognizing it as Korean is equal to or lower than the predetermined threshold, the selecting unit 344 selects the English And the user recognition contents recognized as Chinese and Chinese are selected. Then, the selection unit 344 outputs the selected user recognition content to the information management unit 342. The selection unit 344 also outputs the recognition result including the selected user recognition content to the display mode determination unit 346.

(Step S70)
The information management unit 342 outputs the acquired user recognition content to the translation server 20 in order to translate the acquired user recognition content.

(Step S72)
The translation server control unit 24 of the translation server 20 acquires the user recognition content via the translation server communication unit 22.

(Step S74)
The translation server control unit 24 translates the acquired user recognition content into preset Japanese. Specifically, the translation server control unit 24

And "What are y'all coffee" in English are translated into Japanese.

(Step S76)
The translation server control unit 24 outputs the user translation content translated in the translation process to the support server 30 via the translation server communication unit 22.

(Step S78)
The information management unit 342 of the support server 30 acquires the user translation content via the support server communication unit 32. The information management unit 342 outputs the acquired user translation content to the display mode determination unit 346.

(Step S80)
When the user translation content is acquired, the display mode determination unit 346 determines the display mode to be displayed on the display unit 46 of the client terminal 40. Specifically, the display mode determination unit 346 causes the user recognition content included in the recognition result acquired in step S68 to be displayed in the first area for the service user, and the user translation content for the service provider. The display mode is determined so that it is displayed in the second area. Then, the display mode determination unit 346 outputs the display information including the determined display mode, the user recognition content, and the user translation content to the display information output unit 348.

(Step S82)
The display information output unit 348 outputs the acquired display information to the client terminal 40 via the support server communication unit 32.

(Step S84)
The display information acquisition unit 448 of the client terminal 40 acquires the display information via the client terminal communication unit 42. The display information acquisition unit 448 outputs the acquired display information to the display control unit 450.

(Step S86: display step)
Upon obtaining the display information, the display control unit 450 refers to the information included in the display information and causes the display unit 46 to display the image. Specifically, the display control unit 450 refers to the display mode included in the display information, causes the user recognition content included in the display information to be displayed in the first area for the service user, and is included in the display information. The user translated content is displayed in the second area for the service provider. An example of the image displayed on the display unit 46 at this time is shown in FIG. FIG. 7 is a diagram showing another example of an image displayed on the display unit 46 in Embodiment 1 of the present invention, (a) is an image displayed on the first area 46a, and (b) is Is an image displayed in the second area 46b.

  As shown in (a) of FIG. 7, in addition to the text 600, the text 602, and the text 604 described above, the image displayed in the first area 46a includes (1) a user recognized as Chinese A text 610 including recognition contents and (2) a text 612 including user recognition contents recognized as English are included.

  Further, as shown in FIG. 7B, in the image displayed in the second area 46b, in addition to the above-described text 700, (1) user recognition content recognized as Chinese is written in Japanese. A text 710 including the user translated content translated into (1) and (2) a text 712 including the user translated content translated into Japanese from the user recognized content recognized as English.

  In the communication support system 1, as shown in (a) of FIG. 7, the user recognition content (first recognition content) may be displayed from the top in the descending order of recognition accuracy. Specifically, in step S80, the display mode determination unit 346 determines the display mode of the user recognition content with reference to the recognition accuracy included in the recognition result acquired in step S68. For example, as described above, a configuration that determines the display mode so that the user recognition content is displayed from the top in the order of high recognition accuracy, or the user recognition content with high recognition accuracy is displayed in a dark color, and the recognition accuracy is There is a configuration in which the display mode is determined so that the low user recognition content is displayed in a light color. Further, as shown in FIG. 7B, the user translation content obtained by translating the user recognition content having high recognition accuracy may be displayed above in accordance with the display mode of the user recognition content. ..

(In addition, if there is a utterance from the service provider)
Next, the service provider looks at the image shown in FIG. 7 (b), determines that the service user uttered "I want coffee", and asks the service user "Hot and nice. The case of uttering "?" Will be described with reference to FIG.

(Steps S2 to S16)
Since the process is the same as the process described above, the description thereof will be omitted.

(Step S18)
The information management unit 342 outputs the provider recognition content included in the acquired recognition result to the translation server 20 in order to translate the provider recognition content. Here, in step S68 described above, since the selection unit 344 has selected the user recognition contents recognized as English and Chinese, the information management unit 342 translates the provider recognition contents into English and Chinese, respectively. The provider recognition content is output to the translation server 20 together with the instruction.

(Step S20)
The translation server control unit 24 of the translation server 20 acquires the provider recognition content via the translation server communication unit 22 and receives an instruction to translate the provider recognition content into English and Chinese respectively.

(Step S22)
The translation server control unit 24 translates the acquired provider recognition content into English and Chinese according to the received instruction. Specifically, the translation server control unit 24 translates "Are you sure it's hot?" Into English and Chinese.

(Steps S24 to S34)
Since the process is the same as the process described above, the description thereof will be omitted. Here, an example of the image displayed on the display unit 46 in step S34 is shown in (a) and (b) of FIG. FIG. 8 is a diagram showing still another example of the image displayed on the display unit 46 in the first embodiment of the present invention. FIG. 8A is an image displayed on the first region 46a, and FIG. ) Is an image displayed in the second area 46b.

  As shown in FIG. 8A, in the image displayed in the first area 46a, in addition to the text 600, the text 602, the text 604, the text 610, and the text 612 described above, the content recognized by the provider is displayed. , (1) a text 620 containing the translated content translated into Chinese, and (2) a text 622 containing the translated content translated into English.

  Further, as shown in FIG. 8B, the image displayed in the second area 46b includes the text 720 including the provider recognition content in addition to the text 700, the text 710, and the text 712 described above. Has been.

(If there is a utterance from the service user)
Next, a case where the service user looks at the image shown in FIG. 8A and utters "favorable" will be described with reference to FIG.

(Step S52 to Step S56)
Since the process is the same as the process described above, the description thereof will be omitted.

(Step S58)
When the information management unit 342 acquires the user voice information, the information management unit 342 acquires the user voice information via the support server communication unit 32 in order to acquire the recognition result of recognizing the user voice content indicated by the user voice information. Output to the recognition server 10. Here, in step S68 described above, since the selection unit 344 has selected the user recognition contents recognized as English and Chinese, respectively, the information management unit 342 instructs to recognize the user voice information as English and Chinese. At the same time, the user voice information is output to the translation server 20.

(Step S60)
The recognition server control unit 14 of the recognition server 10 acquires the user voice information via the recognition server communication unit 12 and receives an instruction to recognize the user voice information as English and Chinese.

(Step S62)
The recognition server control unit 14 recognizes the voice content indicated by the acquired user voice information as English and Chinese. Specifically, the recognition server control unit 14 recognizes “Haoda” as English and Chinese.

(Step S64)
The recognition server control unit 14 outputs the recognition result including the user recognition content indicating the recognized content and the recognition accuracy indicating the certainty of the recognition processing in step S62 to the support server 30 via the recognition server communication unit 12. To do. Specifically, the recognition server control unit 14 recognizes “haoda” as Chinese, and the recognition result has high recognition accuracy, and the user recognition content is also “favorable”. On the other hand, the recognition server control unit 14 has low recognition accuracy in the recognition result of recognizing “Hoda” in English.

(Step S66)
The information management unit 342 of the support server 30 acquires the recognition result via the support server communication unit 32. The information management unit 342 outputs the acquired recognition result to the selection unit 344.

(Step S68)
The selection unit 344 refers to the recognition result and selects the user recognition content to be displayed. Specifically, when the recognition accuracy of recognizing “Haoda” as Chinese is higher than a predetermined threshold and the recognition accuracy of recognizing as “English” is less than or equal to a predetermined threshold, the selection unit 344 uses the usage recognized as Chinese respectively. Person recognition content is selected. Then, the selection unit 344 outputs the selected user recognition content to the information management unit 342. The selection unit 344 also outputs the recognition result including the selected user recognition content to the display mode determination unit 346.

(Step S70 to Step S86)
Since the process is the same as the process described above, the description thereof will be omitted. Here, examples of the image displayed on the display unit 46 in step S86 are shown in FIGS. 8C and 8D. 8C is an image displayed in the first area 46a, and FIG. 8D is an image displayed in the second area 46b.

  As shown in FIG. 8C, in the image displayed in the first area 46a, in addition to the text 610, the text 612, the text 620, and the text 622 described above, user recognition recognized as Chinese is recognized. Included is a text 630 containing the content.

  Further, as shown in FIG. 8D, the image displayed in the second area 46b is recognized as Chinese in addition to the text 700, the text 710, the text 712, and the text 720 described above. The text 730 including the user translated content in which the user recognized content is translated into Japanese is included.

  As described above, the communication support system 1 according to the present embodiment has the first area 46a for the service user (first user) and the second area 46b for the service provider (second user). The display unit 46, the voice input unit 48, and the control unit (the recognition server control unit 14, the support server control unit 34, and the client terminal control unit 44) are provided, and the voice of the service user is shown via the voice input unit 48. Acquires user voice information (first voice information) and recognizes the user voice content (first voice content) indicated by the user voice information as each of a plurality of languages (English, Chinese, and Korean) The recognition process is performed to perform the selection process of selecting the user recognition content to be displayed from the user recognition content (first recognition content) indicating the recognition content recognized as each of the plurality of languages, and The user recognition content is displayed in the first area 46a of the display unit 46. With this configuration, in the communication support system 1, even when the utterance of the service user is recognized as a wrong language, the recognition content recognized as a language other than the wrong language is also presented to the user. Therefore, the communication support system 1 can facilitate communication between users who use different languages.

  Further, in the communication support system 1 according to the present embodiment, the first user may be a service user and the second user may be a service provider. With this configuration, in the communication support system 1, the service provider (for example, a store clerk) can smoothly communicate with a service user (for example, a customer of a store) whose language to be used cannot be specified. You can

  Further, in the communication support system 1 according to the present embodiment, the first user may be a service provider and the second user may be a service user. With this configuration, in the communication support system 1, each of a plurality of service providers (for example, store clerks) who use different languages can smoothly communicate with service users (for example, store customers). You can

  Further, in the communication support system 1 according to the present embodiment, in the selection process, the user recognition content to be displayed is selected with reference to the recognition accuracy indicating the accuracy of the recognition process recognized in each of the plurality of languages. With this configuration, the communication support system 1 presents the user with the recognition content recognized with the recognition accuracy whose recognition accuracy is higher than the threshold value. Therefore, since the recognition content recognized as a language that is unlikely to be the language used by the user is not displayed, it is possible to facilitate communication between users who use different languages.

  Further, in the communication support system 1 according to the present embodiment, the display mode for displaying the user recognition content to be displayed in the first area 46a of the display unit 46 is determined with reference to the recognition accuracy. With this configuration, in the communication support system 1, for example, the recognition contents having high recognition accuracy can be displayed conspicuously, or the recognition contents can be displayed in the order of high recognition accuracy. Therefore, the communication support system 1 can inform the user of which language has the higher recognition accuracy.

  In addition, in the communication support system 1 according to the present embodiment, the user translation content (first translation content) obtained by translating the user recognition content to be displayed is acquired, and the user translation content is displayed in the second section of the display unit 46. It is displayed in the area 46b. With this configuration, the communication support system 1 can translate the utterance content of the service user into the language used by the service provider and present the translated content. Therefore, the communication support system 1 can facilitate communication between users who use different languages.

  Further, in the communication support system 1 according to the present embodiment, the provider voice information (second voice information) indicating the voice of the service provider is acquired via the voice input unit 48, and the provider indicated by the provider voice information. Recognition processing for recognizing voice content (second voice content) is performed, and provider translation content (second translation) obtained by translating the provider recognition content (second recognition content) recognized by the recognition processing into a plurality of languages. Content) is acquired and the provider translation content is displayed in the first area 46a of the display unit 46. With this configuration, the communication support system 1 can translate the utterance content of the service provider into a plurality of languages including the language used by the service user and present the translated content. Therefore, the communication support system 1 can facilitate communication between users who use different languages.

  Note that, in the present embodiment, the user recognition content having the recognition accuracy higher than the predetermined threshold is selected as the configuration for selecting the user recognition content, but the integrated value of the recognition accuracy and the threshold may be compared. ..

  For example, when the user voice information indicating the voice uttered by the service user first is recognized as English, Chinese, and Korean, the recognition accuracies are “30”, “50”, and “10”, respectively. In this case, in step S68 described above, the selection unit 344 selects all the user recognition contents as display targets regardless of the recognition accuracy. In the following description, the recognition accuracy when recognized as the language A and the language A recognition accuracy.

  Then, in the recognition process of recognizing the user voice information indicating the voice uttered by the service user next, the English recognition accuracy, the Chinese recognition accuracy, and the Korean recognition accuracy are “15”, “60”, and If it is "5", in step S68 described above, the selecting unit 344 first calculates the integrated English recognition accuracy by adding the English recognition accuracy in the current recognition processing to the English recognition accuracy in the previous recognition processing. That is, the selection unit 344 adds the English recognition accuracy “15” in the current recognition processing to the English recognition accuracy “30” in the previous recognition processing to calculate the integrated English recognition accuracy “45”.

  Subsequently, the selection unit 344 performs the same processing in each language to calculate the integrated Chinese language recognition accuracy “110” and the integrated Korean language recognition accuracy “15”. Then, the selection unit 344 selects the user recognition content in which the calculated integrated value is recognized with a recognition accuracy higher than a predetermined threshold value. For example, when the threshold value in this case is “40”, the integrated English recognition accuracy “45” and the integrated Chinese language recognition accuracy “110” are higher than the threshold values, so the selection unit 344 is recognized as English and Chinese. Select the user recognition content.

  Further, when the service user speaks, the user voice information indicating the voice uttered by the service user is recognized as English or Chinese. Then, when the English recognition accuracy and the Chinese recognition accuracy are "5" and "50", respectively, the selection unit 344 calculates the respective recognition accuracy as the already calculated integrated English recognition accuracy and integrated Chinese recognition accuracy. Add to. Specifically, the selection unit 344 calculates the integrated English recognition accuracy “50” and the integrated Chinese recognition accuracy “160”. For example, when the threshold value in this case is “80”, the cumulative Chinese language recognition accuracy “160” is higher than the threshold value, and thus the selection unit 344 selects the user recognition content recognized as Chinese language.

  As described above, the communication support system 1 may be configured to compare the integrated value of the recognition accuracy with the threshold value and select the user recognition content to be selected. In the case of this configuration, the language used by the service user can be selected more preferably, so that communication between users who use different languages can be made smoother.

  It should be noted that the method of adding the recognition accuracy as the integrated value has been described as an example, but the same effect is obtained even if the configuration is such that the integration is performed instead of the addition, or if the average or the weighted average is calculated. Can be obtained. In addition, for the first utterance of the service user, an explanation has been given by taking as an example a configuration in which all user recognition contents are selected as display targets regardless of the recognition accuracy. The configuration may be such that all user recognition contents are selected as display targets regardless of the recognition accuracy. For example, when the predetermined number of times is set to three times, all user recognition contents are selected as display targets up to three times for the utterance by the service user regardless of the recognition accuracy (integrated value). Then, for the fourth and subsequent utterances by the service user, the integrated value may be compared with the threshold value, and the user recognition content recognized with a higher recognition accuracy than the threshold value may be selected.

  Further, in order to compare the recognition accuracy of the recognition processing by the client terminal 40 with the recognition accuracy of the recognition processing by the recognition server 10, the recognition accuracy of the recognition processing by the client terminal 40 and the recognition accuracy of the recognition processing by the recognition server 10 are compared. It may be normalized. The recognition accuracy for each language may also be normalized. With this configuration, the same threshold value can be used regardless of whether the recognition process is performed by the client terminal 40, the recognition server 10, or the recognition process of any language. it can.

[Embodiment 2]
Another embodiment of the present invention will be described with reference to FIGS. 9 to 11.

  As described above, the recognition process of recognizing the voice content included in the voice information may be executed by the client terminal 40 or the recognition server 10. Therefore, in the present embodiment, a method for facilitating communication between users who use different languages more effectively by storing a database of utterance contents considered to be frequently used in the client terminal 40 will be described. In the present embodiment, the case where the service user is a customer of the store, the service provider is a store clerk, and the client terminal 40 is installed in the store will be described as an example.

(Database stored in terminal storage unit 52)
FIG. 9 is an example of a database (recognition content candidate list) stored in the terminal storage unit 52 according to the second embodiment of the present invention. As shown in FIG. 9, in the terminal storage unit 52, the utterance contents which are translated into English, Chinese, and Japanese respectively, which are considered to be frequently used in the place where the client terminal 40 is installed ( Hereinafter, the “recognition corpus” is stored in association with each other.

(When the service user speaks)
In the present embodiment, a case where a service user makes a speech will be described with reference to FIG. FIG. 10 is a flowchart showing the flow of processing of the client terminal 40 according to the second embodiment of the present invention. In the present embodiment, the recognition process in step S62 described above is executed in the client terminal 40. First, as described with reference to FIG. 6, the voice information acquisition unit 442 acquires user voice information, and outputs the acquired user voice information to the voice recognition unit 444.

(Step S90)
Similar to step S62 described above, the voice recognition unit 444 recognizes the voice content indicated by the acquired user voice information as a plurality of languages. For example, the pronunciation "Woshan Yao Coffee" issued by the service user is recognized as English, Chinese, and Korean.

(Step S92)
Subsequently, the voice recognition unit 444 determines whether or not, among the recognition contents recognized as each language, there is a recognition contents that matches the recognition corpus stored in the terminal storage unit 52.

  For example, the voice recognition unit 444 determines whether or not there is a recognition corpus that matches the recognition content “What are y'all coffee” that recognizes “Woshan Yao coffee” as English. Since there is no recognition corpus that matches “What are y'all coffee” in the terminal storage unit 52, it is determined that there is no recognition corpus that matches the recognition content “What are y'all coffee” in the speech recognition result.

  Subsequently, the voice recognition unit 444 recognizes “Woshang Yao Coffee” as Chinese.

It is determined whether there is a recognition corpus that matches with. In the terminal storage unit 52, there is a Chinese recognition corpus "Impression [Drink]" in which the item "No." is associated with "6", and the item "No." is associated with "201" in China. Word recognition corpus

There is. Therefore, the voice recognition unit 444 determines the recognition content.

It is determined that there is a recognition corpus that matches with.

  Similarly, the voice recognition unit 444 also determines whether or not the recognition content recognized as another language matches the recognition corpus stored in the terminal storage unit 52.

(Step S94)
When it is determined in step S92 that "there is no recognition content that matches the recognition corpus stored in the terminal storage unit 52 among the recognition contents recognized as each language" (step S92: NO), the voice recognition unit 444 A recognition result including the user recognition content indicating the recognized content and the recognition accuracy indicating the accuracy of the recognition process in step S90 is output to the support server 30 via the client terminal communication unit 42. In this case, the support server 30 executes the processing from step S68 onward in FIG. Further, in this case, for example, when the recognition accuracy acquired by the support server 30 is lower than the predetermined recognition accuracy, the client terminal 40 may be requested to output the user voice information, and the processing from step S56 may be executed. ..

(Step S96)
On the other hand, when it is determined in step S92 that "there are recognition contents that match the recognition corpus stored in the terminal storage unit 52 among the recognition contents recognized as each language" (step S92: YES), the voice recognition unit 444 selects the Japanese corpus corresponding to the matched recognition corpus (in other words, the Japanese recognition corpus associated with the matched recognition corpus in the database shown in FIG. 9).

  For example, the voice recognition unit 444, the voice recognition unit 444,

Since it is determined that there is a recognition corpus that matches

Japanese recognition corpus associated with "I want coffee" (more specifically, item "No." is associated with "6" "I want [Drink]" and item " No. ”is associated with“ 201 ”and is“ coffee ”).

(Step S98)
The voice recognition unit 444 outputs the matched recognition corpus as the user recognition content and the selected Japanese recognition corpus as the user translation content to the display information acquisition unit 448, and the display information acquisition unit 448 performs the display processing. To execute.

  In the display processing, the voice recognition unit 444 outputs the user recognition content and the user translation content to the support server 30 via the client terminal communication unit 42, and the display information acquisition unit 448 causes the client terminal communication unit 42 to operate. The configuration may be such that display information is acquired from the support server 30 via the support server 30 and the acquired display information is referred to. In this case, the support server 30 executes the process of step S80 described above.

  FIG. 11 shows an example of an image displayed on the display unit 46 when the process of step S98 is executed. FIG. 11 is a diagram showing an example of an image displayed on the display unit 46 in Embodiment 2 of the present invention, (a) is an image displayed on the first area 46a, and (b) is It is an image displayed in the second area 46b.

  As shown in FIG. 11A, the image displayed in the first area 46a includes the text 640 including the user recognition content recognized as Chinese. Then, as shown in FIG. 11B, in the image displayed in the second area 46b, the text 740 including the user translation content obtained by translating the user recognition content recognized as Chinese into Japanese. It is included.

  As described above, in the communication support system 1 according to the present embodiment, the user recognition content to be displayed is selected by referring to the predetermined recognition content candidate list (database). With this configuration, in the communication support system 1, the processing by the recognition server 10 can be omitted. Therefore, the communication support system 1 can reduce the amount of communication between devices. Further, when the recognition content candidate list also includes the user translation content obtained by translating the selected user recognition content, the processing by the translation server 20 can be omitted.

  In addition, in step S90 described above, the voice recognition unit 444 may calculate a score indicating the degree of matching with the recognition corpus, in addition to the processing described above. In this case, similarly to the above-described embodiment, the selection unit 344 determines whether the score is higher than the threshold value. Then, the selection unit 344 selects the recognition content recognized by the score higher than the threshold. When there are a plurality of recognition contents recognized by the score higher than the threshold value, in step S96, the voice recognition unit 444 selects a Japanese corpus corresponding to each of the plurality of recognition contents. In this case, the display mode determination unit 346 preferably has a configuration in which the user recognition content and the user translation content are displayed in order from the highest score. With this configuration, the communication support system 1 can more effectively prevent erroneous recognition of the language used by the service user.

[Embodiment 3]
Another embodiment of the present invention will be described with reference to FIG.

  In the above-described embodiment, if there is no user recognition content recognized with a recognition accuracy higher than a predetermined threshold value, the user recognition content will not be displayed. Therefore, in the present embodiment, the sequence diagram of FIG. 6 described above is used for a configuration in which a threshold value (hereinafter, referred to as “candidate threshold value”) whose recognition accuracy is lower than a predetermined threshold value (hereinafter, referred to as “adoption threshold value”) is set. It will be explained using.

(Step S52 to Step S66)
Since the process is the same as the process described above, the description thereof will be omitted.

(Step S68)
Similar to the above-described processing, the selection unit 344 selects the user recognition content to be displayed by referring to the recognition result and determining whether it is higher than the adoption threshold. Here, for example, when the recognition accuracies when recognized as English, Chinese, and Korean are “30”, “50”, and “10”, respectively, and the adoption threshold is “60”, the selection unit This means that there is no user recognition content that 344 selects as the user recognition content to be displayed. In this case, the selection unit 344 determines whether or not the recognition accuracy when recognized as English, Chinese, and Korean is higher than the candidate threshold value. For example, when the candidate threshold is “5”, the selection unit 344 selects the user recognition content higher than the candidate threshold. Then, the selection unit 344 outputs the selected user recognition content to the information management unit 342. The selection unit 344 also outputs the recognition result including the selected user recognition content to the display mode determination unit 346.

(Steps S70 to S78)
Since the process is the same as the process described above, the description thereof will be omitted.

(Step S80)
When the user translation content is acquired, the display mode determination unit 346 determines the display mode to be displayed on the display unit 46 of the client terminal 40. Specifically, the display mode determination unit 346 determines the display mode indicating that the recognition accuracy is low because the recognition accuracy of the user recognition content acquired in step S68 is less than or equal to the adoption threshold and higher than the candidate threshold. As an example of the display mode indicating that the recognition accuracy is low, (1) characters are displayed thinly, and (2) texts that imply low recognition accuracy are added, such as “maybe” and “Did you mean”. , And so on. Then, the display mode determination unit 346 outputs the display information including the determined display mode, the user recognition content, and the user translation content to the display information output unit 348.

(Step S82 to Step S86)
Since the process is the same as the process described above, the description thereof will be omitted.

  12A and 12B show examples of images displayed on the display unit 46 in step S86. FIG. 12 is a diagram showing an example of an image displayed on the display unit 46 in Embodiment 3 of the present invention, (a) is an image displayed on the first area 46a, and (b) is It is an image displayed in the second area 46b.

  As shown in (a) of FIG. 12, in the image displayed in the first area 46a, (1) text 652 including the user recognition content recognized as Chinese, (2) recognition as English The text 654 including the user recognition content, and (3) the text 656 including the user recognition content recognized as Korean are included. Further, the first area 46a includes text 650 that implies that the recognition accuracy is low.

  Further, as shown in FIG. 12B, the image displayed in the second area 46b includes (1) the user translation content obtained by translating the user recognition content recognized as Chinese into Japanese. Included text 752, (2) Text 754 including user translated content translated into Japanese from user recognized content recognized as English, and (3) Translated user recognized content recognized as Korean into Japanese. A text 756 including the translated content of the user is included. The second area 46b also includes text 750 that implies that the recognition accuracy is low.

(When there is no recognition accuracy higher than the candidate threshold)
Further, a case where the recognition accuracy when recognized as English, Chinese, and Korean in step S68 described above is equal to or less than the candidate threshold value will be described using the sequence diagram of FIG. 6 described above.

(Step S52 to Step S66)
Since the process is the same as the process described above, the description thereof will be omitted.

(Step S68)
The selection unit 344 selects the user recognition content to be displayed by referring to the recognition result and determining whether it is higher than the adoption threshold or the candidate threshold. Here, for example, when the recognition accuracies when recognized as English, Chinese, and Korean are “10”, “20”, and “5”, respectively, and the candidate threshold is “30”, the selection unit This means that there is no user recognition content that 344 selects as the user recognition content to be displayed. In this case, the selection unit 344 outputs information indicating that there is no recognition content recognized with the recognition accuracy higher than the candidate threshold value to the display mode determination unit 346.

(Steps S70 to S78)
The process of steps S70 to S78 is not executed, and the process proceeds to step S80.

(Step S80)
The display mode determination unit 346 determines the display mode to be displayed on the display unit 46 of the client terminal 40 with reference to the information acquired in step S68. Specifically, the display mode determination unit 346 indicates that the information acquired in step S68 does not include the recognition content recognized with the recognition accuracy higher than the candidate threshold, and thus the display indicating that the utterance content cannot be recognized is displayed. The mode is decided. Examples of the display mode indicating that the utterance content cannot be recognized include (1) displaying a text indicating that the utterance cannot be recognized, and (2) displaying an image for selecting a language. Then, the display mode determination unit 346 outputs the display information including the determined display mode to the display information output unit 348.

(Step S82 to Step S86)
Since the process is the same as the process described above, the description thereof will be omitted.

  Examples of images displayed on the display unit 46 in step S86 are shown in (c) and (d) of FIG. 12C is an image displayed in the first area 46a, and FIG. 12D is an image displayed in the second area 46b.

  As shown in (c) of FIG. 12, the image displayed in the first area 46a includes text 664 indicating that the image cannot be recognized. The first area 46a also includes a manipulator 662 that receives an operation for selecting a language from a service user, and a text 660 that indicates that the language can be selected by operating the manipulator. ..

  Further, as shown in (d) of FIG. 12, the image displayed in the second area 46b includes the text 760 indicating that the image cannot be recognized. The second area 46b also includes a manipulator 762 that receives an operation for selecting a language from a service provider, and a text 764 that indicates that the language can be selected by operating the manipulator. ..

  The operator may be a displayed GUI (Graphic User Interface) as shown in (c) and (d) of FIG. 12, or may be a switch shape or the like without being limited to a button shape. Good. Alternatively, the operation unit 50 (physical buttons, switches, etc.) provided for the service provider may be used. Further, in the images shown in FIGS. 12A and 12B, the language may be selected by touching the text displayed in each language. For example, in the image shown in FIG. 12A, the client terminal 40 has a configuration in which the language used by the service user is determined to be Chinese when an operation signal indicating that the text 652 is selected is acquired. May be.

  Then, in the communication support system 1, when the language is selected, the language for recognizing the utterance of the service user is determined to be the selected language. Therefore, for example, when the communication support system 1 acquires an operation signal indicating that the text 652 is selected in the image shown in FIG. 12A, the text 752 is displayed in the image shown in FIG. When the operation signal indicating that the flag is selected is acquired, or when the operation signal indicating that the Chinese flag is selected is acquired in the image shown in (c) or (d) of FIG. 12, communication support In the system 1, the language used by the service user is determined to be Chinese.

  As described above, in the communication support system 1 according to the present embodiment, any one of a plurality of languages can be selected as the language according to the input received through the operators (the operation unit 50, the operator 662, and the operator 762). It is determined whether to recognize the user voice content indicated by the user voice information. Therefore, in the communication support system 1, even if the recognition accuracy by the recognition processing is low due to the voice uttered by the service user being small or noise, the service user or the service provider can speak the language. By prompting the operation to be selected, it is possible to facilitate communication between users who use different languages.

[Embodiment 4]
Another embodiment of the present invention will be described with reference to FIG.

  In the above-described embodiment, as shown in (a) of FIG. 4, the display order of the provider translation content obtained by translating the utterance content first uttered by the service provider is not particularly limited, but in the present embodiment, A configuration for displaying the provider translation content according to a predetermined condition will be described with reference to the sequence diagram of FIG. Further, in the present embodiment, the display mode determination unit 346 is configured to refer to the number-of-times information indicating the number of times the recognition process is performed for each language used in the above-described recognition process.

(Steps S2 to S26)
Since the process is the same as the process described above, the description thereof will be omitted.

(Step S28)
The display mode determination unit 346 determines the display mode to be displayed on the display unit 46 of the client terminal 40 when the provider translation content is acquired. Here, the display mode determination unit 346 refers to the number-of-times information for each language and determines the display mode according to the referred number-of-times information. For example, when the number-of-times information acquired by the display mode determination unit 346 indicates that English is used 200 times, Chinese is used 100 times, and Korean is used 50 times in the recognition process, the most frequently used English is used. Is displayed above the first area 46a, and below that, the provider translations translated into Chinese, which are used most frequently, are displayed below the first area 46a. The display mode is determined so as to display the provider translation content translated into Korean. Then, the display mode determination unit 346 outputs the display information including the determined display mode, the provider recognition content, and the provider translation content to the display information output unit 348.

(Steps S30 to S34)
Since the process is the same as the process described above, the description is omitted.

  Examples of images displayed on the display unit 46 in step S34 are shown in FIGS. FIG. 13 is a diagram showing an example of an image displayed on the display unit 46 in Embodiment 4 of the present invention, (a) is an image displayed on the first region 46a, and (b) is It is an image displayed in the second area 46b.

  As shown in (a) of FIG. 13, in the image displayed in the first area 46a, the text 670 including the provider translation content translated into English is displayed on the upper side, and below that, the Chinese language is displayed. A text 672 including the translated content of the provider translated is displayed, and further below that, a text 674 including the translated content of the provider translated into Korean is displayed.

  Further, as shown in FIG. 13B, the image displayed in the second area 46b includes the text 770 including the contents of the recognition by the provider.

(When using a captured image of a service user)
As another example of this embodiment, a configuration using a captured image of a service user will be described. In the communication support system 1 in this example, the display mode determination unit 346 can refer to the determination result of determining the language used by the service user by referring to the captured image of the service user. The process of determining the language used by the service user by referring to the captured image of the service user may be executed by any of the units included in the communication support system 1. It may be configured such that a device different from that makes a determination and the communication support system 1 obtains the determination result.

  In step S28 described above, the display mode determination unit 346 determines the display mode to be displayed on the display unit 46 of the client terminal 40 when the provider translation content is acquired. Here, the display mode determination unit 346 refers to the captured image of the service user, refers to the determination result of determining the language used by the service user, and determines the display mode according to the referenced determination result. .. For example, if the determination result acquired by the display mode determining unit 346 is the determination result that the language used by the service user is English, the display mode determining unit 346 determines that the provider translation has been translated into English. The display mode is determined so that the content is displayed above the first area 46a. Then, the display mode determination unit 346 outputs the display information including the determined display mode, the provider recognition content, and the provider translation content to the display information output unit 348. Even in this example, the image shown in FIG. 13 is displayed on the display unit 46.

  As described above, the communication support system 1 according to the present embodiment displays the translated content obtained by translating the uttered content of the first service provider according to a predetermined condition. With this configuration, in the communication support system 1, the translated content translated into the language that the service user is supposed to use can be displayed conspicuously in the first area 46a, for example. Further, the communication support system 1 can easily make the service user understand that this system is a system for translating the utterance content.

[Embodiment 5]
Another embodiment of the present invention will be described with reference to FIG.

  In the above-described embodiment, the communication support system 1 does not recognize the user recognition content not selected in the selection processing in step S68 by the language in the subsequent processing (in the above-described embodiment 1, Since the recognition content recognized as Korean was not selected in the selection processing, the recognition processing for recognizing the speech content of the service user as Korean was not performed in the processing after the selection processing). In the present embodiment, a configuration will be described in which the recognition process is continuously executed even if the language of the user recognition content is not selected in the selection process in step S68.

  First, in the communication support system 1, the selection unit 344 selects the user recognition content in which the voice content indicated by the user voice information is recognized as Chinese in the selection process. Therefore, the user recognition content recognized as a language other than Chinese is not displayed in the first area 46a of the display unit 46. On the other hand, as described above, in the recognition processing, the voice content indicated by the user voice information is recognized as English, Chinese, and Korean.

  At this time, an example of the image displayed on the display unit 46 is shown in FIGS. FIG. 14 is a diagram showing an example of an image displayed on the display unit 46 in the fifth embodiment of the present invention, (a) is an image displayed on the first area 46a, and (b) is It is an image displayed in the second area 46b.

  As shown in FIG. 14A, the image displayed in the first area 46a includes the text 680 including the user recognition content recognized as Chinese. Further, as shown in FIG. 14B, in the image displayed in the second area 46b, the text 780 including the user translation content obtained by translating the user recognition content recognized as Chinese into Japanese. include.

  The subsequent processing will be described with reference to FIG. 6 described above.

(Step S52)
Similar to the above-described processing, the voice information acquisition unit 442 of the client terminal 40 performs an acquisition process of acquiring user voice information indicating the voice uttered by the service user via the voice input unit 48. Here, for example, when a service user who uses English, which is a language other than the determined language (Chinese), utters “I want tea”, the voice information acquisition unit 442 uses the usage indicating “I want tea”. Person voice information.

(Steps S54 to S60)
Since the process is the same as the process described above, the description is omitted.

(Step S62)
As described above, the recognition server control unit 14 recognizes the voice content indicated by the acquired user voice information as English, Chinese, and Korean. Specifically, the recognition server control unit 14 recognizes the pronunciation “I Want To” as English, Chinese, and Korean.

(Steps S64 to S66)
Since the process is the same as the process described above, the description is omitted.

(Step S68)
Since the selection unit 344 has selected the user recognition content recognized as Chinese as the display target user recognition content, first, the user recognition content recognized as Chinese as the display target user recognition content. Select. Here, when the recognition accuracy recognized as another language is higher than the recognition accuracy recognized as Chinese, the selection unit 344 also selects the user recognition content recognized as the other language. For example, the user recognition content that recognizes "I want tea" as Chinese is "Iwan 7", the recognition accuracy is "20", and the user recognition content that recognizes "I want tea" as English is "I want". "tea", the recognition accuracy is "50", and the user recognition content that recognizes "I want tea" as Korean is

When the recognition accuracy is “10”, the selection unit 344 selects the user recognition content recognized as English as the user recognition content to be displayed in addition to the user recognition content recognized as Chinese. To do.

(Steps S70 to S78)
Since the process is the same as the process described above, the description is omitted.

(Step S80)
When the user translation content is acquired, the display mode determination unit 346 determines the display mode to be displayed on the display unit 46 of the client terminal 40. Here, based on the recognition accuracy of the language (Chinese) of the user recognition content selected as the display target so far, the display mode determination unit 346 determines the language of the user recognition content not selected as the display target so far (English. ) The display mode indicating that the recognition accuracy is higher is determined. As an example of a display mode indicating that the recognition accuracy of the language of the user recognition content selected so far as the display target is higher than the recognition accuracy of the language of the user recognition content not selected as the display target so far, (1) The characters are displayed thinly (displayed in a display mode different from the others), (2) “Midoshite” and “Did you mean”, which have not been selected as display targets so far, but the recognition accuracy is high. For example, you can add text that implies a high price. Then, the display mode determination unit 346 outputs the display information including the determined display mode, the user recognition content, and the user translation content to the display information output unit 348.

(Step S82 to Step S86)
Since the process is the same as the process described above, the description thereof will be omitted.

  Examples of images displayed on the display unit 46 in step S86 are shown in (c) and (d) of FIG. FIG. 14C is an image displayed in the first area 46a, and FIG. 14D is an image displayed in the second area 46b.

  As shown in (c) of FIG. 14, in the image displayed in the first area 46a, in addition to the text 680 described above, (1) a text 682 including user recognition content recognized as Chinese, (2) A text 684 indicating that the recognition accuracy is higher than Chinese recognition accuracy and a text 686 including the user recognition content recognized as English are included. Further, it may be configured to further include a button 688 for accepting an operation to delete the display of the text 684 and the text 686.

  Further, as shown in (d) of FIG. 14, in addition to the text 780 described above, (1) user recognition content recognized as Chinese is displayed in the image displayed in the second area 46b in Japanese. The user translation contents 782 translated into (1), (2) a text 784 implying that the recognition accuracy in Chinese is higher than the recognition accuracy, and the user translation contents 786 translated from the user recognition contents recognized as English into Japanese are included. ing. In addition, a configuration may further include a button 788 that accepts an operation to delete the display of the text 784 and the text 786.

  Here, as described in the third embodiment, when the operation signal acquisition unit 446 acquires an operation signal indicating an operation to change the determined language via the operator, the communication support system 1 Then, the language used by the service user may be changed to a language different from the language of the user recognition content that has been selected as a display target until now. For example, when the service user performs an operation of touching the text 686 on the image shown in FIG. 14C, the communication support system 1 displays the user recognition content recognized as English in the selection process. It may be changed to be selected.

  In addition, if the recognition accuracy of the language of the user recognition content that has not been selected as the display target is higher than the recognition accuracy of the language of the user recognition content that has been selected as the display target, If the number of times (for example, three times) is reached, the communication support system 1 may change the language used by the service user to the language of the user recognition content that has not been selected as the display target until now.

  As described above, in the communication support system 1 according to the present embodiment, the process of recognizing the user voice information as the relevant language is executed in the recognition process even if the language is no longer selected as the user recognition content to be displayed. .. Therefore, in the communication support system 1, for example, (1) when the service user was speaking in Chinese, there were many false recognitions, so the service user changed the language used to English ( 2) Communication is performed even when the language used by the service user has changed, such as when the service user who uses English speaks while talking to the service user who uses Chinese. Can be smoothed.

[Sixth Embodiment]
Another embodiment of the present invention will be described with reference to FIGS.

  In the above-described embodiment, the recognition process is performed without referring to the voice content uttered prior to the recognized voice content, but in the present embodiment, the voice content uttered before the recognized voice content is recognized. The configuration to be referred to will be described. In the present embodiment, the recognition processing in step S62 described above is executed in the client terminal 40, as in the second embodiment described above.

(Utterance sentence database and response database stored in terminal storage unit 52)
FIG. 15 is an example of a database (response content candidate list) stored in the terminal storage unit 52 according to the sixth embodiment of the present invention, (a) is an example of a utterance sentence database, and (b) is a response. It is an example of a database.

(Utterance sentence database)
The terminal storage unit 52 stores an utterance sentence database including utterance contents that are considered to be often used in the place where the client terminal 40 is installed. As shown in FIG. 15A, the utterance sentence database is associated with the item “utterance sentence ID”, the item “utterance sentence type”, and the item “utterance sentence corpus”.

  The item “utterance sentence ID” stores an utterance sentence ID that is an identification number for identifying the associated utterance sentence.

  The item “utterance sentence type” stores an utterance sentence type that is an identification number for identifying the category to which the utterance sentence belongs.

  The item “utterance sentence corpus” stores utterance sentences that are considered to be frequently used. In the utterance sentence database shown in FIG. 15A, a Japanese utterance sentence is stored in the item “utterance sentence corpus”, but the Japanese utterance sentence is stored in English, Chinese, and Korean. The utterance sentence translated to is also stored in association.

(Response database)
Further, the terminal storage unit 52 stores a response database in which a certain utterance sentence and a utterance sentence which is a response to the utterance sentence are associated with each other in the place where the client terminal 40 is installed. As shown in FIG. 15B, the response database is associated with the item “response sentence ID”, the item “utterance sentence type”, the item “condition”, and the item “response utterance sentence type”.

  The item “response sentence ID” stores an identification number for identifying the associated response utterance sentence type.

  Similar to the item “utterance sentence type” stored in the utterance sentence database, the item “utterance sentence type” stores an identification number for identifying the category to which the utterance sentence belongs.

  The item “condition” stores a condition for selecting the associated response utterance sentence type.

  The item “response utterance sentence type” stores an identification number for identifying the category to which the utterance sentence corpus uttered as a response belongs.

(Processing flow of the client terminal 40)
The processing flow of the client terminal 40 in this embodiment will be described with reference to FIG. FIG. 16 is a flowchart showing the processing flow of the client terminal 40 according to the sixth embodiment of the present invention.

  First, according to the sequence diagram of FIG. 3 or FIG. 5 described above, the client terminal 40 acquires the provider voice information indicating “What are you looking for?” From the service provider, and recognizes the provider voice information. The provider recognition content is translated into English, Chinese, and Korean, and the provider recognition content and the provider translation content are displayed. At this time, the image displayed on the display unit 46 is the image of FIG. 4 described above.

  Subsequently, as described with reference to FIG. 6, the voice information acquisition unit 442 acquires the user voice information and outputs the acquired user voice information to the voice recognition unit 444.

(Step S90)
Similar to step S62 described above, the voice recognition unit 444 recognizes the voice content indicated by the acquired user voice information as a plurality of languages. For example, the pronunciation "Woshan Yao Coffee" issued by the service user is recognized as English, Chinese, and Korean.

(Step S100)
Subsequently, the voice recognition unit 444 determines whether or not there are a plurality of user recognition contents having a predetermined recognition accuracy or higher among the user recognition contents recognized as each language. For example, the voice recognizing unit 444 has (1) recognition accuracy of recognizing "Woshan Yao coffee" as English, (2) recognition accuracy of recognizing as Chinese, and (3) recognition accuracy of recognizing as Korean. It is determined whether or not the recognition accuracy is equal to or higher than the predetermined recognition accuracy.

(Step S102)
When it is determined in step S100 that “there are a plurality of user recognition contents having a certain recognition accuracy or higher” (step S100: YES), the voice recognition unit 444 causes the speech recognition unit 444 to immediately (or before, same below) utterance. It is determined whether or not the utterance sentence corpus that matches the provider recognition content of is stored in the utterance sentence database.

  For example, the voice recognition unit 444 determines whether or not the utterance sentence corpus that matches the provider recognition content “Looking for something?” Of the immediately preceding utterance is stored in the utterance sentence database. In the utterance sentence database shown in FIG. 15A, the utterance sentence corpus associated with the utterance sentence ID “1” is “Are you looking for something?”. It is determined that the utterance sentence corpus that matches the provider recognition content "Are you looking for something?" Is stored in the utterance sentence database.

(Step S104)
When it is determined in step S102 that “there is a utterance sentence corpus that matches the provider recognition content of the immediately preceding utterance” (step S102: YES), the voice recognition unit 444 matches the provider recognition content of the immediately preceding utterance. The utterance sentence type associated with the utterance sentence corpus is selected. For example, the voice recognition unit 444 selects the utterance sentence type “1” associated with the utterance sentence corpus that matches the provider recognition content “there you looking for?” Of the utterance immediately before.

(Step S106)
Subsequently, the voice recognition unit 444 selects, in the response database, the response utterance sentence type having the matching condition from the response utterance sentence types associated with the utterance sentence type selected in step S104. For example, the voice recognition unit 444 refers to the condition associated with the response utterance sentence type associated with the utterance sentence type “1” selected in step S104 in the response database illustrated in FIG. 15B. Since the conditions associated with the response utterance sentence type associated with the utterance sentence type “1” are all “− (no condition)”, the voice recognition unit 444 determines that the response utterance sentence types “2”, “3”, Select "4".

(Step S108)
Then, the voice recognition unit 444 determines whether or not there is a utterance sentence corpus that matches the user recognition content in the utterance sentence corpus associated with the response utterance sentence type selected in step S106 in the utterance sentence database. ..

  For example, the voice recognition unit 444 is associated with the utterance sentence types that are the response utterance sentence types “2”, “3”, and “4” selected in step S106 in the utterance sentence database shown in FIG. From the utterance corpus, it is determined whether or not there is a utterance corpus that matches the user recognition content “What are y'all coffee” that recognizes “Woshan Yao coffee” in English. Since there is no utterance sentence corpus that matches the user recognition content “What are y'all coffee” in the utterance sentence database, the voice recognition unit 444 subsequently determines that the user who has recognized “Woshan Yao Coffee” as Chinese. Recognition content

It is determined whether or not there is a utterance sentence corpus that matches with.

  Although not shown in the utterance sentence database shown in FIG.

Matches the utterance sentence corpus associated with the utterance sentence ID “6”, the voice recognition unit 444 determines that there is a utterance sentence corpus that matches the user recognition content. The same processing is executed for the Korean user recognition content, and it is determined that there is no utterance sentence corpus that matches the Korean user recognition content.

(Step S110)
When it is determined in step S108 that “there is a utterance sentence corpus that matches the user recognition content” (step S110: YES), the voice recognition unit 444 determines the user recognition content that matches the utterance sentence corpus in step S108. , Select as the user recognition content to be displayed.

(Step S112)
The voice recognition unit 444 displays the selected user recognition content and the Japanese utterance sentence corpus associated with the utterance sentence corpus that matches the user recognition content in the utterance sentence database as the user translation content and display information. The information is output to the acquisition unit 448, and the display information acquisition unit 448 executes display processing. The display process may be configured to acquire the display information from the support server 30 as in the process described in step S98 of the second embodiment.

(Step S114)
When it is determined in the above-described step S100 that "there is no plurality of user recognition contents having a certain recognition accuracy or higher" (step S100: NO), "the utterance sentence that matches the provider recognition content of the immediately previous utterance" in step S102. If it is determined that there is no corpus (step S102: NO), or if there is no utterance sentence corpus that matches the user recognition content in step S108 (step S108: NO), the voice recognition unit 444. Outputs the recognition result including the user recognition content and the recognition accuracy to the support server 30 via the client terminal communication unit 42 to the support server 30. When the recognition result is acquired, the support server 30 executes the processing of step S68 and the subsequent steps described above.

  An example of an image displayed on the display unit 46 in this embodiment is shown in FIG. FIG. 17 is a diagram showing an example of an image displayed on the display unit 46 in the sixth embodiment of the present invention, (a) is an image displayed on the first area 46a, and (b) is It is an image displayed in the second area 46b, (c) is an image displayed in the first area 46a, and (d) is an image displayed in the second area 46b.

  For example, when the process of step S114 is executed, such as when it is determined in step S100 that “there is no plurality of user recognition contents having a predetermined recognition accuracy or higher” (step S100: NO), (a) of FIG. As shown in FIG. 7, the image displayed in the first area 46a includes (1) a text 690 including the translated content translated into English, and (2) a translated content translated into Chinese. In addition to the text 692 and (3) the text 694 containing the translated content translated into Korean, (1) the text 696 containing the user recognized content as Chinese, (2) the user recognized as English The text 697 including the recognition content and (3) the text 698 including the user recognition content recognized as Korean are included.

  On the other hand, the image displayed in the first area 46a in the display process in step S112 includes the text 696 in addition to the text 690, the text 692, and the text 694 described above, as shown in (c) of FIG. Text 697 and text 697 are not included.

  In addition, in the image displayed in the second area 46b, as shown in (b) of FIG. 17, in addition to the text 790 including the contents of provider recognition, (1) user recognition recognized as Chinese Text 796 containing the user translated content translated into Japanese, (2) Text 797 containing the user translated content translated into Japanese from the user recognized content recognized as English, and (3) As Korean It contains the user translation contents that are translated from the recognized user recognition contents into Japanese.

  On the other hand, the image displayed in the second area 46b in the display processing in step S112 includes the text 796 in addition to the above-described text 790, as shown in FIG. Text 798 is not included.

  As described above, in the communication support system 1 according to the present embodiment, in the process of selecting the user recognition content in which the service user's utterance content is recognized, the utterance content before the service user utters is referred to and displayed. Select the user recognition content of. Therefore, the communication support system 1 can select the user recognition content that suits the flow of conversation and can recognize the user in the language used by the service user, so that communication between users who use different languages can be improved. Can be smooth.

[Embodiment 7]
Another embodiment of the present invention will be described with reference to FIGS. 18 and 19.

  In the above-described embodiment, the communication in which the service provider and the service user have a conversation has been described as an example, but a case where the service provider is the client terminal 40 will be described.

(Processing flow of the client terminal 40)
The process flow of the client terminal 40 in this embodiment will be described with reference to FIG. FIG. 18 is a flowchart showing the processing flow of the client terminal 40 according to the seventh embodiment of the present invention.

(Step S90)
The voice recognition unit 444 recognizes the voice content indicated by the acquired user voice information as a plurality of languages. For example, the pronunciation "Woshan Yao Coffee" issued by the service user is recognized as English, Chinese, and Korean.

(Step S120)
The voice recognition unit 444 selects the user recognition content to be displayed by referring to the recognition accuracy indicating the accuracy of the recognition processing in step S90. For example, when the recognition accuracy of recognizing “Woshan Yao Coffee” as Chinese is higher than a predetermined threshold and the recognition accuracy of recognizing as English and Korean is equal to or lower than the predetermined threshold, the selection unit 344 recognizes as Chinese. User recognition content

Select.

(Step S122)
Subsequently, the voice recognition unit 444 determines whether or not the utterance sentence database stored in the terminal storage unit 52 has the utterance sentence corpus that matches the user recognition content selected in step S120. For example, in the example of the utterance sentence database shown in FIG. 15A, the user recognition content selected in step S120.

Matches the utterance sentence corpus “Imoko [Drink]” associated with the utterance sentence ID “6”, the voice recognition unit 444 determines that there is a utterance sentence corpus that matches.

(Step S124)
When it is determined in step S122 that “there is a matching utterance sentence corpus” (step S122: YES), the voice recognition unit 444 selects the utterance sentence type associated with the matching utterance sentence corpus. For example, user recognition content

Matches the utterance sentence corpus “Imoko [Drink]” associated with the utterance sentence ID “6”, the voice recognition unit 444 causes the speech recognition unit 444 to associate the utterance sentence type “3” with the utterance sentence ID “6”. Select.

(Step S126)
The voice recognition unit 444 refers to the response database and selects a response utterance sentence type that satisfies the condition from the response utterance sentence types associated with the utterance sentence type selected in step S124. For example, the conditions associated with the utterance sentence type “3” selected in step S124 are (1) [Drink] is in stock, (2) [Drink] is out of stock, and (3) Other, ( 1) When [Drink] satisfies the stock, the voice recognition unit 444 selects the response utterance sentence type “201” associated with the condition.

(Step S128)
The voice recognition unit 444 refers to the utterance sentence database and selects the utterance sentence corpus to be displayed as the provider translation content from the utterance sentence corpus associated with the response utterance sentence type selected in step S126. For example, the voice recognition unit 444 selects “[Drink] present or less” from the utterance sentence corpus associated with the utterance sentence type “201”.

(Step S130)
The voice recognition unit 444 outputs the user recognition content selected in step S120 and the provider translation content selected in step S128 to the display information acquisition unit 448, and the display information acquisition unit 448 executes the display process. The display process may be configured to acquire the display information from the support server 30 as in the process described in step S98 of the second embodiment.

(Step S132)
On the other hand, when it is determined in step S122 that “there is no matching utterance sentence corpus” (step S122: NO), the voice recognition unit 444 displays the translation content of the provider indicating that the recognition cannot be performed in the display information acquisition unit. Then, the display information acquisition unit 448 executes display processing. The display process may be configured to acquire the display information from the support server 30 as in the process described in step S98 of the second embodiment.

  FIG. 19 shows an example of the image displayed on the display unit 46 in step S130. FIG. 19 is a diagram showing an example of an image displayed on the display unit 46 in the seventh embodiment of the present invention, (a) is an example of the displayed image, and (b) is the displayed image. Is another example of.

  As shown in (a) of FIG. 19, in the display unit 46, the upper side is the first area 46a, (1) the text 800 including the user recognition content recognized as Chinese, and (2) the recognition as English. A text 802 including the recognized user recognition content and (3) a text 804 including the user recognition content recognized as Korean are included. Further, the display unit 46 includes the text 810 including the content of the utterance of the provider selected in step S128 described above, with the lower side as the second area 46b.

  Further, for example, in step S90, the speech content uttered by the service user is recognized as English, Chinese, and Korean, and in step S122, there is a speech corpus that matches the user recognition content recognized as Chinese. If there is no utterance corpus that matches the user recognition contents recognized as English, Korean, and English, in addition to the above-described text 800, text 802, text 804, and text 810, as shown in FIG. The second area 46b indicates that the text 800, the text 802, the text 804, and the text 820, the text 822, the text 824, the text 830, and the text 820 that cannot be recognized as English are included, respectively. The configuration may include the English text 832 and the Korean text 834 indicating that the text cannot be recognized as Korean.

  As described above, in the communication support system 1 according to the present embodiment, the user recognition is performed by referring to the predetermined utterance sentence database and response database (list of response content candidates) corresponding to the user recognition content to be displayed. The response content to the content is determined, the user recognition content to be displayed is displayed in the first area 46a of the display unit 46, and the response content is displayed in the second area 46b of the display unit 46. With this configuration, in the communication support system 1, the client terminal 40 can respond to the utterance of the service user as a service provider.

  Further, in the communication support system 1 according to the present embodiment, when the predetermined utterance sentence database and the response database do not have the response content corresponding to the user recognition content, a message indicating that the user voice content cannot be recognized is displayed. To do. Therefore, the communication support system 1 can notify the user that the user's utterance could not be recognized.

[Embodiment 8]
Another embodiment of the present invention will be described with reference to FIG.

  In the present embodiment, an image displayed when the communication support system 1 recognizes that the utterance content of the service user includes a product name such as “coffee” will be described. In the present embodiment, the terminal storage unit 52 stores a database including information about products, and the voice recognition unit 444 can refer to the database.

  When the voice recognition unit 444 determines in step S122 described above that the user recognition content selected in step S120 includes the product name, the voice recognition unit 444 acquires information about the product from the terminal storage unit 52.

  For example, in step S122, the user recognition content selected in step S120

Matches the utterance sentence corpus “Isoyo [Drink]” associated with the utterance sentence ID “6”, and “[Drink]” is

If the voice recognition unit 444 determines that

The information about the information is acquired from the terminal storage unit 52. Then, in step S130, in addition to the selected user recognition content and the selected provider translation content, it is acquired.

The information regarding the output is output to the display information acquisition unit 448. In this case, an example of the image displayed on the display unit 46 is shown in FIG. FIG. 20 is a diagram showing an example of an image displayed on the display unit 46 in the eighth embodiment of the present invention.

  As shown in FIG. 20, on the display unit 46, (1) a text 840 including the user recognition content recognized as Chinese, (2) a text 842 including the user recognition content recognized as English, and ( 3) The text 844 including the user recognition content recognized as Korean is included. Further, the image displayed by the display unit 46 includes information 846 regarding the product determined to be included in the user recognition content in step S122.

  As described above, in the communication support system 1 according to the present embodiment, when the user recognition content includes the product name, the information about the product is displayed on the display unit 46. Therefore, the communication support system 1 can present the service user with information on the product name issued by the service user.

  In addition, although the case where the service provider is the client terminal 40 has been described in the present embodiment, as described in the first to sixth embodiments, even when the service provider is, for example, a clerk, communication is performed. The support system 1 may display the information about the product on the display unit 46 (at least one of the first region 46a and the second region 46b).

[Embodiment 9]
The control blocks of the recognition server 10, the translation server 20, the support server 30, and the client terminal 40 (in particular, the recognition server control unit 14, the translation server control unit 24, the support server control unit 34, and the client terminal control unit 44) are integrated circuits. It may be realized by a logic circuit (hardware) formed in an (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit).

  In the latter case, the recognition server 10, the translation server 20, the support server 30, and the client terminal 40 have a CPU that executes the instructions of a program that is software that realizes each function, and the program and various data are a computer (or CPU). A ROM (Read Only Memory) or a storage device (these are referred to as a “recording medium”) recorded in a readable manner, a RAM (Random Access Memory) for expanding the program, and the like are provided. Then, the computer (or CPU) reads the program from the recording medium and executes the program to achieve the object of the present invention. As the recording medium, a “non-transitory tangible medium”, for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

[Embodiment 10]
In each of the above-described embodiments, an example in which a plurality of devices (recognition server 10, translation server 20, support server 30, and client terminal 40) are used has been described, but each function of each device is provided by one device or more devices. It may be configured to be included in the device. For example, the database (including the utterance sentence database and the response database) stored in the terminal storage unit 52 in the above-described embodiment may be stored in the support server 30 or the recognition server 10. ..

  Further, in each of the above-described embodiments, an example in which a plurality of servers (recognition server 10, translation server 20, and support server 30) are used has been described, but each function of each server may be one server or more servers. The configuration may be provided. Then, when a plurality of servers are applied, each server may be managed by the same business operator or may be managed by different business operators.

[Embodiment 11]
Each block of the recognition server 10, the translation server 20, the support server 30, and the client terminal 40 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or a CPU (Central Processing). Unit) and may be realized by software. In the latter case, each of the recognition server 10, the translation server 20, the support server 30, and the client terminal 40 can be configured using a computer (electronic computer) as shown in FIG.

  FIG. 21 is a block diagram exemplifying the hardware configuration of a computer 910 that can be used as the recognition server 10, the translation server 20, the support server 30, and the client terminal 40. The computer 910 includes an arithmetic unit 912, a main storage device 913, an auxiliary storage device 914, an input / output interface 915, and a communication interface 916 that are connected to each other via a bus 911. The arithmetic unit 912, the main storage device 913, and the auxiliary storage device 914 may be, for example, a CPU, a RAM (random access memory), and a hard disk drive. The input / output interface 915 is connected to an input device 920 for the user to input various information to the computer 910 and an output device 930 for the computer 910 to output various information to the user. The input device 920 and the output device 930 may be built in the computer 910 or may be connected (externally attached) to the computer 910. For example, the input device 920 may be a keyboard, a mouse, a touch sensor, etc., and the output device 930 may be a display, a printer, a speaker, etc. Further, a device having both functions of the input device 920 and the output device 930, such as a touch panel in which a touch sensor and a display are integrated, may be applied. The communication interface 916 is an interface for the computer 910 to communicate with an external device.

  The auxiliary storage device 914 stores various programs for operating the computer 910 as the recognition server 10, the translation server 20, the support server 30, and the client terminal 40. Then, the arithmetic unit 912 expands the program stored in the auxiliary storage device 914 onto the main storage device 913 and executes the instructions included in the program, thereby causing the computer 910 to operate the recognition server 10 and the translation server 20. , The support server 30, and the client terminal 40 function as each unit. The recording medium for recording information such as programs provided in the auxiliary storage device 914 may be any computer-readable “non-transitory tangible medium”, and examples thereof include tape, disk, card, semiconductor memory, and programmable logic. It may be a circuit or the like.

  Further, the program may be acquired from outside the computer 910, and in this case, it may be acquired via an arbitrary transmission medium (communication network, broadcast wave, or the like). The present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention. Furthermore, new technical features can be formed by combining the technical means disclosed in each of the embodiments.

1 Communication Support System 2 Network 10 Recognition Server 12 Recognition Server Communication Unit 14 Recognition Server Control Unit (Control Unit)
20 translation server 22 translation server communication unit 24 translation server control unit (control unit)
30 Support Server 32 Support Server Communication Unit 34 Support Server Control Unit (Control Unit)
342 Information management unit 344 Selection unit 346 Display mode determination unit 348 Display information output unit 40 Client terminal 42 Client terminal communication unit 44 Client terminal control unit (control unit)
442 voice information acquisition unit 444 voice recognition unit 446 operation signal acquisition unit 448 display information acquisition unit 450 display control unit 46 display unit 46a first area 46b second area 48 voice input unit 48a service user side voice input unit 48b service Provider side voice input unit 50 Operation unit 52 Terminal storage unit

Claims (12)

A communication support system comprising: a display section having a first area for a first user and a second area for a second user; a voice input section; and a control section,
The control unit is
Obtaining first voice information indicating the voice of the first user via the voice input unit,
A recognition process is performed for recognizing the first voice content indicated by the first voice information as each of a plurality of languages,
From the first recognition content indicating the recognition content recognized as each of the plurality of languages, the recognition accuracy indicating the accuracy of the recognition process recognized in each of the plurality of languages is referred to, and each of the plurality of languages is referenced. Among the recognition contents recognized as, the selection process of selecting the recognition content in the language recognized with a recognition accuracy higher than a predetermined threshold as the first recognition content to be displayed,
As the first recognition content of the display target, the recognition content in the language recognized with the recognition accuracy higher than the predetermined threshold value is displayed in the first area of the display unit,
A communication support system characterized by that. The control unit is
If the content selected as the first recognition content by the selection processing is different from the content currently displayed in the first area after the voice input is started, the first processing is performed by the selection processing. The content displayed in the first area of the display unit is updated so that the content selected as the recognition content of the display unit is displayed in the first area of the display unit. Support system. The control unit is
With reference to the recognition accuracy, a display mode in which the first recognition content of the display target is displayed in the first area of the display unit is determined,
The communication support system according to claim 1 or 2, characterized in that. The control unit is
Acquire the first translation content obtained by translating the first recognition content of the display target,
Displaying the first translation content in the second area of the display unit,
Communication support system according to any one of claim 1 to 3, characterized in that. The control unit is
Acquiring second voice information indicating the voice of the second user via the voice input unit,
A recognition process for recognizing the second voice content indicated by the second voice information is performed,
Acquiring second translation content obtained by translating the second recognition content recognized by the recognition processing into the plurality of languages,
Displaying the second translation content in the first area of the display unit,
Communication support system according to any one of claim 1 to 4, characterized in that. Further equipped with controls
The control unit is
According to the input received via the operator, it is determined which language of the plurality of languages is to recognize the first voice content indicated by the first voice information,
Communication support system according to any one of claim 1 to 5, characterized in that. The control unit is
The response content for the first recognition content is determined by referring to a predetermined response content candidate list corresponding to the first recognition content of the display target,
The first recognition content of the display target is displayed in the first area of the display unit, and the response content is displayed in the second area of the display unit,
Communication support system according to any one of claim 1 to 3, characterized in that. The control unit is
When there is no response content for the first recognition content in the predetermined response content candidate list,
The communication support system according to claim 7 , wherein a message indicating that the first voice content cannot be recognized is displayed. The first user is a service user and the second user is a service provider,
Communication support system according to any one of claims 1-8, characterized in that. The first user is a service provider and the second user is a service user.
Communication support system according to any one of claims 1-8, characterized in that. An acquisition step of acquiring first voice information indicating the voice of the first user;
A recognition step of recognizing the first voice content indicated by the first voice information as each of a plurality of languages;
From the first recognition content indicating the recognition content recognized as each of the plurality of languages, the recognition accuracy indicating the accuracy of the recognition process recognized in each of the plurality of languages is referred to, and each of the plurality of languages is referenced. A selection step of selecting, as the first recognition content to be displayed, the recognition content in the language recognized with a recognition accuracy higher than a predetermined threshold value among the recognition content recognized as
As a first recognition content of the display target, a display step of displaying recognition content in a language recognized with a recognition accuracy higher than the predetermined threshold value in a first area for a first user , respectively .
A communication support method comprising: A program for causing a computer to function as a communication support system including a display unit having a first area for a first user and a second area for a second user, a voice input unit, and a control unit. So, in the above control unit,
An acquisition process for acquiring first voice information indicating the voice of the first user via the voice input unit;
Recognition processing for recognizing the first voice content indicated by the first voice information as each of a plurality of languages,
From the first recognition content indicating the recognition content recognized as each of the plurality of languages, the recognition accuracy indicating the accuracy of the recognition process recognized in each of the plurality of languages is referred to, and each of the plurality of languages is referenced. A selection process of selecting, as the first recognition content to be displayed, the recognition content in the language recognized with a recognition accuracy higher than a predetermined threshold value out of the recognition content recognized as
As the first recognition content of the display target, a display process of displaying recognition content in a language recognized with a recognition accuracy higher than the predetermined threshold value in the first area of the display unit,
A program characterized by causing to execute.