JP2014150442A - Telephone conversation system and telephone conversation relay method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to understand speech content of a telephone conversation partner more easily.SOLUTION: The telephone conversation system comprises: a first telephone conversation relay section for receiving voice and video transmitted from a first telephone conversation terminal; a first voice recognition section for converting voice received by the first telephone conversation relay section into text data to generate text video which indicates the text data; a second telephone conversation relay section for receiving the voice and video transmitted from a second telephone conversation terminal; a second voice recognition section for converting the voice received by the second telephone conversation relay section into the text data to generate text video which indicates the text data; and a synthesis section for synthesizing the video received by the first telephone conversation relay section, the video received by the second telephone conversation relay section, the text video generated by the first voice recognition section, and the text video generated by the second voice recognition section to generate a synthesized video.

Description

  The present invention relates to a technology of a call device for making a call.

  In recent years, ICT (Information and Communication Technology) services have been developed. As a specific example, for example, a videophone terminal has been proposed that can convey a user's feelings and impressions to a partner in an easily understandable manner (see Patent Document 1).

JP 2009-112027 A

However, the spread of ICT services remains limited. For example, it is hard to say that ICT services are sufficiently popular among elderly people.
Elderly people have various handicap due to aging. This has hindered the use of the ICT service. For example, there is a problem that it is difficult to hear the content of the other party's utterance due to a decline in hearing.

  In view of the above circumstances, an object of the present invention is to provide a technique that makes it easier to understand the utterance content of a call partner.

  One aspect of the present invention is a first call relay unit that receives voice and video transmitted from a first call terminal, converts voice received by the first call relay unit into text data, and converts the text data into A first voice recognition unit that generates a text image to represent, a second call relay unit that receives voice and video transmitted from the second call terminal, and a voice received by the second call relay unit is converted into text data A second voice recognition unit that generates a text video representing the text data, a video received by the first call relay unit, a video received by the second call relay unit, and the first voice recognition A synthesis unit that generates a synthesized video by synthesizing the text video generated by the unit and the text video generated by the second voice recognition unit, Call relay unit transmits the synthesized image to the first call terminal, the second call relay unit transmits the combined image to the second call terminal, a call system.

  One aspect of the present invention is the call system described above, wherein a first input unit that receives an input of a text image generated by the first voice recognition unit, and a voice and a video received by the first call relay unit A second input unit that accepts input, a third input unit that accepts input of voice and video received by the second call relay unit, and a second input unit that accepts input of text video generated by the second voice recognition unit A four-input unit, and the synthesizing unit generates the synthesized video by arranging each video input to the first to fourth input units in a predetermined screen area.

  One aspect of the present invention is a first call receiving step for receiving audio and video transmitted from a first call terminal, and converting voice received by the first call receiving step into text data, A first voice recognition step for generating a text image to represent, a second call reception step for receiving voice and video transmitted from the second call terminal, and converting the voice received by the second call reception step into text data A second voice recognition unit that generates a text video representing the text data, a video received by the first call reception step, a video received by the second call reception step, and the first voice recognition The text image generated by the step is synthesized with the text image generated by the second speech recognition step. And a first transmission step for transmitting the synthesized video to the first call terminal, and a second transmission step for transmitting the synthesized video to the second call terminal. It is a relay method.

  According to the present invention, it is possible to more easily understand the utterance content of the other party.

1 is a system configuration diagram of a call system 100. FIG. 6 is a schematic diagram illustrating an outline of processing of a combining unit 60. FIG. It is the schematic which shows the specific example of a synthetic | combination image | video. 4 is a sequence diagram showing a specific example of a flow of processing when a call session is established in the call system 100. FIG. 4 is a sequence diagram showing a specific example of a call processing flow in call system 100. FIG. It is a system configuration figure of the 1st modification (call system 100a) of call system 100. It is a system configuration figure of the 2nd modification (call system 100b) of call system 100. It is the schematic which shows the specific example of the synthetic | combination image | video produced | generated in the telephone system 100b.

Hereinafter, a call system according to an embodiment of the present invention will be described.
FIG. 1 is a system configuration diagram of the call system 100. The call system 100 includes two call terminals 10 (10-1, 10-2) and a relay device 90. The call terminal 10 and the relay device 90 are connected to each other via a network 11 (11-1, 11-2) so that bidirectional communication is possible.

The call terminal 10 is operated by a user who makes a call. The call terminal 10 includes a voice input unit, an imaging unit, a voice output unit, and a display unit.
The voice input unit is a voice input device such as a microphone or a receiver, and inputs the voice of the user (speaker) of the call terminal 10. The voice input unit may be an interface for connecting the voice input device to the call terminal 10. In this case, the voice input unit inputs the voice signal generated by the voice input device to the call terminal 10.

  The imaging unit is an imaging device such as a camera, and images the face of the user (speaker) of the call terminal 10. The imaging unit may be an interface for connecting the imaging device to the call terminal 10. In this case, the imaging unit inputs the video signal generated by the imaging device to the call terminal 10.

  The audio output unit is an audio output device such as a speaker, and outputs the uttered voice of the user who interacts with the call terminal 10. The audio output unit may be an interface for connecting the audio output device to the call terminal 10. In this case, the voice output unit generates an electrical signal representing the speech voice and outputs it to the voice output device.

  The display unit is an image display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro Luminescence) display. The display unit displays the video data generated by the relay device 90. The display unit may be an interface for connecting the image display device to the call terminal 10. In this case, a display part produces | generates the video signal for displaying the video data produced | generated by the relay apparatus 90, and outputs a video signal to the image display apparatus connected to self.

  The call terminal 10 enables communication with the relay device 90 by performing communication with the predetermined protocol via the network 11. For example, the call terminal 10 establishes a call session with the relay device 90 by operating based on SIP (Session Initiation Protocol). The call terminal 10-1 and the call terminal 10-2 can talk with each other by establishing a call session with the relay device 90. That is, when the call terminal 10-1 and the call terminal 10-2 each establish a call session with the relay device 90, the audio input by the audio input unit of the call terminal 10-1 and the video taken by the imaging unit Is output at the call terminal 10-2. Similarly, the voice input by the voice input unit of the call terminal 10-2 and the video taken by the imaging unit are output at the call terminal 10-1. Therefore, the user of the call terminal 10-1 and the user of the call terminal 10-2 can make a call while watching the video respectively.

  The relay device 90 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus, and executes a relay program. The relay device 90 executes the relay program, thereby performing a plurality of call relay units 20 (20-1, 20-2), a plurality of splitters 30 (30-1, 30-2), and a plurality of voice recognition units 40 ( 40-1, 40-2), and functions as a device including the synthesizing device 70. Note that all or part of the functions of the relay device 90 may be realized by using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). The relay program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system.

  The call relay unit 20 (20-1, 20-2) enables communication with the call terminal 10 by performing communication with the predetermined protocol via the network 11. For example, the call relay unit 20 establishes a call session with the call terminal 10 by operating based on SIP.

  The call relay unit 20-1 establishes a call session with the call terminal 10-1 via the network 11-1. The call relay unit 20-1 outputs the voice received from the call terminal 10-1 to the splitter 30-1. The call relay unit 20-1 outputs the video received from the call terminal 10-1 to the second input unit 52. The call relay unit 20-2 establishes a call session with the call terminal 10-2 via the network 11-2. The call relay unit 20-2 outputs the voice received from the call terminal 10-2 to the splitter 30-2. The call relay unit 20-2 outputs the video received from the call terminal 10-2 to the third input unit 53. In addition, the call relay unit 20-1 transmits the synthesized video and audio output by the synthesis unit 60 to the call terminal 10-1 via the network 11-1. The call relay unit 20-2 transmits the synthesized video and audio output by the synthesis unit 60 to the call terminal 10-2 via the network 11-2.

  The splitter 30 (30-1, 30-2) distributes the sound output by the call relay unit 20 to a plurality of output destinations. The splitter 30-1 distributes the voice output by the call relay unit 20-1 to the voice recognition unit 40-1 and the second input unit 52. The splitter 30-2 distributes the voice output by the call relay unit 20-2 to the voice recognition unit 40-2 and the third input unit 53.

  The voice recognition unit 40 (40-1, 40-2) converts the content of the input voice into text data. Then, the voice recognition unit 40 generates a video (text video) displaying characters representing text data. The voice recognition unit 40-1 generates a text video based on the voice distributed from the splitter 30-1 and outputs it to the first input unit 51. The voice recognition unit 40-2 generates a text video based on the voice distributed from the splitter 30-2 and outputs the text video to the fourth input unit 54.

  The combining device 70 includes a first input unit 51, a second input unit 52, a third input unit 53, a fourth input unit 54, and a combining unit 60. The first input unit 51 inputs the text image output from the voice recognition unit 40-1 to the synthesis unit 60. The second input unit 52 inputs the audio distributed from the splitter 30-1 and the video output from the call relay unit 20-1 to the synthesis unit 60. The third input unit 53 inputs the audio distributed from the splitter 30-2 and the video output from the call relay unit 20-2 to the synthesis unit 60. The fourth input unit 54 inputs the text image output from the voice recognition unit 40-2 to the synthesis unit 60.

  The synthesizing unit 60 generates a synthesized video by synthesizing each video input by the first input unit 51 to the fourth input unit 54. The synthesizing unit 60 outputs the synthesized video to both the call relay unit 20-1 and the call relay unit 20-2. The synthesizing unit 60 outputs the voice input by the second input unit 52 to the call relay unit 20-2, and outputs the voice input by the third input unit 53 to the call relay unit 20-1.

  FIG. 2 is a schematic diagram showing an outline of processing of the synthesis unit 60. The synthesizing unit 60 divides one video plane into a plurality of areas, and generates a synthesized video by arranging video and text video in each area. In the specific example shown in FIG. 2, one image plane is divided into four areas.

  The synthesizing unit 60 arranges a text image related to the voice on the side of one call terminal 10 (for example, the call terminal 10-1) in the first area 81 located in the lower left. That is, the synthesis unit 60 arranges the text image input by the first input unit 51 in the first area 81. The synthesizing unit 60 arranges an image on the side of one call terminal 10 (for example, the call terminal 10-1) in the second region 82 located at the upper left. That is, the composition unit 60 arranges the video input by the second input unit 52 in the second region 82.

  The synthesizing unit 60 arranges the video on the other call terminal 10 (for example, call terminal 10-2) side in the third area 83 located in the upper right. That is, the composition unit 60 arranges the video input by the third input unit 53 in the third region 83. The synthesizing unit 60 arranges a text image related to the voice on the other call terminal 10 (for example, the call terminal 10-2) side in the fourth area 84 located in the lower right. That is, the composition unit 60 arranges the text image input by the fourth input unit 54 in the fourth area 84.

  FIG. 3 is a schematic diagram illustrating a specific example of a composite video. A video is arranged in each of the first area 81 to the fourth area 84 to generate a composite video as shown in FIG. As shown in FIG. 3, the face of the user of the call terminal 10-1 is displayed in the second area 82, and the utterance content of the user of the call terminal 10-1 is displayed as text in the first area 81. Further, the face of the user of the call terminal 10-2 is displayed in the third area 83, and the utterance content of the user of the call terminal 10-2 is displayed as text in the fourth area 84.

  FIG. 4 is a sequence diagram showing a specific example of the flow of processing when a call session is established in the call system 100. FIG. 4 shows a process flow until a call session is established between the call terminal 10 operated by Mr. A and the call terminal 10 operated by Mr. B.

  First, Mr. A operates the call terminal 10 to input a call instruction to Mr. B (step S101). The call terminal 10 that has received the call instruction to Mr. B transmits a call request to the call connection device (step S102). The call request includes identification information indicating a plurality of call terminals 10 that establish a call session. The call request transmitted in step S102 includes the identification information of Mr. A's call terminal 10 and the identification information of Mr. B's call terminal 10.

  When the call connection device receives the call request, the call connection device makes a call to each call terminal 10 indicated by the identification information included in the call request (steps S103 and S104). Furthermore, the call connection device also makes a call to the call relay unit 20 associated with each call terminal 10 that is the call destination. That is, the call connection device places a call to the Mr. A side call relay unit 20 and the Mr. B side call relay unit 20 (steps S105 and S106).

  The Mr. A call terminal and the Mr. B call terminal that have received a call from the call connection device output a ring tone to notify the user that the call is being received. When the user goes off-hook by operating the call terminal 10, the call terminal 10 responds to the call connection device (steps S107 and S108).

  The Mr. A side call relay unit 20 and the Mr. B side call relay unit 20 that have received a call from the call connection device automatically transition to the off-hook state in response to the incoming call, and respond to the call connection device (step S109). , S110).

  A call session is established by performing the above V-shaped call processing between Mr. A's call terminal 10 and Mr. A's call relay unit 20 (step S111). In addition, a call session is established by performing the above V-shaped call processing between Mr. B's call terminal 10 and Mr. B's call relay unit 20 (step S112). The Mr. A-side call relay unit 20 and the Mr. B-side call relay unit 20 are connected via the combining unit 60. Therefore, the Mr. A call terminal 10 and the Mr. B call terminal 10 are in a state in which a call can be made.

  FIG. 5 is a sequence diagram showing a specific example of a call processing flow in the call system 100. FIG. 5 shows a processing flow of a call performed between the call terminal 10 operated by Mr. A and the call terminal 10 operated by Mr. B.

  The Mr. A call terminal 10 transmits the input Mr. A's voice and video to the Mr. A side call relay unit 20 (step S201). The Mr. A side call relay unit 20 outputs the received voice to the Mr. A side voice recognition unit 40 via the splitter 30 (step S202). In addition, the Mr. A side call relay unit 20 outputs the received voice and video to the synthesizing device 70 (step S203). The Mr. A side voice recognition unit 40 performs a voice recognition process on the voice output from the Mr. A side call relay unit 20 to generate a text image (step S204). The Mr. A side voice recognition unit 40 outputs the generated text image to the synthesizing device 70 (step S205).

  The Mr. B call terminal 10 transmits the input Mr. B's voice and video to the Mr. B side call relay unit 20 (step S206). The Mr. B side call relay unit 20 outputs the received voice to the Mr. B side voice recognition unit 40 via the splitter 30 (step S207). In addition, the Mr. B side call relay unit 20 outputs the received voice and video to the synthesizing device 70 (step S208). The B-side voice recognition unit 40 performs a voice recognition process on the voice output from the B-side call relay unit 20 to generate a text image (step S209). The Mr. B side voice recognition unit 40 outputs the generated text image to the synthesis device 70 (step S210).

  The synthesizing device 70 includes the voice and video output from the Mr. A side call relay unit 20, the voice and video output from the Mr. B side call relay unit 20, and the text video output from the Mr. A side voice recognition unit 40. The synthesized video is generated by synthesizing the text video output from the Mr. B side voice recognition unit 40 (step S211).

  The synthesizing device 70 outputs the synthesized video and the voice output from the Mr. B side call relay unit 20 to the Mr. A side call relay unit 20 (step S212). The Mr. A side call relay unit 20 transmits the synthesized video and audio output by the synthesizing device 70 to the Mr. A call terminal 10 (step S213). The Mr. A call terminal 10 displays the received composite video and outputs a sound (step S214).

  The synthesizing device 70 outputs the synthesized video and the voice output from the Mr. A side call relay unit 20 to the Mr. B side call relay unit 20 (step S215). The Mr. B side call relay unit 20 transmits the synthesized video and audio output by the synthesizing device 70 to the Mr. B call terminal 10 (step S216). The Mr. B call terminal 10 displays the received composite video and outputs a sound (step S217).

  In the call system 100 configured as described above, when a call is made using the call terminal 10, the content of the other party's utterance is displayed on the screen of the call terminal 10 as a text image. Therefore, even if the user is deaf, it is possible to more easily understand the utterance content of the other party.

  In addition, there is a problem that the elderly can not think of speech due to the deterioration of the oral cavity and throat. For such a problem, in the above-described call system 100, the content of the utterance is displayed as a text image to the other party. For this reason, even a user who cannot clearly speak an utterance can more accurately convey the utterance content to the other party according to the performance of the voice recognition unit 40.

<Modification>
FIG. 6 is a system configuration diagram of a first modification (call system 100a) of the call system 100. The call system 100a differs from the call system 100 in that it includes a conversion unit 41 (41-1, 41-2).

The voice recognition unit 40 in the call system 100a does not generate a text image and outputs text data as a result of voice recognition to the conversion unit 41. The conversion unit 41 converts (translates) the sentence of the text data output by the voice recognition unit 40 into a sentence in another language designated by the user of the call terminal 10. The conversion part 41 produces | generates the image | video (text image | video) which displayed the character showing the text data after conversion. Then, the conversion unit 41 outputs the generated text video to the synthesis device 70. For example, the conversion unit 41-1 converts the X language into the Y language, and the conversion unit 41-2 converts the Y language into the X language.
With this configuration, it is possible to smoothly perform conversations with users in other languages.

  FIG. 7 is a system configuration diagram of a second modification of the call system 100 (call system 100b). In the call system 100b, the display mode of the text video generated by the voice recognition unit 40 is different from the display mode of the text video in the call system 100.

  FIG. 8 is a schematic diagram illustrating a specific example of a composite video generated in the call system 100b. As shown in FIG. 8, the text image displayed in the first area 81 and the text image displayed in the fourth area 84 have different positions in the height direction of characters. In the text image representing the content uttered earlier on the time axis, characters are arranged at a higher position, and in the text image representing the content uttered later on the time axis, characters are arranged at a lower position.

It may be determined by the voice recognition unit 40-1 and the voice recognition unit 40-2 which of the call terminal 10-1 and the call terminal 10-2 is uttered first. That is, the speech recognition unit 40 (40-1 and 40-2) transmits a signal (flag signal) indicating that speech recognition has been performed to another speech recognition unit 40 every time speech recognition is performed. After receiving the flag signal, the voice recognition unit 40 generates a text image in which text is arranged in a predetermined high region until the flag signal is transmitted. On the other hand, after transmitting the flag signal, the voice recognition unit 40 generates a text image in which text is arranged in a predetermined low position area until the flag signal is received.
By being configured in this way, each user who is making a call can easily determine which of the displayed utterance contents has most recently spoken.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 100 ... Call system, 10 ... Call terminal, 11 ... Network, 20 ... Call relay part, 30 ... Splitter, 40 ... Voice recognition part, 51 ... First input part, 52 ... Second input part, 53 ... Third input part 54 ... fourth input unit, 60 ... synthesis unit, 70 ... synthesis device

Claims (3)

A first call relay unit that receives audio and video transmitted from the first call terminal;
A first voice recognition unit that converts voice received by the first call relay unit into text data and generates a text image representing the text data;
A second call relay unit for receiving audio and video transmitted from the second call terminal;
A second voice recognition unit that converts the voice received by the second call relay unit into text data and generates a text image representing the text data;
Video received by the first call relay unit, video received by the second call relay unit, text video generated by the first voice recognition unit, and generated by the second voice recognition unit A synthesizing unit that generates a synthesized video by synthesizing the text video,
With
The first call relay unit transmits the composite video to the first call terminal;
The second call relay unit transmits the composite video to the second call terminal. A first input unit that receives input of a text image generated by the first voice recognition unit;
A second input unit for receiving input of audio and video received by the first call relay unit;
A third input unit that receives input of audio and video received by the second call relay unit;
A fourth input unit for receiving an input of a text image generated by the second voice recognition unit;
Further comprising
The call system according to claim 1, wherein the synthesis unit generates the synthesized video by arranging each video input to the first to fourth input units in a predetermined screen area. A first call reception step for receiving audio and video transmitted from the first call terminal;
A first voice recognition step of converting the voice received by the first call receiving step into text data and generating a text image representing the text data;
A second call receiving step for receiving audio and video transmitted from the second call terminal;
A second voice recognition step of converting the voice received by the second call receiving step into text data and generating a text image representing the text data;
The video received by the first call receiving step, the video received by the second call receiving step, the text video generated by the first voice recognition step, and the second voice recognition step A synthesis step of generating a synthesized video by synthesizing the text video;
A first transmission step of transmitting the composite video to the first call terminal;
A second transmission step of transmitting the composite video to the second call terminal;
A call relay method comprising:

Images