JP2023084986A - Display control system, display control method, and program - Google Patents

Abstract

To provide a display control system, a display control method, and a program, capable of timely displaying a translation result of voice to be inputted.SOLUTION: An input relay unit 80 receives voice data representing voice inputted by a speaker. The input relay unit 80 receives a confirmation request output in response to a prescribed operation performed by the speaker. A character string relay unit 84 controls such that a translation of the voice represented by the voice data received before reception of the confirmation request starts upon the reception of the confirmation request as a trigger. A display control unit 50 causes a display unit to display a screen in which an image obtained by superimposing a character string representing a translation result of the voice represented by the voice data received before the reception of the confirmation request, on an image photographed by a photographing unit, is arranged.SELECTED DRAWING: Figure 8A

Description

Patent Law Article 30, Paragraph 2 application filed June 15, 2021, https://sourcenext. co. jp/, https://sourceext. co. jp/pressrelease_html/JS/2021/2021061503/ September 9, 2021, https://pockettalk. jp/details/subtitles/, https://pockettalk. com/software/subtitles/verify/

The present invention relates to a display control system, display control method and program.

2. Description of the Related Art There is a technique for displaying an image obtained by superimposing a character string representing a result of speech translation on an image captured by a capturing unit. As an example of such a technique, Patent Document 1 discloses a video signal of video data in which character information obtained by translating speech data of a speaker is superimposed on a video signal obtained by imaging a speaker. A teleconferencing system for displaying is described.

There is also a technology that triggers the absence of recognizable speech input for several seconds to start translating the speech that has been input up to that point.

JP 2015-153408 A

In the technique described in Patent Document 1, when there is no recognizable speech input for several seconds as a trigger, translation of the speech that has been input up to that point is started. It takes a certain amount of time until the translation result is displayed. Therefore, the participants of the video conference cannot grasp the speech translation result in a timely manner.

The present invention has been made in view of the above problems, and one of its objects is to provide a display control system, a display control method, and a program capable of displaying a translation result of input speech in a timely manner.

A display control system according to the present invention includes: audio data reception means for receiving audio data representing audio input by a speaker; and confirmation request reception for receiving a confirmation request output in response to a predetermined operation performed by the speaker. means, translation control means for controlling, triggered by acceptance of the confirmation request, to start translating the voice represented by the audio data received by the acceptance of the confirmation request; translation result display control means for causing a display unit to display a screen on which an image superimposed with a character string representing a translation result of the voice represented by the voice data received before the acceptance of the confirmation request is arranged.

In one aspect of the present invention, the display unit displays a screen in which an image obtained by superimposing a character string representing a voice recognition result of the voice represented by the voice data on the image captured by the imaging unit is displayed on the display unit. display control means, wherein the voice recognition result display control means adds the voice recognition result of the voice represented by the received voice data to the image captured by the imaging unit before receiving the confirmation request. A screen on which an image superimposed with a character string representing the object is displayed on the display unit.

Further, in one aspect of the present invention, the translation result display control means adds a character string representing a voice recognition result of the voice represented by the voice data received until the confirmation request is received to the image captured by the capturing unit. and a character string representing the translation result of the voice represented by the voice data received before the confirmation request is received.

Further, in one aspect of the present invention, it further includes an image output unit for outputting an image obtained by superimposing a character string on the image captured by the capturing unit to a video conference system, wherein the translation result display control means controls the display of the TV The screen generated by the conference system is displayed on the display unit.

Further, in one aspect of the present invention, the voice data receiving means receives from the terminal the voice data representing voice input by the speaker to the terminal, and the confirmation request receiving means sends the confirmation request to the terminal. The confirmation request transmitted from the terminal in response to a predetermined operation performed by the speaker is received, and the translation result display control means displays the translation result of the speech represented by the received speech data until the reception of the confirmation request. The translation result display control means causes the display unit of the terminal to display the character string representing A display unit of the client device displays a screen on which an image superimposed with a character string representing the character string is arranged.

Alternatively, the voice data receiving means receives from the client device the voice data representing a voice input to the client device by the speaker, and the confirmation request receiving means receives from the client device the confirmation request received by the speaker. The translation result display control means accepts the confirmation request transmitted from the client device in response to a predetermined operation, and the translation result display control means adds the voice data received by the reception of the confirmation request to the image captured by the imaging unit. A screen on which an image superimposed with a character string representing the translation result of the represented voice is displayed on the display unit of the client device.

In one aspect of the present invention, the translation control means controls to start translating the voice represented by the voice data received by the time the confirmation request is received into a plurality of languages, and controls the translation result display. means for displaying, on the display unit, a screen in which an image obtained by superimposing a character string representing a translation result of the voice represented by the voice data for each of the plurality of languages on the image captured by the capturing unit is arranged; display.

Further, a display control method according to the present invention includes the steps of: receiving voice data representing voice input by a speaker; receiving a confirmation request output in response to a predetermined operation performed by the speaker; a step of controlling the reception of a confirmation request as a trigger to start translating the voice represented by the received audio data before the reception of the confirmation request; and causing a display unit to display a screen on which an image superimposed with a character string representing a translation result of the voice represented by the received voice data is arranged.

Further, the program according to the present invention includes a procedure for accepting speech data representing speech input by a speaker, a procedure for accepting a confirmation request output in response to a predetermined operation performed by the speaker, and a procedure for accepting the confirmation request. is used as a trigger to start translating the voice represented by the voice data received before the acceptance of the confirmation request; causes the computer to display, on the display unit, a screen on which an image on which a character string representing the translation result of the speech represented by is superimposed is arranged.

1 is a diagram showing an example of the overall configuration of a video conference translation system according to an embodiment of the present invention; FIG. It is a figure which shows an example of the back surface of the terminal which concerns on one Embodiment of this invention. It is a figure which shows an example of a structure of the terminal which concerns on one Embodiment of this invention. It is a figure which shows an example of a structure of the client apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of a structure of the relay apparatus which concerns on one Embodiment of this invention. It is a figure showing an example of composition of a sound processing system concerning one embodiment of the present invention. It is a figure which shows an example of a teleconference screen. It is a figure which shows an example of a speech-recognition result image. It is a figure which shows an example of a teleconference screen. It is a figure which shows an example of a translation result image. 1 is a functional block diagram showing an example of functions implemented by a terminal, a relay device, and an audio processing system according to an embodiment of the present invention; FIG. 3 is a functional block diagram showing an example of functions implemented in a client device according to one embodiment of the present invention; FIG. FIG. 4 is a flow diagram showing an example of the flow of processing performed in the relay device according to one embodiment of the present invention; FIG. 4 is a flow diagram showing an example of the flow of processing performed in the relay device according to one embodiment of the present invention; FIG. 4 is a flow diagram showing an example of the flow of processing performed in the client device according to one embodiment of the present invention; It is a figure which shows an example of a teleconference screen. FIG. 10 is a diagram showing an example of the configuration of a client device according to a modification of one embodiment of the present invention;

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing an example of the overall configuration of a video conference translation system 1 according to this embodiment. FIG. 2 is a diagram showing an example of the back surface of the terminal 10 according to this embodiment. FIG. 3A is a diagram showing an example of the configuration of the terminal 10 according to this embodiment. FIG. 3B is a diagram showing an example of the configuration of the client device 12 according to this embodiment. FIG. 3C is a diagram showing an example of the configuration of the relay device 14 according to this embodiment. FIG. 3D is a diagram showing an example of the configuration of the audio processing system 16 according to this embodiment.

As shown in FIG. 1, the teleconference translation system 1 according to this embodiment includes a terminal 10, a client device 12, a relay device 14, a voice processing system 16, and a teleconference system 18. FIG. The terminal 10, client device 12, relay device 14, audio processing system 16, and video conference system 18 are connected to a computer network 20 such as the Internet. Therefore, the terminal 10 , the client device 12 , the relay device 14 , the audio processing system 16 and the teleconference system 18 can communicate with each other via the computer network 20 .

A terminal 10 according to the present embodiment is a computer used by a user who participates in a video conference such as a remote conference. As shown in FIG. 3A, the terminal 10 according to this embodiment includes, for example, a processor 10a, a storage unit 10b, a communication unit 10c, an operation unit 10d, an imaging unit 10e, a touch panel 10f, a microphone 10g, and a speaker 10h.

The processor 10a is a program-controlled device such as a microprocessor that operates according to a program installed in the terminal 10, for example.

The storage unit 10b is, for example, a storage element such as ROM or RAM. The storage unit 10b stores programs and the like executed by the processor 10a.

The communication unit 10c is a communication interface for exchanging data with the relay device 14 via the computer network 20, for example. Here, the communication unit 10c may include a wireless communication module that communicates with a computer network 20 such as the Internet via a mobile phone line including a base station. The communication unit 10c may also include a wireless LAN module that communicates with a computer network 20 such as the Internet via a Wi-Fi (registered trademark) router or the like.

The operation unit 10d is, for example, an operation member such as a button or a touch sensor that outputs the content of an operation performed by a user to the processor 10a. In FIG. 1, as an example of the operation unit 10d, a translation button 10da that is pressed when inputting a speech to be translated, a power button 10db for turning the power on and off, and the volume of the speech output from the speaker 10h. A volume adjustment section 10dc for making adjustments is shown. The translation button 10da is arranged below the touch panel 10f provided on the front surface of the terminal 10 . A power button 10db and a volume control section 10dc are arranged on the right side of the terminal 10 .

The photographing unit 10e is, for example, a photographing device such as a digital camera. As shown in FIG. 2, the terminal 10 according to the present embodiment is provided with an imaging unit 10e on the back.

The touch panel 10f is, for example, a combination of a touch sensor and a display such as a liquid crystal display or an organic EL display. The touch panel 10f is provided on the front surface of the terminal 10, and displays a screen generated by the processor 10a.

The microphone 10g is, for example, an audio input device that converts received audio into electrical signals. Here, the microphone 10g may be a dual microphone built into the terminal 10 and equipped with a noise canceling function that makes it easy to recognize human voices even in a crowd.

The speaker 10h is, for example, an audio output device that outputs audio. Here, the speaker 10h may be a dynamic speaker that is built into the terminal 10 and that can be used even in noisy places.

The client device 12 according to this embodiment is a general computer such as a smart phone, tablet terminal, or personal computer. As shown in FIG. 3B, the client device 12 according to this embodiment includes, for example, a processor 12a, a storage unit 12b, a communication unit 12c, an operation unit 12d, a photographing unit 12e, a display 12f, a microphone 12g, and a speaker 12h. .

The client device 12 according to the present embodiment is used by a user who uses the terminal 10 during a teleconference such as a remote conference. That is, in this embodiment, the user of the terminal 10 and the user of the client device 12 are the same.

The processor 12a is a program-controlled device such as a CPU that operates according to a program installed in the client device 12, for example.

The storage unit 12b is, for example, a storage element such as ROM or RAM, a solid state drive, a hard disk drive, or the like. The storage unit 12b stores programs and the like executed by the processor 12a.

The communication unit 12c is, for example, a communication interface such as a network board or a wireless LAN module. The communication unit 12c exchanges data with the relay device 14 and the video conference system 18 via the computer network 20, for example.

The operation unit 12d is a user interface such as a keyboard and a mouse, for example, and receives user's operation input and outputs a signal indicating the content of the input to the processor 12a.

The photographing unit 12e is, for example, a photographing device such as a digital video camera. The photographing unit 12e is arranged at a position where the user of the client device 12 can be photographed. The photographing unit 12e according to this embodiment is capable of photographing moving images.

The display 12f is a display device such as a liquid crystal display or an organic EL display, and displays various images according to instructions from the processor 12a.

The microphone 12g is, for example, an audio input device that converts received audio into electrical signals.

The speaker 12h is an audio output device that outputs audio, for example.

In the present embodiment, the relay device 14 receives, for example, voice data representing voice input to the terminal 10, a voice recognition result character string representing the voice recognition result of the voice, a translation result character string representing the translation result of the voice, etc. It is a computer system such as a server computer that relays. It should be noted that the teleconference translation system 1 may include one relay device 14 or may include a plurality of relay devices 14 . As shown in FIG. 3C, the relay device 14 according to this embodiment includes, for example, a processor 14a, a storage unit 14b, and a communication unit 14c.

The processor 14a is a program control device such as a CPU that operates according to a program installed in the relay device 14, for example.

The storage unit 14b is, for example, a storage element such as ROM or RAM, a solid state drive, a hard disk drive, or the like. Programs and the like executed by the processor 14a are stored in the storage unit 14b.

The communication unit 14c is, for example, a communication interface such as a network board. The communication unit 14c exchanges data with the terminal 10, the client device 12, and the audio processing system 16 via the computer network 20, for example.

The speech processing system 16 is, for example, a computer system such as a server computer that performs speech recognition of speech represented by received speech data and speech processing such as translation of the speech. The audio processing system 16 may be composed of one computer, or may be composed of a plurality of computers. As shown in FIG. 3D, the speech processing system 16 according to this embodiment includes, for example, a processor 16a, a storage unit 16b, and a communication unit 16c.

The processor 16a is a program-controlled device such as a CPU that operates according to programs installed in the audio processing system 16, for example.

The storage unit 16b is, for example, a storage element such as ROM or RAM, a solid state drive, a hard disk drive, or the like. The storage unit 16b stores programs and the like executed by the processor 16a.

The communication unit 16c is, for example, a communication interface such as a network board. The communication unit 16c exchanges data with the relay device 14 via the computer network 20, for example.

The video conference system 18 is, for example, a general video conference system that realizes a video conference such as a remote conference by a plurality of participants. In this embodiment, for example, it is assumed that the client device 12 is installed with client software related to the video conference system 18 that operates in cooperation with the video conference system 18 .

In this embodiment, a teleconference such as a remote conference in which a plurality of participants including the users of the terminals 10 and the client devices 12 participate has already been held by the function of the teleconference system 18 .

Further, in the present embodiment, by performing a predetermined operation on the terminal 10 by the user in advance, a pre-translation language, which is the language of the speech input to the terminal 10, and a post-translation language, which is the language into which the speech is translated. language is set. In the following description, it is assumed that Japanese is set as the pre-translation language and English is set as the post-translation language.

Further, in the present embodiment, the voice input through the microphone 10g during the period from when the user presses a predetermined button (here, for example, the translation button 10da) provided on the terminal 10 with his finger until he releases it is , voice recognition processing is executed. Also, when the user releases his or her finger from the translation button 10da as a trigger, the translation process is executed for the voice input through the microphone 10g during the period from when the user presses the translation button 10da with his finger to when he/she releases the translation button 10da. be. Hereinafter, the state in which the translation button 10da is pressed is called an input ON state, and the state in which the translation button 10da is not pressed is called an input OFF state.

In the present embodiment, for example, during the input ON state, speech recognition processing is sequentially performed on voices input from the timing when the input OFF state changes to the input ON state to the present time. Then, a speech recognition result character string, which is a character string representing the speech recognition result for the speech, is displayed on the display 12f of the client device 12 and also on the touch panel 10f of the terminal 10. FIG.

FIG. 4 is a diagram showing an example of a teleconference screen 30, which is a screen of a teleconference such as a remote conference, displayed on the display 12f of the client device 12. As shown in FIG. As shown in FIG. 4, in the present embodiment, for example, a teleconference screen including a superimposed image 32 in which a voice recognition result character string is superimposed on a photographed image of a user who is a speaker who has performed voice input to the terminal 10. 30 is displayed on the display 12f. A photographed image according to the present embodiment is, for example, an image photographed by the photographing unit 12e. Note that the captured image according to the present embodiment may be an image captured by the imaging unit 10e.

FIG. 5 is a diagram showing an example of the voice recognition result image 34 displayed on the touch panel 10f of the terminal 10. As shown in FIG. As shown in FIG. 5, in this embodiment, the same character string as the character string arranged on the teleconference screen 30 shown in FIG.

In the present embodiment, as described above, while the terminal 10 is in the input-on state, voice input is sequentially performed from the timing when the terminal 10 changes from the input-off state to the input-on state to the present time. Recognition processing is performed. Then, the voice recognition result character string displayed on the touch panel 10f or the display 12f is updated each time the voice recognition process is executed.

Then, when the user releases the translation button 10da and the terminal 10 enters an input off state, the terminal 10 transmits a confirmation request to the relay device 14. FIG. Then, the final speech recognition process is executed for the speech input while the terminal 10 is in the input ON state. Then, a translation process is performed on a speech recognition result character string representing the result of the speech recognition process, and a translation result character string is generated by translating the speech recognition result character string. Here, for example, a translation result character string, which is an English character string, is generated by translating a voice recognition result character string, which is a Japanese character string.

The speech recognition character string and the translation result character string thus generated are displayed on the display 12f of the client device 12 and also on the touch panel 10f of the terminal 10. FIG.

For example, as shown in FIG. 6, a television in which a superimposed image 32 is arranged in which a voice recognition result character string and a translation result character string are superimposed on a photographed image of a user who is a speaker who has performed voice input to the terminal 10. A conference screen 30 is displayed on the display 12f.

Further, as shown in FIG. 7, the same character string as the speech recognition result character string arranged on the teleconference screen 30 shown in FIG. 6 and the translation result character string arranged on the teleconference screen 30 shown in FIG. A translation result image 36 in which the same character string as the column is arranged is displayed on the touch panel 10f.

For convenience of explanation, FIG. 6 shows the teleconference screen 30 on which the translation result character string is easy to see. Depending on the photographed image, the displayed translation result character string may be difficult to see, and the user, who is the speaker, may not be able to accurately grasp the translation result.

In this embodiment, as shown in FIG. 7, a translation result image 36 in which the same character string as the translation result character string shown in FIG. 6 is arranged is displayed on the touch panel 10f of the terminal 10. FIG.

In this manner, according to the present embodiment, the user can accurately grasp the translation result of the speech input by the user.

4 and 6 show the teleconference screen 30 on which the voice recognition result character string is easily visible for convenience of explanation. Depending on the background image (here, for example, a photographed image), the displayed speech recognition result character string may be difficult to see, and the user, who is the speaker, may not be able to accurately grasp the speech recognition result.

In this embodiment, as shown in FIG. 5, a speech recognition result image 34 in which the same character string as the speech recognition result character string shown in FIG. 4 is arranged is displayed on the touch panel 10f of the terminal 10. FIG. 7, a translation result image 36 in which the same character string as the speech recognition result character string shown in FIG. 6 is arranged is displayed on the touch panel 10f of the terminal 10. FIG.

In this manner, according to the present embodiment, the user can accurately grasp the speech recognition result of the speech input by the user.

Further, in the present embodiment, the reception of the confirmation request by the relay device 14 triggers the translation of the speech represented by the received audio data until the reception of the confirmation request. In this way, the lack of recognizable speech input for a few seconds triggers the translation of the previously input speech, and the The time it takes for the speech to be translated is shortened. Thus, according to this embodiment, the translation result of the input speech can be displayed in a timely manner.

The functions of the videoconference translation system 1 according to the present embodiment and the processing executed by the videoconference translation system 1 will be further described below.

FIG. 8A is a functional block diagram showing an example of functions implemented by the terminal 10, the relay device 14, and the voice processing system 16 according to this embodiment. FIG. 8B is a functional block diagram showing an example of functions implemented in the client device 12 according to this embodiment.

Note that the terminal 10, the relay device 14, and the audio processing system 16 according to the present embodiment do not need to implement all the functions shown in FIG. 8A, and functions other than the functions shown in FIG. It doesn't matter if Further, the client device 12 according to the present embodiment does not need to implement all the functions shown in FIG. 8B, and may implement functions other than the functions shown in FIG. 8B.

As shown in FIG. 8A, the terminal 10 according to the present embodiment functionally includes, for example, an operation input reception unit 40, a voice input reception unit 42, a voice buffer 44, an input transmission unit 46, a character string reception unit 48, A display control unit 50 is included. The operation input reception unit 40 is mainly implemented by the processor 10a, the operation unit 10d, and the touch panel 10f. The voice input receiving unit 42 is mainly implemented by the processor 10a and the microphone 10g. The audio buffer 44 is implemented mainly in the storage unit 10b. The input transmission unit 46 and the character string reception unit 48 are mainly implemented by the communication unit 10c. The display control unit 50 is mainly implemented by the processor 10a and the touch panel 10f.

The functions described above are implemented by causing the processor 10a to execute a program including commands corresponding to the functions described above, which is installed in the terminal 10, which is a computer. This program is supplied to the terminal 10 via a computer-readable information storage medium such as an optical disk, magnetic disk, magnetic tape, magneto-optical disk, flash memory, or the like, or via the Internet.

As shown in FIG. 8B, the client device 12 according to the present embodiment functionally includes, for example, a voice input reception unit 60, a character string reception unit 62, a captured image acquisition unit 64, a superimposed image generation unit 66, a video conference A client unit 68, an audio output control unit 70, and a display control unit 72 are included. The voice input reception unit 60 is mainly implemented by the processor 12a and the microphone 12g. The character string receiving unit 62 is implemented mainly by the communication unit 12c. The photographed image acquisition unit 64 is mainly implemented by the processor 12a and the photographing unit 12e. The superimposed image generator 66 is implemented mainly by the processor 12a. The videoconference client unit 68 is mainly implemented by the processor 12a and the communication unit 12c. The audio output control unit 70 is mainly implemented by the processor 12a and the speaker 12h. The display control unit 72 is mainly implemented by the processor 12a and the display 12f.

The functions described above are implemented by causing the processor 12a to execute a program containing commands corresponding to the functions described above, which is installed in the client device 12, which is a computer. This program is supplied to the client device 12 via a computer-readable information storage medium such as an optical disk, magnetic disk, magnetic tape, magneto-optical disk, flash memory, or the like, or via the Internet.

As shown in FIG. 8A, the relay device 14 according to the present embodiment functionally includes an input relay unit 80, a voice buffer 82, and a character string relay unit 84, for example. The input relay unit 80 and the character string relay unit 84 are mainly implemented by the communication unit 14c. The audio buffer 82 is implemented mainly in the storage unit 14b.

The functions described above are implemented by causing the processor 14a to execute a program containing commands corresponding to the functions described above, which is installed in the relay device 14, which is a computer. This program is supplied to the relay device 14 via a computer-readable information storage medium such as an optical disk, magnetic disk, magnetic tape, magneto-optical disk, flash memory, or the like, or via the Internet.

As shown in FIG. 8A, the speech processing system 16 according to this embodiment functionally includes, for example, a speech recognition unit 90 and a translation unit 92 . The speech recognition unit 90 and the translation unit 92 are mainly implemented by the processor 16a and the communication unit 16c.

The functions described above are implemented by causing the processor 16a to execute a program containing commands corresponding to the functions described above, which is installed in the audio processing system 16, which is a computer. This program is supplied to the audio processing system 16 via a computer-readable information storage medium such as an optical disk, magnetic disk, magnetic tape, magneto-optical disk, flash memory, or the like, or via the Internet.

In this embodiment, the operation input reception unit 40 of the terminal 10 receives an operation input to the terminal 10 such as an operation of pressing the translation button 10da with the user's finger or an operation of releasing the finger from the translation button 10da.

In this embodiment, for example, the voice input reception unit 42 of the terminal 10 receives voice input by the speaker via the microphone 10g while the terminal 10 is in the input ON state.

The audio buffer 44 of the terminal 10 stores, in this embodiment, audio data representing audio input via the microphone 10g, for example.

The input transmission unit 46 of the terminal 10 transmits to the relay device 14, for example, an operation signal corresponding to the operation input received by the operation input reception unit 40 in this embodiment.

Also, in the present embodiment, the input transmission unit 46 transmits, for example, voice data representing voice input to the terminal 10 to the relay device 14 .

In the present embodiment, for example, the input transmission unit 46 transmits a communication start request to the relay device 14 in response to the terminal 10 changing from the input-off state to the input-on state. Voice data representing voice input through the microphone 10g during the period from when the terminal 10 changes from the input-off state to the input-on state until the communication between the relay device 14 and the terminal 10 is established. is stored in the audio buffer 44 .

Then, when communication is established between the relay device 14 and the terminal 10 (that is, when the terminal 10 is connected to the relay device 14), the input transmission unit 46 transmits the voice data accumulated in the voice buffer 44. It is transmitted to the relay device 14 . In general, for example, the audio data stored in the audio buffer 44 and representing audio with a length of 2 seconds is transmitted in about 0.1 seconds.

After all the audio data accumulated in the audio buffer 44 has been transmitted to the relay device 14, the input transmission unit 46 expresses the audio received by the audio input reception unit 42 while the terminal 10 is in the input ON state. Packets of audio data are streamed to the relay device 14 . In this case, the audio data packets are directly transmitted to the relay device 14 in real time without being accumulated in the audio buffer 44 . Note that the voice data packet may include pre-translation language data indicating the pre-translation language and post-translation language data indicating the post-translation language.

The input relay unit 80 of the relay device 14 receives, for example, audio data transmitted from the input transmission unit 46 in this embodiment. The input relay unit 80 then transmits the received voice data to the voice recognition unit 90 of the voice processing system 16 . For example, the input relay unit 80 receives packets of audio data streamed from the input transmission unit 46 and transmits the packets to the audio recognition unit 90 of the audio processing system 16 .

In addition, in this embodiment, the speech processing system 16 may include a plurality of speech recognition units 90 each associated with a different language. Then, the input relay unit 80 may transmit the received voice data to the voice recognition unit 90 associated with the translated language.

In this embodiment, upon receiving a packet transmitted from the input transmission unit 46, the input relay unit 80 temporarily stores the packet in the audio buffer 82. FIG. The input relay unit 80 then transmits the packets stored in the audio buffer 82 to the audio recognition unit 90 of the audio processing system 16 . By doing so, even if a communication error occurs in the communication between the voice processing system 16 and the relay device 14, it is possible to retry packet transmission.

The speech recognition unit 90 of the speech processing system 16 receives packets of speech data transmitted from, for example, the input relay unit 80 of the relay device 14 in this embodiment.

Then, in this embodiment, the speech recognition unit 90 of the speech processing system 16 executes speech recognition processing on the speech represented by the received speech data, and generates a speech recognition result character string representing the speech recognition result of the speech. to generate Here, for example, every time the voice recognition unit 90 receives a packet of voice data, voice recognition processing is performed on the voice data received from when the terminal 10 is connected to the relay device 14 until the packet is received. may be executed to generate a speech recognition result string.

Then, the speech recognition unit 90 of the speech processing system 16 transmits, for example, the speech recognition result character string generated by the speech recognition unit 90 to the relay device 14 in this embodiment. Here, when the speech recognition processing is executed sequentially, the generated speech recognition result character string may be transmitted to the relay device 14 each time the speech recognition result character string is generated.

The character string relay unit 84 of the relay device 14 receives, for example, the speech recognition result character string described above in this embodiment.

Then, in response to the terminal 10 changing from the input-on state to the input-off state, the input transmission unit 46 transmits a confirmation request to the relay device 14 . If there is audio data accumulated in the audio buffer 44 when the input-on state changes to the input-off state, the input transmission unit 46 transmits the audio data accumulated in the audio buffer 44 to the relay device 14 . , and then transmits a confirmation request to the relay device 14 . Also, if there is no audio data accumulated in the audio buffer 44 when the input-on state changes to the input-off state, the input transmission unit 46 immediately transmits a determination request to the relay device 14 . In general, when the input-on state changes to the input-off state, there is often no audio data stored in the audio buffer 44, and at the timing of the change from the input-on state to the input-off state, Almost all audio data has been transmitted.

In this embodiment, when voice is input to the terminal 10 for a predetermined period of time (for example, 30 seconds), acceptance of voice may be terminated at that timing and a confirmation request may be transmitted.

In the present embodiment, the input relay unit 80 of the relay device 14 receives, for example, a confirmation request output in response to a predetermined operation performed by the speaker (here, for example, an operation of releasing the translation button 10da). For example, the input relay unit 80 of the relay device 14 receives a confirmation request transmitted from the input transmission unit 46 when the speaker releases the translation button 10da.

In this embodiment, for example, the character string relay unit 84 of the relay device 14 starts translating the speech represented by the received voice data by the reception of the confirmation request by the input relay unit 80 as a trigger. to control. For example, in response to the input relay unit 80 receiving the confirmation request, the character string relay unit 84 of the relay device 14 receives voice data received from when the terminal 10 was connected to the relay device 14 to when the confirmation request was received. to the translation unit 92 of the speech processing system 16 .

In addition, in this embodiment, the speech processing system 16 may include a plurality of translation units 92 each associated with a different language. Then, the character string relay unit 84 may transmit the speech recognition character string to the translation unit 92 associated with the post-translation language.

The translation unit 92 of the speech processing system 16 receives the speech recognition result character string transmitted by the character string relay unit 84 in this embodiment, for example. Then, the translation unit 92 of the speech processing system 16 translates the received speech recognition result character string. Then, the translation unit 92 generates a translation result character string representing the result of the translation processing.

Then, in the present embodiment, for example, the translation unit 92 transmits the translation result character string generated as described above to the relay device 14 .

Further, in the present embodiment, the character string relay unit 84 of the relay device 14, for example, transmits the speech recognition result character string representing the speech recognition result of the speech represented by the above-described speech data to the communication unit 10c of the terminal 10 and the client device 12. 12c. For example, upon receiving the speech recognition result character string from the speech recognition unit 90 of the speech processing system 16 , the character string relay unit 84 transmits the speech recognition result character string to both the terminal 10 and the client device 12 .

Further, in the present embodiment, the character string relay unit 84 of the relay device 14, for example, transmits the translation result character string representing the translation result of the voice represented by the above-described voice data to the communication unit 10c of the terminal 10 and the communication unit 12c of the client device 12. to both. For example, the character string relay unit 84 transmits the translation result character string to both the terminal 10 and the client device 12 in response to receiving the translation result character string from the translation unit 92 of the speech processing system 16 .

The character string receiving unit 48 of the terminal 10 receives, for example, the speech recognition result character string transmitted from the relay device 14 in this embodiment.

Further, the character string receiving unit 48 of the terminal 10 receives, for example, the translation result character string transmitted from the relay device 14 in this embodiment.

The display control unit 50 of the terminal 10 causes the display unit (for example, the touch panel 10f) of the terminal 10 to display the voice recognition result character string received by the character string receiving unit 48, for example. The display control unit 50 also causes the display unit (for example, the touch panel 10f) of the terminal 10 to display the translation result character string received by the character string receiving unit 48, for example.

Here, as shown in FIG. 7, the display control unit 50 generates a translation result image 36, which is an image in which both the speech recognition result character string and the translation result character string received by the character string receiving unit 48 are arranged. may Then, the display control unit 50 may display the translation result image 36 on the touch panel 10f.

In the present embodiment, the display control unit 50 may cause the touch panel 10f to display the character string received by the character string receiving unit 48 in a different color from the single-color background. By doing so, the user can more accurately grasp the result of translation of the voice input by the user, the result of voice recognition, and the like.

The voice input reception unit 60 of the client device 12 receives the user's voice input via the microphone 12g in this embodiment, for example. Then, the voice input reception unit 60 outputs voice data representing the input voice to the teleconference client unit 68 .

The character string receiving unit 62 of the client device 12 receives, for example, the speech recognition result character string transmitted from the relay device 14 in this embodiment.

Further, the character string receiving unit 62 of the client device 12 receives, for example, the translation result character string transmitted from the relay device 14 in this embodiment.

The captured image acquisition unit 64 acquires a captured image, which is an image captured by the imaging unit 12e in this embodiment, for example.

In the present embodiment, the superimposed image generation unit 66 generates, for example, the superimposed image 32, which is an image obtained by superimposing the speech recognition result character string received by the character string receiving unit 62 on the above-described captured image. In addition, in the present embodiment, the superimposed image generation unit 66 generates, for example, the superimposed image 32 which is an image in which the translation result character string received by the character string receiving unit 62 is superimposed on the above-mentioned photographed image.

Here, as shown in FIG. 6, the superimposed image generation unit 66 superimposes both the translation result character string and the voice recognition result character string received by the character string reception unit 62 on the above-described photographed image. An image 32 may be generated.

Then, the superimposed image generation unit 66 outputs the generated superimposed image 32 to the videoconference client unit 68 in this embodiment, for example.

In this embodiment, for example, the teleconference client unit 68 of the client device 12 cooperates with the teleconference system 18 to execute various processes related to the teleconference.

The videoconference client unit 68 may output, to the videoconference system 18, a superimposed image 32 in which the character string received by the character string receiving unit 62 is superimposed on the above-described photographed image, for example. For example, the videoconference client unit 68 may output the superimposed image 32 received from the superimposed image generator 66 to the videoconference system 18 .

Further, the teleconference client unit 68 may output voice data received from the voice input receiving unit 60 to the teleconference system 18, for example.

The videoconference client unit 68 outputs the videoconference screen 30 shown in FIGS. 4 and 6, which is generated by the videoconference system 18 in this embodiment, to the display control unit 72, for example.

Also, in the present embodiment, the videoconference client unit 68 outputs, for example, audio data representing the voice of the speaker in the videoconference generated by the videoconference system 18 to the audio output control unit 70 .

In this embodiment, the audio output control unit 70 of the client device 12 causes the speaker 12h to output the audio represented by the audio data received from the video conference client unit 68, for example.

In this embodiment, for example, the display control unit 72 of the client device 12 displays a screen on which an image obtained by superimposing a character string representing the voice recognition result of the voice represented by the voice data on the image captured by the capturing unit 12e is arranged. Display on 12f. Here, the display control unit 72 arranges an image obtained by superimposing a character string representing the voice recognition result of the voice represented by the received voice data on the image captured by the capturing unit 12e before accepting the confirmation request. The displayed screen may be displayed on the display 12f. For example, the display control unit 72 of the client device 12 causes the display 12f of the client device 12 to display a screen in which an image obtained by superimposing the character string received by the character string receiving unit 62 on the captured image is arranged. .

Further, in the present embodiment, the display control unit 72 arranges an image obtained by superimposing a character string representing the translation result of the voice represented by the voice data received before the acceptance of the confirmation request on the image captured by the capturing unit 12e. The displayed screen is displayed on the display 12f. For example, the display control unit 72 of the client device 12 causes the display 12f of the client device 12 to display a screen in which an image obtained by superimposing the translation result character string received by the character string receiving unit 62 on the above-mentioned photographed image is arranged.

Here, as shown in FIG. 6, the display control unit 72 arranges a superimposed image 32 in which both the translation result character string and the voice recognition result character string received by the character string receiving unit 62 are superimposed on the photographed image. The teleconference screen 30 may be displayed on the display 12f.

Further, the display control unit 72 may cause the display 12f to display a screen generated by the video conference system 18. FIG. For example, the display control unit 72 may display the teleconference screen 30 received from the teleconference client unit 68 on the display 12f.

An example of the flow of audio data relay processing executed by the relay device 14 will now be described with reference to the flowchart shown in FIG.

In this processing example, the input relay unit 80 monitors reception of a communication start request transmitted from the input transmission unit 46 of the terminal 10 (S101).

When the input relay unit 80 receives the communication start request from the input transmission unit 46 of the terminal 10, the input relay unit 80 establishes communication between the relay device 14 and the terminal 10 (S102).

Then, the input relay unit 80 monitors reception of audio data packets (S103). When the input relay unit 80 receives the audio data packet, it stores the received packet in the audio buffer 82 (S104).

Then, the input relay unit 80 transmits the packets stored in the audio buffer 82 in the process shown in S104 to the audio recognition unit 90 of the audio processing system 16, and returns to the process shown in S103.

The processes shown in S103 to S105 are continued until the process shown in S207, which will be described later, is executed.

Next, an example of the flow of character string relay processing executed by the relay device 14 will be described with reference to the flowchart shown in FIG.

In this processing example, the character string relay unit 84 monitors reception of the speech recognition result character string transmitted from the speech recognition unit 90 of the speech processing system 16 (S201). When the character string relay unit 84 receives the speech recognition result character string, it transmits the received speech recognition result character string to the character string reception unit 62 of the client device 12 (S202).

Then, the character string relay unit 84 confirms whether or not the input relay unit 80 has received the confirmation request (S203). If reception of the confirmation request is not confirmed (S203: N), the process returns to S201. If the reception of the confirmation request is confirmed (S203: Y), the character string relay unit 84 performs voice processing on the speech recognition result character string representing the speech recognition result of the speech represented by the voice data received up to the reception of the confirmation request. It is transmitted to the translation section 92 of the system 16 (S204).

Then, the character string relay unit 84 receives a translation result character string obtained by translating the speech recognition result character string transmitted in the processing shown in S203, which is transmitted from the translation unit 92 of the speech processing system 16 (S205).

Then, the character string relay unit 84 outputs the confirmation flag, the translation result character string received in the processing shown in S205, and the speech recognition result character string representing the speech recognition result of the speech represented by the speech data received up to the reception of the confirmation request. is sent to the character string receiving unit 62 of the client device 12 (S206).

Then, the character string relay unit 84 disconnects the communication between the relay device 14 and the terminal 10 (S207), and the processing shown in this processing example ends. By executing the process shown in S207, the processes shown in S103 to S105 are also terminated.

Next, an example of the flow of processing for generating the superimposed image 32 executed by the client device 12 will be described with reference to the flowchart shown in FIG. In this example of processing, the processing shown in the following S301 to S305 is repeatedly executed at the frame rate at which the photographing unit 12e photographs the photographed image. In this embodiment, for example, the processes shown in S301 to S305 may be executed at intervals of 1/30 second. Note that the execution interval of the processes shown in S301 to S305 may be an interval longer (or shorter) than 1/30 second. Also, the execution interval may be adjustable by the user.

First, the captured image acquisition unit 64 acquires the captured image in the frame (S301).

Then, the superimposed image generation unit 66 checks whether the character string reception unit 62 has received the confirmation flag after the timing when the process shown in S202 was last executed (S302).

When it is confirmed that the confirmation flag has not been received (S302: N), the superimposed image generating unit 66 adds the latest voice received by the character string receiving unit 62 to the captured image acquired in the processing shown in S301. A superimposed image 32 on which the recognition result character string is superimposed is generated (S303).

When it is confirmed that the confirmation flag has been received (S302: Y), the superimposed image generation unit 66 adds the latest speech recognition result received by the character string reception unit 62 to the captured image acquired in the processing shown in S301. A superimposed image 32 is generated in which the character string and the latest translation result character string are superimposed (S304).

Then, the superimposed image generation unit 66 outputs the superimposed image 32 generated by the processing shown in S303 or S304 to the videoconference client unit 68 (S305), and returns to the processing shown in S301.

In this embodiment, the displayable area within the captured image may be settable by the user. For example, the displayable area may be selectable from upper, lower, entire, and the like. Also, the displayable area for the speech recognition result character string and the displayable area for the translation result character string may be set separately. For example, FIGS. 4 and 6 show an example of the teleconference screen 30 when the lower part is set as the displayable area for the speech recognition result character string. Also, FIG. 6 shows an example of the teleconference screen 30 when the entire area is set as the displayable area for the translation result character string.

Further, in this embodiment, a character string in a language such as Japanese that does not have a space to separate words may be line-wrapped at a predetermined number of characters. Also, a character string in a language such as English that has spaces separating words may be word-wrapped for a predetermined number of characters.

Also, in order to improve readability, the character size of the translation result character string may be larger than the character size of the speech recognition result character string.

Moreover, in the present embodiment, both the translation result character string and the voice recognition result character string need not be superimposed on the captured image. For example, when the translation result character string is superimposed on the captured image, the speech recognition result character string may not be superimposed on the captured image.

Further, in the present embodiment, the character size of the speech recognition result character string may be a fixed size, and the character size of the translation result character string may be a variable size.

In this case, the maximum character size of characters included in the translation result character string may be a size obtained by multiplying the height of the screen by a predetermined ratio. Then, the character size of the translation result character string may be made smaller as the number of characters in one line increases.

Note that the character size of the speech recognition result character string may be variable. Also, the character size of the translation result character string may be a fixed size.

Also, in the present embodiment, the number of displayable characters corresponding to the size of the displayable area may be predetermined. When a voice recognition result character string with a number of characters larger than the number of characters that can be displayed is superimposed on the captured image, the voice recognition result character string is reduced to fit within the height of the displayable area and then superimposed on the captured image. It may be superimposed. Also, when a translated character string with more characters than the number of characters that can be displayed is superimposed on the captured image, the translated character string is reduced to fit within the height of the displayable area and then superimposed on the captured image. may

Further, in the present embodiment, the character string relay unit 84 is triggered by the fact that the reception of the voice data packet by the input relay unit 80 is interrupted for a predetermined time (for example, 1.5 seconds). may be controlled to start translating the speech represented by . For example, the character string relay unit 84 of the relay device 14 connects the terminal 10 to the relay device 14 when the input relay unit 80 stops receiving voice data packets for a predetermined time (for example, 1.5 seconds). A speech recognition character string representing the speech recognition result of the speech represented by the speech data received up to the present may be transmitted to the translation unit 92 of the speech processing system 16 .

Also, a list (log) of the speech recognition result character strings and the translation result character strings may be displayed on a screen (for example, a browser) other than the teleconference screen 30 . This log may be stored in a storage medium such as the storage unit 12b of the client device 12. FIG. Also, a translation result character string obtained by translating the voice recognition result character string, which is translated into a language different from the above-described post-translation language, may be displayed on the browser.

Also, the functions of the terminal 10 of the video conference translation system 1 may be implemented in the client device 12 .

For example, as shown in FIG. 12, the client device 12 may have a function of displaying a translation button 94 on the display 12f. Then, in addition to the teleconference screen 30, a translation button 94 may be displayed on the display 12f. Then, for example, each time the speaker performs a predetermined operation such as a click operation on the translation button 94, the client device 12 may switch between the input-on state and the input-off state. Then, the voice recognition result character string and the translation result character string for the voice input while the input is on may be displayed on the teleconference screen 30 .

FIG. 13 is a functional block diagram showing an example of functions implemented in the client device 12 according to the modification of the embodiment described with reference to FIGS. 1 to 11. As shown in FIG. It is not necessary for the client device 12 according to the present embodiment to implement all the functions shown in FIG. 13, and functions other than the functions shown in FIG. 13 may be installed.

As shown in FIG. 13 , the client device 12 according to the modification includes functionally, for example, an operation input reception unit 40 , a voice buffer 44 , an input transmission unit 46 , a voice input reception unit 60 , a character string reception unit 62 . , a captured image acquisition unit 64, a superimposed image generation unit 66, a video conference client unit 68, an audio output control unit 70, and a display control unit 72. The operation input reception unit 40 is mainly implemented by the processor 12a and the operation unit 12d. The audio buffer 44 is implemented mainly in the storage unit 12b. The input transmission unit 46 and the character string reception unit 62 are mainly implemented by the communication unit 12c. The voice input reception unit 60 is mainly implemented by the processor 12a and the microphone 12g. The photographed image acquisition unit 64 is mainly implemented by the processor 12a and the photographing unit 12e. The superimposed image generator 66 is implemented mainly by the processor 12a. The videoconference client unit 68 is mainly implemented by the processor 12a and the communication unit 12c. The audio output control unit 70 is mainly implemented by the processor 12a and the speaker 12h. The display control unit 72 is mainly implemented by the processor 12a and the display 12f.

The functions described above are implemented by causing the processor 12a to execute a program containing commands corresponding to the functions described above, which is installed in the client device 12, which is a computer. This program is supplied to the client device 12 via a computer-readable information storage medium such as an optical disk, magnetic disk, magnetic tape, magneto-optical disk, flash memory, or the like, or via the Internet.

In this embodiment, for example, the operation input reception unit 40 displays a translation button 94 on the display 12f. Then, the operation input receiving unit 40 receives an operation input such as an operation of clicking the translation button 94 in this embodiment.

The voice input reception unit 60 receives the user's voice input via the microphone 12g in this embodiment, for example. Then, the voice input reception unit 60 outputs voice data representing the input voice to the teleconference client unit 68 .

In this embodiment, for example, the input transmission unit 46 transmits a communication start request to the relay device 14 in response to the client device 12 changing from the input off state to the input on state. Then, the voice representing the voice input through the microphone 12g during the period from when the client device 12 changes from the input-off state to the input-on state until the communication between the relay device 14 and the terminal 10 is established. The data is stored in the audio buffer 44 as well as being output to the videoconference client portion 68 .

Also, in response to the client device 12 changing from the input-on state to the input-off state, the input transmission unit 46 transmits a determination request to the relay device 14 .

Other functions of the audio buffer 44 and the input transmission unit 46 are the same as the functions described above with reference to FIG. 8A, so description thereof will be omitted. The functions of the character string receiving unit 62, the captured image acquiring unit 64, the superimposed image generating unit 66, the video conference client unit 68, the audio output control unit 70, and the display control unit 72 are the same as those described above with reference to FIG. 8B. Since it is the same as the function, the explanation is omitted. In addition, in the modification, the relay device 14 does not transmit the character string to the terminal 10 .

As in the examples shown in FIGS. 12 and 13, the input relay unit 80 may receive from the client device 12 voice data representing voice input to the client device 12 by the speaker. Also, the input relay unit 80 may receive a confirmation request transmitted from the client device 12 in response to a predetermined operation performed by the speaker on the client device 12 .

Then, the display control unit 72 displays a screen in which an image obtained by superimposing a character string representing the translation result of the voice represented by the voice data received until the confirmation request is received on the image captured by the capturing unit 12e is displayed on the client device. 12 may be displayed on the display 12f.

Also, in this embodiment, a plurality of languages may be set as post-translation languages. Then, the character string relay unit 84 may perform control so that the translation of the voice represented by the voice data received before the confirmation request is received into the set multiple languages is started. In this case, for example, the character string relay unit 84 may transmit the speech recognition character string to a plurality of translation units 92 each associated with a plurality of post-translation languages.

Then, the display control unit 72 may cause the display 12f to display a screen in which an image obtained by superimposing the translation result character strings for each of the plurality of set languages on the captured image is arranged.

For example, in the lower part of the photographed image, the translated result character string obtained by translating the speech recognition result character string into English is displayed, and in the upper part of the photographed image, the speech recognition result character string is displayed as the Chinese translation result character string. may be displayed.

Then, when it is confirmed that the translation result character strings for all post-translation languages have been displayed, these translation result character strings may be erased from the screen.

It should be noted that the present invention is not limited to the above-described embodiments.

For example, the division of roles among the terminal 10, the client device 12, the relay device 14, the audio processing system 16, and the video conference system 18 is not limited to what has been described above. For example, the speech recognition result character string may be translated by the speech processing system 16 without the speech recognition result character string passing through the relay device 14 .

For example, the client device 12 may receive voice data transmitted from the terminal 10 to the relay device 14 from the relay device 14 . Then, the client device 12 may output the voice data received from the relay device 14 to the teleconference system 18 instead of the voice data representing the voice input via the microphone 12g.

Moreover, the specific character strings and numerical values described above and the specific character strings and numerical values in the drawings are examples, and the present invention is not limited to these character strings and numerical values.

1 translation system for video conference, 10 terminal, 10a processor, 10b storage unit, 10c communication unit, 10d operation unit, 10da translation button, 10db power button, 10dc volume control unit, 10e photographing unit, 10f touch panel, 10g microphone, 10h speaker , 12 client device, 12a processor, 12b storage unit, 12c communication unit, 12d operation unit, 12e photographing unit, 12f display, 12g microphone, 12h speaker, 14 relay device, 14a processor, 14b storage unit, 14c communication unit, 16 audio processing system 16a processor 16b storage unit 16c communication unit 18 video conference system 20 computer network 30 video conference screen 32 superimposed image 34 speech recognition result image 36 translation result image 40 operation input reception unit 42 voice input reception unit 44 voice buffer 46 input transmission unit 48 character string reception unit 50 display control unit 60 voice input reception unit 62 character string reception unit 64 captured image acquisition unit 66 superimposed image generation unit 68 Teleconference client unit 70 voice output control unit 72 display control unit 80 input relay unit 82 voice buffer 84 character string relay unit 90 voice recognition unit 92 translation unit 94 translation button.

Claims (9)

voice data receiving means for receiving voice data representing voice input by a speaker;
Confirmation request receiving means for receiving a confirmation request output in response to a predetermined operation performed by the speaker;
translation control means for controlling, with the reception of the confirmation request as a trigger, to start translating the speech represented by the received audio data until the reception of the confirmation request;
Translation result display control for displaying on the display unit a screen in which a character string representing the translation result of the voice represented by the voice data received before the acceptance of the confirmation request is superimposed on the image captured by the capturing unit. means and
A display control system comprising: voice recognition result display control means for causing the display unit to display an image obtained by superimposing a character string representing a voice recognition result of the voice represented by the voice data on the image captured by the imaging unit. ,
The voice recognition result display control means superimposes a character string representing the voice recognition result of the voice represented by the received voice data on the image captured by the imaging unit before accepting the confirmation request. to display a screen on which is arranged on the display unit;
The display control system according to claim 1, characterized by: The translation result display control means adds a character string representing a voice recognition result of the voice represented by the voice data received until the acceptance of the confirmation request to the image captured by the imaging unit, and causing the display unit to display a screen on which an image in which both character strings representing the translation result of the voice represented by the voice data received are superimposed is arranged;
3. The display control system according to claim 1, wherein: an image output unit that outputs an image obtained by superimposing a character string on the image captured by the capturing unit to a video conference system;
The translation result display control means causes the screen generated by the video conference system to be displayed on the display unit.
4. The display control system according to any one of claims 1 to 3, characterized by: the voice data receiving means receives from the terminal the voice data representing voice input by the speaker to the terminal;
the confirmation request receiving means receives the confirmation request transmitted from the terminal in response to a predetermined operation performed by the speaker on the terminal;
The translation result display control means causes a display unit of the terminal to display a character string representing a translation result of the voice represented by the voice data received before the acceptance of the confirmation request,
The translation result display control means displays a screen in which an image obtained by superimposing a character string representing a translation result of the voice represented by the voice data received until the confirmation request is received on the image captured by the imaging unit is arranged. displayed on the display unit of the client device,
5. The display control system according to any one of claims 1 to 4, characterized in that: the voice data receiving means receives from the client device the voice data representing voice input by the speaker to the client device;
the confirmation request receiving means receives the confirmation request transmitted from the client device in response to a predetermined operation performed by the speaker on the client device;
The translation result display control means displays a screen in which an image obtained by superimposing a character string representing a translation result of the voice represented by the voice data received until the confirmation request is received on the image captured by the imaging unit is arranged. display on the display unit provided in the client device;
5. The display control system according to any one of claims 1 to 4, characterized in that: The translation control means controls to start translating the voice represented by the voice data received by the time the confirmation request is received into a plurality of languages,
The translation result display control means provides a screen on which an image obtained by superimposing a character string representing a translation result of the voice represented by the voice data for each of the plurality of languages on the image captured by the capturing unit is arranged. is displayed on the display unit,
7. The display control system according to any one of claims 1 to 6, characterized by: receiving speech data representing speech input by a speaker;
a step of receiving a confirmation request output in response to a predetermined operation performed by the speaker;
a step of controlling the reception of the confirmation request as a trigger to start translating the speech represented by the received audio data before the reception of the confirmation request;
a step of displaying on a display unit a screen in which an image captured by a capturing unit is superimposed with a character string representing a translation result of the voice represented by the voice data received before the acceptance of the confirmation request;
A display control method comprising: a procedure for accepting speech data representing speech input by a speaker;
a procedure for receiving a confirmation request output in response to a predetermined operation performed by the speaker;
A procedure for controlling the reception of the confirmation request as a trigger so that the translation of the speech represented by the received audio data is started by the reception of the confirmation request;
A procedure for displaying on a display unit a screen in which an image captured by an imaging unit is superimposed with a character string representing a translation result of the voice represented by the voice data received until the confirmation request is received,
A program characterized by causing a computer to execute

Images