JP2022114044A - Remote conference device, remote conference method, remote conference terminal and remote conference system - Google Patents

Abstract

To provide a technology that enables a remote conference with a plurality of participants to proceed more smoothly.SOLUTION: A remote conference device is provided, comprising: a speech detection unit which detects speech from sound data received from a first remote conference terminal: a speech information generating unit which inserts speech start data indicating that the speech has started on the basis of the detection of the speech from the sound data, into the front of the sound data; a speed adjustment unit which adjusts a first speed of the sound data on the basis of the insertion of the speech start data; and transmission control unit which controls the data transmission unit to transmit the sound data after the first speed has been adjusted to a second remote conference terminal after controlling the data transmission unit to transmit the speech start data to the second remote conference terminal.SELECTED DRAWING: Figure 1

Description

The present invention relates to a teleconference device, a teleconference method, a teleconference terminal, and a teleconference system.

In recent years, with the development of ICT (Information and Communication Technology) and changes in work styles, the establishment of satellite offices with a small number of people and the shift to telework are rapidly progressing. When multiple workers are dispersed in remote locations, they may not be able to know each other's situation in detail as compared to when they are gathered in one office. Therefore, it may become difficult for each worker to communicate smoothly with other workers in remote locations.

Therefore, a telework system has been proposed, in which multiple offices in remote locations are interconnected by video, sound, and information from various sensors, enabling smooth communication with users in remote locations. (For example, see Non-Patent Document 1).

In this system, cameras and microphones are installed at multiple locations in an office, and video data and sound data obtained from these cameras and microphones are transmitted to other offices away from the office. A user in the other office can freely switch the camera of interest among a plurality of cameras installed in remote locations, and each time the user switches cameras, a microphone installed near the camera to be switched to is used. Sounds collected by are played.

This allows the user to know the status of the remote location in real time. Then, users in remote locations can communicate with each other after understanding each other's situations.

JP 2013-017027 A Japanese Unexamined Patent Application Publication No. 2012-215600 JP 2017-135669 A

Nonaka et al., “Office Communication System Using Multiple Image/Sound/Sensor Information”, Human Interface Society Research Report Vol. Kano et al., “Study of sound image localization for making remote multi-person conversation easier to understand,” Information Processing Society of Japan Research Report, Vol.2014-GN-91, NO. 45, 2014. Morita, Itakura, "Expansion and Compression of Speech on the Time Axis Using Overlapping Addition Method (PICOLA) with Pointer Movement Amount Control and Its Evaluation", Proceedings of Acoustical Society of Japan, 1986.

However, it is desired to provide a technique that enables a remote conference with a plurality of participants to proceed more smoothly.

In order to solve the above problem, according to one aspect of the present invention, a speech detection unit for detecting speech from sound data received from a first teleconference terminal; an utterance information generating unit that inserts utterance start data indicating that the utterance has started before the sound data based on; and adjusts a first speed of the sound data based on the insertion of the utterance start data and a data transmission unit to transmit the speech start data to the second teleconference terminal, and then transmit the sound data after the first speed adjustment to the second teleconference terminal. a transmission control unit for controlling the data transmission unit to transmit to a terminal.

The speech information generating unit inserts the speech start data before the audio data and video data received together with the audio data from the first teleconference terminal, based on the detection of the speech; The speed adjustment unit adjusts the first speed of the sound data and the second speed of the video data to match the first speed after the adjustment based on the insertion of the speech start data. , the transmission control unit, after controlling the data transmission unit to transmit the speech start data to the second remote conference terminal, the sound data after the first speed is adjusted; The data transmission unit may be controlled to transmit the video data after the second speed adjustment to the second teleconference terminal.

The speed adjustment unit adjusts the first speed by making the first speed higher than before the insertion of the speech start data until the delay of the sound data due to the insertion of the speech start data is eliminated. You may

The speed adjustment unit may adjust the first speed by gradually lowering the first speed higher than before the speech start data is inserted until the delay is eliminated. .

The speed adjustment unit may adjust the first speed by deleting a part of the sound data after the speech start data is inserted.

The speed adjusting unit deletes the sound data of the non-speech section from the sound data after the insertion of the utterance start data, and fills the section vacated by the deletion with the sound data after the non-speech section. The first speed may be adjusted.

According to another aspect of the present invention, detecting an utterance from sound data received from a first remote conference terminal; inserting speech start data indicating that speech has started before the sound data; adjusting a first speed of the sound data based on the insertion of the speech start data; After controlling the data transmission unit to transmit to the second teleconference terminal, the data transmission unit is controlled to transmit the sound data after the first speed is adjusted to the second teleconference terminal. A teleconferencing method is provided, comprising:

Further, according to another aspect of the present invention, a message indicating that the speech has started is inserted before the sound data based on the detection of the speech from the sound data transmitted from another teleconference terminal. an acquisition unit that acquires from the remote conference device speech start data, which is data indicating the speech start data, and acquires sound data after the first speed is adjusted based on the insertion of the speech start data from the remote conference device; Based on the acquisition of the start data, the presentation unit is controlled to present the presentation data indicating that the speech has started, and the sound data after the first speed is adjusted is presented. and a presentation control section that controls the presentation section.

Further, according to another aspect of the present invention, the acquisition unit, based on the detection of the speech, before the audio data and the video data transmitted together with the audio data from the other teleconference terminal, The inserted speech start data is acquired from the remote conference device, and based on the insertion of the speech start data, the first speed is adjusted, and the first speed is adjusted. The video data after the second speed has been adjusted to match the speed is acquired from the remote conference device, and the presentation control unit presents the presentation data based on the acquisition of the speech start data. and control the presentation unit to present the sound data after the first speed adjustment and the video data after the second speed adjustment. good.

The presentation data may include predetermined image data, and the presentation control unit may control the monitor so that the predetermined image data is displayed as the presentation data.

The presentation data may include predetermined sound data, and the presentation control unit may control the speaker so that the predetermined sound data is displayed as the presentation data.

Further, according to another aspect of the present invention, a message indicating that the speech has started is inserted before the sound data based on the detection of the speech from the sound data transmitted from another teleconference terminal. obtaining from the teleconference device speech start data, which is data indicating the speech start data, and obtaining from the teleconference device sound data after the first speed has been adjusted based on the insertion of the speech start data; Based on the acquisition of the data, the presenting unit is controlled to present the presentation data indicating that the speech has started, and the sound data after the first speed is adjusted is presented. and controlling a presentation unit.

According to another aspect of the present invention, there is provided a teleconference system comprising a first teleconference terminal, a teleconference device, and a second teleconference terminal, wherein the first teleconference terminal is a transmission control unit for controlling a data transmission unit to transmit sound data, the teleconference device comprising: a speech detection unit for detecting speech from sound data received from the first teleconference terminal; an utterance information generating unit that inserts utterance start data indicating that the utterance has started based on the detection of the utterance from the sound data before the sound data; and based on the insertion of the utterance start data, After controlling a speed adjustment unit for adjusting the first speed of the sound data and a data transmission unit to transmit the speech start data to the second teleconference terminal, after the first speed is adjusted a transmission control unit for controlling the data transmission unit to transmit the sound data of to the second remote conference terminal, wherein the second remote conference terminal transmits the speech start data from the remote conference device an acquisition unit that acquires the sound data after the first speed is adjusted from the remote conference device; and a presentation indicating that the speech has started based on the acquisition of the speech start data. a presentation control unit for controlling a presentation unit to present data; and a presentation control unit for controlling the presentation unit to present sound data after the first speed has been adjusted. be.

As described above, according to the present invention, a technology is provided that enables a remote conference with a plurality of participants to proceed more smoothly.

1 is a block diagram showing an example of the configuration of a remote conference system according to an embodiment of the invention; FIG. It is a block diagram showing an example of the hardware constitutions of the device concerning the embodiment of the present invention.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

In addition, in the present specification and drawings, a plurality of constituent elements having substantially the same or similar functional configurations are distinguished by attaching different alphabets after the same reference numerals. However, when there is no particular need to distinguish between a plurality of components having substantially the same or similar functional configurations, only the same reference numerals are used.

(0. Background)
First, the background of the embodiments of the present invention will be described. As described above, in Non-Patent Document 1, a plurality of remote offices are connected to each other by video, sound, various sensor information, etc., and smooth communication with users in remote locations is possible. A telework system that enables By using the method described in Non-Patent Document 1, it is possible to allow the user to experience the current conditions of various remote locations with a rich presence.

However, when the user works by telework (for example, when the user works at home), it may be difficult to install equipment for creating a sense of presence. Alternatively, network bandwidth may be limited. Because of the potential for such problems, when a plurality of participants hold a conference, a system (teleconference system) that aggregates the video and audio of each of the participants is generally used.

However, when many people participate in a teleconference, it may be difficult for each participant to know the situation and atmosphere of other participants. In particular, when other participants turn off the video transmission function to secure network bandwidth, it is easy for each participant to fall into a situation where it is difficult for each participant to know the situation and mood of the other participants. Become.

In such a situation, at least a part of the temporal sections of speeches of two or more participants may overlap (hereinafter simply referred to as "overlap of speech sections"). This may make it difficult to hear the voices (spoken voices) of each of two or more participants. In addition, each time the utterance segments overlap, two or more participants interrupt their utterances, which may hinder the smooth progress of the teleconference.

In order to solve this problem, Patent Document 1 proposes a method of performing sound image localization processing on the voices of the participants so that the uttered voices of two or more participants can be heard from different directions. According to such a method, even if the utterance segments overlap, it becomes easier to hear the utterances of two or more participants. In addition, non-patent document 2 explains the effect that can be enjoyed by giving a sense of direction to the way the sound is heard.

However, there is a limit to the type of direction in which the sound image can be localized (the resolution of the sound image localization process), so if many participants speak at the same time, it will be difficult to hear the voices of many participants. . Moreover, even if the number of participants speaking at the same time is small, it is still difficult to distinguish between voices spoken by two or more participants at the same time. Therefore, even if the voices uttered simultaneously by a small number of participants can be heard, the participants may feel stress.

Therefore, this specification mainly proposes a technique for reducing the possibility that the utterance segments of a plurality of participants overlap. By reducing the possibility that the utterance segments of the plurality of participants overlap, the teleconference of the plurality of participants can proceed more smoothly.

(1. Details of embodiment)
Next, details of embodiments of the present invention will be described.

(1-1. System configuration)
FIG. 1 is a block diagram showing an example configuration of a remote conference system according to an embodiment of the present invention. Referring to FIG. 1, a remote conference system 1 according to an embodiment of the present invention has a remote conference server 10, N (N is an integer equal to or greater than 2) remote conference clients 20, and a network 40. FIG. The teleconference server 10 and N teleconference clients 20 are connected to a network 40 and can communicate with each other via the network 40 . Teleconference server 10 may function as an example of a teleconference device. Also, the teleconference client 20 can function as an example of a teleconference terminal.

Each of the N teleconference clients 20 is used by a corresponding participant (user). As an example, there is a one-to-one correspondence between the teleconference client 20 and the user. In the following description, when distinguishing the N remote conference clients 20, the N remote conference clients 20 are referred to as remote conference clients 20-1 to 20-N.

Further, in the embodiment of the present invention, it is mainly assumed that the teleconference client 20 is a PC (Personal Computer). However, the form of the remote conference client 20 is not limited. For example, the teleconference client 20 may be a mobile terminal such as a smart phone, or may be a voice input/output device installed at home or the like. For example, the audio input/output device may be an AI (Artificial Intelligence) speaker or the like.

(1-2. Functional Configuration of Teleconference Server)
Referring to FIG. 1 , remote conference server 10 includes data receiver 110 , decoder 120 , speech detector 130 , speech information generator 140 , media speed adjuster 150 , encoder 160 and data transmitter 170 . Media speed adjuster 150 may function as an example of a speed adjuster. Encoder 160 may function as an example of a transmission controller.

(1-3. Functional Configuration of Teleconference Client)
Referring to FIG. 1, the teleconference client 20 includes an encoder 210, a data transmitter 220, a data receiver 230, a decoder 240 and a video/audio reproducer 250. FIG. A microphone 31 and a camera 32 are connected to the encoder 210 . Encoder 210 may serve as an example of an acquirer. In the following description, when distinguishing between the N remote conference clients 20, the microphones 31 and cameras 32 connected to the encoders 210 of the N remote conference clients 20 are referred to as microphones 31-1 to 31-N. and cameras 32-1 to 32-N.

A speaker 33 and a monitor 34 are connected to the video/audio reproducing unit 250 . Speaker 33 and monitor 34 may each function as an example of a presentation unit. The video/audio reproduction unit 250 can function as an example of a presentation control unit. In the following description, when N teleconference clients 20 are to be distinguished, the speaker 33 and monitor 34 connected to the video/audio reproduction units 250 of the N teleconference clients 20 will be referred to as the speaker 33-1. . . . 33-N and monitors 34-1 through 34-N.

(1-4. Operation example of remote conference system)
Next, an operation example of the remote conference system 1 according to the embodiment of the present invention will be described with reference to FIG.

In the following description, it is assumed that the remote conference client 20-1 has the sound data transmission function turned on. In such a case, sound data obtained by the microphone 31-1 of the remote conference client 20-1 is transmitted to the remote conference clients 20-2 to 20-N other than the remote conference client 20-1 via the remote conference server 10. transmitted and output by speakers 33-2 to 33-N. As a result, the participant using the teleconference client 20-1 can make his/her own speech sound heard by the other participants using the teleconference clients 20-2 to 20-N.

Similarly, it is assumed that each of the remote conference clients 20-2 to 20-N has the sound data transmission function turned on. As a result, the participants using the teleconference clients 20-2 to 20-N can hear their own uttered voices to other participants. In this way, the conference progresses by having the participants listen to each other's spoken voices. However, the sound data transmission function does not necessarily have to be turned on in all of the remote conference clients 20-1 to 20-N. For example, among the remote conference clients 20-1 to 20-N, there may be one or more remote conference clients 20 whose sound data transmission function is turned off.

Also, in the following description, it is assumed that the remote conference client 20-1 has the video data transmission function turned on. In such a case, video data obtained by the camera 32-1 of the remote conference client 20-1 is sent to the remote conference clients 20-2 to 20-N other than the remote conference client 20-1 via the remote conference server 10. transmitted and output by monitors 34-2 through 34-N. As a result, the participant using the teleconference client 20-1 can show the other participants using the teleconference clients 20-2 to 20-N each of the participants using the teleconference clients 20-2 to 20-N. The video data obtained by the camera 32-1 of the remote conference client 20-1 may also be transmitted to the remote conference client 20-1 via the remote conference server 10 and output by the monitor 34-1.

Similarly, it is assumed that each of the teleconference clients 20-2 to 20-N has the video data transmission function turned on. As a result, the participants using the teleconference clients 20-2 to 20-N can show the other participants a video of themselves as the subject. By showing the video images among the participants in this manner, each participant can easily grasp the situation and atmosphere of the other participants. However, it is not necessary for all of the teleconference clients 20-1 to 20-N to have the video data transmission function turned on. For example, some or all of the teleconference clients 20-1 to 20-N may have their video data transmission function turned off.

In order to simplify the explanation below, an example in which audio data and video data are transmitted from the teleconference client 20-1 to each of the teleconference clients 20-2 to 20-N via the teleconference server 10 will be mainly described. explain. At this time, the remote conference client 20-1 functions as a first remote conference terminal, and each of the remote conference clients 20-2 to 20-N functions as a second remote conference terminal. However, such an example can be similarly applied to the case where audio data and video data are transmitted to the teleconference server 10 from the teleconference clients 20-2 to 20-N.

(Encoder 210)
First, in the teleconference client 20-1, the encoder 210 encodes the video data obtained by the microphone 31-1 and the sound data obtained by the camera 32-1 using an arbitrary codec.

(Data transmission unit 220)
The data transmission unit 220 transmits the video data and audio data encoded by the encoder 210 to the remote conference server 10 via the network 40 .

(Data receiving unit 110)
In the teleconference server 10, the data receiving unit 110 receives encoded video data and audio data from the teleconference client 20-1 via the network 40. FIG.

(decoder 120)
The decoder 120 decodes the encoded video data and audio data using an arbitrary codec to obtain decoded video data and audio data.

(Utterance detection unit 130)
The speech detection unit 130 attempts to detect speech from the sound data decoded by the decoder 120 . More specifically, the speech detection unit 130 detects whether or not the participant has started speaking based on the decoded sound data.

Note that the speech detection method is not limited to a specific method. As an example of an utterance detection method, a method is used in which it is determined that an utterance has started when a segment determined from sound data changes from a non-speech segment such as noise to a speech segment using speech segment detection technology. may be For example, as a method of detecting a speech interval, a method of determining a speech interval based on amplitude fluctuations or changes in acoustic features may be used, or the method disclosed in Patent Document 2 may be used. good too.

(Utterance information generation unit 140)
Speech information generation section 140 generates speech start data indicating that speech has started, based on detection of speech from sound data by speech detection section 130 (that is, based on detection of the start of speech). is inserted before the audio and video data. The speech start data can serve as a trigger signal for causing the teleconference clients 20-2 to 20-N to present presentation data indicating that speech has started. Examples of presentation data are described in detail later.

Note that, when video data is not transmitted from the remote conference client 20-1, the speech information generation unit 140 converts the speech start data into the sound data based on the detection of speech from the sound data by the speech detection unit 130. should be inserted before Also, it is desirable that the time length of the speech start data is set to the extent that the delay time of the sound data does not become too large (for example, it may be set between 100 milliseconds and 2000 milliseconds). .

(Media speed adjustment unit 150)
The media speed adjustment unit 150 adjusts the speed (first speed) of sound data based on the speech start data inserted by the speech information generation unit 140 . More specifically, by inserting the speech start data before the sound data, the sound data is delayed by at least the temporal length of the speech start data. Therefore, the media speed adjustment unit 150 adjusts the speed of the sound data until the delay in the sound data is eliminated (that is, until the delay in the sound data is recovered by the time length of the speech start data). It is desirable to adjust the rate of the audio data by making it higher than it was before the data was inserted (ie by doubling the rate of the audio data).

Here, the method of adjusting the speed of sound data is not limited to a specific method. For example, as a method for adjusting the speed of sound data, there is speech speed conversion that changes the speech speed while maintaining the voice quality of each person, represented by PICOLA (Pointer Interval Control OverLap and Add) shown in Non-Patent Document 3. techniques can be used.

In addition, there are no specific restrictions on when and to what level the speed of the sound data is adjusted. For example, if the speed of the sound data is increased too much, the participants listening to the sound may feel uncomfortable. Therefore, the media speed adjustment unit 150 may suppress the speed of sound data to a certain level or less. As an example, the media speed adjustment unit 150 adjusts the speed of the sound data so that the speed of the sound data after adjustment is between 1.1 and 1.5 times the speed of the sound data before the speech start data is inserted. may be adjusted.

Further, when the delay of the sound data is eliminated, the media speed adjustment unit 150 restores the speed of the sound data to the speed of the sound data before the speech start data is inserted. At this time, if the speed of the sound data suddenly returns, there is a possibility that the participants who are listening to the sound will feel uncomfortable. Therefore, the media speed adjustment unit 150 may gradually decrease the speed of the sound data while increasing the speed of the sound data from before the insertion of the speech start data until the delay of the sound data is eliminated.

As an example, the media speed adjustment unit 150 may adjust the speed of the sound data so that the speed of the sound data immediately after the speech start data is 1.5 times the speed of the sound data before the speech start data is inserted. good. Then, the media speed adjustment unit 150 adjusts the speed of the sound data so that as the delay time of the sound data becomes shorter, the speed of the sound data becomes 1.3 times and 1.1 times the speed of the sound data before the speech start data is inserted. The data rate may be gradually reduced.

Also, the media speed adjustment unit 150 may delete part of the sound data after the speech start data is inserted. This can also eliminate the delay in sound data. For example, the media speed adjustment unit 150 deletes the sound data of the non-speech section detected by the speech detection unit 130 from the sound data after the speech start data is inserted, and inserts the sound data after the non-speech section into the section vacated by the deletion. You can fill in the data.

In addition, the media speed adjustment unit 150 adjusts the speed of the video data (second speed) so as to match the speed of the audio data after being adjusted (first speed) based on the insertion of the speech start data. . As a result, the other participants can perceive the voice and video of the participant who has spoken without any sense of incongruity. The media speed adjustment unit 150 may adjust the speed of the video data by increasing the speed of the video data (that is, by doubling the speed of the video data) in the same manner as the speed of the audio data.

In addition, when the sound data of the non-speech section is deleted, the media speed adjustment unit 150 may also delete the video data of the non-speech section, and fill the space left by the deletion with the video data after the non-speech section. . At this time, the media speed adjusting unit 150 may perform, for example, morphing processing on the video data in order to suppress the sense of incongruity that occurs in the video data before and after the deleted section. Note that the media speed adjustment unit 150 need not adjust the speed of the video data when the video data is not transmitted from the teleconference client 20-1.

(Encoder 160)
The encoder 160 encodes the speech start data inserted before the audio data and the video data by the speech information generating section 140 using an arbitrary codec. Encoder 160 then outputs the encoded speech start data to data transmitting section 170 . This controls the transmission of the encoded speech start data by the data transmission unit 170 .

Subsequently, the encoder 160 encodes the audio data and video data whose speed has been adjusted by the media speed adjustment unit 150 using an arbitrary codec. Encoder 160 then outputs the encoded audio data and video data to data transmitting section 170 . This controls the transmission of the encoded audio data and video data by the data transmission unit 170 .

(Data transmission unit 170)
The data transmission unit 170 transmits the speech start data encoded by the encoder 160 via the network 40 to each of the remote conference clients 20-2 to 20-N other than the remote conference client 20-1. Subsequently, the data transmission unit 170 transmits the audio data and video data encoded by the encoder 160 to each of the teleconference clients 20-2 to 20-N other than the teleconference client 20-1 via the network 40. do.

(Data receiving unit 230)
In the teleconference clients 20-2 to 20-N, the data reception unit 230 receives the encoded speech start data transmitted from the teleconference server 10. FIG. Subsequently, the data receiving unit 230 receives the encoded audio data and video data transmitted from the remote conference server 10 .

(Decoder 240)
Decoder 240 decodes the encoded speech start data received by data receiving section 230 using an arbitrary codec to obtain decoded speech start data. Further, the decoder 240 decodes the encoded audio data and video data received by the data receiving unit 230 using an arbitrary codec to obtain decoded audio data and video data.

(Video/audio reproduction unit 250)
The video/audio reproducing unit 250 controls to present the presentation data indicating the start of speech based on the fact that the speech start data decoded by the decoder 240 is obtained. Here, the type of presentation data is not limited to specific presentation data. For example, presentation data may include predetermined image data (still image data or moving image data). Still image data may include icons. Also, moving image data may include animation. In such a case, the video/audio reproducing unit 250 controls the monitor 34 so that a predetermined image is displayed as an example of presentation data.

Alternatively, the presentation data may include predetermined sound data. In such a case, the video/audio reproducing unit 250 controls the speaker 33 so that predetermined sound data is output as an example of presentation data.

Further, the video/audio reproduction unit 250 reproduces the decoded audio data and video data based on the fact that the audio data and the video data after being decoded by the decoder 240 are obtained. The video/audio reproducing unit 250 outputs the reproduced sound data to the speaker 33 . As a result, the speaker 33 outputs sound data. Also, the video/audio reproducing unit 250 outputs the reproduced video data to the monitor 34 . As a result, the monitor 34 outputs video data.

The operation example of the remote conference system 1 according to the embodiment of the present invention has been described above.

(2. Hardware configuration)
FIG. 2 shows a teleconference server 10 and a teleconference client 20 according to an embodiment of the present invention (hereinafter, the teleconference server 10 and the teleconference client 20 may be referred to as "apparatus according to this embodiment" without distinction). ) is a block diagram showing an example of a hardware configuration.

Note that the remote conference server 10 and the remote conference client 20 do not need to have all of the following hardware configurations (for example, the remote conference server 10 does not need to have a sensor directly). A hardware module capable of realizing the functional configuration of each device may be appropriately limited and provided.

Referring to FIG. 2, the device according to this embodiment includes a bus 801, a CPU (Central Processing Unit) 803, a ROM (Read Only Memory) 805, a RAM (Random Access Memory) 807, a storage device 809, a communication interface 811, a sensor 813 , an input device 815 , a display device 817 and a speaker 819 .

A CPU 803 executes various processes in the apparatus according to this embodiment. The ROM 805 also stores programs and data for causing the CPU 803 to execute processing in the apparatus according to this embodiment. The RAM 807 also temporarily stores programs and data when the CPU 803 executes processing.

A bus 801 interconnects the CPU 803 , ROM 805 and RAM 807 . A storage device 809 , a communication interface 811 , a sensor 813 , an input device 815 , a display device 817 and a speaker 819 are also connected to the bus 801 . Bus 801 includes, for example, multiple types of buses. As an example, bus 801 includes a high speed bus connecting CPU 803, ROM 805 and RAM 807, and one or more other buses that are slower than the high speed bus.

A storage device 809 stores data to be temporarily or permanently stored in the device according to this embodiment. The storage device 809 may be, for example, a magnetic storage device such as a hard disk, or may be EEPROM (Electrically Erasable and Programmable Read Only Memory), flash memory, MRAM (Magnetoresistive Random Access Memory). , FeRAM (Ferroelectric Random Access Memory) and PRAM (Phase change Random Access Memory).

A communication interface 811 is communication means provided in the device according to the present embodiment, and communicates with an external device via a network (or directly). The communication interface 811 may be an interface for wireless communication, in which case it may include, for example, a communication antenna, RF circuits and other circuits for processing communication. Also, the communication interface 811 may be an interface for wired communication, and in this case may include, for example, a LAN terminal, a transmission circuit, and other circuits for communication processing.

Sensor 813 is, for example, a camera, microphone, biosensor, other sensor, or a combination thereof. A camera captures an image of a subject, and includes, for example, an optical system, an image sensor, and an image processing circuit. A microphone picks up ambient sound, converts the sound into an electrical signal, and converts the electrical signal into digital data.

The input device 815 is a touch panel, a mouse, a line-of-sight detection device such as a camera, a microphone, or the like. The display device 817 displays an output image (that is, a display screen) from the device according to the present embodiment, and is realized using, for example, a liquid crystal, an organic EL (Organic Light-Emitting Diode), a CRT (Cathode Ray Tube), or the like. can be The speaker 819 outputs sound, converts digital data into an electric signal, and converts the electric signal into sound.

The hardware configuration example of the apparatus according to the present embodiment has been described above.

(3. Summary)
As described above, the remote conference server 10 according to the embodiment of the present invention includes the speech detection unit 130 that detects speech from sound data received from the remote conference client 20-1, and the speech detection unit 130 that detects speech from the sound data. a speech information generating unit 140 that inserts speech start data indicating that speech has started before the sound data based on the fact that the speech has started; and adjusts the first speed of the sound data based on the insertion of the speech start data. After controlling the media speed adjustment unit 150 and the data transmission unit 170 to transmit the speech start data to the remote conference clients 20-2 to 20-N, the audio data after the first speed adjustment is transmitted to the remote conference client. and an encoder 160 that controls the data transmission unit 170 to transmit to the clients 20-2 to 20-N.

According to such a configuration, speech start data is delivered before sound data. As a result, the participant can quickly notice the speech of the other participants, so that even if he/she tries to speak, he or she can stop speaking just before the speech. Therefore, it is less likely that one participant will interrupt another participant's speech (that is, it is less likely that the utterance segments of a plurality of participants will overlap). As a result, it becomes possible to smoothly proceed with a teleconference involving a plurality of participants.

The reason why the speech start data is distributed before the sound data enables the participants to quickly notice the speech of the other participants is as follows. That is, when a participant hears a sound coming from a remote location where another participant exists and recognizes that the other participant has spoken, the participant listens to the sound for a certain length (for example, 1 second or more). Only after that, it is usually possible to recognize that the sound contains speech by other participants. Therefore, it is easy for a participant to start speaking by the time he or she recognizes that another participant has spoken. is likely to occur.

On the other hand, as in the embodiment of the present invention, by presenting speech start data as visual information to the participants, it is possible to recognize that other participants have spoken based on the sound coming from a remote location. Compared to the case, the participants can quickly recognize that the other participants have started speaking. Even when speech start data is presented as predetermined sound data (for example, special sound data prepared in advance), participants can quickly recognize that other participants have started speaking. (for example, it becomes possible to recognize in a few milliseconds). Note that the special sound data is not limited as long as it is a sound other than a human voice, and may be a predetermined warning sound.

Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. It is understood that these also naturally belong to the technical scope of the present invention.

For example, as described above, embodiments of the present invention reduce the likelihood of overlapping speech segments of multiple participants. However, it is conceivable that a plurality of participants may speak at the same time and the teleconference client 20 may receive a plurality of speech start data from the teleconference server 10 . In such a case, the video/audio reproducing unit 250 may perform sound image localization processing such as the method disclosed in Patent Document 3, and reproduce the uttered voices of a plurality of participants with a sense of direction.

1 remote conference system 10 remote conference server 110 data receiver 120 decoder 130 speech detector 140 speech information generator 150 media speed adjuster 160 encoder 170 data transmitter 20 remote conference client 210 encoder 220 data transmitter 230 data receiver 240 decoder 250 video/audio playback unit 31 microphone 32 camera 33 speaker 34 monitor 40 network

Claims (13)

a speech detection unit that detects speech from sound data received from the first teleconference terminal;
an utterance information generation unit that inserts utterance start data indicating that the utterance has started, before the sound data, based on the detection of the utterance from the sound data;
a speed adjustment unit that adjusts a first speed of the sound data based on the insertion of the speech start data;
After controlling the data transmission unit to transmit the speech start data to the second teleconference terminal, the sound data after the first rate is adjusted is transmitted to the second teleconference terminal. a transmission control unit that controls the data transmission unit;
A remote conference device. The speech information generating unit inserts the speech start data before the audio data and video data received together with the audio data from the first teleconference terminal, based on the detection of the speech;
The speed adjustment unit adjusts the first speed of the sound data and the second speed of the video data to match the first speed after the adjustment based on the insertion of the speech start data. ,
After controlling the data transmission unit to transmit the speech start data to the second remote conference terminal, the transmission control unit controls the sound data after the first rate is adjusted, and the controlling the data transmission unit to transmit the video data after the speed adjustment of 2 to the second teleconference terminal;
The teleconference device according to claim 1. The speed adjustment unit adjusts the first speed by making the first speed higher than before the insertion of the speech start data until the delay of the sound data due to the insertion of the speech start data is eliminated. do,
3. The teleconference device according to claim 1 or 2. The speed adjustment unit adjusts the first speed by gradually lowering the first speed higher than before the speech start data is inserted until the delay is eliminated.
The teleconference device according to claim 3. The speed adjustment unit adjusts the first speed by deleting part of the sound data after the speech start data is inserted.
3. The teleconference device according to claim 1 or 2. The speed adjusting unit deletes the sound data of the non-speech section from the sound data after the insertion of the utterance start data, and fills the section vacated by the deletion with the sound data after the non-speech section. adjusting the first speed;
The remote conference device according to claim 5. detecting speech from sound data received from the first teleconference terminal;
inserting speech start data indicating that the speech has started based on the detection of the speech from the sound data before the sound data;
adjusting a first rate of the sound data based on the insertion of the start-of-speech data;
After controlling the data transmission unit to transmit the speech start data to the second teleconference terminal, the sound data after the first rate is adjusted is transmitted to the second teleconference terminal. controlling the data transmitter;
A teleconferencing method comprising: Speech start data, which is data indicating that the speech has started and is inserted before the sound data based on detection of the speech from sound data transmitted from another teleconference terminal, is sent from the teleconference device. an acquisition unit that acquires, from the teleconference device, sound data after the first speed has been adjusted based on the insertion of the speech start data;
Based on the acquisition of the speech start data, the presentation unit is controlled to present the presentation data indicating that the speech has started, and the sound data after the first speed is adjusted is presented. a presentation control unit that controls the presentation unit such that
A remote conference terminal. The acquisition unit acquires the speech start data inserted before the audio data and the video data transmitted together with the audio data from the other remote conference terminal based on the detection of the speech. The sound data after the first speed is adjusted, and the second speed is adjusted to match the first speed after the adjustment, obtained from the device and based on the insertion of the speech start data. obtaining the video data after the video data from the teleconference device;
The presentation control unit controls the presentation unit to present the presentation data based on the acquisition of the speech start data, and controls the presentation unit to present the presentation data, and the sound data after the first speed is adjusted and the first speed. controlling the presentation unit to present the video data after the speed of 2 has been adjusted;
The teleconference terminal according to claim 8. the presentation data includes predetermined image data;
The presentation control unit controls the monitor so that the predetermined image data is displayed as the presentation data.
The teleconference terminal according to claim 8 or 9. the presentation data includes predetermined sound data;
The presentation control unit controls the speaker so that the predetermined sound data is displayed as the presentation data.
The teleconference terminal according to claim 8 or 9. Speech start data, which is data indicating that the speech has started and is inserted before the sound data based on detection of the speech from sound data transmitted from another teleconference terminal, is sent from the teleconference device. obtaining from the teleconference device sound data obtained and having a first speed adjusted based on the insertion of the speech start data;
Based on the acquisition of the speech start data, the presentation unit is controlled to present the presentation data indicating that the speech has started, and the sound data after the first speed is adjusted is presented. and controlling the presentation unit to
A teleconferencing method comprising: A teleconference system comprising a first teleconference terminal, a teleconference device, and a second teleconference terminal,
The first teleconference terminal,
A transmission control unit that controls the data transmission unit to transmit sound data,
The teleconference device is
a speech detection unit that detects speech from sound data received from the first teleconference terminal;
an utterance information generation unit that inserts utterance start data indicating that the utterance has started, before the sound data, based on the detection of the utterance from the sound data;
a speed adjustment unit that adjusts a first speed of the sound data based on the insertion of the speech start data;
After controlling the data transmission unit to transmit the speech start data to the second teleconference terminal, the sound data after the first rate is adjusted is transmitted to the second teleconference terminal. a transmission control unit that controls the data transmission unit;
with
The second teleconference terminal,
an acquisition unit that acquires the speech start data from the teleconference device and acquires the sound data after the first speed is adjusted from the teleconference device;
Based on the acquisition of the speech start data, the presentation unit is controlled to present the presentation data indicating that the speech has started, and the sound data after the first speed is adjusted is presented. a presentation control unit that controls the presentation unit such that
A remote conference system.

Images