JP7290366B2 - COMMUNICATION TERMINAL, REMOTE CONFERENCE METHOD AND PROGRAM - Google Patents

Description

The present invention relates to a communication terminal, teleconference method and program.

2. Description of the Related Art In recent years, it has become possible to hold a conference between a plurality of communication terminals provided at distant points via a network. In a teleconference system that holds such a teleconference, it is not necessary for the participants of the conference to gather in one conference room. can be done.

In relation to such technology, Patent Literature 1 discloses a communication control device that provides a realistic conference while considering the load on the communication line. Further, Patent Literature 2 discloses a conference system that is highly expressive, easy to speak, and rich in interactivity without impairing the reproduction of existing speech even if a new speech is made during the reproduction of existing speech.

JP 2010-239393 A JP-A-2001-230773

In teleconferencing, it may be difficult to achieve complete real-time communication due to communication delays and the like. In addition, since the faces of the other participants may not be visible, it may be difficult to understand the behavior of the other participants. In such a case, an utterance conflict may occur in which another participant speaks while another participant is speaking (uttering). When a speech collision occurs, the participant who speaks late may refrain from speaking. In this case, the dissatisfaction of the participant who spoke late (that is, the speech collision occurred) may increase. Therefore, if a speech collision occurs, there is a risk that the teleconference will not proceed smoothly.

An object of the present disclosure is to solve such problems, and to provide a teleconference system, a communication terminal, a teleconference method, and a program that enable a teleconference to proceed smoothly. be.

A teleconferencing system according to the present disclosure includes speech determination means for determining whether the voice of each of a plurality of participants in a remote conference indicates an utterance or indicates a backtracking; is controlled so that it is output on the communication terminal of each of the participants, and when another participant speaks while a participant among the plurality of participants is speaking, the other participant voice output control means for controlling to suppress the output of the utterances of the participants; counting means for counting the number of first utterances, which are utterances whose output is suppressed, for each of the plurality of participants; and number display control means for controlling such that the number of times is displayed on the communication terminal of each of the plurality of participants.

Further, the communication terminal according to the present disclosure includes speech determination means for determining whether the user's voice indicates an utterance or a backtracking in a teleconference in which the user of the communication terminal participates; The voice of each of the participants is output at the communication terminal, and the voice of the user is output at the first communication terminal, which is the communication terminal of each of the plurality of participants, and the plurality of participants When the user speaks while one of the participants is speaking, voice output is controlled to suppress the output of the user's speech at the first communication terminal. control means; counting means for counting the number of first utterances, which are utterances whose output is suppressed, for a user of the communication terminal; and control to display the number of times on the first communication terminal. and a control means for displaying the number of times performed.

Further, the remote conference method according to the present disclosure determines whether the voice of each of the plurality of participants in the remote conference indicates an utterance or indicates a backtracking, and the voice of each of the plurality of participants control so that the communication terminal of each of the participants is output, and when another participant speaks while a certain participant among the plurality of participants is speaking, the other participant control to suppress the output of the utterance of the plurality of participants, the number of first utterances, which is the utterance whose output is suppressed, is counted for each of the participants, and the display of the number of times is displayed for each of the plurality of participants Control is performed as in the communication terminal.

Further, the program according to the present disclosure includes a function of determining whether the voice of each of a plurality of participants in a teleconference indicates an utterance or a backtracking, and control so that the communication terminal of each of the participants is output, and when another participant speaks while a certain participant among the plurality of participants is speaking, the other participant a function of controlling to suppress the output of an utterance of a participant; a function of counting the number of first utterances, which are utterances whose output is suppressed, for each of the participants; and a function of controlling the communication terminal of each participant.

According to the present disclosure, it is possible to provide a teleconference system, a communication terminal, a teleconference method, and a program that enable a teleconference to proceed smoothly.

1 is a diagram showing a remote conference system according to an embodiment of the present disclosure; FIG. 4 is a flow chart showing a teleconferencing method executed by the teleconferencing system according to the embodiment of the present disclosure; 1 illustrates a remote conference system according to a first embodiment; FIG. 1 is a diagram showing a configuration of a communication terminal according to Embodiment 1; FIG. 1 is a diagram showing a configuration of a teleconference device according to Embodiment 1; FIG. 4 is a diagram exemplifying participant information according to the first embodiment; FIG. 4 is a flow chart showing a teleconference method executed by the teleconference system according to the first embodiment; FIG. 10 is a diagram showing a teleconference system according to a second embodiment; FIG. FIG. 10 is a diagram illustrating a state in which speech state information is transmitted and received in the teleconferencing system according to the second embodiment; FIG. 11 is a block diagram showing the configuration of an utterance state detection unit according to the second exemplary embodiment; FIG. FIG. 10 is a diagram illustrating conference information according to the second embodiment; FIG. FIG. 10 is a diagram showing the configuration of a conference control unit according to the second embodiment; FIG. 9 is a flow chart showing a teleconference method executed by the teleconference system according to the second embodiment; FIG. 10 is a diagram illustrating a conference image displayed on each communication terminal in a teleconference according to the second embodiment; FIG. 10 is a diagram illustrating a conference image displayed on each communication terminal in a teleconference according to the second embodiment;

(Overview of Embodiments According to the Present Disclosure)
Prior to describing the embodiments of the present disclosure, an outline of the embodiments of the present disclosure will be described. FIG. 1 is a diagram showing a remote conference system 1 according to an embodiment of the present disclosure. The remote conference system 1 realizes a remote conference (web conference). A remote conference is held using communication terminals of a plurality of participants. The teleconference system 1 can be implemented by, for example, a computer. The teleconference system 1 may be realized by each communication terminal of participants in the teleconference, or may be realized by a server or the like that manages the teleconference. Also, the teleconferencing system 1 may be realized by a plurality of devices such as a server and a communication terminal.

The teleconference system 1 has an utterance determination section 2 , an audio output control section 4 , a count section 6 , and a number display control section 8 . The utterance determination unit 2 has a function as utterance determination means. The audio output control section 4 has a function as an audio output control means. The counting section 6 has a function as counting means. The frequency display control section 8 has a function as frequency display control means.

FIG. 2 is a flow chart showing a remote conference method executed by the remote conference system 1 according to the embodiment of the present disclosure. The utterance determination unit 2 determines whether the voices of the plurality of participants in the teleconference represent utterances or backtracks (step S12). A determination method will be described in subsequent embodiments. Here, "utterance" is sound (utterance) corresponding to words (language) with meaningful content. On the other hand, a "backtrack" is a voice (utterance) corresponding to a word that has no meaning in itself. In this specification, the terms "utterance" and "backhand" are paired with each other.

The audio output control unit 4 performs control so that the audio of each of the multiple participants is output from the communication terminals of each of the multiple participants. The audio output control unit 4 suppresses the output of the other participant's speech when another participant speaks while a certain participant among the plurality of participants is speaking. Control is performed (step S14). In other words, the voice output control unit 4 suppresses the output of other participants' utterances (collision utterances) when an utterance collision occurs. In addition, hereinafter, the later utterance (the utterance that caused the utterance collision) may be referred to as "conflicting utterance". Therefore, a conflicting utterance is an utterance whose output is suppressed. Note that suppressing the output of conflicting utterances means, for example, not outputting conflicting utterances from the communication terminals of the participants, but is not limited to this.

In the present embodiment, "utterance collision" means that when one participant is speaking, another participant speaks. does not mean that is output at the same time on each communication terminal. Note that in this embodiment, the output of later utterances among the multiple participants' utterances may be suppressed. Therefore, in the present embodiment, the participant who uttered the conflicting utterance can recognize that the "speech conflict" has occurred, but the other participants may not recognize it. In other words, the participant who made the colliding utterance made the utterance while the other participant's utterance was being output from his/her own communication terminal, so he/she can grasp that the utterance collision has occurred. On the other hand, since the output of the colliding utterance is suppressed at each communication terminal, participants other than the participant who uttered the colliding utterance may not recognize that the utterance collision has occurred.

The counting unit 6 counts the number of utterances whose output is suppressed (conflicting utterance; first utterance) for each participant (step S16). The number display control unit 8 performs control so that the number of times is displayed on the communication terminals of a plurality of participants (step S18). As a result, each participant can grasp which participant's utterance collision occurs more often.

Here, it can be said that a participant who has made a large number of conflicting utterances (has a large number of occurrences of utterance conflict) is a participant who wants to speak. Therefore, by displaying on the communication terminal of a participant in the teleconference that the number of conflicting utterances is large, the other participants can recognize that the participant wants to make an utterance. As a result, other participants can perform actions such as prompting the participant to speak or waiting until the participant speaks. Therefore, the dissatisfaction that the participant cannot speak even if he/she wants to speak can be reduced. Therefore, the teleconference system 1 according to the present embodiment enables the teleconference to proceed smoothly.

(Embodiment 1)
Hereinafter, embodiments will be described with reference to the drawings. For clarity of explanation, the following descriptions and drawings are omitted and simplified as appropriate. Moreover, in each drawing, the same elements are denoted by the same reference numerals, and redundant description is omitted as necessary.

FIG. 3 is a diagram showing the remote conference system 20 according to the first embodiment. The teleconference system 20 has a plurality of communication terminals 30 and a teleconference device 100 . A communication terminal 30 may be provided for each remote conference participant. The plurality of communication terminals 30 and the remote conference device 100 are connected to each other via the network 22 so as to be able to communicate with each other. Network 22 may be wired, wireless, or a combination of wired and wireless. The network 22 may be the Internet or a LAN (Local Area Network).

The communication terminal 30 is, for example, a computer owned by a participant. The communication terminal 30 is, for example, a personal computer (PC) and a mobile terminal such as a smart phone or a tablet terminal. When a participant participates in a remote conference, the communication terminal 30 transmits audio data indicating the voice (speech or backtracking) uttered by the participant to the remote conference device 100 via the network 22 . Also, the communication terminal 30 receives voice data representing voices (utterances or backtracking) of other participants from the teleconference device 100 via the network 22 . Then, the communication terminal 30 outputs the voice corresponding to the voice data so that the participant who is the user of the communication terminal 30 can listen to it.

The remote conference device 100 is, for example, a computer such as a server. The teleconference device 100 manages teleconferences. The teleconference device 100 receives voice data from each participant's communication terminal 30 and transmits the voice data to a plurality of communication terminals 30 . In this case, the teleconference device 100 does not have to transmit the voice data to the communication terminal 30 that transmitted the voice data (the same applies to other embodiments). In Embodiment 1, the term "speech" can also mean "speech data representing speech" as a processing target in information processing.

FIG. 4 is a diagram showing the configuration of the communication terminal 30 according to the first embodiment. The communication terminal 30 has a control unit 32, a storage unit 34, a communication unit 36, and an interface unit 38 (IF: Interface) as main hardware components. The control unit 32, storage unit 34, communication unit 36, and interface unit 38 are interconnected via a data bus or the like.

The control unit 32 is a processor such as a CPU (Central Processing Unit), for example. The control unit 32 has a function as an arithmetic device that performs control processing, arithmetic processing, and the like. The storage unit 34 is, for example, a storage device such as memory or hard disk. The storage unit 34 is, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 34 has a function of storing control programs, arithmetic programs, and the like executed by the control unit 32 . The storage unit 34 also has a function of temporarily storing processing data and the like. Storage unit 34 may include a database.

The communication unit 36 performs processing necessary for communicating with devices such as the remote conference device 100 that constitute the remote conference system 20 . Communication unit 36 may include communication ports, routers, firewalls, and the like. The interface unit 108 is, for example, a user interface (UI). The interface unit 108 has an input device such as a keyboard, touch panel, or mouse, and an output device such as a display or speaker. The interface unit 108 receives a data input operation by a user (operator) and outputs information to the user. Further, the interface unit 108 may have a sound collecting device such as a microphone and an imaging device such as a camera as input devices. Also, at least part of interface section 108 does not need to be physically integrated with communication terminal 30 . At least part of the interface section 108 may be connected to the communication terminal 30 by wire or wirelessly.

In addition, the communication terminal 30 has an audio acquisition unit 42, an audio transmission unit 44, an audio reception unit 46, an audio output unit 48, a display information reception unit 52, and an image display unit 54 as components. The audio acquisition unit 42, the audio transmission unit 44, the audio reception unit 46, the audio output unit 48, the display information reception unit 52, and the image display unit 54 may be implemented by the hardware configuration described above, or may be implemented by software. may be

The speech acquisition unit 42 acquires speech uttered by the user of the communication terminal 30 who is a participant in the teleconference. The voice acquisition unit 42 may acquire voice using a sound collector, which is the interface unit 38 . The voice transmission unit 44 transmits the acquired user's voice (voice data) to the teleconference device 100 via the network 22 . The audio transmission unit 44 may transmit audio (audio data) through the communication unit 36 .

The voice receiving unit 46 receives voices (audio data) of a plurality of participants in the remote conference from the remote conference device 100 via the network 22 . The voice receiving unit 46 may receive voice (audio data) through the communication unit 36 . The voice output unit 48 outputs voices of a plurality of participants so that the user of the communication terminal 30 can listen to them. The audio output unit 48 may output audio through a speaker that is the interface unit 38 .

The display information receiving unit 52 receives display information from the teleconference device 100 via the network 22 . Here, the display information is information indicating information displayed by the interface section 38 of the communication terminal 30 . Display information will be described later. The display information receiving section 52 may receive the display information through the communication section 36 . The image display unit 54 displays an image corresponding to the received display information. The image display section 54 may display an image on the display that is the interface section 38 .

FIG. 5 is a diagram showing the configuration of the remote conference device 100 according to the first embodiment. The remote conference device 100 has a control unit 102, a storage unit 104, a communication unit 106, and an interface unit 108 as main hardware components. The control unit 102, storage unit 104, communication unit 106 and interface unit 108 are interconnected via a data bus or the like.

The control unit 102 is a processor such as a CPU, for example. The control unit 102 has a function as an arithmetic device that performs analysis processing, control processing, arithmetic processing, and the like. The storage unit 104 is, for example, a storage device such as memory or hard disk. The storage unit 104 is, for example, ROM or RAM. The storage unit 104 has a function of storing a control program, an arithmetic program, and the like executed by the control unit 102 . The storage unit 104 also has a function of temporarily storing processing data and the like. Storage unit 104 may include a database.

The communication unit 106 performs processing necessary for communicating with other devices such as the communication terminal 30 via the network 22 . Communication unit 106 may include communication ports, routers, firewalls, and the like. The interface unit 108 is, for example, a user interface (UI). The interface unit 108 has an input device such as a keyboard, touch panel, or mouse, and an output device such as a display or speaker. The interface unit 108 receives data input operations by the operator and outputs information to the operator.

The teleconferencing device 100 according to the first embodiment includes, as components, a participant information storage unit 110, an audio reception unit 112, an utterance determination unit 120, an audio output control unit 130, a number count unit 140, and a display unit. and a control unit 150 . The voice output control section 130 has a speech collision determination section 132 and a speech output suppression section 134 . The display control unit 150 has a count display control unit 152 and an icon display control unit 154 . Note that the teleconference device 100 does not have to be physically composed of one device. In this case, each component described above may be implemented by a plurality of physically separate devices.

The participant information storage section 110 has a function as a participant information storage means. The voice receiving unit 112 has a function as voice receiving means. The utterance determination unit 120 corresponds to the utterance determination unit 2 shown in FIG. The utterance determination unit 120 has a function as utterance determination means. The audio output control section 130 corresponds to the audio output control section 4 shown in FIG. The audio output control section 130 has a function as audio output control means. The number counting section 140 corresponds to the counting section 6 shown in FIG. The number counting unit 140 has a function as number counting means. The display control unit 150 has a function as display control means.

The speech collision determination unit 132 also functions as a speech collision determination means. The speech output suppression unit 134 functions as speech output suppression means. The number display control unit 152 corresponds to the number display control unit 8 shown in FIG. The number display control unit 152 has a function as number display control means. The icon display control unit 154 has a function as icon display control means.

Each component described above can be realized by executing a program under the control of the control unit 102, for example. More specifically, each component can be implemented by control unit 102 executing a program stored in storage unit 104 . Further, each component may be realized by recording necessary programs in an arbitrary non-volatile recording medium and installing them as necessary. Moreover, each component may be implemented by any combination of hardware, firmware, and software, without being limited to being implemented by program software. Also, each component may be implemented using a user-programmable integrated circuit such as an FPGA (field-programmable gate array) or a microcomputer. In this case, this integrated circuit may be used to implement a program composed of the above components. These are the same for other embodiments described later.

The participant information storage unit 110 stores participant information, which is information about participants in the teleconference.
6 is a diagram exemplifying participant information according to the first embodiment; FIG. FIG. 6 illustrates participant information corresponding to a teleconference in which four participants A to D participate. The participant information includes the identification information of each participant, the participation status of each participant, and the number of collisions of each participant.

Here, "participation status" indicates how each participant is currently participating in the teleconference. The participation state is determined by an utterance determination unit 120 and an utterance collision determination unit 132, which will be described later. In the example of FIG. 6, the participant A speaks while the participant B speaks. In other words, participant A has caused an utterance collision. Therefore, the participation state of participant A is "speech collision", and the participation state of participant B is "speaking". Participant C is backtracking, and participant D is not speaking. Therefore, the participation state of participant C is "backhand", and the participation state of participant D is "no voice".

Also, the "number of collisions" indicates the number of times each participant has caused an utterance collision, that is, the number of colliding utterances of each participant. The number of collisions is counted by the number counting unit 140, which will be described later. In the example of FIG. 6, the number of collisions of participant A is one. As described above, since participant A has caused a speech collision, the number of collisions has been updated from 0 to 1. Also, the number of collisions for participant B is two, the number of collisions for participant C is one, and the number of collisions for participant D is zero.

The voice receiving unit 112 receives voices (audio data) of participants who are users of the respective communication terminals 30 from each communication terminal 30 via the network 22 . The voice receiving unit 112 receives the voice of the participant transmitted by the voice transmitting unit 44 of the communication terminal 30 (voice data) through the communication unit 106 . As a result, the voices of the participants A to D are received.

The utterance determining unit 120 analyzes the voice received by the voice receiving unit 112 and performs voice recognition processing for each of the plurality of participants. Then, the utterance determination unit 120 determines whether the voice of each participant indicates utterance or backtracking. In other words, the speech determination unit 120 determines whether or not each participant is speaking (speaking or backtracking).

Specifically, the utterance determination unit 120 performs processing such as acoustic analysis and natural language processing to analyze words included in speech. Then, the utterance determination unit 120 determines whether or not the speech includes meaningful words (subject, predicate, object, etc.). In other words, the utterance determination unit 120 determines whether or not the speech includes words other than meaningless words (interjections, etc.). If the speech contains a meaningful word, the speech determination unit 120 determines that the speech is “speech”. On the other hand, if the speech contains only meaningless words (such as interjections), the utterance determination unit 120 determines that the speech is a "backtrack." Note that the utterance determination unit 120 may determine whether or not the received voice includes human voice. If the voice does not include human voice, the utterance determination unit 120 does not need to determine whether the voice is background sound and whether it is an utterance or a backtracking.

The audio output control unit 130 performs control so that the audio of each of the multiple participants is output from the communication terminals 30 of the multiple participants. Specifically, the audio output control unit 130 transmits the received audio (audio data) to the communication terminals 30 of the respective participants via the network 22 by the communication unit 106 . As a result, the sound is output by the sound output unit 48 of each communication terminal 30 . Therefore, participants AD can listen to the voices of other participants. In addition, the audio output control unit 130 may perform mixing processing so that when voices of a plurality of participants are uttered at the same time, the voices of the participants are not interrupted. However, in the present embodiment, as will be described later, when a speech collision occurs, the output of the voice in which the speech collision has occurred is suppressed. On the other hand, when the voice corresponds to the backtracking, the voice output control section 130 transmits the voice to the communication terminals 30 of the respective participants. As a result, the voice output unit 48 of each communication terminal 30 outputs the backtrack of the participant.

The speech collision determination unit 132 determines whether or not a speech collision has occurred for each of the plurality of participants. Specifically, when the speech determination unit 120 determines that a certain participant is speaking, the speech collision determination unit 132 performs , determine whether another participant has started speaking. The speech collision determination unit 132 causes another participant (participant who speaks later) to cause speech collision when another participant starts speaking while a certain participant is speaking. I judge that. The utterances of the other participants that caused the utterance collision are referred to as colliding utterances. In the example of FIG. 6, since participant A started speaking while participant B was speaking, the speech collision determination unit 132 determines that participant A has caused a speech collision, Participant A's utterance is determined as a conflicting utterance.

The speech output suppression unit 134 performs control for suppressing the output of conflicting speech. Specifically, the speech output suppression unit 134 performs control so as not to transmit conflicting speech (audio data) to the communication terminals 30 of a plurality of participants. As a result, each communication terminal 30 does not receive the collision utterance (voice data), so that the communication terminal 30 does not output the collision utterance. Therefore, in the example of FIG. 6, at each communication terminal 30, participant A's speech (collision speech) does not interfere with listening to participant B's speech. Alternatively, the speech output suppression unit 134 may control each communication terminal 30 so that the collision speech is output at a low volume. For example, the utterance output suppression unit 134 reduces the volume of the conflicting utterance so that the volume of the conflicting utterance (in the example of FIG. 6, the utterance of participant B) is reduced to the extent that it does not interfere with listening to the voice of the conflicting utterance. Data may be processed. Then, the voice output control section 130 may transmit the processed voice data to each communication terminal 30 . As a result, in the example of FIG. 6, each communication terminal 30 outputs the speech of the participant A at a very low volume that does not interfere with listening to the speech of the participant B. FIG.

The number-of-times counting unit 140 counts the number of occurrences of speech collision for each of the plurality of participants. In other words, the number counting unit 140 counts the number of conflicting utterances for each of a plurality of participants (communication terminals 30). As a result, the number of collisions illustrated in FIG. 6 is counted.

The display control unit 150 controls what kind of image is displayed on each communication terminal 30 for each of the plurality of participants. Specifically, the display control unit 150 generates display information indicating an image to be displayed on each communication terminal 30 . The display control unit 150 then transmits the generated display information to each communication terminal 30 . Also, the display control unit 150 may generate display information according to the participant information stored in the participant information storage unit 110 . The display control unit 150 may transmit display information for displaying a message indicating that another participant is speaking to the communication terminal 30 of the participant who has caused the speech collision. Further, the display control unit 150 may generate display information including participant information and an instruction to display according to the participant information. In this case, the communication terminal 30 generates an image to be displayed on the interface section 28 of the communication terminal 30 according to the display information.

The frequency display control unit 152 performs control so that each communication terminal 30 displays the number of conflicting utterances for each of a plurality of participants. Specifically, the number display control unit 152 generates display information indicating how many times each participant has collided. Then, the display control unit 150 transmits the display information to the plurality of communication terminals 30 so that the number of collisions of each participant is displayed on the plurality of communication terminals 30 . In the example of FIG. 6, the number of collisions for participant A is one, the number of collisions for participant B is two, and the number of collisions for participant C is one. and that the number of collisions for participant D is 0. Thereby, each participant can grasp the number of collisions of all the participants. Therefore, each participant can grasp which participant wants to speak.

Note that the number of collisions display control unit 152 may display the number of collisions greater than a predetermined threshold in a display form that is more conspicuous than the number of collisions equal to or less than the threshold. In other words, when the number of collisions of a certain participant is greater than a predetermined threshold, the number of collisions display control unit 152 displays the number of collisions in a display form that is more conspicuous than the number of collisions of other participants. You may do so. The number of times display control unit 152 generates display information including an instruction for displaying the number of collisions in that display form. For example, the frequency display control unit 152 may display the number of collisions below the threshold in black and the number of collisions exceeding the threshold in red. This allows each participant to more reliably grasp which participant wants to speak.

Further, the number of times display control unit 152 may cause each communication terminal 30 to display the number of collisions, which is the largest among the number of collisions of a plurality of participants, in a display form that is more conspicuous than the display of other number of collisions. good. The number of times display control unit 152 generates display information including an instruction to display the number of collisions in that display form. For example, the frequency display control unit 152 may display the highest number of collisions in red and the other number of collisions in black. As a result, each participant can reliably grasp which participant's number of collisions is greater than the number of collisions of other participants. This makes it possible to reliably grasp which participant relatively wants to speak more.

In addition, the number of times display control unit 152 may cause each communication terminal 30 to display the number of collisions that is significantly larger than the number of other collisions in a display form that is more conspicuous than the display of the number of other collisions. For example, the number display control unit 152 subtracts the number of collisions of each of the other participants from the number of collisions of the first participant among the plurality of participants. Then, when the values obtained by the subtraction are all larger than a predetermined threshold value, the number display control unit 152 displays the number of collisions of the first participant more than the number of collisions of the other participants. may be displayed in a conspicuous display form. The number of times display control unit 152 generates display information including an instruction to display the number of collisions in that display form. For example, the number display control unit 152 may display the number of collisions of the first participant in red and the number of collisions of the other participants in black. As a result, each participant can reliably grasp which participant's number of collisions is significantly higher than that of other participants. This makes it possible to more reliably grasp which participant relatively wants to speak more.

The icon display control unit 154 performs control so that face icons corresponding to each of the multiple participants are displayed on the communication terminals 30 of the multiple participants. Icon display control unit 154 generates display information including an instruction to display a face icon. In the example of FIG. 6, four face icons corresponding to participants A to D are displayed on communication terminal 30 .

Here, the icon display control unit 154 may generate display information such that each face icon operates in accordance with the participation state of the corresponding participant. Specifically, the icon display control unit 154 may display the face icon of the participant who made the conflicting utterance so as not to move the face icon. On the other hand, the icon display control unit 154 may display the face icon so as to operate the face icon of the participant who made an utterance other than the conflicting utterance. In addition, the icon display control unit 154 may display the face icon so as to move the face icon of the participant who gave the backtrack.

For example, the icon display control unit 154 may display the face icon of the participant (participant D in the example of FIG. 6) who does not speak (utterance or backhand) so that the mouth of the face icon is closed. good. Further, the icon display control unit 154 may display the face icon of the participant (participant B in the example of FIG. 6) who has made an utterance other than the conflicting utterance so that the mouth of the face icon is open. Alternatively, the icon display control unit 154 may display the face icon such that the mouth of the face icon of the participant who made the utterance other than the collision utterance opens and closes. In addition, the icon display control unit 154 may display the face icon of the participant (participant C in the example of FIG. 6) who gave the backtrack so that the mouth of the face icon is open. Alternatively, the icon display control unit 154 may display the face icon such that the mouth of the face icon of the participant who made the backtracking opens and closes. On the other hand, the icon display control unit 154 may display the face icon of the participant who made the conflicting utterance (participant A in the example of FIG. 6) so that the mouth of the face icon remains closed.

Thereby, each participant can see the face icon displayed on each communication terminal 30 and grasp which participant is speaking. Further, each participant can recognize that the participant who gave the backtrack has backtracked, even if the communication terminal 30 of the participant who has backtracked is set to mute. In addition, since the face icon of the participant who caused the speech collision does not move, each participant is prevented from being annoyed by the speech collision.

FIG. 7 is a flow chart showing a remote conference method executed by the remote conference system 20 according to the first embodiment. The processing shown in FIG. 7 is mainly executed by the teleconference device 100. FIG. The remote conference device 100 starts a remote conference (step S102). At this time, the display information generated by the display control unit 150 indicates that the mouth of the face icon is in a closed state (a state in which the face icon is not operating) and that the number of collisions is 0 for all participants. It indicates that

Next, the voice receiving unit 112 receives the voice of the participant X (step S104). Here, when participants A to D are participating in the teleconference as shown in FIG. 6, participant X (and participant Y to be described later) is one of participants A to D. FIG. Then, as described above, the utterance determination unit 120 determines whether the voice of the participant X indicates utterance or backtracking (step S106). If the voice of the participant X does not indicate an utterance (that is, indicates a backtracking) (NO in step S108), the voice output control unit 130 causes the backtracking of the participant X to be output from each communication terminal 30. Control is performed (step S112). Further, the display control unit 150 (icon display control unit 154) performs control so that the face icon of the participant X is displayed on each communication terminal 30 so as to operate (step S114).

On the other hand, if the voice of the participant X indicates an utterance (YES in S108), the utterance collision determination unit 132 determines whether or not the participant Y different from the participant X is already speaking ( step S120). If participant Y is not speaking (NO in S120), no speech collision has occurred because no one else is speaking when participant X speaks. Therefore, the voice output control unit 130 performs control so that the voice of the participant X is output from each communication terminal 30 (step S122). Further, the display control unit 150 (icon display control unit 154) performs control so that the face icon of the participant X is displayed on each communication terminal 30 so as to operate (step S124). At this time, the display control unit 150 may perform control so that each communication terminal 30 displays a message indicating that the participant X is speaking.

On the other hand, if participant Y is speaking (YES in S120), participant X's speech causes a speech collision. Therefore, the voice output control unit 130 (speech output suppression unit 134) performs control so as to suppress the output of the speech of the participant X (step S132). In addition, the number counting unit 140 increments the number of collisions of the participant X by one (step S134). As a result, the number of collisions of the participant X in the participant information stored in the participant information storage unit 110 is updated. Further, the display control unit 150 (the number of times display control unit 152) performs control so that the display of the number of collisions of the participant X is updated (step S136). In addition, the display control unit 150 controls the communication terminal 30 of the participant X so that "another participant is speaking" is displayed (step S138).

(Embodiment 2)
Next, Embodiment 2 will be described with reference to the drawings. For clarity of explanation, the following descriptions and drawings are omitted and simplified as appropriate. Moreover, in each drawing, the same elements are denoted by the same reference numerals, and redundant description is omitted as necessary. Embodiment 2 differs from Embodiment 1 in that the functions of the teleconference apparatus 100 according to Embodiment 1 can be implemented in each communication terminal.

FIG. 8 is a diagram showing a remote conference system 200 according to the second embodiment. Teleconference system 200 has a plurality of communication terminals 201A to 201D and conference server 220 . Communication terminals 201A to 201D are connected to a network such as the Internet. The communication terminals 201A to 201D and the conference server 220 are communicably connected to each other via a network. Although four communication terminals 201 are shown in FIG. 8, the number of communication terminals 201 may be any number equal to or greater than two.

A plurality of communication terminals 201A-201D each have a conference execution system 202, a camera 203, a microphone 204, a display 205, and a speaker 206. Conference execution system 202 functions to conduct teleconferences. The camera 203 can photograph the appearance (face, etc.) of the user of the communication terminal 201 . Microphone 204 can collect the voice of the user of communication terminal 201 . The display 205 can display images relating to the teleconference. The speaker 206 can output the voices of the participants of the remote conference (users of the communication terminals 201A to 201D).

The conference execution system 202 has an utterance state detection unit 207, a conference information reception unit 208, a conference control unit 209, and a conference information transmission unit 210 as components. Each communication terminal 201 can have the hardware configuration of the communication terminal 30 according to the first embodiment described above. Description of each component of the communication terminal 201 will be given later.

The communication terminal 201 transmits voice information indicating the voice of the user of the communication terminal 201 to the conference server 220 . The communication terminal 201 also detects the user's speech state and transmits speech state information indicating the detected speech state to the conference server 220 . Here, the "utterance state" indicates whether each participant is uttering or backtracking. Note that the speech state may indicate that the participant is silent.

When the conference server 220 receives the voice information and the speech state information from each communication terminal 201, the conference server 220 performs mixing processing on the voice information of each user (participant in the teleconference). Then, the conference server 220 transmits the mixed voice information and the utterance state information to the plurality of communication terminals 201 . By transmitting audio information that has undergone mixing processing, audio can be stably output from speaker 206 in each communication terminal 201 .

FIG. 9 is a diagram illustrating a state in which speech state information is transmitted and received in the remote conference system 200 according to the second embodiment. 201 A of communication terminals (communication terminal A) transmit the speech state information of the user A of 201 A of communication terminals to the conference server 220. FIG. Communication terminal 201B (communication terminal B) transmits the speech state information of user B of communication terminal 201B to conference server 220 . The communication terminal 201C (communication terminal C) transmits the speech state information of the user C of the communication terminal 201C to the conference server 220 . Communication terminal 201D (communication terminal D) transmits the speech state information of user D of communication terminal 201D to conference server 220 .

The communication terminal 201A also receives the speech state information of all members (users A to D) from the conference server 220. FIG. Similarly, the communication terminals 201B-201D receive speech state information of all users (users A-D) from the conference server 220. FIG. Each communication terminal 201 may receive the speech state information of all users other than the user from the conference server 220 . For example, the communication terminal 201A may receive speech state information of users BD from the conference server 220. FIG.

FIG. 10 is a block diagram showing the configuration of the speech state detection unit 207 according to the second embodiment. The utterance state detection unit 207 corresponds to the utterance determination unit 2 shown in FIG. 1 and the utterance determination unit 120 shown in FIG. In other words, the utterance state detection unit 207 has a function as utterance determination means. The utterance state detection unit 207 has a voice input unit 222 , a voice detection unit 223 , a language recognition unit 224 , and an utterance presence/absence determination unit 225 .

Audio input unit 222 receives audio signals collected by microphone 204 (audio signals of the user of communication terminal 201). The voice detection unit 223 detects voice information from the voice signal. The language recognition unit 224 performs speech recognition processing, acoustic analysis, natural language processing, etc., and recognizes meaningful language (subject, predicate, object, etc.) from speech information.

The speech presence/absence determination unit 225 determines whether the audio information corresponds to speech or backtracking. When a language (meaningful word) is recognized from the voice information, the speech presence/absence determination unit 225 determines that the voice information corresponds to speech. If no language is recognized from the voice information, the utterance presence/absence determining unit 225 determines that the voice information corresponds to backtracking. If human voice is not recognized from the voice information, the utterance presence/absence determination unit 225 may determine that the voice information corresponds to "silent" (a state in which neither utterance nor backtracking is performed). The utterance state detection unit 207 generates utterance state information according to the determination result by the utterance presence/absence determination unit 225 . Note that the speech state information may be generated by the conference control unit 209 .

The conference information receiving section 208 and the conference information transmitting section 210 are connected to the conference server 220 via a network. The conference information receiving unit 208 receives the conference information of the users of the communication terminals 201A to 201D from the conference server 220. FIG. The conference information transmission unit 210 transmits the conference information of the user of the communication terminal 201 to the conference server 220 . For example, the communication terminal 201A transmits user A's conference information to the conference server 220 .

FIG. 11 is a diagram exemplifying conference information according to the second embodiment. The conference information includes face icon display information, speech state information, voice information, and collision count information. Also, the conference information may include identification information of the corresponding user (communication terminal 201). The face icon display information is information indicating how to display the face icon of the corresponding user. The number of collisions information is information indicating the number of collisions of the corresponding user. Note that the conference information transmitted by the conference information transmission unit 210 does not necessarily include all of the information shown in FIG. 11 . Also, the conference information received by the conference information receiving unit 208 does not necessarily include all of the information shown in FIG.

The conference control section 209 generates conference information to be transmitted by the conference information transmission section 210 . In other words, the conference control unit 209 determines which information among the information illustrated in FIG. 11 is to be transmitted as the conference information. Here, the conference control section 209 uses the conference information received by the conference information receiving section 208 to generate conference information to be transmitted by the conference information transmitting section 210 . Also, the conference control unit 209 uses the conference information received by the conference information receiving unit 208 to display an image of the conference on the display 205 . Also, the conference control unit 209 uses the conference information received by the conference information receiving unit 208 to cause the speaker 206 to output sound.

FIG. 12 is a diagram showing the configuration of the conference control unit 209 according to the second embodiment. The conference control unit 209 has an audio output control unit 211 , a number counting unit 215 and a display control unit 216 . The voice output control section 211 has a speech collision determination section 212 and a speech output suppression section 214 . The display control section 216 has a count display control section 217 and an icon display control section 218 . The conference control unit 209 may be configured so that the processing that the remote conference device 100 according to the first embodiment performs for each participant is performed only for the user of the corresponding communication terminal 201 .

The audio output control section 211 corresponds to the audio output control section 4 shown in FIG. 1 and the audio output control section 130 shown in FIG. The audio output control unit 211 has a function as audio output control means. The speech collision determination unit 212 corresponds to the speech collision determination unit 132 shown in FIG. The speech collision determination unit 212 has a function as speech collision determination means. Speech output suppression section 214 corresponds to speech output suppression section 134 shown in FIG. The speech output suppression unit 214 functions as speech output suppression means. The number counting section 215 corresponds to the counting section 6 shown in FIG. 1 and the number counting section 140 shown in FIG. The number counting unit 215 has a function as counting means. A display control unit 216 corresponds to the display control unit 150 shown in FIG. The display control unit 216 has a function as display control means. The number display control unit 217 corresponds to the number display control unit 8 shown in FIG. 1 and the number display control unit 152 shown in FIG. The frequency display control unit 217 has a function as frequency display control means. The icon display control unit 218 corresponds to the icon display control unit 154 shown in FIG. The icon display control unit 218 has a function as icon display control means.

The audio output control unit 211 performs control so that the audio of each of the multiple participants in the teleconference is output from the corresponding communication terminal 201 . Further, the voice output control unit 211 performs control so that the voice of the user of the corresponding communication terminal 201 is output from the communication terminal 201 (first communication terminal) of each of the multiple participants. For example, in the communication terminal 201A, the voice output control unit 211 performs control so that the voice of the user A is output from the communication terminals 201 of the respective participants. The audio output control section 211 may have substantially the same function as the audio output control section 130 .

Speech collision determination section 212 determines whether or not a speech collision has occurred for the user of corresponding communication terminal 201 . For example, in the communication terminal 201A, the speech collision determination unit 212 determines whether user A's speech causes a speech collision. The utterance collision determination unit 212 uses the conference information regarding other users received by the conference information reception unit 208 to determine whether user A's utterance is made during the other user's utterance period. The speech collision determination section 212 may have substantially the same function as the speech collision determination section 132 .

The speech output suppressing unit 214 performs control for suppressing the output of the conflicting speech at the communication terminal 201 (first communication terminal) of each of the plurality of participants when the user of the corresponding communication terminal 201 generates a conflicting speech. I do. For example, in the communication terminal 201A, the speech output suppressing unit 214 controls the output of the conflicting speech in the communication terminals 201 (first communication terminals) of the respective participants when the user A makes a conflicting speech. control. The speech output suppression unit 214 may have substantially the same function as the speech output suppression unit 134 .

Number-of-times counting section 215 counts the number of occurrences of speech collision for the user of corresponding communication terminal 201 . For example, in the communication terminal 201A, the number counting unit 215 counts the number of times that user A has had a speech collision. The number counting section 215 may have substantially the same function as the number counting section 140 .

The display control unit 216 controls what kind of image is displayed on the communication terminal 201 (first communication terminal) of each of the plurality of participants for the user of the corresponding communication terminal 201 . For example, in the communication terminal 201A, the display control unit 216 controls what kind of image about the user A is displayed on the communication terminal 201 (first communication terminal) of each of the plurality of participants. The display control section 216 may have substantially the same function as the display control section 150 .

The frequency display control unit 217 performs control so that the number of conflicting utterances of the user of the corresponding communication terminal 201 is displayed on the communication terminal 201 (first communication terminal) of each of the multiple participants. For example, in the communication terminal 201A, the frequency display control unit 217 performs control so that the number of conflicting utterances of the user A is displayed on the communication terminal 201 (first communication terminal) of each of the multiple participants. The number display control section 217 may have substantially the same function as the number display control section 152 .

The icon display control unit 218 performs control so that the face icon corresponding to the user of the corresponding communication terminal 201 is displayed on the communication terminal 201 (first communication terminal) of each of the multiple participants. For example, in the communication terminal 201A, the icon display control unit 218 performs control so that the face icon corresponding to the user A is displayed on the communication terminal 201 (first communication terminal) of each of the multiple participants. Icon display control section 218 may have substantially the same function as icon display control section 154 .

FIG. 13 is a flowchart showing a remote conference method executed by the remote conference system 200 according to the second embodiment. The teleconference method according to FIG. 13 is mainly executed by the conference execution system 202 of each communication terminal 201. FIG. In the following description, the processing of the communication terminal 201A will be described as appropriate, but the same applies to the other communication terminals 201 as well.

First, the conference execution system 202 is activated (step S201). At this time, the number of collisions of all participants in the teleconference is zero. Also, the face icons of all participants in the teleconference have their mouths closed. Then, the utterance state detection unit 207 (voice input unit 222) inputs a voice signal from the microphone 204 of the communication terminal 201A (step S202). The voice detection unit 223 determines whether or not there is user A's voice. (Step S203).

If it is determined that there is no user A's voice (NO in S203), the conference control unit 209 generates user A's conference information corresponding to this determination and transmits it to the conference server 220 (step S204). Then, the processing flow returns to S202. Specifically, the conference control unit 209 generates conference information including speech state information indicating silence and face icon display information indicating a face icon with an open mouth, and transmits the conference information to the conference server 220 . The conference server 220 transmits this conference information to the communication terminals 201A-201D. As a result, the face icon of the user A whose mouth is not open is displayed on the display 205 of each communication terminal 201 . Since the conference information does not include voice information, the speaker 206 of each communication terminal 201 does not output the voice of the user A. FIG. Examples of face icons will be described later.

In addition, in the process of S204, the utterance state detection unit 207 generates utterance state information indicating silence. Also, the icon display control unit 218 of the display control unit 216 generates face icon display information indicating a face icon whose mouth is not open. Also, the audio output control unit 211 determines not to include the audio information in the conference information. Note that the meeting information may include collision number information indicating that the number of collisions is zero. At this time, the number display control unit 217 may generate collision number information indicating that the number of collisions has not increased.

On the other hand, if it is determined that there is user A's voice (YES in S203), the language recognition unit 224 performs the language recognition described above (step S205). Then, the utterance presence/absence determination unit 225 determines whether or not there is a language in the voice information (step S206). That is, the utterance presence/absence determination unit 225 determines whether or not the language is recognized from the voice information. If there is no language (NO in S206), the utterance presence/absence determination unit 225 determines that the voice information of user A corresponds to backtracking.

Then, the conference control unit 209 generates user A's conference information corresponding to this determination, and transmits it to the conference server 220 (step S207). Then, the processing flow returns to S202. Specifically, the conference control unit 209 generates speech state information indicating backtracking, face icon display information indicating a face icon with an open mouth, and conference information including voice information, and transmits the generated conference information to the conference server 220 . . The conference server 220 transmits this conference information to the communication terminals 201A-201D. As a result, the face icon of user A with an open mouth is displayed on the display 205 of each communication terminal 201 . Also, the speaker 206 of each communication terminal 201 outputs the user A's voice (backhand).

In addition, in the process of S207, the utterance state detection unit 207 generates utterance state information indicating backtracking. Also, the icon display control unit 218 of the display control unit 216 generates face icon display information indicating a face icon with an open mouth. Also, the audio output control unit 211 determines to include the audio information in the conference information. Note that the meeting information may include collision number information indicating that the number of collisions has not increased. At this time, the number display control unit 217 may generate collision number information indicating that the number of collisions has not increased.

On the other hand, if there is a language (YES in S206), the speech presence/absence determination unit 225 determines that there is speech in the voice information of user A (step S208). At this time, the utterance collision determination unit 212 of the conference control unit 209 determines whether or not there is an utterance from another user (step S209). In other words, the utterance collision determination unit 212 uses the received conference information (voice information and utterance state information) of other users to determine whether or not another user has spoken before user A speaks. do. In other words, the utterance collision determination unit 212 determines whether user A's utterance causes an utterance collision.

If there is no utterance from another user (YES in S209), the conference control unit 209 determines that user A's utterance does not cause an utterance collision. Then, the conference control unit 209 generates user A's conference information corresponding to this determination, and transmits it to the conference server 220 (step S210). Then, the processing flow returns to S202. Specifically, the conference control unit 209 generates speech state information indicating an utterance, face icon display information indicating a face icon with an open mouth, and conference information including voice information, and transmits the conference information to the conference server 220 . . The conference server 220 transmits this conference information to the communication terminals 201A-201D. As a result, the face icon of user A with an open mouth is displayed on the display 205 of each communication terminal 201 . Also, the speaker 206 of each communication terminal 201 outputs the voice (utterance) of the user A. FIG. At this time, the conference information may include display information indicating that the user A is busy. In this case, the display 205 of each communication terminal 201 displays a message indicating that the user A is busy. As a result, each user can grasp who is speaking, which makes it easier to create the minutes.

In addition, in the process of S210, the utterance state detection unit 207 generates utterance state information indicating utterance. Also, the icon display control unit 218 of the display control unit 216 generates face icon display information indicating a face icon with an open mouth. Also, the audio output control unit 211 determines to include the audio information in the conference information. Note that the meeting information may include collision number information indicating that the number of collisions has not increased. At this time, the number display control unit 217 may generate collision number information indicating that the number of collisions has not increased.

On the other hand, if there is an utterance from another user (NO in S209), the conference control unit 209 determines that user A's utterance has caused an utterance collision. Then, the conference control unit 209 causes the display 205 of the communication terminal 201A to display a message such as "another user is busy" (step S211). Then, the conference control unit 209 generates user A's conference information corresponding to this determination, and transmits it to the conference server 220 (step S212). Then, the processing flow returns to S202. Specifically, the conference control unit 209 includes utterance state information indicating an utterance (collision utterance), face icon display information indicating a face icon whose mouth is not open, and collision count information obtained by incrementing the number of collisions by one. It generates conference information and transmits it to the conference server 220 . The conference server 220 transmits this conference information to the communication terminals 201A-201D. As a result, the face icon of the user A whose mouth is not open is displayed on the display 205 of each communication terminal 201 . Also, the number of collisions of user A, which is incremented by one, is displayed on the display 205 of each communication terminal 201 . Since the conference information does not include voice information, the speaker 206 of each communication terminal 201 does not output the voice of the user A. FIG.

In addition, in the process of S212, the utterance state detection unit 207 generates utterance state information indicating the utterance (collision utterance). Also, the icon display control unit 218 of the display control unit 216 generates face icon display information indicating a face icon whose mouth is not open. Also, the speech output suppression unit 214 of the voice output control unit 211 determines not to include the voice information in the conference information. In addition, the number display control unit 217 generates collision number information indicating that the number of collisions has increased by one.

14 and 15 are diagrams illustrating conference images 230 displayed on each communication terminal 201 in the teleconference according to the second embodiment. In the conference image 230, a face icon 231 and the number of collisions 232 corresponding to each user are displayed near the user name of each user. Therefore, the face icon 231A and the number of collisions 232A are displayed in the vicinity of the user A's user name. Similarly, a face icon 231B and the number of collisions 232B are displayed near User B's user name. A face icon 231C and the number of collisions 232C are displayed in the vicinity of the user C's user name. A face icon 231D and the number of collisions 232D are displayed in the vicinity of the user D's user name. In the example of FIG. 14, the number of collisions 232A indicates zero, the number of collisions 232B indicates two, the number of collisions 232C indicates one, and the number of collisions 232D indicates zero. Note that the conference image 230 may have display areas 230a-230d that display the face icon 231 and the number of collisions 232 for each of the users A-D.

Also, in the conference image 230 illustrated in FIG. 14, the user B is speaking. Therefore, a message 234 indicating that User B is speaking is displayed near User B's face icon 231B. Also, the mouth of the face icon 231B of user B is open. In addition, the user C is backtracking. Therefore, the mouth of the face icon 231C of user C is open. Also, user A and user D are silent. Therefore, the mouth of user A's face icon 231A and the mouth of user D's face icon 231D are closed. Also, since user B is speaking, each communication terminal 201 outputs user B's speech. Also, since user C is making a backhand, each communication terminal 201 outputs user C's backhand.

FIG. 15 illustrates a case where user A's utterance causes an utterance collision in the state of the conference image 230 exemplified in FIG. When user A speaks later than user B while user B speaks, user A's speech is determined to be a collision speech. At this time, user A's communication terminal 201A displays a message 236 indicating that another user (user B) is speaking. Also, the number of collisions 232A for user A is updated from 0 to 1. Note that the mouth of the face icon 231A of the user A is closed because the user A's utterance is a conflicting utterance. Although the message 236 is displayed only on the user A's communication terminal 201A, the conference images 230 displayed on the communication terminals 201 of each user other than the message 236 may be the same.

(Effects of this embodiment)
The effects of this embodiment will be described below.
In recent years, there has been an increase in the number of participants holding teleconferences while staying at home. In holding remote conferences, the use of remote conferences using the Internet environment at home is increasing. In this case, there is a delay due to the Internet environment at home, so there is a possibility that multiple participants' speeches will overlap (speech collisions) or that they will refrain from speaking to each other. It may not proceed smoothly. Also, when participating in a teleconference at home, participants often participate in the teleconference using only voice for the purpose of preventing privacy issues and Internet line congestion. In that case, there is a problem that the facial expression of the other party cannot be read during conversation. Furthermore, there is a problem that it is difficult for the speaker to hear backtracking by setting mute except when he or she speaks in order to prevent environmental sounds from entering. In addition, in technology that uses a mechanism to indicate that a person is speaking to someone with audio information, even if only a backlog is given, it is considered that the person is speaking, so if there are many conference participants, it is difficult to understand who is speaking. There is also the problem of

In the teleconferencing system according to the present embodiment, when another participant speaks later while a certain participant is speaking, the speech of the other participant who speaks later is It is configured to suppress output from the communication terminals of the participants. As a result, each participant is prevented from listening to the conflicting speech (the speech of the other participant who spoke later) on the communication terminal, so that the teleconference progresses smoothly.

Furthermore, the teleconference system according to the present embodiment is configured to count the number of collisions for each participant who has caused a speech collision, and display the number of collisions on each communication terminal. Therefore, each participant can grasp which participant has the most speech collisions. This allows each participant to give an awareness of which participant wants to speak. As a result, other participants can perform actions such as prompting the participant to speak or waiting until the participant speaks. Therefore, the teleconference system according to the present embodiment enables the teleconference to proceed smoothly.

Further, the remote conference system according to the present embodiment is configured so that the number of collisions is displayed on each participant's communication terminal for each of a plurality of participants. Thereby, each participant can grasp the number of collisions of each participant.

Further, the remote conference system according to the present embodiment is configured to display a message such as "another user is busy" on the communication terminal of the participant who speaks late. Therefore, it is possible to make the participant who made the conflicting utterance aware of the occurrence of the utterance conflict.

In addition, the remote conference system according to the present embodiment outputs the backtracking to each participant's communication terminal even if another participant backtracks while a certain participant is speaking. is configured as As a result, the participant (speaker) who is speaking can feel secure that the other participant is listening to the speech.

Further, in the remote conference system according to the present embodiment, when a certain participant backtracks, the communication terminal of each participant displays a face icon with an open mouth corresponding to the participant who backtracked. is configured as As a result, even if the communication terminal of the participant who made the backtracking is set to mute, the speaker can grasp that there is a participant who backtracked, so that he/she can feel secure that he or she is being heard. be able to.

Further, the teleconferencing system according to the second embodiment is configured such that, when a speech collision occurs, the voice information of the collision speech is not transmitted from the communication terminal to the conference server. This can reduce the load on the network.

(Modification)
It should be noted that the present invention is not limited to the above embodiments, and can be modified as appropriate without departing from the scope of the invention. For example, the multiple embodiments described above are mutually applicable. For example, the functions of the remote conference device 100 according to the first embodiment may be realized by the communication terminal 201 according to the second embodiment. Also, the functions of the communication terminal 201 according to the second embodiment may be realized by the remote conference apparatus 100 according to the first embodiment.

Also, in the flowchart described above, the order of each process (step) can be changed as appropriate. Also, one or more of a plurality of processes (steps) may be omitted. For example, in FIG. 7, the order of the processing of S112 and the processing of S114 may be reversed. Similarly, in FIG. 13, the order of the processing of S211 and the processing of S212 may be reversed. Moreover, in FIG. 7, the processing of S114, S124, and S138 may be omitted. Similarly, the processing of S211 may be omitted.

In the above-described embodiment, the number of times display control unit performs control so that the number of collisions of each participant (user) is displayed on the communication terminals of a plurality of participants. Not limited. The number of times display control unit does not need to display the number of times of collision itself on a plurality of communication terminals. For example, the number display control unit may cause a plurality of communication terminals to display levels corresponding to the number of collisions. For example, the number of times display control unit displays on each communication terminal level C if the number of collisions is 2 or less, level B if the number of collisions is 3 to 4, and level A if the number of collisions is 5 or more. You may let Further, the number display control unit may cause each communication terminal to display a warning when the number of collisions of a certain participant exceeds a threshold. Further, for example, the number display control unit operates the face icon of the participant whose number of collisions has increased in such a manner as to indicate that the participant wants to speak (the color of the face icon turns red, etc.). You can display it on your terminal.

Also, the number of collisions may continue to increase each time a speech collision occurs while the teleconference is being executed, or may be reset during the teleconference. For example, the collision count may be reset when the corresponding participant makes a non-collision utterance a predetermined number of times. Also, for example, the number of collisions may be reset by the corresponding participant operating the communication terminal.

Also, in the second embodiment, each communication terminal 201 generates face icon display information for a corresponding user, but the configuration is not limited to this. For example, each communication terminal 201 may generate a face icon of user A using speech state information about user A transmitted from communication terminal 201A.

Further, in the above-described embodiment, the face icon of each user (participant) is displayed on each of a plurality of communication terminals during the teleconference, but the present invention is not limited to such a configuration. An image of each user's face captured by the camera 203 or the like may be displayed on each of a plurality of communication terminals. However, when an image of the user's face is displayed, it is possible that both the mouth of the user making the backtracking and the mouth of the user making the conflicting utterance are moving in the image. Therefore, other users may not be able to visually distinguish between backtracking and conflicting utterances. On the other hand, in the present embodiment, the face icon is displayed on each communication terminal, and the mouth of the face icon of the user who is making a conflicting utterance is closed, and the mouth of the face icon of the user who is backtracking is open. I'm trying Therefore, in the present embodiment, it is possible to visually distinguish between backtracking and conflicting utterances. Furthermore, in the teleconferencing system according to the present embodiment, each communication terminal does not transmit video information, so it is possible to grasp the user's speech state while reducing the load on the network.

In the above examples, the programs can be stored and delivered to computers using various types of non-transitory computer readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (eg, flexible discs, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical discs), CD-ROMs (Read Only Memory), CD-Rs, CD-R/W, semiconductor memory (eg, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be delivered to the computer on various types of transitory computer readable medium. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can deliver the program to the computer via wired channels, such as wires and optical fibers, or wireless channels.

Some or all of the above-described embodiments can also be described in the following supplementary remarks, but are not limited to the following.
(Appendix 1)
an utterance determination means for determining whether the voice of each of the plurality of participants in the teleconference indicates an utterance or an acknowledgment;
Control is performed so that the voice of each of the plurality of participants is output from the communication terminal of each of the plurality of participants, and when one of the plurality of participants is speaking, another participant voice output control means for controlling to suppress the output of the other participant's utterance when the other participant utters;
counting means for counting the number of first utterances, which are utterances whose output is suppressed, for each of the plurality of participants;
a number display control means for controlling so that the number of times is displayed on the communication terminal of each of the plurality of participants;
Teleconferencing system with
(Appendix 2)
The frequency display control means performs control so that the number of times of the first utterance for each of the plurality of participants is displayed on the communication terminal of each of the plurality of participants.
The teleconferencing system according to appendix 1.
(Appendix 3)
The number display control means causes the communication terminal to display the number of times greater than a predetermined threshold in a display form that is more conspicuous than the display of the number of times less than or equal to the threshold.
The teleconferencing system according to appendix 2.
(Appendix 4)
The number display control means causes the communication terminal to display the number of times that is the largest among the number of times of the plurality of participants in a display form that is more conspicuous than the display of other numbers of times.
The teleconferencing system according to appendix 2.
(Appendix 5)
The voice output control means, when the participant makes a backtracking, performs control so that the backtracking is output from the communication terminals of each of the plurality of participants.
5. The remote conference system according to any one of Appendices 1 to 4.
(Appendix 6)
Icon display control means for controlling so that face icons corresponding to each of the plurality of participants are displayed on the communication terminals of each of the plurality of participants;
further having
The icon display control means displays the face icon so as not to operate the face icon corresponding to the other participant who has made the first utterance, and the face icon which has made an utterance other than the first utterance. displaying the face icon to activate the face icon corresponding to the participant;
6. The remote conference system according to any one of Appendices 1 to 5.
(Appendix 7)
The icon display control means displays the face icon so as to operate the face icon corresponding to the participant when the participant backtracks.
The teleconferencing system according to appendix 6.
(Appendix 8)
a communication terminal,
speech determination means for determining whether the user's voice indicates an utterance or a backtracking in a teleconference in which the user of the communication terminal participates;
controlling so that the voice of each of the plurality of participants in the teleconference is output from the communication terminal, and the voice of the user is output from the first communication terminal, which is the communication terminal of each of the plurality of participants; suppressing output of the user's speech at the first communication terminal when the user speaks while one of the plurality of participants is speaking an audio output control means for controlling;
counting means for counting the number of first utterances, which are utterances whose output is suppressed, for the user of the communication terminal;
a count display control means for controlling so that the display regarding the count is made on the first communication terminal;
A communication terminal having
(Appendix 9)
The number display control means performs control so that the number of times the user of the communication terminal utters the first utterance is displayed on the first communication terminal.
The communication terminal according to appendix 8.
(Appendix 10)
The voice output control means performs control so that, when the user of the communication terminal makes a backhand, the backhand is output from the first communication terminal;
The communication terminal according to appendix 8 or 9.
(Appendix 11)
icon display control means for controlling so that a face icon corresponding to the user of the communication terminal is displayed on the first communication terminal;
further having
The icon display control means displays the face icon so as not to operate the face icon when the user of the communication terminal utters the first utterance, and the user of the communication terminal utters an utterance other than the first utterance. displaying the face icon so as to operate the face icon when the utterance of
11. The communication terminal according to any one of Appendices 8 to 10.
(Appendix 12)
The icon display control means causes the face icon to be displayed so as to operate the face icon when the user of the communication terminal responds.
The communication terminal according to appendix 11.
(Appendix 13)
determining whether the audio of each of the multiple participants in the teleconference indicates speech or backtracking;
controlling so that the voice of each of the plurality of participants is output at each communication terminal of each of the plurality of participants;
When another participant speaks while one of the plurality of participants is speaking, controlling to suppress the output of the speech of the other participant,
Counting the number of first utterances, which are utterances whose output is suppressed, for each of the participants,
controlling so that the number of times is displayed on the communication terminal of each of the plurality of participants;
Teleconferencing method.
(Appendix 14)
controlling so that the number of times of the first utterance for each of the plurality of participants is displayed on the communication terminal of each of the plurality of participants;
The teleconference method according to appendix 13.
(Appendix 15)
causing the communication terminal to display the number of times greater than a predetermined threshold in a display form that is more conspicuous than the display of the number of times less than or equal to the threshold;
15. The teleconference method according to appendix 14.
(Appendix 16)
causing the communication terminal to display the number of times, which is the largest among the number of times of the plurality of participants, in a display form that is more conspicuous than the display of the other number of times;
15. The teleconference method according to appendix 14.
(Appendix 17)
When the participant makes a backtracking, control is performed so that the backtracking is output at the communication terminal of each of the plurality of participants;
17. The remote conference method according to any one of Appendices 13 to 16.
(Appendix 18)
controlling so that face icons corresponding to each of the plurality of participants are displayed on the communication terminals of each of the plurality of participants;
displaying the face icon so as not to operate the face icon corresponding to the other participant who made the first utterance;
displaying the face icon so as to operate the face icon corresponding to the participant who made an utterance other than the first utterance;
18. The teleconference method according to any one of Appendices 13 to 17.
(Appendix 19)
displaying the face icon so as to operate the face icon corresponding to the participant when the participant backtracks;
19. The teleconferencing method according to appendix 18.
(Appendix 20)
A remote conference method executed by a communication terminal,
Determining whether the user's voice indicates utterance or backtracking in a teleconference in which the user of the communication terminal participates,
controlling so that the voice of each of the plurality of participants in the teleconference is output from the communication terminal, and the voice of the user is output from the first communication terminal, which is the communication terminal of each of the plurality of participants;
suppressing output of the user's speech at the first communication terminal when the user speaks while one of the plurality of participants is speaking take control,
counting the number of first utterances, which are utterances whose output is suppressed, for the user of the communication terminal;
controlling so that the number of times is displayed on the first communication terminal;
Teleconferencing method.
(Appendix 21)
controlling so that the number of times the user of the communication terminal utters the first utterance is displayed on the first communication terminal;
21. The teleconferencing method according to appendix 20.
(Appendix 22)
When the user of the communication terminal makes a backtracking, control is performed so that the backhandling is output at the first communication terminal;
22. The teleconference method according to appendix 20 or 21.
(Appendix 23)
controlling so that a face icon corresponding to the user of the communication terminal is displayed on the first communication terminal;
displaying the face icon so as not to operate when the user of the communication terminal utters the first utterance;
displaying the face icon so as to operate the face icon when the user of the communication terminal utters an utterance other than the first utterance;
23. The teleconference method according to any one of appendices 20 to 22.
(Appendix 24)
displaying the face icon so as to operate the face icon when the user of the communication terminal backtracks;
24. The teleconferencing method according to appendix 23.
(Appendix 25)
the ability to determine whether the voices of each of a plurality of participants in a teleconference represent utterances or backtracking;
Control is performed so that the voice of each of the plurality of participants is output from the communication terminal of each of the plurality of participants, and when one of the plurality of participants is speaking, another participant a function of controlling to suppress the output of the other participant's speech when the participant speaks;
A function of counting the number of first utterances, which are utterances whose output is suppressed, for each of the participants;
A function of controlling so that the number of times is displayed on the communication terminal of each of the plurality of participants;
A program that makes a computer realize
(Appendix 26)
A program for executing a teleconference method executed by a communication terminal,
a function of determining whether the user's voice indicates an utterance or a backtracking in a teleconference in which the user of the communication terminal participates;
controlling so that the voice of each of the plurality of participants in the teleconference is output from the communication terminal, and the voice of the user is output from the first communication terminal, which is the communication terminal of each of the plurality of participants; suppressing output of the user's speech at the first communication terminal when the user speaks while one of the plurality of participants is speaking functions to control and
A function of counting the number of first utterances, which are utterances whose output is suppressed, for the user of the communication terminal;
a function of controlling so that the number of times is displayed on the first communication terminal;
A program that makes a computer realize

1 Teleconference system 2 Utterance determination unit 4 Audio output control unit 6 Count unit 8 Number display control unit 20 Teleconference system 22 Network 30 Communication terminal 42 Audio acquisition unit 44 Audio transmission unit 46 Audio reception unit 48 Audio output unit 52 Display information reception Unit 54 Image display unit 100 Teleconference device 110 Participant information storage unit 112 Voice receiving unit 120 Speech determination unit 130 Voice output control unit 132 Speech collision determination unit 134 Speech output suppression unit 140 Number counting unit 150 Display control unit 152 Number display control Unit 154 Icon display control unit 200 Remote conference system 201 Communication terminal 202 Conference execution system 207 Speech state detection unit 208 Conference information reception unit 209 Conference control unit 210 Conference information transmission unit 211 Voice output control unit 212 Speech collision determination unit 214 Speech output suppression Unit 215 Number counting unit 216 Display control unit 217 Number display control unit 218 Icon display control unit 220 Conference server 222 Voice input unit 223 Voice detection unit 224 Language recognition unit 225 Speech presence/absence determination unit

Claims (8)

a communication terminal,
speech determination means for determining whether the user's voice indicates an utterance or a backtracking in a teleconference in which the user of the communication terminal participates;
controlling so that the voice of each of the plurality of participants in the teleconference is output from the communication terminal, and the voice of the user is output from the first communication terminal, which is the communication terminal of each of the plurality of participants; suppressing output of the user's speech at the first communication terminal when the user speaks while one of the plurality of participants is speaking an audio output control means for controlling;
counting means for counting the number of first utterances, which are utterances whose output is suppressed, for the user of the communication terminal;
a count display control means for controlling so that the display regarding the count is made on the first communication terminal;
In a communication terminal having
The number display control means performs control so that the number of times the user of the communication terminal utters the first utterance is displayed on the first communication terminal,
The voice output control means performs control so that, when the user of the communication terminal makes a backhand, the backhand is output from the first communication terminal;
communication terminal. icon display control means for controlling so that a face icon corresponding to the user of the communication terminal is displayed on the first communication terminal;
further having
The icon display control means is
displaying the face icon so as not to operate when the user of the communication terminal utters the first utterance;
displaying the face icon so as to operate the face icon when the user of the communication terminal utters an utterance other than the first utterance;
The communication terminal according to claim 1 . The icon display control means causes the face icon to be displayed so as to operate the face icon when the user of the communication terminal responds.
A communication terminal according to claim 2 . A remote conference method executed by a communication terminal,
Determining whether the user's voice indicates utterance or backtracking in a teleconference in which the user of the communication terminal participates,
controlling so that the voice of each of the plurality of participants in the teleconference is output from the communication terminal, and the voice of the user is output from the first communication terminal, which is the communication terminal of each of the plurality of participants;
suppressing output of the user's speech at the first communication terminal when the user speaks while one of the plurality of participants is speaking take control,
counting the number of first utterances, which are utterances whose output is suppressed, for the user of the communication terminal;
controlling so that the number of times is displayed on the first communication terminal ;
controlling so that the number of times the user of the communication terminal utters the first utterance is displayed on the first communication terminal;
Teleconferencing method. When the user of the communication terminal makes a backtracking, control is performed so that the backhandling is output at the first communication terminal;
The teleconference method according to claim 4 . controlling so that a face icon corresponding to the user of the communication terminal is displayed on the first communication terminal;
displaying the face icon so as not to operate when the user of the communication terminal utters the first utterance;
displaying the face icon so as to operate the face icon when the user of the communication terminal utters an utterance other than the first utterance;
The teleconference method according to claim 4 or 5 . displaying the face icon so as to operate the face icon when the user of the communication terminal backtracks;
The teleconference method according to claim 6 . A program for causing a communication terminal to execute the teleconference method according to any one of claims 4 to 7 .

Images