JP2011077839A - Information sharing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress the number of terminals possible to directly attain two-way communication of video or sound or the like with a server on the basis of user's utterance request detection and reduce transmission delay, communication failure, and server's load. <P>SOLUTION: A server is connected with a plurality of information terminals via network and the information terminals are grouped into one or plural terminal groups according to the predetermined set condition. One terminal group includes one representative terminal possible to directly receive shared information from the server and a participating terminal for receiving distribution of shared information from the representative terminal. The information terminal is provided with an utterance detection section for detecting an utterance request of users and an utterance request section for transmitting utterance request information to a server. The server is provided with a two-way communication control section that switches a communication state between the information terminal and the server to a two-way communication state possible to directly and mutually transmit and receive the shared information when the server receives the utterance request information. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an information sharing system, and more particularly, to an information sharing system that transmits information such as common video and audio between a plurality of information terminals connected via a network.

Conventionally, a communication system for transmitting information such as video and audio between a plurality of information terminals via a network, a voice conference and a video conference between a plurality of information terminals at remote locations via a dedicated server, etc. A communication system is used.
For example, in a conventional video conference system, a multipoint connection unit (MCU: Multi-point Control Unit) and a server having an MCU function include an information terminal installed at each point and an information terminal for each participant. Connected via a network such as the Internet, LAN, or a dedicated line, information such as video, audio, and documents input for each information terminal is accumulated in the MCU, and each piece of information is decoded (decoded) and mixed in the MCU. A series of processes such as (combining) and encoding (encoding) are performed, and the information is distributed to each information terminal as common information.
In general, it is assumed that participants in a conference using each information terminal perform a conference while watching a common video and listening to a common voice.

  Further, in Patent Document 1, in a multipoint conference system in which a plurality of conference terminals are connected via a multipoint connection device (MCU), the multipoint connection device manages the terminals participating in the conference for each group. Upstream data transmission from the terminal to the multipoint connection device is performed only by the representative terminal of the group, while downlink data transmission from the multipoint connection device to the conference terminal is performed to all conference participation terminals, and When switching the representative terminal of the group, the connection request signal for setting the uplink session from the conference terminal is exchanged via the MCU, thereby switching to another conference terminal in the same group. There has been proposed a session management method for a multipoint conference system in which the number of sessions managed by the point conference device is suppressed.

  Also, in Patent Document 2, information on the importance level of a communication session and information on necessity of recovery corresponding to a communication source terminal and a communication destination partner are set in advance and stored in a session importance level table. When a connection request for a session with high importance is received, the lowest priority of existing sessions is selected and disconnected, and information on the disconnected communication session is stored in the session recovery table, and session resources are stored. A gateway device has been proposed that automatically recovers a session according to information on a communication session stored in the session recovery table 140 as soon as a space is available.

  In Patent Document 3, when at least one of a plurality of conference terminals having a function of mixing audio data transmitted from other conference terminals is included, Prior to the event, the multipoint audio connection device causes the conference terminal having the mixing function to be preferentially selected as a child terminal that establishes a direct communication session with the multipoint audio connection device, and the conference terminal participating in the remote conference There has been proposed a remote conference system that avoids an increase in processing load associated with a multipoint audio connection device that mixes audio data transmitted from each conference terminal as the number increases.

JP 2005-80212 A JP 2009-49550 A JP 2009-1000015 A

However, in a multipoint conference system using a conventional MCU, as the number of information terminals simultaneously connected to the MCU increases, traffic transmitted and received on the communication line between the MCU and each information terminal increases. Communication failures such as information transmission delays and communication errors due to noise are likely to occur, and smooth conferences may not be possible.
Furthermore, in the MCU, as the number of connected information terminals increases, the amount of information to be processed such as encoding (encoding) and decoding (decoding) increases, so the load on the MCU increases, This causes a delay in information transmission to each information terminal.

  Moreover, in the system described in Patent Document 1, the number of sessions between the MCU and each conference terminal can be suppressed, but downlink data transmission from the MCU to the conference terminal is performed from the MCU to each conference terminal. Therefore, if the number of conference terminals increases, traffic for this data transmission in the downstream direction increases and information to be processed on the MCU side increases, so problems such as information transmission delay are likely to occur. As a result, the load on the MCU also increases.

  Moreover, in the gateway apparatus of patent document 2, since connection / disconnection of a session is switched according to the importance of a communication session within the range of the predetermined maximum number of sessions, the maximum number of sessions may be exceeded. When an information terminal is connected, there may be a case where a requested conference cannot be smoothly performed by simply switching between session connection and disconnection.

  Further, in the system described in Patent Document 3, since the connection form of the conference terminal with respect to the multipoint audio connection apparatus has a tree structure, even if an increase in processing load on the multipoint audio connection apparatus can be avoided, mixing is performed. If at least one conference terminal having a function is required and the number of conference terminals connected to the conference terminal having a mixing function increases, the processing load on the conference terminal having the mixing function increases, and audio data There is a risk that a delay in the meeting will occur and smooth progress of the conference may not be possible.

  Therefore, the present invention has been made in consideration of the above circumstances, and among the plurality of information terminals connected to the server, the information terminal of the user who has requested to speak is preferentially designated as the server. Information that is processed by the server, reducing the number of information terminals that can communicate directly with the server and reducing the number of information terminals that are in direct communication with the server and reducing the occurrence of communication failures such as communication errors. It is an object of the present invention to provide an information sharing system that can reduce the load on the server by reducing the amount.

In the present invention, a server and a plurality of information terminals are connected via a network, terminal information input in the information terminal is transmitted to the server, and the server uses terminal information transmitted from each information terminal. The information sharing system generates shared information and distributes the shared information to the information terminals, wherein the plurality of information terminals are grouped into one or a plurality of terminal groups based on predetermined setting conditions in advance. One terminal group includes one representative terminal that can directly receive the shared information from the server, and a participating terminal that receives distribution of the shared information from the representative terminal. Each information terminal is a user The speech detection unit that detects that the speech request has been made, the speech request unit that transmits information indicating that the speech request has been made to the server, and the operation status of the information terminal according to the notification from the server A terminal mode switching unit that changes the communication status between the information terminal and the server when the server receives the message request information from the information terminal, and the terminal information and the shared information. A two-way communication control unit that switches to a two-way communication state capable of direct transmission / reception between each other, and there is only one information terminal that is in a two-way communication state with the server and is allowed to speak by a user at the same time. An information sharing system managed by a server is provided.
This suppresses an increase in the number of information terminals that are directly connected to the server and enter a two-way communication state, thereby reducing information transmission delays, reducing communication failures due to communication errors, etc. The load on the distribution process can be reduced.

In addition, when there is a first information terminal that has made a request for speech and the communication state with the server is a two-way communication state, the speech detection unit of the second information terminal has a new one. When a speech request is detected, the speech request unit of the second information terminal transmits information indicating the new speech request to the server, and the server receives the speech request information from the second information terminal. The bidirectional communication control unit cancels the bidirectional communication state with the first information terminal and sets both the communication state between the second information terminal and the server that has made a new speech request. It is characterized by switching to a bidirectional communication state.
According to this, even if an information terminal that makes a new request for speech appears, the number of information terminals in the two-way communication state does not increase, information transmission delay and communication failure do not increase, and the server The load will not increase.

Further, when the speech request is detected by the user at the speech detection units of the plurality of information terminals almost simultaneously, the bidirectional communication control unit of the server transmits the messages transmitted from the plurality of information terminals, respectively. Of the request information, switch the communication state with the information terminal that sent the earliest received speech request information to the two-way communication state, or check multiple received speech request information and use it at the earliest time The communication state with the information terminal that is determined to have detected the user's speech request is switched to the bidirectional communication state.
Thereby, control of a user's right to speak can be made appropriate and a smooth conference can proceed.

The information terminal further includes an input unit that inputs an input signal by a user operation, a video input unit that inputs video captured by a camera, and an audio input unit that inputs audio from a microphone. When the speech detection unit detects at least one of the input signal indicating the speech request, the input voice indicating the speech request, or the input video indicating the speech request, the user makes a speech request. It is characterized by judging.
According to this, a user's speech request can be detected appropriately.

  Further, the setting condition for grouping is connection base information in which each information terminal is arranged, and the server includes a storage unit that stores the connection base information for each information terminal. . Here, the connection base information means a place name, a base ID described later, and the like.

The representative terminal belonging to the one terminal group receives the shared information from the server by streaming delivery. Thereby, one-way communication of shared information can be performed from the server to the information terminal.
The representative terminal belonging to the one terminal group performs streaming distribution of the shared information received from the server to the participating terminals belonging to the terminal group.
The representative terminal belonging to the one terminal group performs streaming delivery of the shared information received from the server to other information terminals not belonging to the terminal group. Thereby, the load concerning the information delivery of a server can be reduced more.

  When the grouped one terminal group is composed of a single information terminal, the information terminal itself is a representative terminal, and the terminal group is a terminal group that does not include a participating terminal. To do. Accordingly, even a single information terminal that is not grouped into the same terminal group as other information terminals can receive shared information from the server by becoming a representative terminal.

In the present invention, a server and a plurality of information terminals are connected via a network, terminal information input in the information terminal is transmitted to the server, and the server transmits terminal information transmitted from each information terminal. An information sharing method of an information sharing system that generates shared information using a message and distributes the shared information to the information terminal, wherein the plurality of information terminals are preliminarily set based on a predetermined setting condition. When one terminal group includes one representative terminal that can directly receive the shared information from the server and a participating terminal that receives distribution of the shared information from the representative terminal, A message detection step in which the information terminal detects that a user has requested a message; a message requesting step in which information indicating that the user has requested the message is transmitted to a server; and the server When the request information is received from the information terminal, the communication state between the information terminal and the server is switched to a bidirectional communication state in which the terminal information and the shared information can be directly transmitted and received, The information terminal that is in the two-way communication state and is allowed to speak by the user at the same time is controlled by the communication control step to be managed and the information terminal that has detected the speech request by the notification from the server. An information sharing method comprising a terminal mode switching step for changing a communication state to a two-way communication state is provided.
Furthermore, the present invention provides a program of an information sharing system for causing a computer to execute each step of the information sharing method.

  According to the present invention, since only one information terminal that is in a two-way communication state with the server and is allowed to speak by a user exists at the same time, the amount of information transmitted and received between the server and the information terminal By reducing the number of information terminals that perform two-way communication directly with the server, it is possible to reduce the transmission delay of information and thereby reduce communication failures due to communication errors, etc. The load on the server information distribution process can be reduced.

It is a schematic block diagram of one Example of the information sharing system of this invention. It is a block diagram of the configuration of an embodiment of the information terminal of the present invention. It is a block diagram of the configuration of an embodiment of the server of the present invention. It is explanatory drawing of the communication state between server terminals (Example 1-1) of 1st Example. It is explanatory drawing of the communication state between server terminals (Example 1-2) of 1st Example. It is explanatory drawing of the communication state between server terminals (Example 1-3) of 1st Example. It is explanatory drawing of the communication state between server terminals (Example 1-4) of 1st Example. It is explanatory drawing of the communication state between server terminals (Example 1-5) of 1st Example. It is explanatory drawing of the communication state between server terminals (Example 1-6) of 1st Example. It is explanatory drawing of the database stored in the memory | storage part of a server. It is explanatory drawing of the change of the terminal database used by 1st Example. It is explanatory drawing of the change of the terminal database used by 1st Example. It is explanatory drawing of the communication state between server terminals (Example 2-1) of 2nd Example. It is explanatory drawing of the communication state between server terminals of Example 2 (Example 2-2). It is explanatory drawing of the communication state between server terminals (Example 2-3) of 2nd Example. It is explanatory drawing of the communication state between server terminals (Example 2-4) of 2nd Example. It is explanatory drawing of the communication state between server terminals (Example 2-5) of 2nd Example. It is explanatory drawing of the communication state between server terminals (Example 2-6) of 2nd Example. It is explanatory drawing of the change of the terminal database used in 2nd Example. It is explanatory drawing of the change of the terminal database used in 2nd Example. It is a flowchart of the mode switching process in a terminal. It is a flowchart of the streaming delivery process in a representative terminal. It is a flowchart of the terminal state detection control process in a server. It is a flowchart of the speech detection request process in a terminal. It is a flowchart of the speech terminal detection control process in a server. It is explanatory drawing of the communication sequence (terminal mode switching) between a server and a terminal. It is explanatory drawing of the communication sequence (utterance detection control) between a server and a terminal. It is explanatory drawing of the operation mode of a terminal, and the communication state between a server and a terminal.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, this invention is not limited by description of the following examples.
<Configuration of Information Sharing System of the Invention>
FIG. 1 shows a schematic configuration diagram of an embodiment of an information sharing system according to the present invention.
Here, an information sharing system (hereinafter also referred to as a conference system) in which information such as video, audio, and documents is shared and a conference is performed between a plurality of information terminals using image audio and the like will be described.

In FIG. 1, the information sharing system of the present invention is mainly composed of one server 1 and a plurality of information terminals (hereinafter also referred to as conference terminals or simply terminals) 2.
The server 1 and each information terminal 2 are connected via a network.
The server 1 is a device having at least a function equivalent to that of a multipoint connection device (MCU) among conference control functions also used in a conventional conference system, and mainly monitors the status of connected terminals, connection control, Various types of information communication control, video and audio encoding, video and audio decoding processing, user authentication, device authentication, communication encryption, and the like are performed.
The terminal 2 is a device installed in a room (such as an office or a conference room) in which a user who participates in a conference is present, and mainly recognizes the user himself / herself through video and audio, It performs input operation confirmation processing, processing conversion of various input information, and communication control processing.

In the information terminal 2, information necessary for the conference is input, and the input information is transmitted to the server. As will be described later, the input terminal information is mainly input signals such as a keyboard and a mouse, input sound from a microphone, and input video from a camera. The input information may be transmitted to the server after being encoded. In addition, encryption processing may be performed in consideration of confidentiality and confidentiality in the communication path.
The server 1 generates shared information using the terminal information transmitted from the terminal as described above, and distributes the shared information to each information terminal.

The server 1 and the terminals 2 (TE1 to TEn) may be connected directly by, for example, a wired communication cable, or may be connected by a wireless communication method such as a so-called wireless LAN.
Further, the server 1 and the terminal 2 may be connected via various networks (LAN (N1), Internet (N2), dedicated communication network (N3), etc.).
When a user who uses a certain terminal (TE1) participates in the conference, the communication line between the terminal TE1 and the server 1 is connected and the link is established, and then the camera or microphone attached to the terminal TE1 is used. The user's image and sound, or documents and drawing data necessary for the conference are transmitted to the server 1.
At this time, a codec of a predetermined standard (for example, MPEG4, H.263, H.264 for video, and G. for audio for video transmission reduction and real-time (low delay) communication). 711, G.722, G.729, etc.), a process (encoding) for compressing and encoding the amount of data to be transmitted is performed.

The server 1 decodes (decodes) the terminal information such as video and audio sent from the terminal TE1 and similar information sent from the other terminals 2 (TE2,... TEn), and then synthesizes (mixing). ) Is performed. Information (composition information) on which this composition processing has been performed becomes shared information. The shared information is distributed toward a plurality of terminals 2 that are currently connected.
In the server 1, before mixing a plurality of received videos, the video sent from each terminal 2 is decoded (decoded), further adjusted in image size (resized), and then shared by one user. A process of synthesizing a plurality of videos for distribution as information is performed.
For example, if four users are participating in a conference, the video data of each person's person image transmitted from the four terminals 2 is decoded and resized so that it can be divided and displayed on one screen. Four images are combined (mixed).
Furthermore, the mixed video after mixing is encoded (encoded) by the server, and then transmitted from the server 1 to the terminal 2 as shared information.

In each terminal 2 to which such mixed shared information has been transmitted, received information is decoded (decoded), and an image or sound is output from a display device or speaker attached to the terminal.
For example, if there are four terminals 2 that are currently connected, the shared information after mixing is distributed from the server 1 to each of the four terminals in real time using a time division multiplexing process or the like. .

  However, when the number of terminals that have been established further increases, for example, when the number of terminals is 20, in order to make the conference proceed smoothly, each of the 20 terminals 2 is similarly real-time. Since it is necessary to open the sound of the shared information by looking at the image or document of the shared information, the server 1 needs to perform information distribution processing at a higher speed.

  In general, ADSL or ISDN communication using a telephone line as a communication line, FTTH communication using an optical fiber, wireless communication, or other communication using a higher-speed dedicated line can be used. Because there is a limit to the amount of communication that can be transmitted in a certain period of time, there is almost no problem in smooth conference proceedings, and the maximum number of terminals that can deliver information in real time without causing problems in information transmission is also used. The upper limit is determined by the line thickness (communication bandwidth), the amount of video and audio data used for communication, the CPU performance of the terminal itself, and the like.

  In the present invention, even when there are a large number of terminals that can participate in a conference of the information sharing system, a connection is established to the server 1 at the same time, and the shared information is distributed directly from the server. The number is reduced as much as possible, and information that maintains real-time characteristics is distributed so that smooth progress of the conference is not hindered.

  The present invention is characterized in that the substantial reduction in the number of terminals connected to the server as described above is realized by the following two processes.

(1) Reduction of the number of connected terminals as seen from the server by grouping of terminals When there are multiple terminals on a network connected to the same router, set a group to which multiple terminals belong, Is set as a representative terminal, and the number of terminals directly communicating with the server is reduced by limiting the number of terminals directly communicating with the server to only the representative terminal.
In principle, terminals other than the representative terminal belonging to one group are allowed to receive the received information from the representative terminal that has received the shared information.

Here, grouping means dividing a plurality of information terminals connected to the network into one or a plurality of terminal groups based on predetermined setting conditions in advance.
One terminal group includes only one terminal or includes two or more terminals. In principle, one terminal group can directly receive shared information from a server. Assume that there is one representative terminal.
Further, when a plurality of terminals belong to one terminal group, one representative terminal as described above and a participating terminal that receives distribution information from the representative terminal are included.
However, as will be described later, in the case of a terminal group composed of a single terminal, the single terminal itself may be a representative terminal. At this time, the terminal group does not include a participating terminal. A group.
In addition, such a single terminal may not be a representative terminal, and at this time, the shared information may be received from a representative terminal belonging to another terminal group.

Various setting conditions for grouping are conceivable and are not limited to one. For example, connection base information where each information terminal is arranged can be used. Examples of connection base information include place names such as prefectures, municipalities, postal codes, addresses, telephone numbers, company names, business office names, organization names, and specific numerical codes based on these. Corresponds to this.
When the base ID is used as the setting condition, terminals having the same base ID are grouped as one terminal group. This geographical base information is stored in advance in the storage unit of the server for each terminal.
The setting conditions may be fixedly determined in advance, but the system administrator or the user may be able to change the settings before starting the conference.
Further, although not described in this paper, it may be determined that the same base is used by using a specific numeric string portion in the IP address of the terminal, that is, domain and network segment information.

The representative terminals belonging to one terminal group directly receive the shared information from the server, and the reception method is assumed to be streaming distribution from the server. This enables efficient one-way communication with a reduced amount of information to be transmitted, compared to the case of performing two-way communication in a normal conference system.
In addition, the shared information received from the server is also distributed in a streaming manner to participating terminals that belong to the same terminal group as the representative terminal.
Further, the shared information received from the server may be stream-distributed to other information terminals that do not belong to the terminal group of representative terminals as in the second embodiment described later.

(2) Switching the terminal state based on speech detection When a user makes a speech request, the terminal used by the user switches to a state in which bidirectional communication of information consisting of images, sounds, etc. is possible with the server. .
When there is a terminal that has already made a speech request (referred to as an existing speech terminal), the existing speech terminal is switched to a terminal that does not have a speech request, and instead, a terminal that has newly made a speech request is replaced with an existing speech terminal. Switch to the terminal.

For example, in the case where there is a first information terminal that has made a request for speech and the communication state with the server is a two-way communication state, speech detection of a second information terminal that is different from this is described later. When the unit detects a new message request, a message request unit (to be described later) of the second information terminal transmits information indicating the new message request (message request information) to the server.
When the server receives this message request information, the server, which will be described later, cancels the two-way communication state with the first information terminal, and the second information terminal and the server requesting a new message The communication state between is switched to the two-way communication state.

  Here, it is assumed that there is always only one existing speech terminal that has a right to make a speech request in the entire system. In addition, a terminal that has newly become an existing speech terminal performs two-way communication of information such as images and sounds with the server, and a terminal that is no longer an existing speech terminal receives streaming distribution from the server or is represented Receive streaming distribution from the terminal.

Moreover, the detection of the speech request by the user may be performed almost simultaneously by the speech detection units of a plurality of information terminals. In this case, the bidirectional communication control unit of the server indicates the communication state with the information terminal that has transmitted the speech request information received earliest among the speech request information respectively transmitted from the plurality of information terminals. What is necessary is just to switch to a communication state.
Alternatively, when the time information at which the user requests to speak is included in the plurality of received speech request information, the respective speech request information is confirmed, and the user's speech request is detected at the earliest time. The communication state with the information terminal determined to have been switched may be switched to the bidirectional communication state.

<Description of terminal operation mode>
Assume that the following four modes are mainly provided as modes of operation states of terminals (also referred to as operation modes or simply modes).
(A) Participating terminal mode (M1)
(B) Representative terminal mode (M2)
(C) Speech request terminal mode (M3)
(D) Presentation terminal mode (M4)

Participating terminal mode (M1) is connected to a network and receives streaming distribution of shared information consisting of images and sounds from a server or a terminal in representative terminal mode (referred to as a representative terminal). This means an operating state where image and sound are not transmitted.
Therefore, a terminal in the participating terminal mode (referred to as a participating terminal) can perform only one-way communication by streaming reception of shared information with a server or a representative terminal.
However, as will be described later, in this participating terminal, when the user speaks, the user is switched to the speech requesting terminal mode (M3), and bidirectional communication of images and sounds with the server can be performed.

The representative terminal mode (M2) is connected to a network, receives streaming distribution of shared information including images and sounds from a server, and is sent to other participating terminals in the same group or other terminals other than the same group. On the other hand, it means an operating state in which streaming distribution of shared information can be performed.
Therefore, there is only one representative terminal in the same group, and a terminal that performs one-way communication by direct streaming reception to the server.
A plurality of terminals belonging to the same group correspond to terminals having the same base ID, for example.

The message request terminal mode (M3) means an operating state that is connected to a network and capable of bidirectional communication of information including images and sounds with a server.
A terminal that can enter the speech request terminal mode is a terminal that is allowed to speak by the server after making a speech request to the server.
In addition, the server is managed by the server so that there is only one terminal that is in a bidirectional communication state with the server and is allowed to speak by the user. This management may be performed by a speech request terminal flag, as will be described later.
A terminal in the speech request terminal mode (hereinafter also referred to as speech terminal mode) is called a speech request terminal or speech terminal.
If speaking is permitted, the above-described participating terminals and representative terminals can also be speaking terminals.

The presentation terminal mode (M4) is an operation mode of a terminal of a user or a chairperson who makes a presentation about a certain theme in a meeting, for example, at a presentation, and the terminal is set in advance as a presentation terminal in the server. ing. As will be described later, a terminal whose “presentation terminal flag” is set to ON in the storage unit of the server is a presentation terminal.
When this announcement terminal is connected to the server, in principle, it performs one-way communication that only transmits images and voices input through a microphone to the server. At this time, shared information such as images from the server is not received.
Further, the image to be transmitted is encoded (encoded) by a method conforming to a predetermined standard, and then transmitted to the server.
In addition, this announcement terminal is switched to perform bidirectional communication with information such as images and sounds after the requesting terminal is connected to the network.

FIG. 28 shows an explanatory diagram of the relationship between these four operation modes and the communication state with the server and other terminals.
In FIG. 28, a terminal (participating terminal) in the participating terminal mode (M1) indicates performing one-way communication that receives streaming distribution from a server or one-way communication that receives streaming distribution from another terminal.
Whether to receive streaming distribution from the server or another terminal is determined depending on the connection form of the participating terminal (terminal in the group or terminal not belonging to the group).
For example, participating terminals belonging to one group receive streaming distribution from the representative terminal of the group.
A single participating terminal that does not belong to any group receives streaming distribution from the server in principle.

The terminal in the representative terminal mode (M2) receives streaming distribution from the server, and further performs streaming distribution to other terminals in the same base (same group) or another base.
The terminal in the message request terminal mode (M3) indicates that bidirectional communication is performed with the server.
Furthermore, when the terminal in the presentation terminal mode (M4) is connected to the server, the terminal performs one-way communication that only transmits data to the server, and thereafter, at the time of the conference in which a speech is performed at another terminal. Two-way communication with the server.

  As described above, the four modes indicating the operation state of the terminal have different communication states with the server and other terminals. In the present invention, each terminal switches its operation mode based on the occurrence of an event such as a request from the server or detection of speech, thereby reducing traffic flowing through the network as much as possible and reducing information transmission delay. The present invention is characterized in that communication troubles are reduced and processing load on video / audio encoding / decoding in the server is reduced as much as possible.

Hereinafter, embodiments of the operation mode switching process of the terminal and the change of the communication state between the server and the terminal at the time of a speech request will be described.
In the following embodiments, a plurality of terminals belonging to one group are assumed to be a plurality of terminals belonging to the same base.
Whether or not a plurality of terminals are at the same site is registered in advance in the storage unit of the server, and a plurality of terminals at the same site all have the same site ID.
In addition, one representative terminal in one group is in a state where power is first turned on in the group and an application program necessary for the conference is activated.
In addition, the terminal notifies the server of the operation state of its own operation mode, and further issues a representative terminal detection request, so that the server performs representative terminal detection processing and determines the representative terminal. The representative terminal is set in the storage unit of the server. When a representative terminal flag described later is “ON”, it indicates that the terminal is set as the representative terminal.

Further, after a representative terminal of a certain group is determined, other terminals belonging to the same group operate as participating terminals. In other words, terminals that belong to the same group and become participating terminals receive streaming distribution from the representative terminal of the same group in principle.
However, as a general rule, a single participating terminal that does not have a representative terminal and does not have other terminals having the same base ID is itself a representative terminal and receives streaming distribution from the server.
Moreover, such a single participating terminal may receive streaming distribution from a representative terminal belonging to another group. In this case, the representative terminal performs streaming distribution to participating terminals belonging to the same group and streaming distribution to a single participating terminal not belonging to the group.
When the representative terminal is disconnected from the network, in principle, any of the other participating terminals in the same group may be the representative terminal.

Also, a terminal that has made a speech request can become a speech terminal by receiving permission from the server.
As described above, if there is no existing speech terminal, the terminal that made the speech request becomes the speech terminal as it is and performs bidirectional communication with the server. On the other hand, when the existing speech terminal already exists, the existing speech terminal is switched to the participating terminal and the terminal that newly requested the speech is switched to the speech terminal in response to a switching request from the server.
In addition, when a user at the representative terminal makes a speech request, the representative terminal becomes a speech terminal and performs two-way communication with the server. However, if there is streaming distribution that has been performed to other participating terminals, the streaming is performed. Distribution is also continued as it is.
Further, when the representative terminal that is the speaking terminal is no longer the speaking terminal, the terminal returns to the terminal in the representative terminal mode, and the communication state with the server is switched from the two-way communication state to the one-way communication state that receives streaming distribution. .

<Description of switching of communication status>
Here, the switching of the communication state from the bidirectional communication to the unidirectional communication is mainly considered to be one of the following two methods.
(1) A predetermined communication sequence is continued with the communication protocol fixed, but a series of signal processing for video and audio transmitted from a terminal (participating terminal or representative terminal) without a request for speech is stopped. Specifically, the server stops the video decoding process (decoding) and the audio decoding process (decoding) for the terminal.
In this case, for example, SIP or H.264 is used as a communication protocol. If a standard such as H.323 or HTTP is adopted, a communication sequence using the communication protocol is continuously performed. This method has an advantage that it is not necessary to separately prepare a communication protocol for streaming delivery between server terminals, and a conventional communication protocol can be used as it is.

(2) The communication protocol is changed to a streaming distribution protocol.
For example, SIP and H.264. 323, the communication protocol such as HTTP is stopped, and the communication protocol is changed to the streaming distribution related to video and audio. Streaming distribution is a communication method in which data such as video and audio is unidirectionally transmitted from a server to a terminal.
For streaming delivery, it is desirable to employ any one of predetermined standards (mms, http, rtmp, rtsp, etc.).

In any of the above methods, data is transmitted and received between the terminal and the server based on a predetermined communication protocol, but video and audio are transmitted unidirectionally from the server to the terminal. Therefore, the traffic flowing through the communication line can be reduced. Therefore, the delay in data transmission can be reduced, and communication failures such as communication errors can be reduced. As a result, the number of terminals connected to the server can be substantially reduced. It becomes possible.
On the contrary, the switching from the one-way communication to the two-way communication may be performed in the reverse process of (1) or (2).

<Explanation of speech request detection>
In this invention, the communication state between the server and the terminal is switched by detecting a user's request for speech.
In general, in a conference system that manages a participant's speaking right, a terminal user (participant) obtains a speaking right and uses a method for requesting a speech and approving / permitting the request. As a method for requesting a speech, for example, the following can be considered.
(1) Request by pressing (speech button) (input operation) (2) Request by inputting "voice" (3) Request by "image recognition"

Here, in the case of (1), for example, one of the input keys provided on the keyboard or the like may be assigned as a command key for requesting speech.
When this command key is pressed, it is determined that an input operation for a speech request has been performed, and the speech request is transmitted to the server.

In the case of (2), for example, a voice uttered by the user toward the microphone may be detected, or a specific word associated with the utterance request may be detected by voice recognition from the uttered voice. .
If it is detected from these input speeches or speech recognition results that a speech request is made, the speech request is transmitted to the server.
In the case of (3), an operation associated in advance as a speech request, for example, an operation of raising a user's hand or an operation of waving may be performed toward the camera.
At this time, if the image input from the camera is recognized and it is detected that the operation of raising the hand matches the operation of the predetermined speech request, the speech request is transmitted to the server.

<Configuration of information terminal>
FIG. 2 shows a configuration block diagram of the information terminal of the present invention.
The information terminal 2 (TE) mainly includes an input unit 11, a video input unit 21, a display unit 31, a voice input unit 41, a voice output unit 45, a communication unit 51, an authentication unit 55, a storage unit 61, a signal processing unit 71, And a control unit 80.

The input unit 11 includes a human detection sensor 15 in addition to a pointing device such as a keyboard 12, a mouse 13, and a tablet 14. The input processing unit 16 receives these inputs and converts them into signals for transmission to the control unit 80. With.
As the human detection sensor 15, for example, a heat detection sensor, an infrared sensor, an ultrasonic sensor, or the like can be used. Instead of the human detection sensor, it may be detected whether there is a person in front of the terminal based on input signals from a camera 22 and a microphone 42 described later.
From the input unit 11, an input signal mainly by a user's operation is input, and this input signal is used for generating the speech request information. For example, when a command key associated with a speech request is pressed and a corresponding input signal is input, speech request information indicating that the user of the information terminal has made a speech request is generated.

  The video input unit 21 is a part for inputting video captured by the camera. For example, a part for inputting video near the terminal or video of a person (user) participating in a meeting in front of the terminal. And a camera 22 and a video processing unit 23 for converting a video shot by the camera into a digital signal. The input video 66 is temporarily stored in the storage unit 61. This input video 66 is also used to generate the above-mentioned speech request information.

  The display unit 31 is a part that displays a user's own image or a person image of another person participating in the conference, and displays a display 32 such as an LCD or CRT and a display signal to be displayed on the display 32. And a display processing unit 33 to be generated.

The sound input unit 41 is a part that inputs sound near the terminal TE from a microphone, and includes a microphone 42 and an input sound processing unit 43 that converts sound input from the microphone into a digital signal. The input voice 67 is temporarily stored in the storage unit 61.
This input voice 67 is also used to generate the above-described speech request information.

  The audio output unit 45 includes a speaker 46 and an output audio processing unit 47. For example, the audio output unit 45 is a part that outputs audio transmitted from the server from the speaker.

The communication unit 51 is a part that performs data transmission and data reception with a server or another terminal TE, and uses a predetermined communication protocol (for example, H.323, SIP, HTTP, etc.) via various networks. Data communication control is performed so that data communication is possible. In the conference system, voice and video communications are performed. Data encryption processing (for example, AES: Advanced Encryption Standard, 3DES: Triple) is performed to conceal data and reduce the amount of data as necessary. -Data Encryption Standard, etc.), data compression processing (also called encoding or encoding), for example, MPEG4, H.263, H.264 for video, G.711, G.722, G.729 for audio. Etc.).
Therefore, the communication unit 51 preferably includes a communication encryption unit 52 and a communication decryption unit 53.
In addition, the communication unit 51 changes the communication state with the server according to the operation state after being changed by the terminal mode switching unit 82.
For example, the state is changed to a bidirectional communication state in which video and audio are transmitted and received bidirectionally, and a one-way communication state in which video and audio are received unidirectionally from the server.

  The authentication unit 55 is a part that performs user authentication and login authentication to the terminal and the conference system, and performs authentication mainly by inputting an ID and a password registered and managed in advance.

The storage unit 61 is a part that stores information such as various types of input information (input video 66, input voice 67, etc.), display information, output information, transmission / reception data, signal processed data, and the like, such as voice recognition and image recognition. A recognition database 62 stored in advance is provided.
The recognition database 62 mainly includes user's voice feature data for performing voice recognition and image feature data for performing image recognition. When a plurality of users use the terminal TE, the recognition information may be stored for each user to recognize the person of the user. Although not shown in this paper, the recognition database 62 may store password information for each user (the data itself is preferably encrypted).
As the storage unit 61, a semiconductor element such as a ROM or a RAM, a storage medium such as a hard disk, a CD, a DVD, or a USB memory is used.
The storage unit 61 also stores a program for executing each function of the terminal TE.

The signal processing unit 71 converts input video and audio into data to be transmitted to the server (also referred to as encoding processing and encoding processing), and displays video and audio transmitted from the server or the like on a display. It is a part that converts (also called decoding processing or decoding processing) into original data that can be output from the display and speaker.
It mainly comprises a video encoding unit 72, a video decoding unit 73, an audio encoding unit 74, and an audio decoding unit 75.
For the signal processing unit 71, for example, a codec LSI capable of executing encoding and decoding conforming to a codec of a predetermined standard is used. Encoding and decoding processing may be realized by a software codec instead of a hardware codec such as an LSI.

The control unit 80 is a part corresponding to a microcomputer including a CPU, a ROM, a RAM, an I / O controller, a timer, and the like. When the CPU reads a program stored in the storage unit 61 into a RAM (main memory) The various functions of the terminal are executed by organically operating various hardware.
In FIG. 2, the terminal state notification unit 81, the terminal mode switching unit 82, and the like are shown as the functions performed by the control unit 80. This is realized by executing predetermined processes using information obtained from the input unit 11, the video input unit 21, the audio input unit 41, and the like based on the program.

The terminal state notification unit 81 is a part that transmits the operation state of the information terminal to the server. For example, state information indicating that one of the four modes (M1, M2, M3, M4) is in the state is generated and transmitted to the server via the communication unit 51.
For example, when a speech request is detected, status information indicating that the terminal has entered the speech request terminal mode is transmitted to the server.

The terminal mode switching unit 82 is a part that changes the operation state of its own terminal according to the notification from the server or the state of its own hardware, for example, a part that changes to one of the above four modes. is there.
For example, when there is a speech request and speech is permitted, the current operating state of the terminal is changed to the speech request terminal mode. By changing the operation state, communication with the server is switched to bidirectional communication.

In addition, the bidirectional communication control unit 89 transmits the input video and input audio to the server via the communication unit 51 according to a predetermined communication protocol, and is shared by the video and audio transmitted from the server. This is the part that receives information.
The representative terminal detection request unit 83 is a part that requests the server to detect a representative terminal in the same base, and receives information on the detected representative terminal from the server.
The speech detection unit 84 is a portion that detects that the user of the terminal has made a speech request by any of the methods described above. That is, when any one or more of an input signal, an input voice, and an input video that mean a speech request is detected, it is determined that the user has requested the speech.
The speech request unit 85 is a part that transmits speech request information indicating that a detected speech request has been made to the server.

The streaming request unit 86 is a part that requests the representative terminal to distribute streaming information (audio and video) to the terminal.
The streaming receiving unit 87 is a part that receives streaming information distributed from a server or a representative terminal.

The streaming delivery unit 88 is a part that transmits streaming information to other terminals when the terminal is a representative terminal.
Here, streaming distribution means that video and audio, which are streaming information, are transmitted in one direction.
An information terminal that is distributed by streaming does not encode or transmit the video and audio data of the terminal itself, so the load on the terminal is less than that in the case of performing bidirectional communication.

The above is the main functional block of the terminal, but the function realized by the control unit 80 is not limited to this.
Since these functions are mainly realized by software, a program for executing other functions is stored in the storage unit 61 as necessary, and the CPU organically operates the hardware based on the program. The function may be executed.
In addition, the program may be provided in a fixed storage in a ROM or the like in advance, but may be stored in a rewritable storage device such as a hard disk for subsequent deletion, addition, change, update, etc. The program may be provided by a recording medium such as a CD or a DVD, or may be downloaded from a server or the like via a network.

<Server configuration>
FIG. 3 shows a configuration block diagram of the server of the present invention.
The server 1 mainly includes an input unit 101, a display unit 111, a communication unit 121, an authentication unit 125, an MCU unit 131, a storage unit 171, a signal processing unit 151, and a control unit 161.

The input unit 101 is a part for inputting various types of information. The input unit 105 converts input from the keyboard 102, mouse 103, tablet 104, and these input devices into signals that can be transmitted to the control unit 161, and Is provided.
The display unit 111 includes a display 112 such as an LCD or a CRT, and a display processing unit 113 that generates a display signal to be displayed on the display.
However, when the server 1 is operated completely unattended, the input unit and the display unit may not be provided.

The communication unit 121 is a part that transmits / receives data to / from the terminal TE via the network, and performs data communication with a predetermined communication protocol (for example, H.323, SIP, HTTP, etc.), video, Audio is encoded with a predetermined codec (for example, MPEG4, H.263, H.264 for video, G.711, G.722, G.729 for audio, etc.), and a predetermined standard (for example, AES) : Advanced Encryption Standard, 3DES: Triple-Data Encryption Standard, etc.).
Similar to the terminal TE, the authentication unit 125 is a part that performs user authentication and login authentication to the terminal and the conference system.

  The MCU unit 131 is a part that executes processing performed in a so-called multipoint connection unit (MCU), and mainly includes a video mixing unit 132 that combines a plurality of video signals, and a video resizing unit 133 that changes the size of the video. , And a voice mixing unit 134 that synthesizes a plurality of voice signals. The MCU unit 131 may be configured in common with the signal processing unit 151.

The storage unit 171 is a part that stores various data used in processing executed by the server, and further specifies information (device ID 173, base ID 174, IP address 175, announcement terminal flag) that is connected to the server. 176, representative terminal flag 177, speech request terminal flag 178) and a user database storing information (user ID 179, site ID 174, IP address 175, mail address 180) for identifying a user who uses the terminal. This is the part that stores the database 181.
The terminal database 172 and the user database 181 are used for two-way communication between the server and the terminal, for transmitting one-way information such as a large volume of audio / video from the server to the terminal, or for detecting the representative terminal. Used in processing, existing speech terminal detection processing, etc.
In addition to this, programs necessary for executing each function of the server are also stored in the storage unit 171.
As the storage unit 171, a ROM, a RAM, a hard disk, a recording medium such as a CD or a DVD, or the like is used.

The signal processing unit 151 performs conversion processing (decoding processing, decoding processing) for temporarily returning the video and audio sent from the terminal to the original signal, and transmits video and audio to a plurality of terminals. This is the part that performs data conversion processing (encoding processing, encoding processing).
Similar to the terminal TE, the signal processing unit 151 mainly includes a video encoding unit 152, a video decoding unit 153, an audio encoding unit 154, and an audio decoding unit 155, and is realized using a codec LSI or a software codec. can do.

The control unit 161 is a part corresponding to a microcomputer including a CPU, a ROM, a RAM, an I / O controller, a timer, and the like. The CPU performs various hardware operations based on a program stored in the storage unit 171. Each function of the server is realized by operating the server.
In FIG. 3, as a function of the control unit 161, a terminal state determination unit 162, a communication method switching unit 163, a representative terminal detection unit 164, an existing speech terminal detection unit 165, a streaming distribution unit 166, and bidirectional communication control Part 167.

The terminal state determination unit 162 is a part that mainly receives state information indicating the mode of the operation state sent from the terminal and determines which mode the terminal is currently in.
The communication method switching unit 163 is a part that switches the communication method for the information terminal based on the operation state of the terminal transmitted by the terminal state notification unit of the information terminal. As communication methods, there are mainly a bidirectional communication state, a one-way communication (streaming distribution) state in which transmission of video and audio and the like is only in the direction from the server to the terminal, and a data communication stop state.
The communication state between the server and the information terminal is changed for each information terminal according to the operation state of the information terminal.

  The representative terminal detection unit 164 is a part that detects whether or not a terminal to be a representative terminal exists in the same base using the base ID 174 stored in the terminal database 172 of the storage unit 171.

  FIG. 10 shows an explanatory diagram of an embodiment of the contents of each database stored in the storage unit of the server.

FIG. 10A shows an example of the terminal database 172 stored in the storage unit of the server.
Here, for each terminal, a device ID 173, a base ID 174, an IP address 175, an announcement terminal flag 176, a representative terminal flag 177, and a speech request terminal flag 178 are stored in association with each other.

FIG. 10B shows an example of the user database 181 stored in the storage unit of the server.
Here, the user name 179, the base ID 174, the mail address 180, and the IP address 175 are stored in association with each other.
Note that the items of information stored in each database are not limited to those described above, and any other necessary items may be additionally stored.

In FIG. 10, a representative terminal flag 177 is information for identifying whether or not the terminal is a representative terminal. When “ON”, it means that the terminal is a representative terminal, and “OFF” Sometimes the terminal is not a representative terminal.
In FIG. 10, of the four terminals, two terminals whose base ID is “Kansai” are terminals of the same base, and among the two terminals included in the same base, the device ID = “G53182” The representative terminal flag 177 of the terminal “” is “ON”, which means that this terminal is the representative terminal.
Here, the same base forms one group, and there is only one representative terminal in one group.

In the grouping, for example, as shown in FIG. 10, terminals installed at relatively close positions are set as one group, and a plurality of terminals belonging to the same base belong to one group. Treat as.
In addition, a plurality of terminals belonging to a network under the same router or the same relay server may be treated as belonging to one group, and a specific numeric string portion of the numeric strings constituting the IP address is the same. Terminals (terminals in the same domain) may be treated as belonging to one group.
In the following description, a plurality of terminals having the same base ID are treated as belonging to the same group.

In the case of FIG. 10, two terminals included in the same site are regarded as terminals belonging to one group, and as viewed from the server, only the representative terminals belonging to the group are connected to the server. Be treated. That is, when information such as video and audio is transmitted from the server, the information is transmitted from the server to only the representative terminals belonging to the same base (the same group), and is transmitted to other participating terminals that are not representative terminals belonging to the same base. Is not sent directly from the server.
In principle, information is transmitted to participating terminals that are not representative terminals by streaming distribution from representative terminals belonging to the same base.

  Further, in principle, the representative terminal in the same base performs streaming distribution only to other participating terminals belonging to the same group. However, when a streaming distribution request is received from a terminal TEn belonging to another base, The terminal may perform streaming delivery to the terminal TEn at another base.

Further, the existing speech terminal detection unit 165 is a portion that detects whether or not there is a terminal that is already a speech terminal, and detects it using the speech request terminal flag 178 in the terminal database.
If there is a terminal whose “utterance request terminal flag” shown in FIG. 10 is “ON”, it indicates that the terminal is a speaking terminal, and “OFF” indicates that it is not a speaking terminal.
In the case of FIG. 10, it is indicated that the terminal TE4 is a speech request terminal.

The streaming distribution unit 166 is a part that transmits shared information including video and audio to the terminal as streaming.
The bidirectional communication control unit 167 is a part that controls bidirectional communication between the server and the terminal according to a predetermined communication protocol, like the bidirectional communication control unit 89 of the terminal described above.
When speech request information is received from an information terminal, the communication state between the information terminal and the server is switched to a bidirectional communication state in which terminal information and shared information can be directly transmitted and received.
The above is the main function of the server.

<First embodiment: description of communication state between server terminals>
Below, the communication state between six servers and a terminal is demonstrated as an Example of the communication state between server terminals.
Here, description will be made assuming that four terminals (TE1 to TE4) are connected to the server 1 via a network. However, the number of connected terminals is not limited to four.
FIG. 4 shows an explanatory diagram (embodiment 1-1) of a communication state between server terminals in the first embodiment.
FIG. 4 shows a state in which only one terminal TE1 is turned on.
It is assumed that the terminal TE1 is a presentation terminal, and the presentation terminal flag of the terminal TE1 is set to “ON” in advance in the server.
In this case, the communication state between the presentation terminal TE1 and the server is a state in which only one-way communication from the terminal TE1 to the server is possible.
That is, the input image (SD1) of the self-portrait photographed with the camera of the terminal TE1 is encoded by the terminal and then transmitted to the server using a predetermined communication protocol. In this one-way communication, only image and audio data transmission from the terminal to the server is possible, and image and audio data transmission from the server to the terminal is not performed.

The other terminals (TE2, TE3, TE4) are still in a power-off state and are not connected to the server.
The terminal TE2 is a single terminal without other terminals belonging to the same base.
Terminals TE3 and TE4 are terminals belonging to the same base and grouped into one group.
In the state of FIG. 4, these three terminals (TE2, TE3, TE4) do not have images to be transmitted (transmission images SD2, SD3, SD4) and further have no reception images (RD2, RD3, RD4).
Further, in FIG. 4, only one terminal TE1 is connected to the server 1, and since it is in a one-way communication state, there is no received image displayed on the display of the terminal TE1.

FIG. 11A shows an explanatory diagram of the states of the three flags of the terminal database in the case of the embodiment 1-1.
As shown in FIG. 11A, only the announcement terminal flag of the terminal TE1 is “ON”, and all other flags are “OFF”.
In addition, the two terminals (TE3, TE4) having “Kansai” in the base ID are treated as terminals belonging to the same base, that is, the same group.

FIG. 5 is an explanatory diagram of a communication state between server terminals (embodiment 1-2) in the first embodiment.
Here, the state where the terminal TE3 is powered on and activated from the state of FIG. 4 is shown. The terminal TE3 is assumed to be in a state where an application program necessary for the conference is activated and communication with the server is possible.
It is assumed that the other terminals TE2 and TE4 are not connected to the server in a power-off state.
The terminal TE3 is assumed to be the first activated terminal in one same site to which the two terminals (TE3, TE4) belong. At this time, the terminal TE3 transmits a representative terminal detection request to the server.
In the state of FIG. 4, if there is no terminal serving as a representative terminal in the same base, the server TE3 notifies that the representative terminal has not been detected. It becomes.
Thereafter, the terminal TE3, which has become the representative terminal, notifies the server that it has become the representative terminal of the same base, and switches the operation mode to the representative terminal mode so that streaming distribution can be received from the server.
The server performs a series of signal processing from decoding to encoding, generates a composite image including only the video SD1 of the terminal TE1 currently being received, and distributes it to the terminal TE3 by streaming. This composite image is one piece of shared information.

The terminal TE3 displays the image RD3 received by streaming delivery on the display. Here, the received image RD3 is obtained by, for example, dividing one screen into four and displaying the self-portrait of the terminal TE1 on the one divided screen (the upper left screen in FIG. 5).
As for the flag in the state of FIG. 5, the representative terminal flag of the terminal TE3 is changed from “OFF” to “ON” as shown in FIG. 11B.
In addition, since the presentation terminal TE1 performs one-way communication with the server, the reception image RD1 from the server is not received, but the terminal TE1 side generates a reception image RD1 including only the self-portrait of the terminal TE1. You may display on a display.

FIG. 6 is an explanatory diagram of a communication state between server terminals (embodiment 1-3) in the first embodiment.
Here, from the state of FIG. 5, the terminals TE2 and TE4 are in a state where the power is turned on and activated.
The representative terminal TE3 receives streaming distribution of the composite image from the server, and is in a state where streaming distribution can be performed to other terminals (TE4) in the same base.
First, since the terminal TE4 belonging to the same base already has the representative terminal TE3, it becomes a participating terminal and receives streaming distribution from the representative terminal TE3. The received image RD3 that is streamed from the server and the received image RD4 that is streamed from the representative terminal TE3 are the same shared information.
Similarly to FIG. 5, the image RD4 sent from the representative terminal TE3 is a four-divided screen image consisting only of the self-portrait of the terminal TE1, and the terminal TE4 displays the received image RD4 on the display.
Since the participating terminal TE4 receives only streaming distribution from the representative terminal TE3 and cannot transmit video or the like to the server itself, there is no transmission image SD4.

Next, since the terminal TE2 is a single terminal, it transmits a representative terminal detection request to the server, and is notified of non-detection. The terminal TE2 that has become the representative terminal receives streaming distribution from the server. Also for terminal TE2, image RD2 received from the server is a four-divided image consisting only of the self-portrait of terminal TE1.
Also, since the terminal TE2 cannot transmit video or the like to the server, there is no transmission image SD2.
On the other hand, in the server, it is set that the terminal TE2 has become the representative terminal, and as shown in FIG. 11C, the representative terminal flag of the terminal TE2 is set to “ON”.
In the state of FIG. 6, all four terminals are in an activated state, but one terminal TE1 is in a one-way communication state in which only transmission is performed to the server, and two terminals ( TE2 and TE3) are in a one-way communication state in which streaming reception is performed with respect to the server, and the other terminal TE4 is a terminal that does not perform communication such as video directly with the server.

However, in the state of FIG. 6, none of the three terminals (TE2, TE3, TE4) can transmit their own images (SD2, SD3, SD4), and the user of the presentation terminal TE1 unilaterally, It can be said that the contents of the announcement are explained to users of other terminals.
In the state of FIG. 6, the communication direction is limited by the state (operation mode) of the terminal, so that the load on the server can be reduced compared to the case where all four terminals perform bidirectional communication with the server. By reducing the traffic flowing through the network, transmission delay can be reduced and the occurrence of communication failures can be reduced.

FIG. 7 is an explanatory diagram of a communication state between server terminals (embodiment 1-4) in the first embodiment.
Here, in the state of FIG. 6, the user of the terminal TE2 performs a speech request operation, and there is a speech request from the terminal TE2 to the server, and the speech is permitted.
The terminal TE2 is also a representative terminal and a speaking terminal. The terminal TE2 that has become the speaking terminal is switched to a state in which bidirectional communication of video and audio is performed with the server.
In the server, as shown in FIG. 12D, the speech request terminal flag of the terminal TE2 is set to “ON”.
Furthermore, since the terminal TE2 can perform bidirectional communication with the server, the terminal TE2 transmits its own image (SD2) to the server.
The server performs a series of signal processing such as decoding, resizing, mixing, and encoding on the video SD1 sent from the terminal TE1 and the video SD2 sent from the terminal TE2, and these two terminals (TE1, TE1) are processed. A composite image composed of the video of TE2) is generated and transmitted to the terminal.

In the state of FIG. 7, it is assumed that the presentation terminal TE <b> 1 is switched to a mode in which bidirectional communication with the server can be performed because another terminal that has become a speech terminal has appeared.
In this state, the composite image is transmitted from the server to the three terminals (TE1, TE2, TE3). Among these, transmission to the representative terminal TE3 is streaming transmission.
Similarly to FIG. 6, the same composite image is transmitted to the participating terminal TE4 by streaming delivery from the representative terminal TE3. As a result, the same composite image is displayed on all four terminals, and the conference is held while viewing the screen on which the self-portrait of the presentation terminal TE1 and the self-portrait of the terminal TE2 of the user currently speaking are displayed separately. Can proceed.
Also in this case, the transmission delay and communication failure can be reduced and the load on the server can be reduced as compared with the case where four terminals communicate with the server in two ways.
In addition, because the communication state between the terminal that requested the speech and the server is switched to bidirectional communication by the speech request, the traffic increases, but the increase can be suppressed to the minimum necessary range and the speech is made. Since the user's self-portrait can be viewed on each terminal, information necessary for the conference can be obtained, and the conference can proceed smoothly.

FIG. 8 is an explanatory diagram of the communication state between the server terminals (Example 1-5) in the first example.
Here, in the state of FIG. 7, the user of the terminal TE3 performs a speech request operation, the speech request is permitted, and the speech terminal is changed from the terminal TE2 to the TE3.
First, the terminal TE2 (existing speech request terminal), which has been a speech terminal until now, returns to the representative terminal (the same state as in FIG. 6), and the communication state with the server also returns to the one-way communication state that receives streaming distribution from the server. .
Accordingly, transmission of the self-portrait SD2 of the terminal TE2 is stopped.
On the other hand, the terminal TE3 becomes a speech terminal and performs bidirectional communication with the server.
Since the terminal TE3 can transmit video and the like to the server, the terminal TE3 transmits the self-portrait SD3 to the server. The server generates an image obtained by combining the self-portraits (TE1, TE3) of the two terminals, and transmits the combined image to the three terminals (TE1, TE2, TE3).

In the server, as shown in FIG. 12E, the message request terminal flag of TE2 is “ON”.
”Is changed to“ OFF ”, and the speech request terminal flag of TE3 is set to“ ON ”.
In addition, the participating terminal TE4 receives the same composite image by streaming delivery from the terminal TE3 which is also the representative terminal. Thereby, in the four terminals, the same received image consisting of the self-portraits of the two terminals (TE1, TE3) is received and displayed on the respective displays.
In the case of FIG. 8, the speaking terminal capable of bidirectional communication is merely changed from the terminal TE2 to the TE3. Therefore, as in the case of FIG. Can reduce the load.

FIG. 9 is an explanatory diagram of the communication state between the server terminals (Example 1-6) in the first example.
Here, in the state of FIG. 8, a user of the terminal TE4 performs a speech request operation, the speech request is permitted, and the speech terminal is changed to the terminal TE4.
First, the terminal TE3 (existing speech request terminal) returns from the speech terminal to the representative terminal, and the communication state with the server is changed to a communication state in which streaming distribution is received from the server.
The self-portrait SD3 cannot be transmitted from the terminal TE3 to the server.
Further, since the terminal TE4 which has been a participating terminal until now becomes a speaking terminal, the terminal TE4 is switched so as to be able to perform two-way communication with the server. That is, the streaming delivery from the terminal TE3 to the terminal TE4 is stopped, and the terminal TE4 enters a state of directly communicating with the server, and can transmit its own image (SD4) to the server.

As shown in FIG. 12F, the server changes the message request terminal flag of the terminal TE3 to “OFF” and sets the message request terminal flag of the terminal TE4 to “ON”.
Further, the server synthesizes the self-portraits of the two terminals (TE1, TE4) and transmits the synthesized video to the four terminals.
Each terminal receives the same received image made up of the self-portraits of the two terminals (TE1, TE4) and displays them on the respective displays.
In the case of FIG. 9, the communication state between the server and the two terminals (TE1, TE4) is bidirectional communication, and the communication state between the server and the two terminals (TE2, TE3) Become.
Therefore, the traffic is less than when all four terminals perform two-way communication, so that transmission delay can be reduced, communication failure can be reduced, and server load can be reduced.

<Description of Communication Sequence of First Embodiment>
FIG. 26 shows a schematic explanatory diagram of an embodiment of a communication sequence when the operation mode of the terminal is switched.
Here, it is assumed that the information terminal TE1 is currently in the participating terminal mode (M1).
FIG. 26A shows a sequence when a representative terminal is detected when a request for detecting a representative terminal is transmitted to the server, and FIG. 26B shows that the representative terminal is not detected and the terminal TE1 itself The sequence for changing the operation mode to the representative terminal is shown.

In FIG. 26 (a), first, in order for the terminal TE1 to confirm whether there is a group to which the terminal TE1 can belong, the current operation mode of the terminal TE1 is the participating terminal mode (M1). A terminal status notification indicating the presence is transmitted, and a representative terminal detection request is further transmitted.
The terminal status notification includes, for example, the device ID of the terminal TE1, the base ID, the ID address, information indicating the participation terminal mode, and the like.

The server that has received the terminal state notification from the terminal TE1 sets all the three flags of the terminal TE1 in the terminal database to “OFF”, and sets the state indicating the participation terminal mode.
Also, the server that has received the representative terminal detection request checks whether there is a representative terminal that can be grouped with the terminal TE1 using the terminal database.
If there is a terminal (TE2) having the same base ID as the terminal TE1 and the representative terminal flag “ON” in the terminal database, these two terminals (TE1, TE2) are assigned to the same group. Can be grouped into Since the representative terminal (TE2) is detected at this time, the server transmits a representative terminal detection notification indicating that the representative terminal is found to the terminal TE1. This notification includes the device ID and IP address of the representative terminal TE2.

The terminal TE1 knows from the representative terminal detection notification from the server that there is a group (same base) to which the terminal TE1 can belong and that the representative terminal of the group is the terminal TE2.
After that, the terminal TE1 transmits a streaming request to the representative terminal TE2 so that the communication state can be received by streaming.
The terminal TE2 that has received the streaming request distributes the received image RD2 sent from the server by streaming distribution to the terminal TE1.
The terminal TE1 receives video, audio, etc. sent from the representative terminal TE2 by streaming delivery, and displays the received image RD1 on the display.
In this case, the terminal TE1 does not receive distribution of video or the like directly from the server, and acquires shared information such as video and audio from the terminal TE2 by one-way communication (streaming reception) by streaming distribution. Thereby, the terminal TE1 can receive the shared information even when the terminal TE1 does not communicate directly with the server.

In FIG. 26B, it is assumed that the server performs the representative terminal detection process, but there is no other terminal that can be grouped with the terminal TE1, and the representative terminal cannot be detected. At this time, an undetected notification of the representative terminal is transmitted from the server to the terminal TE1.
The terminal TE1 that has received this non-detection notification switches itself to the representative terminal mode, and transmits a terminal state notification indicating that the terminal TE1 has shifted to the representative terminal mode to the server. The terminal TE1 is in a state where it can receive streaming delivery from the server.
Upon receiving this notification, the server detects that the operation mode of the terminal TE1 has been changed to the representative terminal mode, and sets the representative terminal flag of the terminal TE1 to “ON”. Thereafter, the server performs streaming delivery to the terminal TE1.
Thereby, the terminal TE1 receives shared information such as video and audio by streaming delivery from the server as a single representative terminal that does not belong to the group.

  The communication sequence in FIG. 26 (a) corresponds to, for example, a communication sequence performed between the terminal TE4 and the server in FIG. 6, and the communication sequence in FIG. 26 (b) is performed between the terminal TE2 and the server in FIG. This corresponds to a communication sequence performed between the two.

FIG. 27 shows a schematic explanatory diagram of an embodiment of a communication sequence regarding speech detection control when a speech request is made at a terminal.
FIG. 27A shows a communication sequence when the user of the terminal TE1 who is in the participating terminal mode and has not yet requested a speech makes a speech request.
FIG. 27B shows a case where a user of another terminal TE2 makes a new speech request in a state where the terminal (existing speech terminal) TE1 that is in the speech requesting terminal mode that currently has the right to speak is present. A communication sequence is shown.
In FIG. 27A, it is assumed that the terminal TE1 is currently in a participating terminal mode and cannot receive shared information such as video directly from the server. At this time, shared information cannot be directly received from the server, but information such as a speech request and a terminal status notification can be transmitted and received between the terminal TE1 and the server via a predetermined communication protocol.

In FIG. 27A, it is assumed that the user of the terminal TE1 inputs a video by an operation such as an input operation, a voice input, or a hand raising meaning a speech request.
At this time, the terminal TE1 detects that a speech request has been made, and transmits data representing the speech request to the server. When the server receives this speech request, the server confirms whether there is a terminal that currently has the right to speak based on the speech request terminal flag in the terminal database (existing speech terminal detection).
Here, if an existing speech terminal whose speech request terminal flag is “ON” is detected, the server transmits a request for switching to a participating terminal to the existing speech terminal (TE2). Receiving this switching request, the existing speech terminal (TE2) switches its own operation mode from the speech request terminal mode to the participation terminal mode.
Thereafter, the existing speech terminal (TE2) changes the communication state with the server from the bidirectional communication state to a one-way communication state in which streaming distribution is received unidirectionally from a representative terminal at the server or the same base (same group). To do.

Further, the server transmits a notification of permission to switch to the speech request terminal mode to the terminal TE1 that has requested speech.
The terminal TE1 that has received this switching permission notification switches its operation mode from the participating terminal mode to the speech requesting terminal mode.
Also, the server changes the terminal TE1 to “ON” and the terminal TE2 to “OFF” for the message request terminal flag in the terminal database.
Thereafter, the server and the terminal TE1 switch the communication state between the two to a state in which shared information such as video can be bidirectionally communicated. As a result, the user's own video and audio can be transmitted from the terminal TE1 to the server, and the shared information including the video of the terminal TE1 can be transmitted from the server to the terminal TE1 according to a predetermined protocol. .
The speech information of the user of the terminal TE1 who has obtained the right to speak is transmitted as shared information to all other terminals connected to the server and participating in the conference.

In FIG. 27B, it is assumed that the terminal TE1 is an existing speech terminal and is in a two-way communication state with the server.
In FIG. 27B, since the terminal TE1 is already an existing speech terminal, the speech request terminal flag of the terminal TE1 in the terminal database is “ON”.
At this time, it is assumed that a user of another terminal TE2 performs an input operation meaning a speech request. In the other terminal TE2, as in the case shown in FIG. 27 (a), a speech request is transmitted to the server, and the server that has received the speech request performs a process of detecting an existing speech terminal.
Since the server detects from the terminal database that the terminal TE1 is an existing speech terminal, the server transmits a request for switching to a participating terminal to the existing speech terminal TE1.

The terminal TE1 that has received this switching request switches its own operation mode to the participating terminal mode, and transmits a terminal state notification indicating that its own terminal has been switched to the participating terminal mode to the server.
Thereafter, since the terminal TE1 enters the participation terminal mode, the terminal TE1 enters a one-way communication state in which streaming distribution is received from a representative terminal at the server or the same base (same group).
On the other hand, the server that has received the terminal state notification (participating terminal mode) from the terminal TE1 switches the communication state with the terminal TE1 to, for example, a one-way communication state in which streaming distribution is performed from the server. In addition, the speech request terminal flag of the terminal TE1 is set to “OFF”. As a result, the communication state between the terminal TE1 and the server becomes a one-way communication state in which video or the like is streamed from the server to the terminal TE1.
The communication sequence related to the speech detection control of the present invention has been described above.

The terminal that has made a speech request enters a two-way communication state with the server, but the number of terminals that can enter the speech request terminal mode is limited to only one of all terminals connected to the server and managed. Thus, the number of terminals capable of bidirectional communication with the server can be reduced.
Therefore, by limiting the terminals that can communicate bidirectionally with the server, traffic flowing through the network can be reduced compared to when all terminals communicate bidirectionally with the server. Reduces failures and server processing load.

<Description of mode switching processing in information terminal>
FIG. 21 shows a flowchart of an embodiment of the mode switching process in the terminal of the present invention.
In step S1, it is assumed that the initial state of the terminal is set to the participating terminal mode. As described above, the participating terminal mode is a state in which streaming distribution of shared information including video and audio can be received.
In step S2, the terminal state notification unit 81 of the terminal notifies the server of the current operation state (operation mode) of the terminal (terminal state notification process).
Since it is currently a participating terminal mode, information indicating that it is in the participating terminal mode is transmitted to the server together with the device ID of the terminal.
In step S3, the representative terminal detection request unit 83 transmits a representative terminal detection request to the server.
In step S4, it is checked whether or not information (representative terminal detection notification) indicating that a representative terminal sent from the server has been received has been received.
If received, the process proceeds to step S5, and if not received or if information indicating that the representative terminal has not been detected (not-detected notification) is received, the process proceeds to step S9.

When the information indicating the detection (detection notification) is received, there is a representative terminal belonging to the same group as this terminal, and the IP address and the like of the representative terminal can be known. In step S5, the streaming request unit 86 The information requesting streaming delivery is transmitted to the representative terminal (streaming request).
The representative terminal can be identified by the device ID, IP address, etc. of the representative terminal included in the detection notification from the server.
In step S6, the streaming receiving unit 87 receives streaming delivery sent from the representative terminal (streaming reception).
In step S7, a representative terminal detection request is transmitted to the server again, as in step S3. This process is for confirming whether the representative terminal is disconnected from the communication line for some reason or is not turned off.
In step S8, it is checked whether a representative terminal detection notification is received from the server. If a detection notification indicating that the same representative terminal is present is received, the process ends.

On the other hand, if no detection notification has been received, or if a non-detection notification from the representative terminal has been received, the process proceeds to step S9.
In step S9, since there is no representative terminal, the terminal mode switching unit 82 switches the operation mode from the participating terminal mode to the representative terminal mode in order to make the terminal itself a representative terminal. Thereby, it switches to the mode which can receive streaming delivery from a server and can perform streaming delivery with respect to another terminal.
In step S10, the terminal state notification unit 81 transmits to the server a terminal state notification indicating that its own operation mode has been switched to the representative terminal mode.
In step S11, the streaming receiving unit 87 receives streaming distribution from the server (streaming reception).
The above is one embodiment of the mode switching process in the terminal. The process in each step corresponds to that shown in the communication sequence of FIG.

FIG. 22 shows a flowchart of the streaming distribution processing of the terminal that has become the representative terminal of the present invention.
In step S21, it is checked whether a streaming distribution request has been received from another terminal. If received, the process proceeds to step S22, where the streaming distribution unit 88 performs streaming distribution of the shared information to the requesting terminal.

<Description of terminal status detection in the server>
FIG. 23 shows a flowchart of the server terminal state detection control process.
In step S101, the terminal state determination unit 162 checks whether or not terminal state information has been received. If received, the process proceeds to step S102, and if not, step S101 is looped.
In step S102, the terminal state determination unit 162 checks whether or not the received terminal state is a participating terminal mode. If it is the participation terminal mode, the process proceeds to step S103, and if not, the process ends.

In step S103, the representative terminal detection unit 164 performs a representative terminal detection process.
Here, first, it is checked whether or not a representative terminal detection request has been transmitted from the terminal. When such a request is received, the terminal database 172 is checked to check the presence or absence of the representative terminal. It is checked whether there is a terminal in the terminal database having the same base ID as the base ID of the terminal that has transmitted the representative terminal detection request. If there is, it is checked whether the representative terminal flag 177 of the terminal is “ON”.

If the representative terminal flag is “ON”, there may be other terminals that belong to the same base (same group) as the terminal that transmitted the representative terminal detection request and that are representative terminals. Detected.
On the other hand, if there are no other terminals having the same base ID, it is detected that there are no terminals belonging to the same group and there is no representative terminal.

In step S104, it is checked whether a representative terminal is detected. If detected, the process proceeds to step S105, and if not, the process proceeds to step S106.
In step S105, a detection notification including the detected device ID and IP address of the representative terminal is transmitted to the terminal that has made the detection request.
On the other hand, in step S106, information indicating that the representative terminal has not been detected (not-detected notification) is transmitted to the terminal that has made the detection request.
The above server processing also corresponds to that shown in the communication sequence of FIG.

<Explanation such as speech detection on the terminal>
FIG. 24 shows a flowchart of speech detection and request processing of the terminal of the present invention.
In step S31, it is assumed that the initial state of the operation mode of the terminal is the participation terminal mode.
In step S32, the speech detection unit 84 checks whether it has detected that the user of the terminal has performed an input operation, speech input, or video input (raising hand operation) meaning speech (speech detection). processing).
If speech is detected, the process proceeds to step S33, and if not, the process ends.
In step S33, the speech request unit 85 transmits information (speech request information) indicating that a speech request has been made to the server (speech request processing).
In the participation terminal mode, the self-portrait or the like cannot be transmitted to the server, but this request information is transmitted to the server by a predetermined communication protocol.

In step S34, when a notification of permission to switch to the speech request terminal mode is received from the server, the terminal mode switching unit 82 switches its own operation mode from the participating terminal mode to the speech request terminal mode. Thereby, this terminal becomes an existing speech terminal.
In step S35, the two-way communication control unit 89 changes the communication state to a state in which two-way communication of audio and video with the server is possible. Thereafter, a self-portrait or input sound captured by the terminal camera can be transmitted from the terminal to the server.
In addition, shared information such as a received image sent from the server can be directly received.
In step S36, it is checked whether or not a status switching request to the participating terminal mode is received from the server. This is a process corresponding to a case where, for example, another terminal requests to speak in a state where the terminal is an existing speaking terminal.

When this switching request is received, the process proceeds to step S37, and when it is not received, the process ends.
In step S37, the terminal mode switching unit 82 switches the operation mode to the participating terminal mode. The terminal in the participation terminal mode enters a one-way communication state in which streaming distribution is received from the server or, if there is a representative terminal, from the representative terminal.
In step S38, the terminal state notification unit 81 notifies the server that the operation mode of the terminal has been changed to the participating terminal mode.
In step S39, the streaming receiver 87 receives shared information such as audio and video sent from the server by streaming delivery.

<Explanation of Talking Terminal Detection in Server>
FIG. 25 shows a flowchart of the speaking terminal detection control process in the server of the present invention.
In step S121, the terminal state determination unit 162 checks whether terminal state information has been received from the terminal. If received, the process proceeds to step S122, and if not, step S121 is looped.
In step S122, if the received terminal state information is a message request that means a shift to the message request terminal mode, the process proceeds to step S123, and if not, the process ends.
A terminal that has transmitted a speech request is referred to as a speech request terminal.
In step S123, the existing speech terminal detection unit 165 performs an existing speech terminal detection process. Here, it is checked whether or not there is a terminal that is “ON” in the speech request terminal flag 178 of the terminal database 172. If there is one in which the request terminal flag 178 is “ON”, it means that there is already a terminal having the right to speak, and an existing speaking terminal has been detected.
On the other hand, if all of the speech request terminal flags 178 are “OFF”, it means that there is no existing speech terminal and has not been detected.

In step S124, it is checked whether or not an existing speech terminal has been detected. If it is detected, the process proceeds to step S127.
In step S125, a notification of permission to switch to the speech request terminal mode is transmitted to the speech request terminal that has requested the speech. Here, the speech request terminal flag of the speech request terminal in the terminal database is set to “ON”.
In step S126, the two-way communication control unit 167 sets the communication state to a state where voice and video can be two-way communicated with the speech request terminal. Thereafter, information such as video is transmitted and received between the server and the speech requesting terminal.

On the other hand, in step S127, a setting is made to shift the existing speech terminal to the participating terminal. Here, the speech request terminal flag 178 of the existing speech terminal in the terminal database is changed from “ON” to “OFF”.
In step S128, the communication method switching unit 163 transmits a request for switching to the participating terminal mode to the existing speech terminal.
In step S129, in response to the switching request, it is checked whether or not a terminal state notification indicating that the terminal has changed to the participating terminal mode has been received from a terminal that has been an existing speaking terminal. If received, the process proceeds to step S130. If not, step S129 is looped.

In step S130, the communication method switching unit 163 performs streaming delivery to the terminal switched to the participating terminal mode.
In step S131, a notification of permission to switch to the speech request terminal mode is transmitted to the terminal that has newly requested speech in step S122. Here, the speech request terminal flag of the new speech request terminal is set to “ON”.
In step S132, the bidirectional communication control unit 167 sets the communication state with the terminal that has newly become the speech requesting terminal to the bidirectional communication state. As a result, the terminals capable of bidirectional communication with the server are changed, but the number of terminals in the bidirectional communication state is not changed.

  Therefore, even if the requesting terminal is changed to another terminal, the number of terminals that communicate bidirectionally with the server is the same, so there is no significant increase or decrease in traffic flowing through the network, and transmission delays or communication failures may occur. There is no increase, and there is no significant change in the processing load on the server.

  Moreover, in this invention, there is no restriction | limiting in the terminal which can request a speech in principle, What is necessary is just a terminal which can be connected to a server and can communicate. In addition, the timing for requesting speech may be possible at any time in principle. If there is a terminal that has newly requested a speech, a terminal capable of bidirectional communication such as video with the server is immediately switched to the terminal that has requested the new speech.

Also, if there is an existing speech terminal, switching the existing speech terminal to the participation terminal mode, for example, cancels bidirectional communication with the server and enters a one-way communication state from the server to the terminal. The self-portrait cannot be transmitted from the terminal that was the terminal.
However, since information such as the self-portrait and input voice from the new speaking terminal is transmitted to all terminals as shared information, the information of the speaking person can be transmitted to all participants, and a smooth conference can be proceeded. Become.
In addition, by detecting the request to speak and limiting the number of terminals that perform two-way communication, the traffic flowing through the network can also be reduced compared to when all terminals communicate with the server. Reduction of transmission delay, occurrence of communication failure, and server processing load can be reduced.

<Second embodiment: description of communication state between server terminals>
FIG. 13 to FIG. 20 are explanatory diagrams of the communication state between the server and the terminal according to the second embodiment of the present invention.
In the first embodiment, when streaming delivery is performed from a representative terminal to another terminal, the terminals that receive the streaming delivery are limited to terminals belonging to the same group (same base).
For example, in FIG. 7, the partner terminal TE4 that performs streaming delivery from the representative terminal TE3 is a terminal that belongs to the same base.
In this second embodiment, grouping is performed for a plurality of terminals belonging to the same base. However, as a terminal to which streaming distribution is performed from a terminal that has become a representative terminal by grouping, terminals belonging to the same group are assigned. In addition, terminals that do not belong to the group and exist as a single terminal that does not have terminals that can be grouped are also targeted.

FIG. 13 is an explanatory diagram of a communication state between server terminals (embodiment 2-1) of the second embodiment.
The communication state in FIG. 13 is the same as that shown in FIG. 4, and only the terminal TE1, which is the presentation terminal, is connected to the server.
In FIG. 14, explanatory drawing of the communication state between server terminals (Example 2-2) of 2nd Example is shown. The communication state shown in FIG. 14 is the same as that shown in FIG. 5, and the terminal TE3 is a representative terminal in the same base.
19 and 20 show the states of the three flags of the terminal database in each communication state of the second embodiment.
The state of the flag in Example 2-1 in FIG. 19A and the state of the flag in Example 2-2 in FIG. 19B are the same as those in FIGS. 11A and 11B, respectively.

FIG. 15 is an explanatory diagram of a communication state between server terminals (embodiment 2-3) of the second embodiment.
FIG. 15 corresponds to FIG. 6 of the first embodiment, but the point that the terminal TE2 does not become a representative terminal, is activated as a participating terminal, and receives streaming distribution from the terminal TE3 that has already been the representative terminal. Different.
In the case of FIG. 15, when the terminal TE2 is turned on and activated, it makes a representative terminal detection request to the server. However, if there is no representative terminal that can be grouped with the terminal TE2 even if the server searches for the representative terminal, the server is already a representative terminal among the terminals connected to the server, regardless of the grouping. Check if there is a terminal.
In FIG. 15, a terminal TE3 that is not the same base as the terminal TE2 but is already a representative terminal can be found.

Therefore, instead of notifying the terminal TE2 that the representative terminal has not been detected, the server transmits a representative terminal detection notification including information indicating that the representative terminal is the terminal TE3 to the terminal TE2. The terminal TE2 that has received the representative terminal detection notification switches to an operation mode in which streaming distribution can be received from the representative terminal TE3.
On the other hand, the server transmits, to the representative terminal TE3, information (streaming distribution request to a terminal outside the group) that requests that the terminal TE2 not belonging to the same base (the same group) perform streaming distribution. .
The terminal TE3 that has received the request information also performs streaming delivery to the terminal TE2 outside the requested group while continuing streaming delivery to the participating terminal TE4 in the same base.
That is, as shown in FIG. 15, after receiving the received image RD3 by streaming delivery from the server, the terminal TE3 performs streaming delivery of the received image RD4 to the terminal TE4, and also receives the received image RD4 to the terminal TE2. RD2 is streamed. Here, the three received images (RD2, RD3, RD4) are the same information as the shared information transmitted from the server.

In Example 2-3, since the terminal TE2 is not a representative terminal but a participating terminal, the representative terminal flag of the terminal TE2 remains “OFF” as illustrated in FIG. 19C.
In the case of FIG. 15, since the terminal TE3 has two counterpart terminals for streaming delivery, the load on the delivery process increases.
However, the server only needs to distribute the streaming to the terminal TE3, and does not have to perform the streaming distribution to the terminal TE2 as shown in FIG. 6, so the load on the streaming distribution process may be small.
That is, in the case of FIG. 15 of the second embodiment, the processing load on the server can be reduced as compared with the first embodiment.

FIG. 16 is an explanatory diagram of a communication state between server terminals (embodiment 2-4) of the second embodiment.
FIG. 16 shows a case where the user of terminal TE2 makes a speech request in the communication state of FIG.
At this time, the terminal TE2 becomes a speaking terminal, and the two terminals (TE1, TE2) together with the presentation terminal TE1 are in a state capable of bidirectional communication with the server.
Here, as shown in FIG. 20D, the speech request terminal flag of the terminal TE2 is set to “ON”.
This communication state is almost the same as that in FIG. 7 except that the terminal TE2 is not a representative terminal. In this case, since there are two terminals for two-way communication with the server, the traffic increases, but the shared information including the self-portraits (SD1, SD2) of the two terminals (TE1, TE2) capable of two-way communication. Will be sent to all terminals.

In FIG. 17, explanatory drawing of the communication state between server terminals (Example 2-5) of 2nd Example is shown.
FIG. 17 shows a case where the user of the terminal TE3 makes a speech request in the communication state of FIG. At this time, the speaking terminal is changed from TE2 to TE3.
Therefore, bidirectional communication is performed between the terminal TE3 and the server.
In addition, the terminal TE2 returns to the original participating terminal, cancels the bidirectional communication state with the server, and receives streaming delivery from the terminal TE3 as in FIG.
Also, as shown in FIG. 20 (e), in the speech request terminal flag, the terminal TE2 is changed to “OFF” and the terminal TE3 is changed to “ON”.
In FIG. 16, there are two terminals that perform two-way communication with the server and one terminal that receives streaming distribution from the server. However, in FIG. 17, there are only two terminals that perform two-way communication with the server, and the server streaming is performed. The load on distribution is eliminated and the load is reduced.

FIG. 18 shows an explanatory diagram of the communication state between server terminals (Example 2-6) of the second example.
FIG. 18 shows a case where the user of terminal TE4 makes a speech request in the communication state of FIG. At this time, the speaking terminal is changed from TE3 to TE4, and bidirectional communication is performed between the terminal TE4 and the server.
On the other hand, the terminal TE3 returns to the original representative terminal and receives streaming distribution from the server. Further, the streaming distribution from the terminal TE3 to the terminal TE2 is continued as it is.
Here, as shown in FIG. 20F, the speech request terminal flag is changed to “OFF” for the terminal TE3 and “ON” for the terminal TE4.
In this case, the communication state between the server and the terminal is two terminals for two-way communication and one terminal for receiving streaming distribution, and the server is in a state where a load similar to that in FIG. 16 is applied.
Also, the terminal TE2 has no other terminals that can be grouped and is not a representative terminal. However, since the terminal TE2 receives streaming distribution from another representative terminal TE3 belonging to one group, the processing load on the server is reduced. be able to.

  As described above, in the case of the second embodiment as well, a message request terminal that can perform two-way communication with a server by streaming distribution from a server or a representative terminal of the same base (same group) and detection of a message request by grouping. Since it is limited to one, all terminals can reduce transmission delay and communication failure compared to the case where two-way communication with a server is performed, and a single terminal having no terminals that can be grouped On the other hand, since streaming delivery is performed from another representative terminal, it is possible to reduce the load required to deliver the shared information of the server.

In reality, the number of terminals connected to the server may be 5 or more, and the operation mode of the terminal changes for each terminal. Therefore, information transmitted from the terminal to the server, server In addition to FIGS. 4 to 9, FIGS. 13 to 18, and the like, various patterns can be considered for the composite image transmitted from the terminal to the terminal.
Also, if all of the terminals connected to the server are capable of bidirectional communication, the amount of data transmitted between the server and each terminal is large, and the amount of data to be processed by the server is large. Although there is a high possibility that a communication failure will occur, if the communication state with the server is changed by detecting the grouping of terminals and the request to speak at the terminal as in the present invention, the connection to the server is established. In addition, the substantial number of terminals that communicate large amounts of information can be reduced, the transmission delay can be reduced and the occurrence of communication failures can be reduced, and the load on the server can be reduced.

1 server 2 information terminal (terminal)
DESCRIPTION OF SYMBOLS 11 Input part 12 Keyboard 13 Mouse 14 Tablet 15 Human detection sensor 16 Input processing part 21 Video input part 22 Camera 23 Video processing part 31 Display part 32 Display 33 Display processing part 41 Audio | voice input part 42 Microphone 43 Input audio | voice processing part 45 Audio | voice output Unit 46 speaker 47 output audio processing unit 51 communication unit 52 communication encryption unit 53 communication decoding unit 55 authentication unit 61 storage unit 62 recognition database 63 input video 64 input audio 71 signal processing unit 72 video encoding unit 73 video decoding unit 74 audio Encoding unit 75 Speech decoding unit 80 Control unit 81 Terminal state notification unit 82 Terminal mode switching unit 83 Representative terminal detection request unit 84 Statement detection unit 85 Statement request unit 86 Streaming request unit 87 Streaming reception unit 88 Streaming distribution unit 89 Bidirectional Communication control unit 01 input unit 111 display unit 121 communication unit 131 MCU unit 151 signal processing unit 161 control unit 162 terminal state determination unit 163 communication method switching unit 164 representative terminal detection unit 165 existing speech terminal detection unit 166 streaming distribution unit 167 bidirectional communication control unit 171 Storage unit 172 Terminal database 173 Device ID
174 Base ID
175 IP address 176 Announcement terminal flag 177 Representative terminal flag 178 Speak request terminal flag 179 User ID
180 E-mail address 181 User database

Claims (11)

A server and a plurality of information terminals are connected via a network,
Transmitting the terminal information input in the information terminal to a server;
The server generates shared information using terminal information transmitted from each information terminal,
An information sharing system for distributing the shared information to the information terminal,
The plurality of information terminals are grouped into one or a plurality of terminal groups based on predetermined setting conditions in advance, and one terminal group includes one representative terminal that can directly receive the shared information from the server And a participating terminal that receives distribution of the shared information from the representative terminal,
Each information terminal, a speech detection unit that detects that the user has made a speech request,
A speech request unit that transmits information indicating that the speech request has been made to the server;
A terminal mode switching unit that changes the operating state of the information terminal according to the notification from the server;
When the server receives the message request information from the information terminal, the communication state between the information terminal and the server can be directly transmitted / received between the terminal information and the shared information. A two-way communication control unit to switch to
An information sharing system, wherein the server manages such that only one information terminal that is in a two-way communication state with the server and is allowed to speak by the user exists at the same time. If there is a first information terminal that has made a request for speech and the communication state with the server is a two-way communication state, the speech detection unit of the second information terminal makes a new speech request. , The message request unit of the second information terminal transmits information indicating the new message request to the server,
When the server receives speech request information from the second information terminal, the bidirectional communication control unit cancels the bidirectional communication state with the first information terminal and makes a new speech request. The information sharing system according to claim 1, wherein the communication state between the second information terminal and the server is switched to a two-way communication state. When detection of a speech request by the user is performed at a speech detection unit of a plurality of information terminals almost simultaneously,
The bidirectional communication control unit of the server sets the communication state with the information terminal that has transmitted the speech request information received earliest among the speech request information transmitted from each of the plurality of information terminals, as the bidirectional communication state. Or switch to
The communication state with the information terminal which is judged that the user's speech request is detected at the earliest time is confirmed, and the two-way speech request information is switched to the bidirectional communication state. 2. Information sharing system. The information terminal further includes an input unit that inputs an input signal by a user operation, a video input unit that inputs video captured by a camera, and an audio input unit that inputs audio from a microphone.
When the speech detection unit detects any one or more of an input signal meaning the speech request, an input voice meaning the speech request, or an input video meaning the speech request, the user makes a speech request The information sharing system according to claim 1, wherein the information sharing system is determined. The setting condition for grouping is connection base information where each information terminal is arranged,
The information sharing system according to claim 1, wherein the server includes a storage unit that stores the connection base information for each information terminal.   6. The information sharing system according to claim 1, wherein representative terminals belonging to the one terminal group receive the shared information by streaming distribution from a server.   7. The information sharing according to claim 1, wherein the representative terminal belonging to the one terminal group performs streaming distribution of the shared information received from the server to the participating terminals belonging to the terminal group. system.   8. The information sharing system according to claim 7, wherein the representative terminal belonging to the one terminal group performs streaming distribution of the shared information received from the server to other information terminals not belonging to the terminal group.   When the grouped one terminal group is composed of a single information terminal, the information terminal itself is a representative terminal, and the terminal group is a terminal group that does not include a participating terminal. The information sharing system according to claim 1. A server and a plurality of information terminals are connected via a network,
Transmitting the terminal information input in the information terminal to a server;
The server generates shared information using terminal information transmitted from each information terminal,
An information sharing method of an information sharing system for delivering the shared information to the information terminal,
The plurality of information terminals are grouped into one or a plurality of terminal groups based on predetermined setting conditions in advance, and one terminal group includes one representative terminal that can directly receive the shared information from the server , Including a participating terminal that receives the distribution of the shared information from the representative terminal,
A speech detection step in which the information terminal detects that the user has requested speech;
A speech requesting step of transmitting information indicating that the speech request has been made to a server;
When the server receives the speech request information from the information terminal, the communication state between the information terminal and the server is changed to a bidirectional communication state in which the terminal information and the shared information can be directly transmitted and received. A communication control step for managing the information terminal so that there is only one information terminal at a time, which is in a two-way communication state with the server and allowed to speak by the user;
An information sharing method comprising: a terminal mode switching step of changing a communication state of an information terminal that has detected a speech request to a bidirectional communication state based on a notification from the server.   A program of an information sharing system for causing a computer to execute each step of the information sharing method according to claim 10.

Images