JP2011114699A - Information sharing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To attain decrease in information transmission delay and communication failure occurrence and to attain reduction in server load by decreasing the practical number of information terminals connected to a server that distributes shared information. <P>SOLUTION: Each of information terminals includes: an idle state setting part for determining whether the information terminal is in an idle state where the information terminal may not be used based on state information indicating that a user is in a usage state; a representative terminal detection request part for transmitting a detection request requesting detection of a representative terminal within the same base to a server when it is determined that the information terminal is in the idle state, and for receiving information about the representative terminal from the server; and a streaming receiving part for receiving streaming information which is distributed from the server or from the representative terminal when it is determined that the information terminal is in the idle state. The server includes a representative terminal detection part for detecting the representative terminal belonging to the same base as a first information terminal when the detection request of the representative terminal is received from the first information terminal, and transmitting information about the representative terminal to the first information terminal. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an information sharing system, and more particularly, to an information sharing system that transmits information such as common video and audio between a plurality of information terminals connected via a network.

Conventionally, a voice communication or video conference is performed between a plurality of information terminals in a remote place through a communication system that transmits information such as video and voice between a plurality of information terminals via a network, and a dedicated server. A communication system is used.
For example, in a conventional video conference system, a multi-point connection unit (MCU: Multi-point Control Unit) and a server having an MCU function include an information terminal installed at each point and an information terminal for each participant. Connected via a network such as the Internet, a LAN, or a dedicated line, information such as video, audio, and documents input for each information terminal is accumulated in the MCU, and the MCU decodes (decodes) and mixes each information (decoding) A series of processing such as synthesis) and encoding (encoding) is performed and distributed to each information terminal as common information.
In general, it is assumed that participants in a conference using each information terminal perform a conference while watching a common video and listening to a common voice.

  Further, without the user himself / herself turning off the main power supply of the remote conference device (information terminal), the motion vector value is obtained from the motion detection mechanism of the video encoding mechanism to determine the user's leaving, A remote conference system with an automatic line disconnection function that can automatically and safely disconnect a line and a power supply has been proposed (see Patent Document 1).

  Further, in a communication system having a presence service that displays information on presence or absence of a user (presence information) on the communication terminal of another user, the user may forget to set or change his / her presence information. In consideration, the captured image taken around the user's communication terminal or the voice acquired around the communication terminal is transmitted as presence information to the other user's communication terminal, and the presence status of the other user is accurately grasped. Presence information processing terminal (see Patent Document 2) and presence information set by the user, presence information of the user is determined based on sound collected by the microphone, as well as presence information. Has been proposed (see Patent Document 3).

Further, in a communication terminal device that transmits / receives data to / from a server device, when presence information indicating the communication terminal device and user status information matches a specific condition, the operation of the terminal device is changed and the communication data is processed. There has been proposed a communication system that can reduce the load on the server side by changing the operation and processing the data on the terminal device side based on presence information (see Patent Document 4). ).
Further, in Patent Document 4, the server device collects terminal group information that meets a set specific condition (for example, a terminal in a specific area) using presence information transmitted from a communication terminal device. Thus, a communication system that automatically creates a new group of terminal devices is disclosed.

JP-A-8-336122 JP 2004-214934 A JP 2007-129411 A International Publication WO2006 / 054778

However, in a multipoint conference system using a conventional MCU, as the number of information terminals simultaneously connected to the MCU increases, traffic transmitted and received on the communication line between the MCU and each information terminal increases. A communication failure such as a communication error such as a delay in information transmission or a noise is likely to occur, and a smooth conference may not be able to proceed.
Furthermore, in the MCU, as the number of connected information terminals increases, the amount of information to be processed such as encoding (encoding) and decoding (decoding) increases, so the load on the MCU increases, This causes a delay in information transmission to each information terminal.

In addition, in the MCU, when centrally managing the operation management of each terminal and the connection and disconnection processing of communication lines based on the information indicating presence or absence sent from the information terminal, the user In some cases, the communication line is disconnected by the MCU, or the power supply to the information terminal of the user is stopped.
In such a case, in order to re-join the conference, the user needs to redo a series of operations such as line connection processing and power-on, which is inconvenient for the user and a heavy operation burden.

In addition, even in the systems described in Patent Documents 1, 2, and 3, even if the operation control of the terminal is automatically performed or the presence status can be accurately grasped, the communication line and power supply that are not intended for the user In some cases, the user needs to perform a restart operation or the like.
In addition, the techniques disclosed in any of the patent documents are not intended to alleviate problems such as transmission delay and communication failure when the number of information terminals connected to the network increases. No significant mitigation measures are taken into account.

  Therefore, the present invention has been made in consideration of the above-described circumstances, and suppresses the number of information terminals that substantially perform information communication with the server among a plurality of information terminals connected to the server. To reduce the delay of information transmission and the occurrence of communication failures such as communication errors, reduce the server load by reducing the amount of information processed by the server, and perform step-by-step control of the operating state of the information terminal It is an object of the present invention to provide an information sharing system that can reduce the operation burden on the user on the information terminal side. Here, the server refers to a device having at least an MCU function among the video conference control functions.

  In the present invention, a server and a plurality of information terminals are connected via a network, information input at the information terminal is transmitted to the server, and the server is shared using information transmitted from each information terminal. An information sharing system for generating information and distributing the shared information to the information terminal, wherein the information terminal generates state information indicating that the user is using the information terminal; An idle state determination unit that determines that the information terminal is in an idle state that may not be used based on the state information generated by the information input unit, and an idle state that is determined In addition, a terminal mode switching unit that changes the operation state in stages corresponding to the current operation state of the information terminal, and a terminal state that transmits the current operation state of the information terminal to the server A representative terminal detection request unit that transmits a detection request for requesting detection of a representative terminal in the same base to the server and receives information on the representative terminal from the server when it is determined that the server is in an idle state. And a streaming receiver that receives streaming information distributed from a server or the representative terminal when it is determined that the server is in an idle state, and the server is based on an operating state transmitted from the information terminal. A communication method switching unit that switches a communication method with the information terminal, and when a request for detecting a representative terminal is received from the first information terminal, from among information terminals belonging to the same base as the first information terminal A representative terminal detecting unit for detecting a representative terminal and transmitting information of the representative terminal to the first information terminal; and a streamer for transmitting streaming information to the information terminal. And a distribution unit according to the operation status of the information terminal, there is provided an information sharing system, characterized in that to change the communication status between the server and each information terminal for each information terminal.

According to this, since the information terminal determined to be in the idle state is in a one-way communication state for receiving streaming information distributed from the server or the representative terminal in the same base, the information terminal between the server and the information terminal The communication state can be set to a state in which the amount of information transmission is small, and further, by reducing the amount of information transmission, it is possible to reduce transmission delay and occurrence of communication failure, thereby reducing the load on the server.
In addition, since the operation state of the information terminal is changed in stages, for example, the operation for returning from the operation state after the change to the operation state before the change can be made relatively easy. The operational burden can be reduced and the restoration can be performed in a short time.

Further, when the representative terminal detection unit can detect the representative terminal, the streaming receiving unit of the first information terminal receives streaming information from the detected representative terminal.
According to this, since the first information terminal in the idle state receives the streaming information from the representative terminal, the shared information such as video is not directly transmitted from the server to the first information terminal, and more transmission is performed. Reduction of delay and occurrence of communication failure can be reduced, and the load on the server can be further reduced.

In addition, when the representative terminal detection unit cannot detect a representative terminal belonging to the same base as the first information terminal, the representative terminal detection unit transmits an undetected notification of the representative terminal to the first information terminal, and When the information terminal receives the non-detection notification, the terminal state notification unit of the first information terminal notifies the server that it is a representative terminal, and then the streaming reception unit receives from the server, Streaming information is received.
According to this, since the first information terminal in the idle state receives streaming information from the server unidirectionally, it is possible to reduce the load on the server, reduce transmission delay, and reduce communication failure. .

In the present invention, the representative terminal in the same base is selected from information terminals set in advance as information terminals belonging to the same group as the first information terminal.
The information terminals belonging to the same group may be any of information terminals having the same base ID, information terminals connected to the same router and belonging to a network under the router, and information terminals in the same domain.
Furthermore, the representative terminal can be an information terminal that has the same base ID as the first information terminal and is not in an idle state.

  The operating state of the information terminal is a first mode in which bidirectional communication with the server is performed, a second mode in which information including audio and video is received from the server and one-way communication is performed, and the main function of the information terminal is stopped. Each time the communication with the server is stopped, and the fourth mode in which the information terminal is turned off. The operation state is changed stepwise in the order from the first mode to the fourth mode.

Here, in the first mode, bidirectional communication of video and audio is performed between the server and the information terminal, whereas in the second mode, one-way transmission of video and audio is performed from the server to the information terminal. Since this is done, the amount of information transmitted is less than in the first mode. In the third and fourth modes, since communication between the server and the information terminal is interrupted, no information is transmitted between the server and the information terminal. Therefore, if the operation state of the information terminal is changed stepwise from the first mode to the second mode, the third mode, and the fourth mode by detecting the idle state, the amount of information transmitted between the information terminal and the server is increased. Since the operation state can be changed in the direction of decreasing the number of information terminals, the substantial number of information terminals connected to the server can be reduced. Thereby, not only the processing load such as video and audio encoding and decoding in the server and the information terminal can be reduced, but also the transmission delay and the occurrence of communication failure can be reduced as the communication transmission amount is reduced.
In addition, every time the idle state is detected, the operation state of the information terminal is changed stepwise in order from the first mode to the fourth mode. Therefore, when the return to the operation state before the change is detected, Compared to when the information terminal is suddenly turned off, the operation can be performed in a shorter time and with easier operation, and the burden on the user can be reduced.

The information input unit includes an input unit that inputs an input signal by a user operation, a video input unit that inputs video captured by a camera, and an audio input unit that inputs audio from a microphone. The state information is generated using an input signal, an input video, and an input sound.
Further, the idle state determination unit continues during a predetermined set time, and there is no input signal input, no input audio input, or no predetermined change in the input video. When the device is detected, it is determined that the information terminal is in an idle state that may not be used.

Here, as the status information indicating that the user is using the information terminal, the presence / absence of the user's operation, the presence / absence of the user's voice input, and the presence / absence of the change of the video captured by the user are used. .
For example, when the user moves the mouse, status information indicating that there is an input signal is generated based on the input signal indicating that the mouse has been moved.
Then, it is determined that there is a possibility that the user is not using the information terminal by comprehensively using the three types of information of the user's operation input, the user's voice input, and the video change. The determination of absence can be made more reliable.

Further, a storage unit storing a user's voice feature data, a recognition database including face image feature data indicating the user's facial features, voice recognition using the input voice and voice feature data, and the input A person recognition unit that performs person recognition for identifying a user using image recognition using video and facial image feature data, and the idle state determination unit continues during a predetermined set time. When the person recognition unit fails in the person recognition unit, it is determined that the information terminal is in an idle state that may not be used.
Thereby, since the idle state specifying the user is determined, it is possible to more reliably determine the absence of the user.

Further, according to the present invention, a server and a plurality of information terminals are connected via a network, information input at the information terminal is transmitted to the server, and the server uses information transmitted from each information terminal. An information sharing method of an information sharing system that generates shared information and distributes the shared information to the information terminal, wherein the information terminal includes status information indicating that the user is using the information terminal. When it is determined that the information input step to be generated, the idle state determination step for determining whether or not the information terminal is in an idle state that may not be used, and the idle state based on the state information In response to the current operating state of the information terminal, the terminal mode switching step for gradually changing to the predetermined operating state and the current operating state of the information terminal to the server A terminal status notifying step for transmitting, a first information terminal determined to be in an idle state, a detection request transmitting step for requesting a server to detect a representative terminal in the same base, and the server including the representative terminal Detecting a representative terminal from among information terminals belonging to the same base as the first information terminal that has transmitted the detection request, and detecting information on the detected representative terminal Transmitting to one information terminal, using the information of the representative terminal received by the first information terminal, a distribution request transmitting step for requesting streaming distribution to the representative terminal, and receiving the representative terminal And a step of transmitting streaming information to the first information terminal based on the distribution request.
The present invention also provides an information sharing system program for causing a computer to execute each step of the information sharing method as described above.

According to the present invention, since the information terminal determined to be in the idle state is in a one-way communication state for receiving streaming information distributed from the server or the representative terminal in the same base, the information terminal is connected between the server and the information terminal. The communication state can be a communication state with a small amount of information transmission, the transmission delay can be reduced, the occurrence of communication failure can be reduced, and the load when the server generates shared information can be reduced. it can.
In addition, since the operating state of the information terminal is changed step by step, the user's operational burden for returning from the operating state after the change to the operating state before the change can be reduced, and the time required for the return can be shortened. it can.

It is a schematic block diagram of one Example of the information sharing system of this invention. It is a block diagram of the configuration of the first embodiment of the information terminal. It is a block diagram of the configuration of the first embodiment of the server. It is explanatory drawing of one Example of the communication state (active mode) between server terminals. It is explanatory drawing of one Example of a communication state (sleep mode) between server terminals. It is explanatory drawing of one Example of the communication state (logoff mode) between server terminals. It is explanatory drawing of one Example of the communication state (shutdown mode) between server terminals. It is explanatory drawing of one Example of the communication sequence between a server and a terminal. It is a flowchart of the mode transition process in the terminal of 1st Example. It is a flowchart of the idle state determination process in the terminal of 1st Example. It is a flowchart of the terminal state judgment control process in the server of 1st Example. It is explanatory drawing of one Example of the database stored in a memory | storage part. It is a structure block diagram of 2nd Example of an information terminal. It is a block diagram of the configuration of the second embodiment of the server. It is a flowchart of the mode transition control process in the server of 2nd Example. It is a flowchart of the idle state determination process in the server of 2nd Example. It is a flowchart of the terminal state control process in the terminal of 2nd Example. It is a structure block diagram of 3rd Example of an information terminal. It is a block diagram of the configuration of the third embodiment of the server. It is explanatory drawing of one Example of the terminal database used by 3rd Example. It is a flowchart of the grouping process in the terminal of 3rd Example. It is a flowchart of the streaming delivery process of the representative terminal of 3rd Example. It is a flowchart of the idle state determination process in the terminal of 3rd Example. It is a flowchart of the terminal state detection control process in the server of 3rd Example. It is explanatory drawing of the communication sequence between the server of 3rd Example, and a terminal. It is explanatory drawing of the communication state (active mode) between server terminals of 3rd Example. It is explanatory drawing of the communication state (sleep mode) between server terminals of 3rd Example. It is explanatory drawing of the communication state (sleep mode) between server terminals of 3rd Example. It is explanatory drawing of the communication state (sleep mode) between server terminals of 3rd Example. It is explanatory drawing of the communication state (sleep mode) between server terminals of 3rd Example.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, this invention is not limited by description of the following examples.
<Configuration of Information Sharing System of the Present Invention>
FIG. 1 shows a schematic configuration diagram of an embodiment of an information sharing system of the present invention.
Here, an information sharing system (hereinafter also referred to as a conference system) in which information such as video, audio, and documents is shared and a conference is performed between a plurality of information terminals using image audio and the like will be described.

In FIG. 1, the information sharing system mainly includes one server 1 and a plurality of information terminals (hereinafter also referred to as conference terminals or simply terminals) 2.
The server 1 and each information terminal 2 are connected via a network.
The server 1 is a device having at least a function equivalent to a multipoint connection device (MCU) among conference control functions also used in a conventional conference system, and mainly monitors the status of connected terminals, connection control, Various types of information communication control, video and audio encoding, video and audio decoding processing, user authentication, device authentication, communication encryption, and the like are performed.
The terminal 2 is a device installed in a room (such as an office or a conference room) in which a user who participates in a conference is present. The terminal 2 mainly recognizes the user himself / herself through video and audio, the operating state of the terminal, It performs input operation confirmation processing, processing and conversion of various input information, and communication control processing.

In the information terminal 2, information necessary for the conference is input, and the input information is transmitted to the server. As will be described later, the input information is mainly input signals from a keyboard and a mouse, input sound from a microphone, and input video from a camera. The input information may be transmitted to the server after being encoded. In addition, encryption processing may be performed in consideration of confidentiality and confidentiality in the communication path.
The server 1 generates shared information using information transmitted to the server as described above, and distributes the shared information to each information terminal.

The server 1 and the terminals 2 (TE1 to TEn) may be connected directly by, for example, a wired communication cable, or may be connected by a wireless communication method such as a so-called wireless LAN.
Further, the server 1 and the terminal 2 may be connected via various networks (LAN (N1), Internet (N2), dedicated communication network (N3), etc.).
When a user who uses a certain terminal (TE1) participates in the conference, the communication line between the terminal TE1 and the server 1 is connected and the link is established, and then the camera or microphone attached to the terminal TE1 is used. The user's image and sound, or documents and drawing data necessary for the conference are transmitted to the server 1.
At this time, a codec of a predetermined standard (for example, MPEG4, H.263, H.264 for video, and G. for audio for video transmission reduction and real-time (low delay) communication). 711, G.722, G.729, etc.), the processing (encoding) for compressing and encoding the amount of data to be transmitted is performed.

The server 1 decodes (decodes) information such as video and audio sent from the terminal TE1 and similar information sent from the other terminals 2 (TE2,... TEn), and then synthesizes (mixing). Perform the process. Information (combining information) obtained by performing the combining process becomes shared information. The shared information is distributed toward a plurality of terminals 2 that are currently connected.
In the server 1, before mixing a plurality of received videos, the video sent from each terminal 2 is decoded (decoded), the image size is adjusted (resized), and then one shared Processing for synthesizing a plurality of videos for distribution as information is performed.
For example, if four users are participating in a conference, the video data of each user's person image transmitted from four terminals 2 is decoded and resized so that it can be divided and displayed on one screen. Four images are combined (mixed).
Furthermore, the mixed video after mixing is encoded (encoded) by the server, and then transmitted from the server 1 to the terminal 2 as shared information.

In each terminal 2 to which such mixed shared information has been transmitted, the received information is decoded (decoded), and an image or sound is output from a display device or speaker attached to the terminal.
For example, if there are four terminals 2 that are currently connected, the shared information after mixing is distributed from the server 1 to each of the four terminals in real time using time division multiplexing processing or the like. .

  However, when the number of terminals that have been established further increases, for example, when the number of terminals is 20, in order to make the conference proceed smoothly, each of the 20 terminals 2 is similarly real-time. Since it is necessary to open the sound of the shared information by looking at the image or document of the shared information, the server 1 needs to perform information distribution processing at a higher speed.

  In general, ADSL or ISDN communication using a telephone line as a communication line, FTTH communication using an optical fiber, wireless communication, or other communication using a higher-speed dedicated line can be used. Because there is a limit to the amount of communication that can be transmitted in a certain period of time, there is almost no problem in smooth conference proceedings, and the maximum number of terminals that can deliver information in real time without causing problems in information transmission is also used. The upper limit is determined by the line thickness (communication bandwidth), the amount of video and audio data used for communication, the CPU performance of the terminal itself, and the like.

  In the present invention, even when there are a large number of terminals that can participate in a conference of the information sharing system, a connection is established to the server 1 at the same time, and the shared information is distributed directly from the server. The number is reduced as much as possible, and information that maintains real-time characteristics is distributed so that smooth progress of the conference is not hindered.

In the present invention, the substantial reduction in the number of terminals connected to the server as described above is realized by the following processing.
(1) Stepwise operation control for each terminal When it is determined that the terminal is substantially in an operation state where the terminal may not be used (hereinafter referred to as an idle state), the mode of the operation state of the terminal is The communication state between the terminal and the server is changed in a predetermined order, and finally the terminal is turned off.
Here, the communication state between the server and each terminal is dynamically changed according to the operation state of the terminal.

(2) Reduction of the number of connected terminals as seen from the server by grouping of terminals When there are multiple terminals on a network connected to the same router, set a group to which multiple terminals belong, Is set as a representative terminal, and the number of terminals directly communicating with the server is reduced by limiting the number of terminals directly communicating with the server to only the representative terminal.
Terminals belonging to a group other than the representative terminal are allowed to receive the received information from the representative terminal that has received the shared information.

In the process of (1) above, whether or not the terminal is in an idle state is determined by, for example, acquiring information about the user himself / herself by image recognition or voice recognition, or operating state of the terminal (power ON / OFF ) Or the presence / absence of an input operation.
Further, the idle state determination process may be performed on either the server side or the terminal side.

Further, in the processing of (1), the mode of operation state of the terminal (also referred to as operation mode) that is changed in stages is, for example, active mode, sleep mode, logoff mode, and shutdown mode (power-off state) in this order. ) Is considered. However, these four modes are examples and are not limited to these.
The first mode corresponds to the active mode, the second mode corresponds to the sleep mode, the third mode corresponds to the logoff mode, and the fourth mode corresponds to the shutdown mode.
For example, when it is determined in the second mode (sleep mode) that it is in the idle state, the mode is switched to the third mode (logoff mode), and in the third mode, it is determined that it is in the idle state. Switch to the fourth mode (shutdown mode).

Here, the active mode means a state in which all the functions installed in the terminal can be executed, for example, means a state in which bidirectional communication of video and audio is performed with the server.
The sleep mode means a state in which only video and audio are received from the server (one-way communication). In order to execute a function that can be executed by the terminal, the user needs It is necessary to switch from the idle mode to the active mode by operating the terminal, changing the camera input image when the user is in front of the terminal, or generating a microphone input sound when the user speaks There is.

In addition, the logoff mode means a state in which a screen prompting the user to log in is displayed on the screen of the terminal. For example, if the user does not perform login authentication, the function of the terminal is executed. It means a state that cannot be made.
The logoff mode is a state in which communication with the server is stopped while the main function of the information terminal is stopped.
The shutdown mode means a state in which the power of the terminal is turned off. In order to operate the terminal, for example, the user must press a power button to start software such as an OS installed in the terminal. Means state.
In the following embodiments, every time an idle state is detected, the operation state of the information terminal is changed from the first mode (active mode) to the second mode (sleep mode), the third mode (logoff mode), and the fourth mode. Change in steps in order of (shutdown mode).

Here, the four modes are different from the operation procedure that must be performed in order for the user to execute a predetermined function, and the time until the user can execute the function.
Regarding the user operation, the active mode is the easiest, whereas the shutdown mode requires the most operation procedures, and it takes the longest time for the user to execute a desired function.
The reason why the four modes are provided in this way is that the user's operation burden necessary for returning the terminal to the active mode for executing a predetermined function is taken into consideration as much as possible.

For example, when it is confirmed that the user is absent as in the past, when the terminal is uniformly turned off, the absence is temporary and the user participates in the conference. Even if there is still a will, the user has to execute all the series of operations starting from the power-on of the terminal, and the operation burden is heavy, and it is difficult to return to the conference.
Therefore, if the absence of a user is confirmed, information communication with the server is temporarily interrupted so that transmission delays and communication failures are reduced, but the absence is not attending the conference (leaves the meeting). If you can't be sure that it is due to, for example, simply put the terminal into sleep mode, you can return to the active mode with a relatively easy operation, so the burden on the user is light and the user can return to the conference in a short time. become able to.
In the following, a configuration for realizing the processes (1) and (2) and an example of specific processing contents will be described.

<Configuration of information terminal>
FIG. 2 shows a configuration block diagram of the first embodiment of the information terminal.
The information terminal 2 (TE) mainly includes an input unit 11, a video input unit 21, a display unit 31, a voice input unit 41, a voice output unit 45, a communication unit 51, an authentication unit 55, a storage unit 61, a signal processing unit 71, And a control unit 80.

Here, the information input unit described above corresponds to a unit including the input unit 11, the video input unit 21, and the audio input unit 41, and generates state information indicating that the user is using the information terminal. It is a part to do.
The input unit 11 includes a human detection sensor 15 in addition to a pointing device such as a keyboard 12, a mouse 13, and a tablet 14. The input processing unit 16 receives these inputs and converts them into signals for transmission to the control unit 80. With.
As the human detection sensor 15, for example, a heat detection sensor, an infrared sensor, an ultrasonic sensor, or the like can be used. Instead of the human detection sensor, it may be detected whether there is a person in front of the terminal based on input signals from a camera 22 and a microphone 42 described later.
From the input unit 11, an input signal mainly by a user's operation is input, and this input signal is used for generating state information. For example, when this input signal is input, state information indicating that the information terminal is in use is generated.

  The video input unit 21 is a part for inputting video in the vicinity of the terminal and video of a person (user) participating in a meeting in front of the terminal. The video 22 and the video captured by the camera are digitally input. And a video processing unit 23 for converting into a signal. The input video 66 is temporarily stored in the storage unit 61. This input video 66 is also used to generate the state information described above.

  The display unit 31 is a part that displays a user's own image or a person image of another person participating in the conference, and displays a display 32 such as an LCD or CRT and a display signal to be displayed on the display 32. And a display processing unit 33 to be generated.

The voice input unit 41 is a part that inputs voice in the vicinity of the terminal TE, and includes a microphone 42 and an input voice processing unit 43 that converts voice input from the microphone into a digital signal. The input voice 67 is temporarily stored in the storage unit 61.
This input voice 67 is also used to generate the state information described above.

  The audio output unit 45 includes a speaker 46 and an output audio processing unit 47. For example, the audio output unit 45 is a part that outputs audio transmitted from the server from the speaker.

The communication unit 51 is a part that performs data transmission and data reception with a server or another terminal TE, and uses a predetermined communication protocol (for example, H.323, SIP, HTTP, etc.) via various networks. Data communication control is performed so that data communication is possible. In the conference system, voice and video communications are performed. Data encryption processing (for example, AES: Advanced Encryption Standard, 3DES: Triple) is performed to conceal data and reduce the amount of data as necessary. -Data Encryption Standard, etc.), data compression processing (also called encoding or encoding), for example, MPEG4, H.263, H.264 for video, G.711, G.722, G.729 for audio. Etc.).
Therefore, the communication unit 51 preferably includes a communication encryption unit 52 and a communication decryption unit 53.
Further, the communication unit 51 changes the communication state with the server according to the operation state after being changed by the terminal mode switching unit 86.
For example, the state is changed to a bidirectional communication state in which video and audio are transmitted and received bidirectionally, and a one-way communication state in which video and audio are received unidirectionally from the server.

  The authentication unit 55 is a part that performs user authentication and login authentication to the terminal and the conference system, and performs authentication mainly by inputting an ID and a password registered and managed in advance.

The storage unit 61 is a part that stores information such as various types of input information (input video 66, input voice 67, etc.), display information, output information, transmission / reception data, signal processed data, and the like, such as voice recognition and image recognition. A recognition database 62 stored in advance for performing person recognition using.
The recognition database 62 mainly includes a user name 63 that identifies a user, voice feature data 64 of the user, and facial image feature data 65 of the user. When a plurality of users use the terminal TE, recognition information (63, 64, 65) is stored for each user. Although not shown in this article, the recognition database 62 may store password information for each user (the data itself is preferably encrypted).
Voice recognition and image recognition are performed by the person recognition unit 84.
As the storage unit 61, a semiconductor element such as a ROM or a RAM, a storage medium such as a hard disk, a CD, a DVD, or a USB memory is used.
The storage unit 61 also stores a program for executing each function of the terminal TE.

The signal processing unit 71 converts input video and audio into data to be transmitted to the server (also referred to as encoding processing and encoding processing), and displays video and audio transmitted from the server or the like on a display. It is a part that converts (also called decoding processing or decoding processing) into original data that can be output from the display and speaker.
It mainly comprises a video encoding unit 72, a video decoding unit 73, an audio encoding unit 74, and an audio decoding unit 75.
For the signal processing unit 71, for example, a codec LSI capable of executing encoding and decoding conforming to a codec of a predetermined standard is used. Encoding and decoding processing may be realized by a software codec instead of a hardware codec such as an LSI.

The control unit 80 is a part corresponding to a microcomputer including a CPU, a ROM, a RAM, an I / O controller, a timer, and the like, and the CPU reads a program stored in the storage unit 61 into a RAM (main memory). The various functions of the terminal are executed by operating various hardware organically.
2 shows an idle state determination unit 81, a terminal state notification unit 85, and a terminal mode switching unit 86 as functions performed by the control unit 80.
The idle state determination unit 81 is a part that determines that the information terminal is in an idle state that may not be used based on the state information generated by the information input unit. An input signal detection unit 83 and a person recognition unit 84 are included.
Each function of these functional blocks (81 to 86) is obtained by the CPU using information obtained from the input unit 11, the video input unit 21, the audio input unit 41, and the like based on the program for each function. This is realized by executing a predetermined process.

The idle state determination unit 81 performs a process of determining the current operation state of the terminal TE connected through the network, and the terminal mode switching unit 86 sets the operation mode to active mode (M1), sleep mode (M2), and logoff. Change to either mode (M3) or shutdown mode (M4).
The idle state means a state where the information terminal may not be used, and a state where the user is not participating in the meeting or may not be participating because of the reason such as being in the middle or leaving. Means. When the idle state is detected, one of the four modes other than the active mode is entered.
As will be described later, the determination of the operation state uses determination of presence / absence of operation input by the input unit 11 and determination by person recognition using input video and input audio.
For example, if there is no operation input such as a mouse, no change in the video input from the camera, and no audio input from the microphone, the idle state continues for a predetermined set time (for example, 10 minutes) It is determined that

The time measuring unit 82 counts a set time for determining whether or not the vehicle is in an idle state. The set time may be fixed as a default value in advance or may be changed by the user. For example, if the set time is set to 10 minutes, the state is continuously monitored for 10 minutes.
The input signal detection unit 83 is a part that detects the presence or absence of an input signal from the input unit 11. For example, if it is detected that there is some character input from the keyboard 12, it is determined that there is an input signal, and it is determined that it is not in the idle state.

The person recognizing unit 84 uses the input video 66 and the input voice 67 to determine whether or not they match the recognition information (63, 64, 65) in the recognition database 62, and the current user This is a part for determining whether or not the user is registered in the recognition database 61 in advance.
In person recognition, a user is specified by using voice recognition using the input voice 67 and the voice feature data 64 and image recognition using the input video 66 and the face image feature data 65.
For example, when the face image data of the user currently participating in the meeting matches the face image data corresponding to the registered user name 63, the user can be identified and the person recognition is determined to be successful. Then, it is determined not to be in an idle state. Conversely, if person recognition fails, it is determined that the information terminal is in an idle state that may not be used.

The terminal state notification unit 85 is a part that transmits the current operation state of the information terminal to the server, and based on the detected idle state, for example, among the four modes (M1, M2, M3, M4) described above State information indicating any state is generated and transmitted to the server via the communication unit 51.
For example, when it is detected that the mode that was in the active mode immediately before becomes the idle state in the subsequent determination, it is determined that the mode is changed from the active mode to the sleep mode, and the terminal indicates the current sleep mode. Send information to the server.

The terminal mode switching unit 86 is a part that changes the mode of the operation state of its own terminal to one of the above four modes, and when it is determined that the terminal is in the idle state, the current operation of the information terminal This is a part that changes the operation state step by step according to the state.
For example, when the idle state determination unit 81 determines that the operation state has changed from the sleep mode to the logoff mode, the current operation state of the terminal is changed to the logoff mode. The change of the operation state is performed in stages corresponding to the current operation state of the terminal.

The above is the main functional block of the terminal, but the functions (81, 85, 86) realized by the control unit 80 are not limited to this.
Since these functions are mainly realized by software, a program for executing other functions is stored in the storage unit 61 as necessary, and the CPU organically operates the hardware based on the program. The function may be executed.
In addition, the program may be provided in a fixed storage in a ROM or the like in advance, but may be stored in a rewritable storage device such as a hard disk for subsequent deletion, addition, change, update, etc. It may be provided by a recording medium such as a CD or a DVD, or may be downloaded from a server or the like via a network.

<Server configuration>
FIG. 3 shows a configuration block diagram of the first embodiment of the server.
The server 1 mainly includes an input unit 101, a display unit 111, a communication unit 121, an authentication unit 125, an MCU unit 131, a storage unit 141, a signal processing unit 151, and a control unit 161.

The input unit 101 is a part for inputting various types of information. The input unit 105 converts input from the keyboard 102, mouse 103, tablet 104, and these input devices into signals that can be transmitted to the control unit 161, and Is provided.
The display unit 111 includes a display 112 such as an LCD or a CRT, and a display processing unit 113 that generates a display signal to be displayed on the display.
However, when the server 1 is operated completely unattended, the input unit and the display unit may not be provided.

The communication unit 121 is a part that transmits / receives data to / from the terminal TE via the network, and performs data communication with a predetermined communication protocol (for example, H.323, SIP, HTTP, etc.), video, Audio is encoded with a predetermined codec (for example, MPEG4, H.263, H.264 for video, G.711, G.722, G.729 for audio, etc.), and a predetermined standard (for example, AES) : Advanced Encryption Standard, 3DES: Triple-Data Encryption Standard, etc.).
Similar to the terminal TE, the authentication unit 125 is a part that performs user authentication and login authentication to the terminal and the conference system.

  The MCU unit 131 is a part that executes processing performed in a so-called multipoint connection unit (MCU), and mainly includes a video mixing unit 132 that combines a plurality of video signals, and a video resizing unit 133 that changes the size of the video. , And a voice mixing unit 134 that synthesizes a plurality of voice signals. The MCU unit 131 may be configured in common with the signal processing unit 151.

The storage unit 141 stores various data used in processing executed by the server, and further stores information (device ID 143, base ID 144, IP address 145) for specifying a terminal connected to the server. This is a part that stores a terminal database 142 and a user database 146 that stores information (user ID 147, site ID 144, IP address 145, mail address 148) that identifies a user who uses the terminal.
The terminal database 142 and the user database 146 are used when bidirectional communication is performed between the server and the terminal or when information such as video and audio is transmitted unidirectionally from the server to the terminal.
In addition to this, programs necessary for executing each function of the server are also stored in the storage unit 141.
As the storage unit 141, a recording medium such as a ROM, a RAM, a hard disk, a CD, or a DVD is used.

The signal processing unit 151 performs conversion processing (decoding processing and decoding processing) for temporarily returning the video and audio sent from the terminal to the original signal, and transmits video and audio to a plurality of terminals. This is the part that performs data conversion processing (encoding processing, encoding processing).
Like the terminal TE, the signal processing unit 151 mainly includes a video encoding unit 152, a video decoding unit 153, an audio encoding unit 154, and an audio decoding unit 155, and is realized using a codec LSI or a software codec. can do.

The control unit 161 is a part corresponding to a microcomputer including a CPU, a ROM, a RAM, an I / O controller, a timer, and the like. The CPU performs various hardware operations based on a program stored in the storage unit 141. Each function of the server is realized by operating the server.
FIG. 3 shows a terminal state determination unit 162 and a communication method switching unit 163 as functions of the control unit 161.

The terminal state determination unit 162 is a part that receives the state information indicating the mode of the operation state sent from the terminal and determines which mode the terminal is currently in.
The communication method switching unit 163 is a part that switches the communication method for the information terminal based on the operation state of the terminal transmitted by the terminal state notification unit of the information terminal. Communication methods mainly include two-way communication status, one-way communication status in which transmission of video and audio, etc. is only from the server to the terminal, data communication suspension status, terminal power-off status, and communication line disconnection In some cases, the terminal is excluded from management.
Further, the communication state between the server and the information terminal is changed for each information terminal according to the operation state of the information terminal.

<Description of terminal operation mode>
In the following, modes of four operation states of the terminal will be described as an example of switching the communication state between the server and the terminal.
FIG. 4 shows an explanatory diagram of an embodiment of the active mode (M1) of the terminal.
Here, a case where four information terminals (TE1 to TE4) are directly connected to the server 1 is shown.
In FIG. 4, it is assumed that the operation states of the four terminals (TE1 to TE4) are all in the active mode.
Here, the active mode M1 means a state in which the terminal TE can perform two-way data communication with the server.
For example, the terminal TE1 indicates that the self-portrait (transmission image data SD1) taken by the camera is transmitted to the server and the reception image RD1 is received from the server.

Here, the received image RD1 is decoded, resized, mixed, and encoded after the server receives the respective own images (SD1, SD2, SD3, SD4) transmitted from the four terminals (TE1 to TE4). This is image data that has been processed so that four self-portraits can be displayed in four on one screen.
Each received image (RD1, RD2, RD3, RD4) is the same image. In FIG. 4, on the screen of the display unit, an image (SD1) photographed by the terminal TE1 at the upper left and a terminal TE2 at the upper right. A received image is shown in which a captured image (SD2), an image captured by the terminal TE3 at the lower left (SD3), and an image captured at the terminal TE4 (SD4) at the lower right are displayed in four divisions.
This received image is one piece of shared information that is shared by terminals connected to the server. As shared information, in addition to this, voice input at any terminal or documents distributed in advance to each terminal Information is also included.

FIG. 5 is an explanatory diagram of an embodiment of the sleep mode (M2) of the terminal.
Here, the sleep mode M2 means a one-way communication state in which one-way communication is performed from the server to the terminal TE regarding the communication of information such as video and audio between the terminal TE and the server.
FIG. 5 shows a state in which only the terminal TE1 is in the sleep mode M2, and the other terminals (TE2, TE3, TE4) are in the active mode M1.

As a condition for the terminal TE1 to switch from the active mode M1 to the sleep mode M2, for example, there is no input from a keyboard or the like of the input unit for a predetermined set time, and there is no change in input video, and voice input is also possible. For example, when it is detected that no state (idle state) continues.
That is, when such an idle state is detected in the active mode M1, the terminal is not immediately shut down, but the operation mode is changed step by step (in this case, the sleep mode M2 is changed). When a user who has left his / her seat returns, it will be easier to return.
In the sleep mode M2, if the user who was sitting down sits in front of the terminal (changes in the input video of the camera and the input voice of the microphone) and performs some input operation, the idle state determination unit 81 and the control unit 80 can automatically return to the active mode.

In addition, the terminal TE1 that has entered the sleep mode M2 does not transmit the self-image SD1, but receives the received image RD1 from the server.
The server performs a series of processes such as decoding, resizing, mixing, and encoding on the own image (SD2, SD3, SD4) sent from the three terminals (TE2, TE3, TE4), Generate a composite image. In this case, the load on the decoding process for the self-portrait SD1 of the terminal TE1 is reduced in the server.
For example, as shown in FIG. 5, a composite image is generated in which the upper left area where the self-portrait of the terminal TE1 was originally displayed is blanked and displayed in four parts. This composite image is transmitted from the server to four terminals, received as received images (RD1, RD2, RD3, and RD4), and displayed on the display unit of each terminal.

FIG. 6 shows an explanatory diagram of an embodiment of the logoff mode (M3) of the terminal.
Here, the logoff mode M3 means a state (communication stop state) in which information communication such as video and audio is not performed between the terminal TE and the server.
In FIG. 6, only the terminal TE1 is in the logoff mode M3, and the other terminals (TE2, TE3, TE4) are in the active mode M1.
The terminal TE1 being in the logoff mode M3 means that, for example, the idle state is continuously detected even if the terminal TE1 that has been in the sleep mode M2 has passed a predetermined set time.
That is, when the idle mode is further detected in the sleep mode M2, the operation mode is changed in stages (here, changed to the logoff mode M3) instead of shutting down the terminal immediately. This makes it easier to return when a user who has left his / her seat in the middle seat returns.

In order for the user to return the terminal to the active mode in the logoff mode M3, for example, user authentication and login operations may be performed. Although an operation procedure is required from the sleep mode M2, it is possible to return to the conference more easily than the shutdown mode M4.
Further, since the terminal TE1 that has entered the logoff mode M3 is in a communication suspension state, the transmission of the own image SD1 and the reception of the received image RD1 are not performed. Although not shown, in this case, a user login screen for the user may be displayed on the display unit 1 of the terminal TE1.
On the other hand, in the other three terminals (TE2, TE3, TE4), the normal operation in the active mode is performed, and the respective self-portraits are transmitted to the server. A reception image obtained by combining the self-portraits of the three terminals is generated on the screen and transmitted from the server to the three terminals. In this case, the load on the decoding process for the self-image SD1 of the terminal TE1 and the transmission of the composite image RD1 is reduced in the server. That is, in the server, the load is reduced in communication processing of only substantially three terminals (TE2, TE3, TE4) excluding the terminal TE1.

FIG. 7 is an explanatory diagram of an example of the shutdown mode (M4) of the terminal.
Here, “shutdown mode” M4 means a power-off state, and is a state in which the line between the terminal and the server is not connected. In principle, the power is turned off automatically on the terminal side, but it may be automatically turned off after receiving an instruction from the server.
In FIG. 7, only the terminal TE1 is in the shutdown mode M4, and the other terminals (TE2, TE3, TE4) are in the active mode M1.
The terminal TE1 enters the shutdown mode M4 means that, for example, after entering the logoff mode M3, it is detected that the terminal is continuously idle for a predetermined set time.
That is, even after the terminal TE1 enters the logoff mode M3, if it is continuously detected that the terminal is in the idle state, it is determined that the user is substantially absent and the terminal is turned off. .
If the user wants to rejoin the conference after shutting down, the terminal is activated by executing normal operation procedures (power-on, login authentication, etc.).

  In FIG. 7, the transmission of the own image and the reception of the received image at each terminal are the same as in the logoff mode of FIG. 6, and bidirectional communication is performed between the three terminals (TE2, TE3, TE4) and the server. .

  As described above, in the case of the sleep mode M2 in FIG. 5, since information communication such as video and audio is only one-way communication from the server to the terminal TE1, the load on communication processing in the server is reduced. In the case of the logoff mode M3 in FIG. 6 and the shutdown mode M4 in FIG. 7, the server does not perform information communication such as video and audio to the terminal TE1 among the four terminals, so the load on the communication processing is reduced. Is done.

In reality, the number of terminals connected to the server may be five or more, and the operation mode of the terminal also changes for each terminal. Therefore, information transmitted from the terminal to the server, server Various patterns other than those shown in FIGS. 4 to 7 can be considered for the composite image transmitted from the terminal to the terminal.
In addition, if all the terminals connected to the server are all in the active mode, the amount of data transmitted between the server and each terminal is large and the amount of data to be processed by the server is large, resulting in transmission delays and communication failures. Although the possibility becomes high, as in this embodiment, the video connected to the server is detected by detecting the state of the terminal where the user may be absent and changing the operation mode step by step. Since the substantial number of terminals that communicate information such as voice can be reduced, it is possible to reduce transmission delay and occurrence of communication failure. Further, when the absence of the user is detected, it is possible to reduce the operation burden on the user for returning to a state where the terminal can be used (active mode) rather than immediately turning off the terminal.

<Communication sequence between information terminal and server>
FIG. 8 shows an explanatory diagram of an embodiment of a communication sequence performed between the information terminal and the server.
FIG. 8A shows an outline of a communication sequence when the terminal TE1 is in the active mode M1.
First, the terminal TE1 determines whether or not the terminal is in an idle state. For example, when it is detected that there is an input from a mouse or a human detection sensor, it is determined that the terminal is not in an idle state.
At this time, terminal state information indicating that the terminal TE1 is currently in the active mode M1 is notified from the terminal TE1 to the server.
On the other hand, if the server confirms the notified terminal state information and determines that the terminal TE1 of the transmission source is in the “active mode”, information such as video and audio is transmitted between the terminal TE1 and the server. Communicate with each other in both directions.

FIG. 8B shows an outline of a communication sequence when the terminal TE1 enters the sleep mode M2.
Here, it is assumed that when the terminal TE1 is in the active mode, it is determined whether or not the terminal TE1 is in the idle state, and it is determined that the terminal TE1 is in the idle state.
At this time, the terminal TE1 shifts the operation mode to the sleep mode M2. Thereafter, the terminal TE1 notifies the server of terminal status information indicating that the terminal TE1 is currently in the sleep mode M2.
On the other hand, the server confirms the notified terminal state information and determines that the terminal TE1 of the transmission source has entered the “sleep mode” M2. And communication with respect to terminal TE1 which became sleep mode is switched from bidirectional | two-way communication to one-way communication.
In other words, information communication such as video and audio is switched to one-way communication in which only transmission from the server to the terminal TE1 is performed.

Here, for the switching to the one-way communication, one of the following two methods is mainly conceivable.
(1) The predetermined communication sequence is continued while the communication protocol is fixed, but a series of signal processing for video and audio transmitted from the terminal TE1 in the sleep mode is stopped. Specifically, the server stops the video decoding process (decoding) and the audio decoding process (decoding) for the terminal TE1.
In this case, for example, SIP or H.264 is used as a communication protocol. If a standard such as H.323 or HTTP is adopted, a communication sequence using the communication protocol is continuously performed.

(2) Change the communication protocol to another communication protocol.
For example, SIP and H.264. 323, the communication protocol such as HTTP is stopped, and the communication protocol is changed to the streaming distribution related to video and audio. Streaming delivery is a communication method in which data such as video and audio is unidirectionally transmitted from the server to the terminal TE1.
For streaming delivery, it is desirable to employ any one of predetermined standards (mms, http, rtmp, rtsp, etc.).

  In any of the above methods, data is transmitted and received between the terminal and the server based on a predetermined communication protocol, but video and audio are transmitted unidirectionally from the server to the terminal. Therefore, the traffic flowing through the communication line can be reduced. Therefore, the delay in data transmission can be reduced, and communication failures such as communication errors can be reduced. As a result, the number of terminals connected to the server can be substantially reduced. It becomes possible.

FIG. 8C shows an outline of a communication sequence when the terminal TE1 enters the logoff mode M3.
Here, it is assumed that the terminal TE1 determines whether or not the terminal TE1 is in the idle state in the sleep mode M2 in FIG. 8B and determines that the terminal TE1 is continuously in the idle state.
At this time, the terminal TE1 shifts the operation mode to the logoff mode M3. When switching to the logoff mode M3, for example, the terminal TE1 specifically performs a process of canceling the user login with the terminal and the server and ending the communication process between the terminal and the server.
Thereafter, the terminal TE1 notifies the server of terminal state information indicating that the terminal TE1 is currently in the logoff mode M3.

On the other hand, the server confirms the notified terminal state information and determines that the terminal TE1 of the transmission source has entered the logoff mode M3. Thereafter, the server stops the one-way communication performed with the terminal TE1, and stops the video and audio communication.
Here, for example, the communication protocol may be continued as it is, and communication such as one-way video and audio may be stopped, or the sequence itself may be stopped according to the communication protocol, and the established link may be stopped. It may be canceled (waiting for user login).

FIG. 8D shows an outline of a communication sequence when the terminal TE1 enters the shutdown mode M4.
Here, it is assumed that when the terminal TE1 is in the logoff mode M3 in FIG. 8C, it is determined whether or not it is in an idle state, and it is determined that the terminal is continuously in an idle state.
At this time, if the communication protocol is continued in the terminal TE1, the server status information indicating that the terminal TE1 shifts to the shutdown mode M4 is notified.
Thereafter, the terminal TE1 automatically performs a series of predetermined shutdown operations, disconnects the line, and disconnects its own power source.
On the other hand, the server side confirms the notified terminal state information and determines that the terminal TE1 has entered the shutdown mode M4. Thereafter, the server performs processing for excluding the terminal TE1 from management, for example, termination of the communication protocol between the server and the terminal TE1, cancellation of user authentication in the terminal TE1 and the server (logoff), etc. Process.

The above is the description of the sequence for the four operation modes of the terminal.
However, the operation modes are not limited to these four, and when other modes other than these four are set, a predetermined sequence set for the other modes may be executed.
As other modes, for example, modes such as video-only communication, audio-only communication, shared document data only communication, bi-directional communication, one-way communication, and the like can be considered.

<Terminal mode transition processing>
FIG. 9 shows a flowchart of the first embodiment of the mode transition process of the information terminal.
Here, as described above, as shown in FIG. 4 to FIG. 8, a process of shifting the terminal mode in stages will be described.
In addition, the terminal and the server are provided with functional blocks as shown in FIGS. 2 and 3, respectively, and the terminal side performs idle state determination processing.

In FIG. 9, first, as shown in step S1, it is assumed that the terminal is in an active mode M1 as an initial state. At this time, as shown in FIG. 8A, the terminal and the server are in a bidirectional communication state.
In step S2, the terminal state notification unit 85 of the terminal confirms the state of the current operation mode of the terminal, and here, notifies the server that it is the active mode M1 (terminal state notification process).

In step S <b> 3, the idle state determination unit 81 of the terminal confirms the presence / absence of an input signal from the input unit 11, the presence / absence of a change in input video from the video input unit 21, the presence / absence of input audio from the audio input unit 41, and the like. Thus, it is determined whether or not the terminal is currently in an idle state (idle state determination process).
Details of the idle state determination processing will be described with reference to the flowchart of FIG.

If it is determined in step S4 that the terminal is in an idle state, the process proceeds to step S5, and if not, the process proceeds to step S14.
In step S5, the terminal mode switching unit 86 executes a process of shifting the operation mode of the terminal to the sleep mode M2.
Here, the operation mode of its own terminal is changed from the active mode M1 to the sleep mode M2. Specifically, for example, processing for switching the communication state in which bidirectional communication has been performed until now to one-way communication for receiving information such as video and audio from the server, streaming reception processing for video and audio, and the like are performed.

In step S6, the terminal state notification unit 85 notifies the server that the sleep mode M2 has been entered.
In step S7, the same idle state determination process as in step S3 is executed.
Here, it is checked whether or not the idle state continues in the sleep mode M2.

If it is determined in step S8 that the idle state continues, the process proceeds to step S9, and if not, the process proceeds to step S15.
Here, the case where this is not the case means, for example, a case where there is an input from a mouse or a human detection sensor. At this time, in step S15, the terminal mode switching unit 86 executes an active mode transition process.
In the active mode transition process, for example, the operation mode of the terminal is changed from the sleep mode M2 to the active mode M1, and the communication state is returned from the one-way communication to the two-way communication.
After step S15, the process proceeds to step S14, and the terminal state notification unit 85 notifies the server that the active mode M1 has been entered.

In step S9, the terminal mode switching unit 86 executes a logoff mode transition process. Here, the operation mode of the terminal is changed from the sleep mode M2 to the logoff mode M3, and although not shown, a process for displaying a user login screen on the terminal screen is performed.
In step S10, the terminal state notifying unit 85 notifies the server that the logoff mode M3 has been entered.
In step S11, the idle state determination unit 81 performs the idle state determination process again.
In step S12, in the state of logoff mode M3, when the idle state is further continued, the process proceeds to step S13, and otherwise, the process proceeds to step S16.

  In step S13, the terminal mode switching unit 86 executes a shutdown mode transition process. Here, the operation mode of the terminal is changed from the logoff mode M3 to the shutdown mode M4. Thereafter, the process proceeds to step S14 to notify the server that the terminal has entered the shutdown mode M4. Thereafter, although not shown in FIG. 9, the terminal itself is automatically turned off. Alternatively, in step S13, after notifying the server to shut down, the power may be automatically turned off and the process may be terminated.

On the other hand, in step S16, since it is detected that the log-off mode M3 is not idle, a login screen is displayed.
At this time, being in an idle state means that the user has performed a so-called login operation using a keyboard or a mouse.
In addition, after the process of step S16, the process proceeds to step S15, and the operation mode of the terminal is changed from the logoff mode M3 to the active mode. Thereafter, in step S14, the server is notified that the operation mode is the active mode.

FIG. 10 shows a flowchart of an embodiment of the idle state determination process performed by the terminal shown in FIG.
In step S21, the time measuring process by the time measuring unit 82 is started. Here, the timer is reset to the initial value (set time).
The initial value is set to an appropriate time for determining whether or not the vehicle is in the idle state, for example, 10 minutes, and the countdown is started. However, an appropriate time for determining whether or not the vehicle is in an idle state may be uniformly fixed at 10 minutes in advance, but this time may vary depending on the user and usage pattern. Therefore, it is preferable that settings can be changed by a user or a system administrator.

In step S <b> 22, the input signal detection unit 83 checks the presence or change of input signals from the input unit 11, the video input unit 21, and the audio input unit 41.
For example, check for operation input from a keyboard or mouse. Also, it is checked whether there is an input signal from the human detection sensor and whether there is no voice input by the user.
Further, it is checked whether or not there is a change that exceeds a predetermined threshold for the video input input from the camera.
As a result of checking these input signals, there is no input signal detection when there is no input signal and input audio input and no change in video input during the set time set as the initial value. It is judged that
On the other hand, when the input signal is input once or more, or when a predetermined change is detected in the video input, it is determined that the input signal is detected.

In step S23, the person recognizing unit 84 performs person recognition of the user by using either one or both of voice recognition and image recognition.
The person recognition process is a process for confirming whether or not the user is a person registered in advance in the recognition database 62 of the storage unit 61 of the terminal.
For example, the input image 66 captured by the camera 22 is compared with the face image feature data stored in the recognition database 62, and matching face image feature data exists in the recognition database 62 by so-called pattern matching. If the user is successfully recognized, it is determined that the current user is correctly identified, and the user is present in front of the terminal and is participating in the conference. .

On the other hand, if the matching face image feature data does not exist in the recognition database 62, the user cannot be specified and the person recognition has failed. In this case, it is determined that there is a possibility that the user does not exist in the shooting range by the camera of the terminal and is not participating in the conference.
However, this person recognition processing is necessary for processing load, processing time, recognition accuracy (when multiple people appear in the camera, or recognition errors occur due to lighting environment, glasses, clothes, etc.) The option may be an option that can be executed by the user as needed without performing the required processing.
In step S24, if no input signal or input voice is detected as a result of steps S22 and S23, or if person recognition fails, it is determined that there is a possibility that the current user is not present in front of the terminal. Then, the process proceeds to step S25.

On the other hand, when any one or more of the input signals are detected, or when the person recognition is successful, or when any of the input signals is present and the person recognition is also successful, the process proceeds to step S27 and the idle state is established. If not, the process ends.
In step S25, it is checked whether or not the set time has elapsed. If the set time has not yet elapsed, the process returns to step S22, and the processes from step S22 to S24 are repeated again.
If the set time has already elapsed, the idle state has been continued for the set time, so the process proceeds to step S26, where it is determined that the terminal is currently in the idle state, and the process ends.

FIG. 11 shows a flowchart of an embodiment of the terminal state determination control process in the server.
In step S31, the terminal state determination unit 162 of the server confirms whether the terminal state information sent from each terminal has been received. If received, the process proceeds to step S32.
Here, the terminal state information includes, for example, a terminal ID for specifying a terminal and data indicating an operation mode (terminal state) of the terminal.
In step S32, it is checked whether or not the data indicating the “terminal state” included in the received terminal state information indicates the active mode M1.
Here, if it is active mode M1, it will progress to step S33 and the communication method switching part 163 will set to the state which can perform audio | voice and image | video bi-directional communication between the terminals. Specifically, for example, the communication method switching unit 163 performs a communication sequence process based on a predetermined communication protocol (SIP, H.323, HTTP, etc.).

In step S32, if it is not the active mode, the process proceeds to step S34.
In step S34, it is checked whether or not the received “terminal state” data indicates the sleep mode M2.
If it is the sleep mode M2, the process proceeds to step S35, and if not, the process proceeds to step S36.
In step S35, the communication method switching unit 163 sets a one-way communication state in which audio and video information is transmitted from the server to the terminal with the terminal.
Specifically, for example, the communication method switching unit 163 switches to a communication protocol based on a streaming distribution standard (mms, http, rtmp, rtsp, etc.) or stops receiving video and audio from the terminal. Perform the process.

In step S36, it is checked whether or not the received “terminal state” data indicates the logoff mode M3.
If it is the logoff mode M3, the process proceeds to step S37, and if not, the process proceeds to step S38.
In step S37, the communication method switching unit 163 stops the audio and video communication between the terminal and the server.

In step S38, it is checked whether or not the received “terminal state” data indicates the shutdown mode M4.
If it is the shutdown mode M4, the process proceeds to step S39, and if not, the process ends. Or you may return to step S31.
In step S39, when it is received that the terminal is shut down, the connection with the terminal is completely disconnected, so that the terminal is set out of management target.
In order to exclude the terminal from the management target, specifically, processing such as termination of a communication sequence based on a predetermined communication protocol may be performed on the terminal.
The processing by the server shown in FIG. 11 is performed for each terminal connected to the server.

FIG. 12 shows an explanatory diagram of an embodiment of the contents of each database stored in the storage unit of the terminal and the server.
FIG. 12A shows an embodiment of the recognition database 62 stored in the storage unit 61 of the terminal shown in FIG.
Here, for each registered user, the user name 63, the user's voice feature data 64, and the face image feature data 65 are stored in association with each other.

FIG. 12B shows an embodiment of the terminal database 142 stored in the storage unit 141 of the server of FIG.
Here, for each terminal, a device ID 143, a base ID 144, and an IP address 145 are stored in association with each other.

FIG. 12C shows an example of the user database 146 stored in the storage unit 141 of the server of FIG.
Here, the user name 147, the site ID 144, the mail address 148, and the IP address 145 are stored in association with each other.
Note that the items of information stored in each database are not limited to those described above, and any other necessary items may be additionally stored.

<Second embodiment of communication between information terminal and server>
The configuration and processing contents of the second embodiment will be described below.
The second embodiment is characterized in that the process for determining whether or not the terminal is currently in an idle state is performed not by the terminal side but by the server.
FIG. 13 shows a configuration block diagram of the second embodiment of the information terminal.
Here, the functional block of the memory | storage part 61 and the control part 81 differs in part from the structure of FIG.
That is, the storage unit 61 does not include the recognition database 62.
In addition, the control unit 81 includes a terminal mode change receiving unit 87 without including the idle state determination unit 81.

The terminal status notification unit 85 of the information terminal transmits status monitoring information including the status information generated by the information input unit including the input unit and the current operation status of the information terminal to the server.
The terminal mode change accepting unit 87 is a part that receives an operation state transition notification transmitted from the server.
The terminal mode switching unit 86 changes the operation state in stages based on the received transition notification.

FIG. 14 is a block diagram showing the configuration of the second embodiment of the server.
Here, the functional block of the memory | storage part 141 and the control part 161 differs in part from the structure of FIG.
The storage unit 141 further includes the same recognition database 149 as that provided in the terminal of FIG.
In addition, the control unit 161 includes a terminal mode control unit 164, and further includes an idle state determination unit 165 provided in the terminal of FIG.

The idle state determination unit 165 performs the same processing as the determination of the idle state performed by the terminal of FIG. 2, but the determination of the idle state is performed by state monitoring transmitted from each connected terminal. This is performed based on information and video and audio information input from the terminal.
When it is determined that the terminal mode control unit 164 is in the idle state, the server mode control unit 164 determines an operation state that the server should shift in a stepwise manner in response to the current operation state included in the state monitoring information transmitted from the terminal. It is a part which sets and transmits the notification of transition to the set operation state to the terminal.
Also in the second embodiment, the communication state between the server and the information terminal is changed for each information terminal according to the operation state of the information terminal.

The recognition database 149 is used in the person recognition process of the idle state determination unit 165, as in the terminal of FIG. 2, and the terminal database 142 and the user database 146 perform bidirectional communication and unidirectional communication between the terminal and the server. It is used when switching between these communication methods.
Of the configurations of FIGS. 13 and 14, the same processing as in FIGS. 2 and 3 is executed for the other functional blocks.

FIG. 15 shows a flowchart of the server mode transition control process in the second embodiment.
Here, mainly, using various information sent from the terminal, it is determined whether or not the terminal is in an idle state, and whether or not to shift the operation mode of the terminal is determined. Processing for notifying the operation mode to be operated is performed.
That is, the server manages the operation mode of each terminal, and each terminal passively changes the setting of the operation mode based on the operation mode notified from the server.
The flow in FIG. 15 is a process performed independently by the server for each terminal.

In step S51, here, as an initial state of the server, it is assumed that communication with the current terminal is in a bidirectional communication state and the terminal is in the active mode.
In step S52, the server idle state determination unit 165 determines whether or not the terminal is in an idle state.
Details of this determination process will be described with reference to the flowchart of FIG.
If it is determined in step S53 that the terminal has entered the idle state, the process proceeds to step S54, and if not, the process ends.
In step S54, since the operation mode of the terminal has been the active mode so far, the terminal mode control unit 164 performs a process of shifting the terminal to the sleep mode (sleep mode transition process).

In step S54, a transition notification requesting to switch the current operation mode to the “sleep mode” M2 is transmitted to the terminal. Furthermore, the communication method switching unit 163 performs a process of switching the current communication state from bidirectional communication to unidirectional communication so that video and audio information can be unidirectionally transmitted from the server to the terminal. The terminal that has received this transition notification switches the operation mode from the active mode M1 to the sleep mode M2, and enters a communication state in which one-way communication is possible.
In step S55, the idle state determination unit 165 performs an idle state determination process as in step S52.
If it is determined in step S56 that the terminal is still in an idle state, the process proceeds to step S57, and if not, the process proceeds to step S61.

In step S57, since it is determined that the current state is the idle state when in the sleep mode M2, the terminal mode control unit 164 performs a process of shifting the terminal to the logoff mode M3 (logoff mode transition process).
Here, a transition notification requesting to switch the operation mode to the “log off mode” M3 is transmitted to the terminal. Further, the communication state is changed from the one-way communication to the communication suspension state.

On the other hand, since it is determined in step S61 that the terminal is not in the idle state, the terminal mode control unit 164 performs an active mode transition process.
For example, when the terminal is in the sleep mode M2, when the state monitoring information indicating that an input signal from a keyboard or a human detection sensor is input from the terminal, it is determined that the terminal is not in the idle state. Then, the process of step S61 is performed.
In step S61, a transition notification requesting that the operation mode be switched from the sleep mode M2 to the active mode M1 is transmitted to the terminal. Further, the communication state is changed from one-way communication to two-way communication.

In step S58, the idle state determination unit 165 performs an idle state determination process.
If it is determined in step S59 that the terminal is still in the idle state, the process proceeds to step S60, and if not, the process proceeds to step S62.
In step S60, since it is determined that the terminal is in the idle state when the terminal is currently in the logoff mode M3, the terminal mode control unit 164 performs processing for shifting the terminal to the shutdown mode M4 (shutdown mode transition processing).

Here, a transition notification requesting to switch the operation mode to the “shutdown mode” M4 is transmitted to the terminal.
The terminal that has received this transition notification automatically performs a predetermined shutdown process to turn off the power.
In addition, the server disconnects the communication line with the terminal and sets the terminal out of management.

On the other hand, if it is determined in step S62 that the user is not in the idle state in the logoff mode M3, it is detected that the user has performed some input operation or the like, or that a video change or voice input has been detected. , Logon transition notification processing.
Here, for example, the terminal is notified that it may have changed from the logoff state to the logon state, and a notification requesting that the operation mode should be shifted to the active mode is transmitted in step S61. To do.
The terminal that has received this notification may accept the user's logon operation and, for example, set the operation mode to the active mode.
Alternatively, since the process of changing from the logoff state to the logon is a process on the terminal side, if it is determined that the server side is not in the idle state in step S59, the process ends without performing the process of step S62. Also good.

FIG. 16 shows a flowchart of idle state determination processing in the server.
Here, the processing in steps S72 and S73 is different from the processing in FIG. 10 in that it is performed based on information transmitted from the terminal.
In step S71, the time measurement unit 166 starts the time measurement process, resets the set time to the initial value, and starts countdown.
In step S72, the terminal state determination unit 162 receives the terminal state monitoring information transmitted from the terminal.
The state monitoring information includes, for example, information indicating the current operation mode of the terminal, information on the input signal input from the input unit of the terminal, video input by the camera of the terminal, and sound input from the microphone. means.

Here, using the received state monitoring information, the input signal detection unit 167 checks whether an input signal that is determined not to be in the idle state is input.
For example, it is determined whether or not there is an input from the human detection sensor, and when it is detected that there is an input from the human detection sensor, it is determined in the subsequent step S74 that the state is not an idle state.
In addition, when it is detected that there is one or more input signals from an input unit such as a keyboard, a mouse, or a tablet of the terminal, it is used for determining that the terminal is not in an idle state.
Furthermore, it is checked whether there is a change in the video sent from the terminal and whether the input audio is sent from the terminal. If it is detected that the video has changed, the input voice is sent. In the case where both the change of the image and the input of the input sound are detected, it is determined that the idle state is not set in any case.
On the other hand, when all the input signals described above are not detected, it is determined that the idle state is set.

In step S73, the person recognizing unit 168 performs a person recognition process using the received video and the received audio sent from the terminal.
Here, the received voice and the voice feature data in the recognition database 149 are compared to perform voice recognition processing, and the received video is compared to the face image feature data in the recognition database 149 to perform image recognition processing.
Each recognition process is the same as the process performed in FIG.
In these recognition processes, when the data that matches the voice and the video exists in the recognition database, it is determined that the person recognition is successful and it is determined not to be in the idle state.
On the other hand, when there is no matching data, it is determined that person recognition has failed, and it is determined that the person is in an idle state.

In step S74, it is determined whether or not the vehicle is in an idle state using the determination results in steps S72 and S73.
Here, if the input signal is detected by the terminal based on the determination result in step S72, if the person recognition is successful in step S73, or if the input signal is detected and the person recognition is successful, Proceeding to step S77, it is determined that the current terminal is not in an idle state.
If no input signal is detected and if person recognition fails, the process proceeds to step S75.
In step S75, it is confirmed whether or not the set time has elapsed. If it has not yet elapsed, the process returns to step S72.
On the other hand, if it has elapsed, the idle state has been continued beyond the set time, so the process proceeds to step S76, where it is determined that the idle state has been reached.

FIG. 17 shows a flowchart of the terminal state control process of the terminal in the second embodiment.
Here, the operation mode of its own terminal is changed mainly based on the content of the migration notification sent from the server.
First, the terminal state notification unit 85 monitors its own terminal state in step S81, and notifies the server of the current state monitoring information acquired by monitoring in step S82.

In step S81, the presence / absence of an input signal from the input unit 11 is checked, and the video input from the camera 22 of the video input unit 21 is stored in the storage unit 61 as the input video 66 and input from the microphone 42. Is stored as the input voice 67.
In step S82, for example, current state monitoring information is generated and transmitted to the server in order to notify the server of information about the presence / absence of an input signal from the input unit, information including input video and input audio.

Next, in step S83, the terminal mode change accepting unit 87 checks whether or not the transition notification information from the server has been received.
If not received, the process returns to step S81. If received, the process proceeds to step S84.

In steps S84, S86, and S88, it is checked which operation mode the content of the received transition notification is.
First, in step S84, if the received transition notification is for requesting transition to the sleep mode M2, the process proceeds to step S85, where the terminal mode switching unit 86 switches the operation mode to the sleep mode M2, and the communication state Is shifted to a state of one-way communication of audio and video.
That is, switching to one-way communication that receives video and audio sent from the server.
At this time, video and audio may not be transmitted from the terminal to the server.

In step S86, if the received transition notification is a request for transition to the logoff mode M3, the process proceeds to step S87.
In step S87, a transition process to the logoff mode similar to step S9 of FIG. 9 described above is performed.
In step S88, if the received transition notification is a request for transition to the shutdown mode M4, the process proceeds to step S89, and the shutdown mode transition process similar to step S13 is performed.

On the other hand, if the received transition notification is not a transition notification to the shutdown mode M4 in step S88, the process proceeds to step S90, and the transition process to the active mode M1 is executed as in step S15. However, this processing may not be executed if the active mode M1 is already set.
When returning from the sleep mode M2 to the active mode M1, processing for switching from one-way communication to two-way communication is performed.

The above is the processing contents of the terminal and the server in the second embodiment.
In the second embodiment, since the determination process of the idle state is performed on the server side, the load for performing the determination process is applied to the server. Since the operation mode is changed, the actual number of terminals connected to the server can be reduced, the server load can be reduced, a smooth conference can be realized by reducing communication delay, etc., and the operation burden on the user can be reduced. Can be reduced.

<Third embodiment, reduction of the number of terminals by grouping>
Here, an embodiment will be described in which the number of terminals connected to the server is substantially reduced by grouping a plurality of terminals and setting one of them as a representative terminal.
FIG. 18 is a block diagram showing the configuration of the third embodiment of the information terminal.
In FIG. 18, the terminal further includes a representative terminal detection request unit 88, a streaming request unit 89, a streaming reception unit 90, and a streaming distribution unit 91.
The representative terminal detection request unit 88 is a part that requests the server to detect a representative terminal in the same base, and receives information on the detected representative terminal from the server.
The streaming request unit 89 is a part that requests the representative terminal to distribute streaming information (audio and video) to the terminal.
The streaming receiving unit 90 is a part that receives streaming information distributed from a server or a representative terminal.

The streaming delivery unit 91 is a part that transmits streaming information to other terminals when the terminal is a representative terminal.
Here, streaming distribution means that video and audio, which are streaming information, are transmitted in one direction.
Since a terminal that performs streaming reception does not encode or transmit video and audio data of the terminal itself, the load on the terminal is less than that in the case of performing bidirectional communication.

FIG. 19 shows a configuration block diagram of a third embodiment of the server.
In FIG. 19, the server further includes a representative terminal detection unit 169 and a streaming distribution unit 170.
The representative terminal detection unit 169 is a part that detects whether or not a terminal to be a representative terminal exists in the same base using the base ID 144 stored in the terminal database 142 of the storage unit 141.
The streaming delivery unit 170 is a part that transmits streaming information from the server to the information terminal.
Streaming distribution from this server is performed only for information terminals that are determined to be representative terminals, and is not performed for information terminals that are not representative terminals.

FIG. 20 shows an explanatory diagram of an embodiment of the terminal database used in the third embodiment. In FIG. 20, the terminal database 142 of the storage unit 141 includes a representative terminal flag 150. The representative terminal flag 150 is information for identifying whether or not the terminal is a representative terminal. When the terminal is “ON”, the terminal is a representative terminal. When the terminal is “OFF”, the terminal Means not a representative terminal.
In FIG. 20, of the four terminals, two terminals whose base ID is “Kansai” are terminals of the same base, and among the two terminals included in the same base, device ID = “G53182” The representative terminal flag 150 of the terminal “” is “ON”, which means that this terminal is the representative terminal.
Here, the same base forms one group, and there is only one representative terminal in one group.

In grouping, terminals installed in relatively close positions are set as one group. For example, as shown in FIG. 20, a plurality of terminals belonging to the same base belong to one group. Treat as.
Further, a plurality of terminals belonging to the network under the same router or the same relay server may be treated as belonging to one group, and a specific numeric string portion of the numeric strings constituting the IP address is the same. Terminals (terminals in the same domain) may be treated as belonging to one group.
In the following description, a plurality of terminals having the same base ID are treated as belonging to the same group.

In the case of FIG. 20, two terminals included in the same base are regarded as terminals belonging to one group. When viewed from the server, only representative terminals belonging to the group are connected to the server. Be treated. That is, when information such as video and audio is transmitted from the server, the information is directly transmitted from the server only to representative terminals belonging to the same base (same group), and directly from the server to terminals that are not representative terminals belonging to the same base. Not sent.
In principle, information is transmitted to terminals that are not representative terminals by streaming distribution from representative terminals that belong to the same base.

Further, when there are a plurality of terminals belonging to the same base, that is, one group, one terminal out of terminals not in an idle state, that is, a terminal currently in the active mode is selected as a representative terminal.
However, if there is no terminal in the active mode among the terminals belonging to one group, one terminal is selected as the representative terminal from the terminals in the idle state (for example, the terminal in the sleep mode). The
Furthermore, if the terminal that was the representative terminal becomes idle, change the other terminal that is in the active mode in the same site to the representative terminal and receive streaming distribution from the changed representative terminal. May be.

Further, when there is only one terminal in the same base, if that terminal is in an idle state, streaming distribution may be received from a representative terminal belonging to another base.
Alternatively, the terminal itself in the idle state may substantially become a representative terminal and receive streaming distribution from the server.
In addition, the representative terminal in the same base performs streaming distribution only to other terminals belonging to the same group in principle. However, when a streaming distribution request is received from a terminal TEn belonging to another base, the representative terminal May perform streaming delivery to the terminal TEn at another base.

First, an outline of information distribution between the terminal and the server in the third embodiment will be described.
FIG. 26 is an explanatory diagram of an active mode that is one of the communication states between server terminals in the third embodiment.
In FIG. 26, it is assumed that four terminals (TE1 to TE4) are connected to one server.
Here, it is assumed that all four terminals (TE1 to TE4) are in the active mode M1, and the terminals TE1 and TE2 belong to the same base, have the same base ID, and are grouped.
Further, it is assumed that none of the four terminals has yet become representative terminals, and therefore, the representative terminal flags of the respective terminals are all “OFF”.

As described above, when all terminals are in the active mode M1, the communication between the server and each terminal is in a two-way communication state, and each terminal (TE1 to TE4) is encoded with the terminal. The self-portraits (SD1 to SD4) are transmitted, and the server receives four self-portraits and executes a series of processes of decoding, resizing, mixing, and encoding, and the combined images (RD1 to RD4) To TE1 to TE4).
Therefore, even when two terminals (TE1, TE2) belong to the same base, the two terminals (TE1, TE2) each perform bidirectional communication with the server.
It can be said that this state is the state where the load is most applied to the server, and transmission delays and communication failures are likely to occur.

FIG. 27 is an explanatory diagram when the terminal TE1 enters the sleep mode M2 in the communication state between server terminals of the third embodiment.
Here, in the terminal TE1, the user is absent and an idle state is detected, the active mode M1 is changed to the sleep mode M2, and direct communication between the terminal TE1 and the server is not performed.
In this case, the terminal TE2 that belongs to the same base (the same group) as the terminal TE1 and is in the active mode becomes the representative terminal of the group.
And terminal TE1 which became sleep mode M2 receives streaming delivery from representative terminal TE2.

The representative terminal TE2 receives the received image RD2 generated by the server, and distributes the received image RD2 as it is to the terminal TE1. That is, the same image RD1 as the received image RD2 is distributed.
Since the terminal TE1 is in the sleep mode M2, the terminal TE1 only receives the information RD1 sent from the terminal TE2 by streaming without transmitting the self-portrait SD1.
Here, since the server does not transmit the self-portrait SD1 of the terminal TE1, the composite image generated by the server does not include the self-portrait SD1 and the self-portrait (SD2) sent from the three terminals (TE2, TE3, TE4). , SD3, SD4) can be divided and displayed on one screen.

In the case of FIG. 27, when viewed from the server, compared to the case of FIG. 26, the number of directly connected and two-way communication terminals is reduced, the amount of communication data performed between the server and the terminal is reduced, and synthesis is performed. Since the number of images to be reduced is reduced from four to three, the amount of data that needs to be decoded by the server is reduced.
That is, there are four terminals that receive the received image that is shared information, but the number of terminals that are directly connected to the server is three, which is substantially reduced. And the occurrence of communication failures is reduced, and the conference can proceed smoothly.

FIG. 28 is an explanatory diagram when both the terminals TE1 and TE2 are in the sleep mode M2 in the server-terminal communication state of the third embodiment.
Here, since all the terminals (TE1, TE2) in the same base are in the sleep mode M2, the terminal TE2 remains streaming as a representative terminal and receives streaming distribution from the server.
Furthermore, the terminal TE1 does not perform direct communication with the server but receives streaming distribution from the representative terminal TE2.
Since both the terminals TE3 and TE4 are in the active mode M1, two-way communication is performed with the server, and the self-portraits (SD3 and SD4) of each terminal are transmitted to the server. The self-portraits (SD1, SD2) from the terminals TE1 and TE2 that have entered the sleep mode M2 are not transmitted to the server.
Accordingly, the received image (RD2, RD3, RD4) sent from the server to the three terminals (TE2, TE3, TE4) includes two own images (SD3, SD4).
The received image RD1 distributed to the terminal TE1 is the same as the image RD2 received by the terminal TE2.

In this case, the number of terminals that perform bidirectional communication with the server is two, the number of terminals that receive streaming distribution from the server is one, and the amount of communication data performed between the server and the terminal is smaller than in the case of FIG.
In addition, since data to be subjected to a series of processing such as decoding by the server also becomes two self-portraits (SD3 and SD4), the load on the processing is reduced.
Therefore, there are still four terminals that receive the received image that is shared information from the server, but there are three terminals that communicate directly with the server, and one of the terminals (TE2) is unidirectional. Since it is a terminal that only performs streaming reception that is communication, transmission delay and communication failure can be reduced, and the load on the server can be reduced.

FIG. 29 shows an explanatory diagram when the terminal TE4 enters the sleep mode M2 in the communication state between server terminals of the third embodiment.
Here, it is assumed that two terminals (TE1, TE2) belonging to the same base and another terminal TE3 are in the active mode M1, and are performing bidirectional communication with the server, respectively.
Only the terminal TE4 is in the sleep mode M2, and it is assumed that there are no other terminals in the same group to which the terminal TE4 belongs.
That is, it is assumed that there is only one terminal TE4 at the same base (same group). In this case, since there is no other terminal to be the representative terminal in the same group, the terminal TE4 itself becomes the representative terminal and receives streaming distribution from the server. Since the terminal 4 is in the sleep mode M2, the received image RD4 is received by streaming from the server without transmitting the own image SD4.
Therefore, since the self-image transmitted to the server is the self-image (SD1, SD2, SD3) from the three terminals (TE1, TE2, TE3), the composite images (RD1 to RD4) transmitted from the server are One self-portrait.

In the case of FIG. 29, when there is only one terminal in the same base (the same group), when that terminal enters the sleep mode M2, the received image RD4 is displayed on the display unit by receiving streaming delivery from the server. Can be displayed.
In addition, since the terminal TE4 that has entered the sleep mode M2 can be switched from bidirectional communication to unidirectional communication that receives streaming distribution, the amount of data transmission can be reduced, the load on the server can be reduced, and transmission delay can be reduced. In addition, communication troubles can be reduced.

FIG. 30 is an explanatory diagram when three terminals TE1, TE2, and TE4 are in a sleep mode in the inter-server terminal communication state of the third embodiment.
Here, as shown in FIG. 28, since all the terminals (TE1, TE2) belonging to the same base are in the sleep mode M2, one terminal TE2 becomes a representative terminal, and streaming distribution is performed from the server. Receive. The other terminal TE1 receives streaming distribution from the representative terminal TE2.

Similarly to the case shown in FIG. 29, the terminal TE4 having only one terminal at the same site is also in the sleep mode M2, but in FIG. 30, the terminal TE4 does not receive streaming delivery from the server. A case is shown in which streaming delivery is received from the representative terminal TE2.
Therefore, the representative terminal TE2 first receives streaming delivery from the server, and further performs streaming delivery to the two terminals (TE1, TE4).
In this case, since the terminal in the active mode M1 is only TE3, the self-image transmitted to the server is only the image SD3 from the terminal TE3, and the composite images (RD1 to RD4) transmitted from the server to the terminal are , Including the self-portrait SD3 of the terminal TE3.

In this case, there are two terminals directly connected to the server, of which one terminal (TE3) performs bidirectional communication, and the other terminal (TE2) is a one-way communication terminal that performs streaming distribution. is there.
Accordingly, the received images (RD1 to RD4) that are shared information are displayed on four terminals, but the number of terminals directly connected to the server is substantially two, and are communicated between the server and the terminals. The amount of data can be reduced.

Also, the self-image received by the server is only from one terminal TE3, and the load on the server for generating the composite image can be reduced.
Therefore, in the case of FIG. 30, the number of terminals directly connected to the server can be further increased by making the other terminal that performs streaming distribution from the representative terminal another terminal belonging to the same base and a terminal belonging to another base. Thus, the load on the server, the transmission delay, and the communication failure can be reduced.

FIG. 25 shows a schematic explanatory diagram of a communication sequence between the terminal and the server in the third embodiment.
FIG. 25A shows a case where the terminal TE1 is in the active mode M1 and the server and the terminal TE1 are performing bidirectional communication. This is the same as in FIG.
FIG. 25B shows a sequence when the terminal TE1 enters the sleep mode M2.
Here, as shown in FIG. 26, it is assumed that the terminal TE1 and the terminal TE2 belong to the same base (the same group). This sequence shows the case of entering the communication state of FIG.

First, a terminal state notification indicating that the terminal TE1 has entered the sleep mode M2 is transmitted from the terminal TE1 that has entered the sleep mode M2 to the server. By this notification, the communication between the terminal TE1 and the server is changed from the bidirectional communication state to the one-way communication state (a state in which video and audio data from the terminal TE1 are not processed on the server side).
Further, the terminal TE1 transmits information requesting the server to detect the representative terminal (transmission of a representative terminal detection request).
The server that has received this detection request uses the terminal database 142 stored in the storage unit of the server to check whether there is a representative terminal that can be grouped with the terminal TE1. In the case of FIG. 26, since the terminals TE1 and TE2 are in the same base, these two terminals are grouped into the same group. Also, assuming that the terminal TE2 is in the active mode M1, the terminal TE2 is detected as a representative terminal.

Since the terminal TE2 can be selected as the representative terminal, the server sets the representative terminal flag of the terminal TE2 in the terminal database 142 to “ON”, and transmits a representative terminal detection notification to the terminal TE1.
In this representative terminal detection notification, the device ID and IP address of the terminal TE2 as the representative terminal are notified.
Thereafter, the terminal TE1 that has received this notification transmits a streaming request to the terminal TE2.

When the terminal TE2 receiving the streaming request detects this request, the terminal TE2 streams the received image RD2 sent from the server to the terminal TE1.
That is, the terminal TE1 does not communicate with the server, and performs one-way communication with the terminal TE2 by streaming delivery including information such as video and audio (streaming reception).
Accordingly, the terminal TE1 that has entered the sleep mode M2 can receive the shared information.

FIG. 25 (c) shows a sequence when no representative terminal is detected by the server. In other words, since the terminal TE1 is in the sleep mode M2, a representative terminal detection request is made to the server, but the representative terminal cannot be detected by the server.
At this time, an undetected notification of the representative terminal is transmitted from the server to the terminal TE1.
Upon receiving this non-detection notification, the terminal TE1 sets itself as a representative terminal, and transmits a terminal state notification (transition to the representative terminal mode) indicating that the terminal TE1 has transitioned to the representative terminal to the server.
Upon receiving this notification, the server detects that the operation mode of the terminal TE1 has become the representative terminal, and sets the representative terminal flag of the terminal TE1 to “ON”.
Thereafter, the server performs streaming delivery to the terminal TE1. Thereby, the terminal TE1 performs streaming reception for receiving the shared information by one-way communication from the server.

FIG. 25C corresponds to the state of FIG.
In this case, since the terminal TE1 is in the sleep mode M2, the communication with the server is switched from bidirectional communication to unidirectional communication, and the unidirectional communication is streaming delivery from the server.
The terminal TE1 that performs streaming reception does not need to decode the received image as in bidirectional communication using a predetermined communication protocol, and the load on the terminal TE1 can be reduced.

FIG. 21 shows a flowchart of terminal grouping processing in the third embodiment.
The grouping process here mainly includes an idle state determination process, a sleep mode transition process, a representative terminal detection request process, and a streaming reception process.
The terminal that has shifted to the sleep mode M2 receives streaming distribution from the representative terminal when there is another terminal that should be the representative terminal in the same base, and if there is no terminal that should be the representative terminal, Becomes the representative terminal and receives streaming distribution from the server.

In step S101 in FIG. 21, it is assumed that the initial state of the terminal is the active mode M1.
In step S102, the terminal state notification unit 85 notifies the server of the current operation mode (active mode) of the terminal.
In step S103, the idle state determination unit 81 determines whether or not the state of its own terminal is in an idle state (idle state determination process).
FIG. 23 shows a detailed flowchart of the idle state determination process of the third embodiment. However, the flow of FIG. 23 is the same as that shown in FIG.

If it is determined in step S104 that the terminal is in an idle state, the process proceeds to step S105, and the terminal mode switching unit 86 performs a sleep mode transition process.
The process for shifting to the sleep mode is the same process as in step S5 described above.
In step S106, a terminal state notification indicating that the current operation mode is the sleep mode M2 is transmitted to the server.

In step S107, the representative terminal detection request unit 88 transmits a representative terminal detection request to the server.
The server that has received this request performs a process of detecting a representative terminal, and when a representative terminal is detected, transmits a notification indicating that the representative terminal has been detected to this terminal. On the other hand, when the representative terminal is not detected, a notification indicating that it has not been detected is transmitted to the terminal.
In step S108, it is checked whether or not a representative terminal has been detected by a notification from the server.
If a notification that a representative terminal has been detected is received, the process proceeds to step S109, and if not, the process proceeds to step S113.

In step S109, the streaming request unit 89 transmits information requesting streaming distribution to the representative terminal.
The representative terminal that has received the request information uses the streaming distribution unit 91 to distribute the streaming information to the terminal.

In step S110, the streaming receiver 90 receives streaming information sent from the representative terminal.
In step S111, the same representative terminal detection request process as in step S107 is performed again. This is a process for checking whether the terminal that has been the representative terminal until now is not the representative terminal or whether the representative terminal has not been changed.
In step S112, if the representative terminal is detected by the notification from the server, the process ends. If not, the process proceeds to step S113.

In step S113, since there is no other terminal that should be the representative terminal, a process of shifting itself to the representative terminal is performed. That is, the operation mode of the terminal is shifted from the sleep mode to the representative terminal mode.
The terminal that has become the representative terminal is a terminal that can perform streaming delivery to other terminals. Also, the terminal that has become the representative terminal receives the streaming information by one-way communication from the server.

In step S114, a terminal status notification indicating that the operation mode of the own terminal has become the representative terminal mode is transmitted to the server.
In step S115, streaming reception is performed to receive streaming information sent from the server.
The information received by streaming is displayed on the display unit of the representative terminal, and the received information is streamed to other terminals in the same base grouped.

FIG. 22 shows a schematic flowchart of the streaming distribution processing of the terminal which is the representative terminal in the third embodiment.
In step S121, it is checked whether a streaming distribution request has been received from another terminal.
If received, the process proceeds to step S122, and streaming distribution of the received streaming information is performed to the terminal that transmitted the request.

FIG. 24 shows a flowchart of the terminal state detection control process in the server of the third embodiment.
In step S131, the terminal state determination unit 162 checks whether or not terminal state information has been received from the terminal.
If it has been received, the process proceeds to step S132, and if not, step S131 is looped.
In step S132, it is checked whether or not the terminal state received from the terminal is in the active mode.
If it is in the active mode, the process proceeds to step S133. If not, the process proceeds to step S134.
In step S133, the communication method switching unit 163 switches the communication state between the server and the terminal to the bidirectional communication state performed in the active mode.
In step S134, it is checked whether or not the received terminal state is the sleep mode.
In the case of the sleep mode, the process proceeds to step S135. Otherwise, the process proceeds to step S139.

In step S135, the representative terminal detection unit 169 performs a representative terminal detection process.
Here, the terminal database 142 of the storage unit 141 is checked to check whether there is another terminal having the same base ID as the base ID 144 to which the terminal belongs.
When there is another terminal having the same base ID, if the terminal is in the active mode, the terminal is set as the representative terminal, and it is determined that the representative terminal has been detected.
That is, the representative terminal flag 150 of the terminal is set to “ON”.
Also, if there is a terminal having the same base ID that already has the representative terminal flag 150 “ON”, it is determined that the representative terminal has been detected.
Further, if there is no other terminal having the same base ID, it is determined that there is no representative terminal.

If a representative terminal is detected in step S136, the process proceeds to step S138, and if not, the process proceeds to step S137.
In step S138, the device ID and IP address of the representative terminal are read from the terminal database 142, a representative terminal detection notification is created, and transmitted to the terminal that has entered the sleep mode.
In step S137, information indicating that the representative terminal has not been detected is transmitted to the terminal that has entered the sleep mode.

In step S139, it is checked whether or not the received terminal state is the representative terminal mode.
If it is the representative terminal mode, the process proceeds to step S140, and if not, the process ends.
In step S140, a process of setting a terminal in the representative terminal mode as a representative terminal is performed. Specifically, the representative terminal flag 150 for the terminal in the terminal database 142 is set to “ON”.
In step S141, the streaming distribution unit 170 performs streaming distribution of shared information such as video and audio after processing such as decoding to the terminal serving as the representative terminal.
Note that the representative terminal flag 150 of the terminal is all set to “OFF” in the initial state and is not shown. However, for example, when the terminal is no longer the representative terminal due to power-off of the terminal, the representative terminal flag of the terminal Is set to “OFF”.

The above is the processing contents of the terminal and server in the third embodiment. When the number of terminals connected to the server is 5 or more, or when the number of terminals belonging to the same base (same group) is 3 or more. Therefore, the grouping pattern is not limited to those shown in FIGS. 26 to 30, and other patterns are also conceivable.
However, in either case, one representative terminal is set in the grouped terminal group, and only the representative terminal is set as a terminal that can be directly connected to the server so as to receive bidirectional communication and streaming distribution from the server. Therefore, the actual number of terminals connected to the server can be reduced. By reducing the number of connected terminals, transmission delay, communication failure, and server load can be reduced. Can do.

1 server 2 information terminal (terminal)
DESCRIPTION OF SYMBOLS 11 Input part 12 Keyboard 13 Mouse 14 Tablet 15 Human detection sensor 16 Input processing part 21 Video input part 22 Camera 23 Video processing part 31 Display part 32 Display 33 Display processing part 41 Audio | voice input part 42 Microphone 43 Input audio | voice processing part 45 Audio | voice output Unit 46 speaker 47 output voice processing unit 51 communication unit 52 communication encryption unit 53 communication decryption unit 55 authentication unit 61 storage unit 62 recognition database 63 user name 64 voice feature data 65 face image feature data 66 input video 67 input voice 71 signal processing unit 72 video encoding unit 73 video decoding unit 74 audio encoding unit 75 audio decoding unit 80 control unit 81 idle state determination unit 82 time measurement unit 83 input signal detection unit 84 person recognition unit 85 terminal state notification unit 86 terminal mode switching Part 87 Terminal mode change acceptance 88 Representative terminal detection request unit 89 Streaming request unit 90 Streaming reception unit 91 Streaming distribution unit 101 Input unit 102 Keyboard 103 Mouse 104 Tablet 105 Input processing unit 111 Display unit 112 Display 113 Display processing unit 121 Communication unit 122 Communication encryption unit 123 Number unit 125 Authentication unit 131 MCU unit 132 Video mixing unit 133 Video resize unit 134 Audio mixing unit 141 Storage unit 142 Terminal database 143 Device ID
144 Location ID
145 IP address 146 User database 147 User ID
148 mail address 149 recognition database 150 representative terminal flag 151 signal processing unit 161 control unit 162 terminal state determination unit 163 communication method switching unit 164 terminal mode control unit 165 idle state determination unit 166 time measurement unit 167 input signal detection unit 168 person recognition unit

Claims (12)

A server and a plurality of information terminals are connected via a network,
Sending information entered at the information terminal to a server;
The server is an information sharing system that generates shared information using information transmitted from each information terminal, and distributes the shared information to the information terminal,
An information input unit for generating state information indicating that the information terminal is in a use state of the information terminal;
Based on the state information generated by the information input unit, an idle state determination unit that determines that the information terminal is in an idle state that may not be used,
A terminal mode switching unit that changes the operation state in stages in response to the current operation state of the information terminal when it is determined that it is in an idle state;
A terminal state notifying unit for transmitting the current operating state of the information terminal to the server;
A representative terminal detection requesting unit for transmitting a detection request for requesting detection of a representative terminal in the same base to the server and receiving information of the representative terminal from the server, when it is determined that it is in an idle state;
A streaming receiving unit that receives streaming information distributed from a server or the representative terminal when it is determined that it is in an idle state;
A communication method switching unit that switches a communication method with the information terminal based on an operation state transmitted from the information terminal;
When a request for detection of a representative terminal is received from the first information terminal, the representative terminal is detected from information terminals belonging to the same base as the first information terminal, and the information on the representative terminal is obtained from the first information terminal. A representative terminal detector for transmitting to the information terminal;
A streaming distribution unit that transmits streaming information to the information terminal,
An information sharing system for changing a communication state between a server and each information terminal for each information terminal according to an operation state of the information terminal.   The streaming receiving unit of the first information terminal receives streaming information from the detected representative terminal when the representative terminal detecting unit can detect the representative terminal. Information sharing system. If the representative terminal detection unit fails to detect a representative terminal belonging to the same base as the first information terminal, a representative terminal non-detection notification is transmitted to the first information terminal,
When the first information terminal receives the non-detection notification, the terminal state notification unit of the first information terminal notifies the server that it is a representative terminal, and then the streaming reception unit The information sharing system according to claim 1, wherein streaming information is received from a server.   The representative terminal in the same base is selected from information terminals set in advance as information terminals belonging to the same group as the first information terminal. Information sharing system.   The information terminals belonging to the same group are any one of an information terminal having the same base ID, an information terminal connected to the same router and belonging to a network under the router, and an information terminal in the same domain. The information sharing system according to claim 4.   6. The information sharing system according to claim 5, wherein the representative terminal is an information terminal that has the same base ID as that of the first information terminal and is not in an idle state. The operating state of the information terminal includes a first mode in which bi-directional communication of video and audio with the server, a second mode in which information including video and audio is received from the server and one-way communication is performed, and main functions of the information terminal. Including a third mode in which communication with the server is stopped and a fourth mode in which the information terminal is powered off,
The operation state of the information terminal is changed stepwise in the order from the first mode to the fourth mode every time the idle state is detected by the idle state determination process. Any information sharing system. The information input unit
An input unit that inputs an input signal by a user operation, a video input unit that inputs video captured by a camera, and an audio input unit that inputs audio from a microphone;
8. The information sharing system according to claim 1, wherein the state information is generated using the input signal, input video, and input sound. The idle state determination unit
If it is detected during the predetermined set time that either the input signal is not input, the input audio is not input, or the input video is not changed, the information 9. The information sharing system according to claim 8, wherein it is determined that the terminal is in an idle state that may not be used. A storage unit storing a recognition database including voice feature data of the user and facial image feature data indicating the features of the user's face;
A person recognition unit for performing person recognition for identifying a user by using voice recognition using the input voice and voice feature data and image recognition using the input video and face image feature data; Prepared,
The idle state determination unit determines that the information terminal is in an idle state that may not be used when person recognition fails by the person recognition unit continuously during a predetermined set time. 10. The information sharing system according to claim 8 or 9, wherein: A server and a plurality of information terminals are connected via a network,
Sending information entered at the information terminal to a server;
An information sharing method of an information sharing system in which the server generates shared information using information transmitted from each information terminal and distributes the shared information to the information terminal,
An information input step in which the information terminal generates state information indicating that the user is using the information terminal;
An idle state determination step for determining whether the information terminal is in an idle state that may not be used based on the state information;
A terminal mode switching step of changing to a predetermined operation state in a stepwise manner in response to the current operation state of the information terminal when it is determined to be in an idle state;
A terminal state notification step of transmitting the current operating state of the information terminal to the server;
A detection request transmission step in which the first information terminal determined to be in an idle state requests the server to detect a representative terminal in the same base;
A step of detecting a representative terminal from among information terminals belonging to the same base as the first information terminal that has transmitted the detection request when the server receives the detection request for the representative terminal; Transmitting information of a representative terminal to the first information terminal;
A distribution request transmission step for requesting streaming distribution to the representative terminal by using the information of the representative terminal received by the first information terminal;
And a step of transmitting the streaming information to the first information terminal based on the received distribution request.   An information sharing system program for causing a computer to execute each step of the information sharing method according to claim 11.

Images