JP2015192230A - Conference system, conference server, conference method, and conference program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow effective band control to be guaranteed when a plurality of conference sessions use limited bandwidth at the same time.SOLUTION: In a conference system in which a first conference server and a plurality of second conference servers are cascade connected, the first conference server comprises first bandwidth control means for controlling communication bandwidth allocated to each of a plurality of conference sessions on the basis of conference information on each conference related to the plurality of conference sessions when the use bandwidth of the plurality of established conference sessions exceeds communication bandwidth with the plurality of second conference servers, and each of the plurality of second conference servers comprises second bandwidth control means for controlling communication bandwidth with a terminal participating in each conference on the basis of the communication bandwidth allocated by the first conference server.

Description

  The present invention relates to a conference system, a conference server, a conference method, and a conference program. For example, the present invention can be applied to a conference system, a conference server, a conference method, and a conference program in a multipoint video conference server on a packet network.

  For example, a multipoint video conferencing server (MCU: Multipoint Control Unit) selects and synthesizes audio and video transmitted from a plurality of terminals on a network and transfers them to each terminal, thereby allowing a video conference between multipoints. It is a device that makes it possible to perform. By configuring the MCU in a cascade configuration including one master server and a plurality of slave servers and connecting the terminals to each slave server, a large-scale video conference in which many terminals participate can be performed.

  In addition, video conferencing communication uses a lot of bandwidth, so network congestion is likely to occur, which causes quality degradation such as sound interruptions and video disturbances due to delay and packet loss. In contrast, video conference terminals and servers have a bandwidth control function that limits the rate of packets to be transmitted to a predetermined value or dynamically changes the transmission rate according to the packet loss rate. Is provided.

  Related to this, there are the following prior arts.

  Non-Patent Document 1 discloses a method in which a terminal dynamically performs transmission rate control (ARC) according to a packet loss rate, a round trip delay time, and the like.

  Also, the technology described in Patent Document 1 uses scalable video coding (SVC) that can encode video at different rates into one video data between the servers configured in the cascade configuration described above. Even when the bandwidths are different, communication is possible with one video data stream.

  In addition, as a technique related to effective use of resources of the conference system, there are the following conventional techniques.

  The technology described in Patent Document 2 is to effectively use conference resources by controlling reception of a request and termination of an existing conference according to the status of an existing conference when a new conference start request is made. is there.

Special table 2011-525770 gazette JP 2009-239714 A

RFC 5348 “TCP Friendly Rate Control (TFRC): Protocol Specification”

  However, these conventional bandwidth control technologies are limited to individual video conference sessions and do not guarantee effective control when multiple video conference sessions use limited bandwidths simultaneously. Absent.

  The simplest method that does not cause congestion in the network is to limit the bandwidth that can be used in each video conference session to be sufficiently small in advance. However, in that case, it is necessary to assume the maximum session to be used at the same time, and there is a problem in that it is not efficient, for example, it is necessary to always reserve a bandwidth that is not actually used. In the technique described in Patent Document 1, different bandwidths can be set between each master server and slave server, but there are similar problems when considering a plurality of sessions.

  Further, although the rate described in Non-Patent Document 1 can be used to perform dynamic rate control for one video conference session, it cannot be coordinated in a plurality of sessions. Therefore, if a specific session occupies most of the bandwidth and the remaining sessions can only be used for low-quality conferences, or if the rate control for multiple sessions is performed in the same way, all sessions increase the transmission rate simultaneously. The problem is that quality is not stable due to lowering or lowering.

  Furthermore, the technology described in Patent Document 2 enables a new conference to be opened by ending an existing conference that is less useful when a new conference start request is made. There is a problem that many conferences cannot be used simultaneously.

  In a cascade configuration, servers are often separated from each other in a network, and congestion is likely to occur. However, when multiple video conferences are held as described above, there is currently no effective bandwidth control method. .

  Therefore, there is a need for a conference system, a conference server, a conference method, and a conference program that can guarantee effective bandwidth control when a plurality of video conference sessions simultaneously use a limited bandwidth.

  In order to solve this problem, the first aspect of the present invention provides a conference system in which a first conference server is connected to a plurality of second conference servers by cascade connection and performs processing related to a plurality of conferences. ) Based on the conference information of each conference related to the plurality of conference sessions, when the first conference server uses the bandwidth of the plurality of conference sessions established exceeds the communication bandwidth between the plurality of second conference servers. And a first band control means for controlling a communication band allocated to each of the plurality of conference sessions, (2) each of the plurality of second conference servers is based on the communication band allocated by the first conference server. And a second band control means for controlling a communication band between terminals participating in each conference.

  According to a second aspect of the present invention, in a conference server that performs processing related to a plurality of conferences in a cascade connection with a plurality of conference servers, the use bandwidth of the plurality of established conference sessions is between the plurality of second conference servers. A conference server comprising bandwidth control means for controlling a communication bandwidth allocated to each of a plurality of conference sessions based on conference information of each conference related to a plurality of conference sessions when the communication bandwidth exceeds .

  According to a third aspect of the present invention, in the conference server cascade-connected to the first conference server serving as the master server, a communication bandwidth is allocated to each of the plurality of conference sessions based on the conference information of each conference related to the plurality of conference sessions. A conference server comprising bandwidth control means for controlling a communication bandwidth with a terminal participating in each conference based on a communication bandwidth assigned to each conference session acquired from one conference server. .

  The fourth aspect of the present invention is a conference method in which the first conference server is connected to a plurality of second conference servers by cascade connection, and performs processing related to a plurality of conferences. (1) The first conference server includes: When the use bandwidth of the plurality of established conference sessions exceeds the communication bandwidth with the plurality of second conference servers, each of the plurality of conference sessions is based on the conference information of each conference related to the plurality of conference sessions. (2) Each of the plurality of second conference servers controls the communication bandwidth between the terminals participating in each conference based on the communication bandwidth allocated by the first conference server. It is the meeting method characterized by doing.

  According to a fifth aspect of the present invention, there is provided a conference program of a conference server that performs processing related to a plurality of conferences in a cascade connection with a plurality of conference servers. When it exceeds the communication bandwidth with the conference server, the bandwidth control unit controls the communication bandwidth allocated to each of the plurality of conference sessions based on the conference information of each conference related to the plurality of conference sessions. It is a featured conference program.

  According to a sixth aspect of the present invention, in the conference program of the conference server that is cascade-connected to the first conference server serving as the master server, the computer is connected to each of the plurality of conference sessions based on the conference information of each conference related to the plurality of conference sessions. To function as a bandwidth control means for controlling the communication bandwidth between the terminals participating in each conference based on the communication bandwidth assigned to each conference session acquired from the first conference server that allocates the communication bandwidth to It is a featured conference program.

  According to the present invention, effective bandwidth control can be ensured when a plurality of video conference sessions simultaneously use a limited bandwidth.

It is a block diagram which shows the structure of the multipoint video conference system which concerns on 1st Embodiment. It is explanatory drawing explaining transmission of the "image P" from the terminal to the direction of a master server via the slave server which concerns on 1st Embodiment. It is explanatory drawing explaining transmission of the "image P" from the master server to the direction of a terminal via the slave server which concerns on 1st Embodiment. It is an internal block diagram which shows the internal structure of MCU of the master server which concerns on 1st Embodiment. It is a block diagram which shows the main structures of the band control function of the control part which concerns on 1st Embodiment. It is a sequence diagram which shows the operation | movement of the band control process in the multipoint conference system which concerns on 1st Embodiment (the 1). It is a sequence diagram which shows the operation | movement of the band control process in the multipoint conference system which concerns on 1st Embodiment (the 2). It is a sequence diagram which shows the operation | movement of the band control process in the multipoint conference system which concerns on 1st Embodiment (the 1). It is a sequence diagram which shows the operation | movement of the band control process in the multipoint conference system which concerns on 1st Embodiment (the 2). It is a block diagram which shows the structure of the encoding part (encoder) of the video signal processing of MCU with which the master server which concerns on 2nd Embodiment is provided. It is explanatory drawing explaining the relationship between the video data of a base layer and the video data of an enhancement layer in the H264 / SVC encoder which concerns on 2nd Embodiment. It is explanatory drawing explaining an example of the multiplexed video data output to the MPEG2-TS production | generation part from the H264 / SVC encoder which concerns on 2nd Embodiment. It is explanatory drawing explaining the relationship between the image quality in 2nd Embodiment, the video data of a base layer, and the video data of an enhancement layer. H. according to the second embodiment. 3 is a flowchart showing an operation of a master server in which an H.264 / SVC encoder is incorporated. It is explanatory drawing explaining an example of the multiplexed video data output to a server slave server from a master server based on Fig.12 (a). It is a block diagram which shows the structure of MCU of the slave server which concerns on 2nd Embodiment. It is explanatory drawing explaining the operation | movement which changes the band allocation in the multipoint video conference system which concerns on 2nd Embodiment (the 1). It is explanatory drawing explaining the operation | movement which changes the band allocation in the multipoint video conference system which concerns on 2nd Embodiment (the 2).

(A) First Embodiment Hereinafter, a first embodiment of a conference system, a conference server, a conference method, and a conference program according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of First Embodiment FIG. 1 is a configuration diagram showing a configuration of a multipoint video conference system according to the first embodiment.

  In FIG. 1, a multipoint video conference system 100 according to the first embodiment includes a master server M, a slave server A, a slave server B, and a plurality of terminals a11, a12, a21, b11, b12, and b21.

  In the multipoint video conference system 100 according to this embodiment, a master server M, a slave server A, and a slave server B are cascade-connected. Further, the number of slave servers, the number of terminals, and the number of conference rooms to be held are not particularly limited.

  Each of the terminals a11, a12, a21, b11, b12, and b21 is a terminal that connects to the slave server A or the slave server B and participates in the conference, and those used in the existing video conference system can be applied. .

  The master server M includes a gatekeeper 31 that performs connection permission and address management of terminals connected to one or more conference rooms to be opened, and an MCU 32 that performs mutual conversion of audio data and video data in a multipoint conference. The master server M combines the video data received from the slave servers A and B for each conference room, and transmits the combined video data to a terminal connected to each conference room.

  Note that the master server M may collect and synthesize audio data from the terminals in each conference room in accordance with the video data, and transmit it. In addition, the terminal that holds each conference room may be connected to the slave server A, the slave server B, the master server M, and the call by using SIP. Calls may be connected using a H.323 compliant system.

  Each of the slave server A and the slave server B is connected to one or a plurality of terminals and is connected to the master server M, and each terminal that participates in each conference room with the audio, video, and data of the held virtual conference room And the master server M.

  The slave server A and the slave server B have a gatekeeper 31 that performs conference connection permission and address management of terminals participating in each conference room to be held. In addition, the slave server A and the slave server B transmit image data received from a terminal connected to each conference room to the master server M, or the synthesized data relating to the conference room received from the master server M is transmitted to each conference room. Or transmitted to a terminal connected to.

  Here, it is assumed that the terminal a11, the terminal a12, and the terminal a21 are connected to the slave server A. Also, it is assumed that the terminal b11, the terminal b12, and the terminal b21 are connected to the slave server B.

  Further, it is assumed that two video conferences of the conference room 1 and the conference room 2 are held on the multipoint video system 100. Assume that terminal a11, terminal a12, terminal b11, and terminal b12 participate in conference room 1, respectively. It is assumed that the terminal a21 and the terminal b21 participate in the conference room 2, respectively.

  When the conference room 1 is being held, the audio data and video data sessions are “between master server M and slave server A”, “between master server M and slave server B”, and “between slave server A and terminal a11”. , “Between slave server A and terminal a12”, “between slave server B and terminal b11”, and “between slave server B and terminal b12”.

  Similarly, when the conference room 2 is being held, the audio data and video data sessions are “between master server M and slave server A”, “between master server M and slave server B”, “slave server A—terminal”. ab "and" between slave server B and terminal b21 ".

  FIG. 2 is an explanatory diagram illustrating transmission of “image P” from the terminal toward the master server M via the slave servers A and B according to the first embodiment.

  In FIG. 2, “P1” of “terminal a11” is transmitted from “terminal a11” to “slave server A”. “P2” of “terminal a12” is transmitted from “terminal a12” to “slave server A”. “P5” of “terminal a21” is transmitted from “terminal a21” to “slave server A”. “P1” of “terminal a11” and “P5” of “terminal a21” are transmitted from “slave server A” to “master server M”.

  Also, “P3” of “terminal b11” is transmitted from “terminal b11” to “slave server B”. “P4” of “terminal b12” is transmitted from “terminal b12” to “slave server B”. “P6” of “terminal b21” is transmitted from “terminal b21” to “slave server B”. “P3” of “terminal b11” and “P6” of “terminal b21” are transmitted from “slave server B” to “master server M”.

  In this embodiment, for the conference room 1, the slave server A transmits the “P1” of the “terminal a11” to the master server M among the “terminal a11” and “terminal a12” to be connected. However, the image transmitted to the master server M is not particularly limited, and “P1” of “terminal a11” and “P2” of “terminal a12” may be transmitted to the master server M. Alternatively, “P2” of “terminal a12” may be transmitted to the master server M. The same applies to the image transmitted to the master server M for the conference room 1 of the slave server B.

  FIG. 3 is an explanatory diagram illustrating transmission of “image P” from the master server M to the terminal via the slave servers A and B according to the first embodiment.

  In FIG. 3, “the composite image of P1 and P3” of the conference room 1 and “the composite image of P5 and P6” of the conference room 2 are transmitted from “master server M” to “slave server A”. “Slave server A” → “terminal a11” is transmitted with “P1 and P3 composite image” of conference room 1, and “slave server A” → “terminal a12” is transmitted with “P1 and P3 of conference room 1”. "Synthesized image" of the conference room 2 is transmitted to "slave server A" → "terminal a21".

  In addition, “the composite image of P1 and P3” of the conference room 1 and “the composite image of P5 and P6” of the conference room 2 are transmitted from “master server M” to “slave server B”. “Slave Server B” → “Terminal b11” is transmitted “P1 and P3 composite image” of conference room 1, and “Slave Server B” → “Terminal b12” is “P1 and P3 composite of conference room 1”. "Image" is transmitted, and "Slave server B"-> "Terminal b21" is transmitted "P5 and P6 composite image" of conference room 2.

  FIG. 4 is an internal configuration diagram illustrating an internal configuration of the MCU 32 of the master server M according to the first embodiment.

  In FIG. 4, the MCU 32 of the master server M includes a control unit 56, data receiving units 51-1 and 51-2, decoding units 52-1 and 52-2, a combining unit 53, and encoding units 54-1 and 54-2. And data transmission units 55-1 and 55-2.

  The MCU 32 included in the master server M includes a video signal processing device 32A that processes a video signal. Note that the MCU 32 may include an audio signal processing device, a data signal processing device, and the like, similarly to the existing MCU.

  The video signal processing unit 32A is, for example, a video processing program according to the embodiment on an information processing apparatus such as a personal computer (not limited to one, but may be configured to be capable of distributed processing of a plurality of units). It may be constructed by installing (including fixed data), and even in that case, it can be expressed functionally as shown in FIG. Further, some or all of the components of the video signal processing device 32A may be constructed as dedicated hardware.

  In FIG. 4, the video signal processing device 32A includes a data receiving unit 51 (51-1 and 51-2), a decoding unit 52 (52-1 and 52-2), a synthesizing unit 53, and an encoding unit 54 (54-1). And 54-2), a data transmission unit 55 (55-1 and 55-2), and a control unit 56. FIG. 4 shows a configuration for realizing one conference session to be held. Therefore, when a plurality of conference sessions are held simultaneously, the configuration of the data reception unit 51, the decoding unit 52, the synthesis unit 53, the encoding unit 54, and the data transmission unit 55 of FIG. It shall be prepared according to the number.

  The data receiving units 51-1 and 51-2 receive video data from the slave servers A and B, and give the received video data to the corresponding decoding units 52-1 and 52-2. The data reception units 51-1 and 51-2 are provided in a number corresponding to the number of slave servers constituting the conference room. For example, in the case of the conference room 1, the data receiving unit 51-1 receives image data from the slave server A, and the data receiving unit 51-2 receives image data from the slave server B. .

  The decoding units 52-1 and 52-2 decode the video data from the data receiving units 51-1 and 51-2 and give the decoded video data to the synthesizing unit 53. The decoding units 52-1 and 52-2 are provided in a number corresponding to the number of data receiving units 51-1 and 51-2.

  The synthesizing unit 53 acquires the video data decoded from the decoding units 52-1 and 52-2, synthesizes a plurality of video data, and supplies the synthesized video data to the encoding units 54-1 and 54-2. In general, the number of the combining units 53 according to the number of conference rooms is provided. Note that the composition unit 53 may compose an image having a layout according to the conference room to be held. For example, when one frame is divided into two and two images are combined, the combining unit 53 inserts the two images acquired from the decoding units 52-1 and 52-2 into the frame and performs image combining. good. The frame layout is not particularly limited, and may be divided into three or more.

  The encoding units 54-1 and 54-2 encode the combined image data from the combining unit 53 and provide it to the data transmission units 55-1 and 55-2. One or more encoding units 54 are created for the conference room. The encoding unit 54 may be newly created or deleted based on the control of the control unit 56. Thus, the encoding units 54-1 and 54-2 can be dynamically changed under the control of the control unit 56. Note that as the encoding parameters instructed by the control unit 56, the video encoding codec type, the image size, the encoding bit rate, the video frame rate, and the frame type (intra frame / inter frame) are applied. The encoding parameters may be applied to some of the parameters described above, or other parameters may be applied, and the number and combination of encoding parameters to be applied are not limited.

  The data transmission units 55-1 and 55-2 transmit the composite image data encoded by the encoding units 54-1 and 54-2 to the slave servers A and B. The data transmission units 55-1 and 55-2, the data reception units 51-1 and 51-2, the number according to the number of slave servers constituting the conference room is provided. For example, in the case of the conference room 1, the data transmission unit 55-1 transmits the composite image data to the slave server A, and the data transmission unit 55-2 transmits the composite image data to the slave server B. .

  The control unit 56 controls the function of the image signal processing device 32A of the MCU 32, and controls the holding of each conference room, encoding processing, and the like. The control unit 56 can be used in each conference session based on the bandwidth between the slave servers A and B to which the master server M is connected and the number of conference sessions flowing in the bandwidth between the slave servers A and B. A bandwidth control unit for allocating bandwidth is included.

  FIG. 5 is a block diagram illustrating a main configuration of the bandwidth control function of the control unit 56 according to the first embodiment. As shown in FIG. 5, the bandwidth control function of the control unit 56 includes a bandwidth control processing unit 65, a conference session number equal allocation processing unit 61, a conference session weight allocation processing unit 62, a conference session participation number allocation processing unit 63, a conference A session holding time allocation processing unit 64 is provided.

  The bandwidth control processing unit 65 controls the allocated bandwidth of the conference session. The bandwidth control processing unit 65 manages a bandwidth that can be used with the slave servers A and B to be connected. The bandwidth control processing unit 65 is used in each conference session based on conference information such as the bandwidth between the slave servers A and B and the number of conference sessions flowing between the slave servers A and B. It limits the bandwidth allocation.

  The bandwidth control processing unit 65 can apply various bandwidth allocation methods. In the first embodiment, the bandwidth control processing unit 65 exemplifies the following four types of bandwidth allocation methods based on conference information relating to a plurality of conference sessions.

  Specifically, the bandwidth control processing unit 65 may apply a conference session number equal allocation processing unit 61, a conference session weight allocation processing unit 62, a conference session participation number allocation processing unit 63, and a conference session holding time allocation processing unit 64. it can. The band control processing unit 65 may apply any one of the four band allocation methods described above, or may combine any one of the four band allocation methods. A plurality of bandwidth allocation methods or all bandwidth allocation methods may be provided.

  The conference session number equal allocation processing unit 61 is a method of equally allocating the bandwidth between the master server M and the slave servers A and B by the number of conference sessions. As a result, even when a plurality of conference sessions are held simultaneously, the bandwidth between the master server M and the slave servers A and B can be shared according to the number of conference sessions. Can do. Specifically, the bandwidth control processing unit 65 recognizes a bandwidth that can be used between the master server M and the slave servers A and B in advance. Moreover, the conference identification information (for example, conference ID etc.) which identifies the conference session currently held is recognized. The conference session number equal allocation processing unit 61 confirms the number of conference sessions based on the conference identification information, and sets the bandwidth of each conference session by dividing the bandwidth between servers by the number of conference sessions.

  The conference session weight assignment processing unit 62 assigns a predetermined weight to the bandwidth used in the plurality of conference sessions, and apportions the bandwidth between the master server M and the slave servers A and B based on the weight. This is a method for setting the bandwidth of each conference session. For example, there is something that should secure a certain band of a conference session such as an executive meeting. In this way, weights are assigned in advance according to the type of conference for which bandwidth should be secured, and the bandwidth of each conference session is set based on the weight corresponding to the type of conference to be held when the conference room is held. . Thereby, the bandwidth of the conference session according to the importance of the conference type can be secured.

  The conference session participation number allocation processing unit 63 divides the bandwidth between the master server M and the slave servers A and B based on the number of participants participating in each conference room, and sets the bandwidth for each conference session. It is. The bandwidth used in a conference session tends to increase with the number of participants in the conference room. Therefore, when there are multiple conference sessions, the bandwidth is allocated fairly by dividing the bandwidth according to the number of participants in each conference room. Assignment can be made. Specifically, the bandwidth control processing unit 65 controls terminal identification information (for example, CALL-ID) participating in each conference room for each conference identification information (conference ID, etc.). Know the number of participants. Therefore, the conference session participation number allocation processing unit 63 can set the bandwidth of each conference session by dividing the bandwidth between the servers by the number of participants.

  When there are a plurality of conference sessions, the conference session holding time allocation processing unit 64 reduces the weight of a plurality of conference sessions whose holding time has passed a predetermined time from the other conference sessions, This is a method for setting the bandwidth of a conference session. That is, it is possible to reduce the bandwidth of a conference with a long holding time and increase the bandwidth of a conference with a short holding time. It should be noted that the time for determining the conference opening time can be set as appropriate, and the weight of the conference session that passes the predetermined time can also be set arbitrarily.

(A-2) Operation of the First Embodiment In the following, in the multipoint conference system 100 according to the first embodiment, when a plurality of conference sessions occur between the master server and the slave server, the state is considered. A process for controlling the master server M so that the bandwidth can be used fairly will be described.

  Here, in order to simplify the description, a bandwidth of 3 Mbps is set for “between master server M and slave server A”, “between master server M and slave server B”, and “between slave servers A and B and the terminal”. It shall be usable. In addition, it is assumed that the data of each conference session uses a maximum bandwidth of 2 Mbps.

  At this time, if only “Conference Room 1” is held, there is no problem even if the session of “Conference Room 1” uses a bandwidth of 2 Mbps. However, when “Conference Room 2” is subsequently held, if the conference session of “Conference Room 1” and the conference session of “Conference Room 2” use 2 Mbps, respectively, “Master Server M-Slave Server A” A total of 4 Mbps data is about to flow between “Between” and “Between master server M and slave server B”. Then, since the bandwidth between “between master server M and slave server A” and “between master server M and slave server B” is 3 Mbps, congestion occurs, and the quality of the two video conferences deteriorates due to packet loss or delay. End up.

  Therefore, the master server M allocates and limits the bandwidth that can be used in each conference session based on the bandwidth that can be used between the slave servers A and B and the number of conference sessions that are flowing between them.

  As described above, any one of the four types of bandwidth allocation methods or a combination of a plurality of bandwidth allocation methods can be applied to the bandwidth allocation method for each conference session in the master server M.

(1) Allocate evenly according to the number of conference sessions (2) Allocate according to a predetermined ratio (weight) (3) Allocate according to the number of participants in the conference (4) Allocate according to the conference opening time When applied in:

  In the method (1), the bandwidth 3 Mbps of “between the master server and the slave server A” and “between the master server and the slave server B” is equally allocated in two conference sessions, so that “conference room 1” and “conference room” Allocate 1.5 Mbps to each conference session of “2”.

  In the method (2), it is assumed that the bandwidth allocation weights of “conference room 1” and “conference room 2” are set to “5” and “1”, respectively. For the weight of the conference room, for example, a setting table in which the weight is set in advance for each conference room according to the importance of the conference room may be used, or the weight can be arbitrarily set when the conference room is held. Anyway. In this case, a bandwidth of 2.5 Mbps is allocated to the conference session of “Conference Room 1”, and a bandwidth of 0.5 Mbps is allocated to the conference session of “Conference Room 2”.

  In the method (3), for example, four people are participating in “conference room 1” and two people are participating in “conference room 2”, so “between master server and slave server A”, “master server and slave server” A bandwidth of 3 Mbps is assigned to the conference session of “Conference Room 1” and a bandwidth of 1 Mbps is assigned to the conference session of “Conference Room 2”.

  In the method (4), for example, a meeting whose holding time exceeds 2 hours is assigned half the weight of a meeting within 2 hours. In this case, if the duration of “Conference Room 1” exceeds 2 hours when “Conference Room 2” is held, a bandwidth of 1 Mbps is allocated to the conference session of “Conference Room 1”, and “Conference Room 2” A bandwidth of 2 Mbps is allocated to the conference session.

  The above-described bandwidth allocation may be performed dynamically during a conference as well as during a new conference.

  For example, in the method (3), the bandwidth of each conference session is allocated according to the number of participants in each conference room. Participation in each conference room can be entered and exited as appropriate, so the bandwidth allocated to each conference session may be dynamically changed each time a participant enters or exits each conference room after the conference is held. .

  Further, for example, in the method (4), the bandwidth allocation may be changed every time the conference time of each conference room exceeds a predetermined time. For example, the bandwidth allocated to each conference session is adjusted so that the weight is reduced every time the conference holding time elapses a predetermined time, such as 30 minutes elapses, 1 hour elapses, 2 hours elapse, etc. It may be changed as desired.

  An example of a combination of the above band allocation methods will be described. For example, the case where the method (2) and the method (3) are combined is illustrated. For example, the bandwidth allocation weight of “conference room 1” of an important meeting such as an executive meeting and “conference room 2” of another meeting is “2” vs. “1”. Then, a bandwidth of 2 Mbps is allocated to the conference session of “conference room 1” such as an executive meeting, and a bandwidth of 1 Mbps is allocated to the conference session of “conference room 2”. After that, another “meeting room 3” is held. At this time, it can be set not to limit the bandwidth to “Conference Room 1”, the bandwidth of “Conference Room 1” is unchanged, and the bandwidth of the conference session assigned to “Conference Room 2” is 1 Mbps. Bands corresponding to the number of participants may be allocated to the conference sessions “2” and “conference room 3”.

  Furthermore, there is a data sharing mechanism in video conferencing, and an additional bandwidth may be required when sharing data. In this case, as an extension of the method (2), control may be performed so that the weight is dynamically increased when data sharing is performed in a conference.

  When the master server M determines the bandwidth allocated to each conference session, the master server M notifies the slave servers A and B and each terminal of the determined bandwidth and changes the bandwidth used in the conference session. This is for example the H.264 video conferencing protocol. When H.323 is used, H.323 is used. It is possible to control by a standard method such as using a flow control command of H.245.

  In the above description, the bandwidths used by the conference session “between master server M and slave server A” and “between master server M and slave server B” are all the same. However, the bandwidths may be different. This can be realized by encoding video data at different rates for each of the slave servers A and B by the master server M.

  Even if the master server M does not strictly recognize the bandwidth that can be used between “between the master server M and the slave server A” and “between the master server M and the slave server B”, the master server M The band can be estimated dynamically by observing a loss or the like, and the same method can be applied even in that case.

  Furthermore, even if the cascade configuration has three or more layers, the same method can be applied between each master server and slave server and between each parent slave server and child slave server.

(A-2-1) Band control processing operation (1)
6 and 7 are sequence diagrams illustrating the operation of the bandwidth control process in the multipoint conference system 100 according to the first embodiment. 6 and 7 illustrate the case where SIP is used as the conference protocol. 6 and 7 exemplify a case where the bandwidth of each conference session is allocated by the method (4) for allocating conference session time.

(Start of conference room 1)
In FIG. 6, first, when the terminal a11 enters the conference room 1, the terminal a11 transmits to the slave server A an INVITE message (initial INVITE request) whose destination is the URI of “conference room 1” (S101). .

  In this INVITE request, session information describing media information such as audio and video used in the video conference is described as SDP (Session Description Protocol). Here, assuming that the bandwidth usable by the terminal a11 is 2 Mbps, the bandwidth information is described in the session information as “b = CT: 2000” (b: bandwidth, CT: Conference Total).

  On the other hand, the slave server A returns a 200 OK response message to the terminal a11 as a response that can participate in the conference room (S102).

  In this 200 OK message, the session information used in the conference session is described as SDP, and since the bandwidth allocated to “conference room 1” is 2 Mbps, “b = CT: 2000” is described as the bandwidth information. When this response reaches the terminal a11, the terminal a11 transmits an ACK message to the slave server A (not shown in FIG. 6), and a conference session between the terminal a11 and the slave server A is established.

  When the first participant (terminal a11) enters the “conference room 1”, the slave server A transmits an INVITE request similar to the request received from the terminal a11 to the master server M (S103). A 200 OK response (S104) is returned to server A. As a result, the slave server A connects to the conference room of the master server M. A session of “Conference Room 1” is established between the slave server A and the master server M, with a bandwidth of 2 Mbps allocated. Since there are a plurality of conference sessions between slave server A and master server M, Call-ID header information is also described in FIG. 6 to distinguish the sessions. Here, it is shown that the message including “Call-ID: 11 @ a” as the Call-ID header information of the conference room 1 relates to the session of the conference room 1 between the slave server A and the master server M. Yes.

  At this time, the master server M records the holding start time of the “conference room 1” and starts measuring the holding time of the “conference room 1”.

  Next, when the terminal a12 enters the “conference room 1”, the same operation as the processing of S101 to S102 is performed, and the conference session between the terminal a12 and the slave server A is established (S105, S106). Similarly, when the terminal b11 and the terminal b12 enter the conference room 1 of the slave server B, the same operation as described above is performed, and the conference session between the terminal b11 and the slave server B, the slave server B and the master server M The conference session between the terminal b12 and the slave server B is established (S117 to S122).

(Start of conference room 2)
The terminal a21 enters the “conference room 2” of the slave server A. When the terminal a21 enters the “conference room 2”, the terminal a21 transmits an INVITE message whose destination is the URI of the “conference room 2” to the slave server A (S107). The server A returns a 200 OK response message to the terminal a21 as a response that allows participation in the conference room (S108).

  Also at this time, since the bandwidth that can be used by the terminal a12 is 2 Mbps, the bandwidth information “b = CT: 2000” is described in the session information of the INVITE request. Since the bandwidth allocated to “Conference Room 2” is 2 Mbps, “b = CT: 2000” is described as bandwidth information in the 200 OK message.

  When the first participant (terminal a21) enters the “conference room 2”, the slave server A transmits an INVITE request similar to the request received from the terminal a21 to the master server M (S109). Returns a 200 OK response (S110) to the slave server A. As a result, a “conference room 2” session is established between the slave server A and the master server M. Note that “Call-ID: 22 @ a” indicates that these messages relate to the session of “conference room 2” between the slave server A and the master server M.

  At this time, since there are a plurality of conference sessions between the master server M and the slave servers A and B, the master server M allocates a bandwidth to each conference session.

(Change of bandwidth allocation on the slave server A side)
Here, it is assumed that, in the method for assigning the holding time of (4), for example, a conference session exceeding 2 hours is allocated with a conference session bandwidth with half the weight of a conference within 2 hours.

  When the session of “Conference Room 2” is established in S110, if the holding time of “Conference Room 1” started in S104 exceeds 2 hours, the master server M changes to the session of “Conference Room 1”. Assigns a bandwidth of “1 Mbps” and “2 Mbps” to the “conference room 2” session. Since the bandwidth allocation of “Conference Room 2” is not changed at “2 Mbps”, the master server M performs only the bandwidth change processing of “Conference Room 1”.

  First, in FIG. 7, the master server M displays to the slave server A an INVITE message (“INVITE request” in FIG. 7) for updating the content of the session “conference room 1”. The INVITE request related to the above is called “re-INVITE request”) (S111). At this time, the master server M transmits a re-INVITE request in which “b = CT: 1000” is described in the SDP of the session information to the slave server A in order to change the bandwidth of the “conference room 1” to 1 Mbps.

  The slave server A that has received this re-INVITE request transmits a 200 OK response together with the SDP described as “b = CT: 1000” as a response (S112). Thereafter, the conference session regarding “conference room 1” between the slave server A and the master server M is exchanged in a bandwidth of 1 Mbps.

  Further, the slave server A transmits a re-INVITE request in which “b = CT: 1000” is described in the SDP of the session information to the terminal a11 and the terminal a12 participating in the “conference room 1”. The bandwidth between A and the terminals (terminal a11, terminal a12) is also changed to 1 Mbps (S113 to S116).

  Accordingly, the master server M can change the bandwidth between the master server M and the slave server A and the bandwidth between the slave server A and the terminal with respect to the “conference room 1”. On the other hand, regarding the “conference room 1”, the master server M does not change the bandwidth allocation at this point in relation to the slave server B. Therefore, the master server M allocates a bandwidth of 2 Mbps to the conference session between the master server M and the slave server B and between the slave server B and the terminal.

  Thereafter, when the terminal b21 enters the “conference room 2” of the slave server B, the bandwidth between the terminal and the slave server B is 2 Mbps, and the bandwidth between the slave server B and the master server M is 2 Mbps. (S123-S132).

  That is, when the terminal b21 enters the conference room 2 of the slave server B (S123, S124), a session of the conference room 2 is established between the slave server B and the master server M (S125, S126). “Call-ID: 22 @ b” indicates that these messages relate to the session “conference room 2” between the slave server B and the master server M.

  At this time, since there are a plurality of conference sessions between the master server M and the slave server B, the master server M allocates a bandwidth to each conference session. Here, in the same way as described above, it is assumed that, for example, a conference exceeding 2 hours is assigned half the weight of a conference within 2 hours by a holding time allocation method.

  Here, the start time of “Conference Room 1” is started in S104, and when the “Conference Room 2” session is established in S110, the holding time of “Conference Room 1” exceeds 2 hours. And Therefore, “1 Mbps” is assigned to the session of “Conference Room 1”, and “2 Mbps” is assigned to the session of “Conference Room 2”.

(Change of bandwidth allocation on the slave server B side)
Since the bandwidth allocation of “Conference Room 2” is not changed at 2 Mbps, the master server M performs only the bandwidth change processing of “Conference Room 1”.

  The master server M transmits an INVITE message (“re-INVITE request”) to the slave server B for updating the content of the session of “conference room 1” (S127). At this time, the master server M transmits a re-INVITE request in which “b = CT: 1000” is described in the SDP of the session information to the slave server B in order to change the bandwidth of the “conference room 1” to 1 Mbps.

  The slave server B that has received this re-INVITE request transmits a 200 OK response together with the SDP described as “b = CT: 1000” as a response (S128). Thereafter, the conference session regarding “conference room 1” between the slave server B and the master server M is exchanged in a bandwidth of 1 Mbps.

  Further, the slave server B transmits a re-INVITE request in which “b = CT: 1000” is described in the SDP of the session information to the terminals b11 and b12 participating in “conference room 1”, and the slave server B The bandwidth between B and the terminal (terminal b11, terminal b12) is also changed to 1 Mbps (S129 to S132).

  As described above, the master server M changes the bandwidth allocation of the conference session related to the “conference room 1” on the slave server B side.

(A-2-2) Band control processing operation (2)
8 and 9 are sequence diagrams illustrating the operation of the bandwidth control process in the multipoint conference system 100 according to the first embodiment. 8 and 9, the conference protocol is H.264. The case where H.323 is used is illustrated. 8 and 9 exemplify a case where the bandwidth of each conference session is allocated by the method (4) for allocating conference session time.

(Start of conference room 1)
When the terminal a11 enters the “conference room 1”, the terminal a11 sends the address of the conference room 1 to the slave server A as a transmission destination. A setup message of 225 call control signaling is transmitted (S201).

  On the other hand, the slave server A transmits a Connect message to the terminal a11 as a response that can participate in the conference room (S202). After this, between the terminal a11 and the slave server A, H. Capability exchange and logical channel opening are performed by 245 signaling (S203), and a video conference session is established between the terminal a11 and the slave server A.

  At this time, the slave server A designates maxBitRate = 2 Mbps so as to limit the transmission bandwidth to the terminal a11 to 2 Mbps. 245 FlowControlCommand message is transmitted (S204). To be precise, the FlowControlCommand is performed on the logical channel, but here, the bandwidth for audio is ignored and the video logical channel is limited to 2 Mbps. The slave server A limits the transmission rate for the terminal a11 to 2 Mbps, and sets the bandwidth used between the slave server A and the terminal a11 to 2 Mbps.

  When the first participant (terminal a11) enters the conference room, the slave server A connects to the conference room of the master server M. At this time, the same Setup message, Connect message, H.B. 245 messages are exchanged, and the session of the conference room 1 is established between the slave server A and the master server M (S205 to S207). Thereafter, the slave server A and the master server M transmit a FlowControlCommand message to each other, and limit the bandwidth between them to 2 Mbps (S208, S209).

  At this time, the master server M records the holding start time of the “conference room 1” and starts measuring the holding time of the “conference room 1”.

  When the terminal a12 enters the conference room 1, the same operation as the processing of S201 to S204 is performed, and the conference session between the terminal a12 and the slave server A is established (S210 to S213). Further, when the terminal b11 and the terminal b12 enter the conference room 1 of the slave server B, the same operation as described above is performed, and the conference session between the terminal b11 and the slave server B, and between the slave server B and the master server M are performed. A conference session is established between the terminal b12 and the slave server B (S227 to S239).

(Start of conference room 2)
When the terminal a21 enters the conference room 2 of the slave server A (S214 to S217), a session of the conference room 2 is established between the slave server A and the master server M (S218 to S222).

  At this time, since there are a plurality of conference sessions between the master server and the slave server, the master server M determines the allocation of the bandwidth for each conference session by the method described above.

(Change of bandwidth allocation on the slave server A side)
Here, it is assumed that, in the method for assigning the holding time of (4), for example, a conference session exceeding 2 hours is allocated with a conference session bandwidth with half the weight of a conference within 2 hours.

  If the session of “Conference Room 1” started in S209 exceeds 2 hours when the session of “Conference Room 2” is established in S222, the master server M changes to the session of “Conference Room 1”. Assigns a bandwidth of “1 Mbps” and “2 Mbps” to the “conference room 2” session. Since the bandwidth allocation of “Conference Room 2” is not changed at “2 Mbps”, the master server M performs only the bandwidth change processing of “Conference Room 1”.

  The master server M transmits a FlowControlCommand message specifying maxBitRate = 1 Mbps to the slave server A in order to limit the bandwidth of the session of “conference room 1” (S223).

  Receiving this FlowControlCommand message, the slave server A transmits the same FlowControlCommand message to each terminal (including the master server M in addition to the terminals a11 and a12) in order to limit the bandwidth of each conference session (S224 to S226). ).

  Accordingly, the master server M can change the bandwidth between the master server M and the slave server A and the bandwidth between the slave server A and the terminal with respect to the “conference room 1”. On the other hand, regarding the “conference room 1”, the master server M does not change the bandwidth allocation at this point in relation to the slave server B. Therefore, the master server M allocates a bandwidth of 2 Mbps to the conference session between the master server M and the slave server B and between the slave server B and the terminal.

  Thereafter, when the terminal b21 enters the “conference room 2” of the slave server B, the bandwidth between the terminal and the slave server B is 2 Mbps, and the bandwidth between the slave server B and the master server M is 2 Mbps. .

  That is, when the terminal b21 enters the conference room 2 of the slave server B (S240 to S243), a session of the conference room 2 is established between the slave server B and the master server M (S244 to S248).

  At this time, since there are a plurality of conference sessions between the master server M and the slave server B, the master server M allocates a bandwidth to each conference session. Here, in the above (4) holding time allocation method, for example, a conference exceeding 2 hours is assigned half the weight of a conference within 2 hours. When the session of “Conference Room 2” is set up and the duration of “Conference Room 1” exceeds 2 hours, the session of “Conference Room 1” includes “1 Mbps” and “Conference Room 2”. Each session is assigned “2 Mbps”.

(Change of bandwidth allocation on the slave server B side)
Since the bandwidth allocation of “Conference Room 2” is not changed at 2 Mbps, the master server M performs only the bandwidth change processing of “Conference Room 1”. The master server M transmits a FlowControlCommand message specifying maxBitRate = 1 Mbps to the slave server B in order to limit the bandwidth of the session of “conference room 1” (S249).

  The slave server B that has received this FlowControlCommand message transmits a similar FlowControlCommand message to each terminal (including the master server M in addition to the terminal b11 and the terminal b12) in order to limit the bandwidth of each conference session (S250 to S252). ).

  Accordingly, the master server M can change the bandwidth between the master server M and the slave server B and the bandwidth between the slave server B and the terminal with respect to the “conference room 1”.

(A-3) Effect of First Embodiment As described above, according to the first embodiment, even when a plurality of video conference sessions are used simultaneously, by controlling the bandwidth allocation of each conference session, By avoiding congestion between the master server and the slave server, the conference can be performed with stable quality.

(B) Second Embodiment Hereinafter, a second embodiment of the conference system, the conference server, the conference method, and the conference program according to the present invention will be described in detail with reference to the drawings.

  In the second embodiment, the codec of the master server M is H.264. A case where the present invention is applied to a multipoint video conference system adopting H.264 / SVC will be exemplified.

  In the first embodiment, the case of changing the bandwidth allocation between the master server M and the slave servers A and B and between the slave servers A and B using the video conference protocol is illustrated.

  On the other hand, the second embodiment is an H.264 code adopted as a codec system. An example in which the bandwidth allocation between the master server M and the slave servers A and B and between the slave servers A and B-terminals is changed using the characteristics of H.264 / SVC.

(B-1) Configuration of Second Embodiment Also in the second embodiment, description will be given using the configuration of the multipoint video conference system 100 of FIG. 1 according to the first embodiment. Also in the second embodiment, the components described in the first embodiment will be described with the same reference numerals as those in the first embodiment.

(B-1-1) Configuration of Master Server M FIG. 10 is a configuration diagram illustrating a configuration of an encoding unit (encoder) of the video signal processing 32A of the MCU 32 included in the master server M according to the second embodiment. In FIG. 10, the encoding unit of the master server M according to the second embodiment includes an H264 / SVC encoder 71 and an MPEG2-TS generation unit 72.

  As shown in FIG. A H.264 / SVC encoder 71 is incorporated. FIG. 10 illustrates a processing configuration in which the H264 / SVC encoder 71 generates a conference room composite image, and the MPEG2-TS generation unit 72 converts the conference room composite image signal into an IP packet.

  H. The H.264 / SVC encoder 71 receives a video signal and generates video data of a base layer and one or more enhancement layers based on the input video signal. The H264 / SVC encoder 71 employs at least one of spatial, temporal, and SNR scalability to generate base layer and enhancement layer video data.

  The H264 / SVC encoder 71 can output hierarchical video data having a plurality of types of resolutions by adopting spatial scalability. The H264 / SVC encoder 71 generates low-resolution video data that is basic in the base layer, and generates high-resolution video data in the enhancement layer. For example, the H264 / SVC encoder 71 generates QCIF (Quarter CIF) standard video data in the base layer, and generates CIF (Common Intermediate Format) standard or VGA (Video Graphics Array) standard video data in the extension layer. . Here, a case using CIF and 4CIF will be described later.

  Further, the H264 / SVC encoder 71 can obtain a plurality of types of hierarchical video data having different frame rates by adopting temporal scalability. The H264 / SVC encoder 71 generates video data with the lowest frame rate, which is the basic in the base layer, and generates video data with a high frame rate in the enhancement layer. For example, the H264 / SVC encoder 71 generates 7.5 fps (frame / rate) video data in the base layer, and generates 15 fps or 30 fps video data in the enhancement layer.

  Further, the H264 / SVC encoder 71 can obtain a plurality of types of hierarchical video data having different image quality by adopting SNR scalability. The H264 / SVC encoder 71 generates video data with the lowest image quality that is basic in the base layer, and generates video data with high image quality in the enhancement layer. For example, the SVC encoder 11 generates video data including a DC component of a DCT conversion coefficient in the base layer, and generates video data including a higher frequency component of the DCT conversion coefficient in a higher extension layer.

  FIG. 11 is an explanatory diagram for explaining a relationship between base layer video data and enhancement layer video data in the H264 / SVC encoder 71 according to the second embodiment.

  FIG. 11A shows an example when scalability is not applied. FIG. 11B shows an example in which time scalability is applied. FIG. 11C shows an example when time and space scalability is applied. A “frame (rectangle)” illustrated in FIG. 11 indicates video data of each frame such as a base layer and each enhancement layer, and an “arrow” indicates a correlation. In FIG. 11, the horizontal direction indicates the time in units of frames, and shows the relationship between each frame and encoding.

  The H264 / SVC encoder 71 generates video data of each enhancement layer by extending the video data of the base layer. That is, as shown by the arrows in FIG. 11, the upper layer data has a correlation depending on the lower layer data.

  FIG. 11A shows an example in which scalability is not adopted, and each frame is encoded without being hierarchized. In FIG. 11A, “code I” indicates a picture (I picture) that is intra-frame encoded, and “code P” indicates a P picture that has been unidirectionally predicted encoded. Each picture has a correlation indicated by an arrow. On the decoding side, when the transmitted I picture is not restored, the subsequent P picture cannot be accurately decoded.

  FIG. 11B shows a base layer in temporal scalability and two layers of enhancement layers. In the base layer indicated by “reference symbol B”, a ¼ frame with respect to the frame rate in FIG. It shows that it is composed of rate video data. It is shown that video data having a frame rate ½ of the frame rate in FIG. 11A is configured by the base layer data and the lower layer extension layer data indicated by “reference E1”. Yes. Furthermore, video data at the same frame rate as in FIG. 9A can be obtained by using up to the enhancement layer data of the upper layer indicated by “reference symbol E2”.

  FIG. 11C shows hierarchical coding based on temporal and spatial scalability, where “code B1” represents high-resolution video data (enhancement layer) compared to low-resolution video data (base layer) represented by “code B0”. ). In addition, “Symbol E11” indicates high resolution video data (enhancement layer) with respect to the low resolution video data (base layer) indicated by “Symbol E10”, and low resolution video data indicated by “Symbol E20”. “Code E21” indicates high resolution video data (enhancement layer) with respect to (base layer). “Code B0” indicates base layer data in temporal and spatial scalability, and when the data of “code B0” is lost on the decoding side, accurate decoding is not possible even using enhancement layer data. Can't do it.

  The H264 / SVC encoder 71 multiplexes the generated base layer data and each enhancement layer data, and outputs the multiplexed data to the MPEG2-TS generating unit 72.

  FIG. 12 is an explanatory diagram illustrating an example of multiplexed video data output from the H264 / SVC encoder 71 according to the second embodiment to the MPEG2-TS generation unit 72.

For example, the H264 / SVC encoder 71 may arrange the base layer and enhancement layer data in the data arrangement shown in FIG. That is, in this case, as shown in FIG. 12, each enhancement layer is arranged following the base layer data. For example, FIG. 12A shows a multiplexing example based on temporal scalability and then spatial scalability.
The data of “symbol B0 (1)”, which is the base layer data of temporal scalability, and is the data of the base layer in spatial scalability, and the data of “code B1 (1)”, which is the data of the enhancement layer, are arranged.

  In addition, data of “symbol E10 (1)” that is data of the first layer of temporal scalability and data of the base layer in spatial scalability and data of “symbol E11 (1)” that is data of the enhancement layer are arranged. Is done.

  Further, the base layer data of temporal scalability, the data of “code E20 (1)” that is the data of the base layer in spatial scalability, the data of “code E21 (1)” that is the data of the enhancement layer, and the base Data of “code E21 (1)” that is layer data and data of “code E22 (1)” that is data of the enhancement layer are arranged.

  The output of the H264 / SVC encoder 71 is given to an MPEG2-TS generator 72, which converts a transmission signal (IP packet) obtained by packetizing the input data into an MPEG standard as a slave server (slave server). A / slave server B).

  FIG. 13 is an explanatory diagram illustrating the relationship between image quality, base layer video data, and enhancement layer video data in the second embodiment. Note that FIG. 13 illustrates the case where the image quality is “CIF” and “4CIF”, for example.

  FIG. 14 is a diagram illustrating H.264 according to the second embodiment. It is a flowchart which shows operation | movement of the master server M in which the H.264 / SVC encoder was incorporated.

  FIG. 15 is an explanatory diagram illustrating an example of multiplexed video data output from the master server M to the server slave servers A and B based on FIG.

  The video signal is input to the H264 / SVC encoder 71. The SVC encoder 71 employs at least one of space, time, and SNR scalability, and hierarchically encodes the input video signal to generate video data of the base layer and each enhancement layer (FIG. 14). S1).

  The H264 / SVC encoder 71 arranges the video data of the base layer and the enhancement layer (S3 in FIG. 14).

  The data output from the H264 / SVC encoder 71 will be described with reference to FIG. FIG. 13 shows the following six examples of image hierarchical coding based on image quality.

(1) A video-encoded signal having a CIF resolution and 7.5 frames per second is composed of “B0”.

(2) A video-encoded signal of CIF resolution and 15 frames per second is composed of “B0” and “E10”.

(3) A video-encoded signal with CIF resolution and 30 frames per second is composed of “B0”, “E10”, and “E20”.

(4) A video-encoded signal with 4 CIF resolution and 7.5 frames per second is composed of “B0” and “B1”.

(5) A video-encoded signal with 4 CIF resolution and 15 frames per second is composed of “B0”, “B1”, “E10”, and “E11”.

(6) A video-encoded signal with 4 CIF resolution and 30 frames per second is composed of “B0”, “B1”, “E10”, “E11”, “E20”, and “E21”.

  As shown in FIG. 15, the video data from the H264 / SVC encoder 71 is sequentially output as shown in FIG. 15, taking FIG. 12A as an example.

(A) In a video-encoded signal with CIF resolution and 7.5 frames per second, data “B0 (1)” is sequentially output.

(B) In a video-encoded signal having a CIF resolution and 15 frames per second, data “B0 (1)” and “E10 (1)” are sequentially output.

(C) In the case of a video-encoded signal with CIF resolution and 30 frames per second, data “B0 (1)”, “E10 (1)”, “E20 (1)”, and “E20 (2)” are sequentially output. Is done.

(D) In a video encoded signal with 4 CIF resolution and 7.5 frames per second, data “B0 (1)” and “B1 (1)” are sequentially output.

(E) In a video encoded signal with 4 CIF resolution and 15 frames per second, data “B0 (1)”, “B1 (1)”, “E10 (1)”, and “E11 (1)” are sequentially output. Is done.

(F) In a video encoded signal with 4 CIF resolution and 30 frames per second, “B0 (1)”, “B1 (1)”, “E10 (1)”, “E11 (1)”, “E20 (1 ) ”,“ E21 (1) ”,“ E20 (2) ”, and“ E21 (2) ”are sequentially output.

  The output of the H264 / SVC encoder 71 is given to the MPEG2-TS generator 72, packetized in accordance with the MPEG standard, and then transmitted as a transmission signal (S4 in FIG. 14).

(B-1-2) Configuration of Slave Servers A and B FIG. 16 is a configuration diagram showing the configuration of MCUs of slave servers A and B according to the second embodiment.

  In FIG. 16, the MCUs of the slave servers A and B according to the second embodiment include a maximum display image quality extraction unit 81, a terminal-side transmission signal control unit 82, and a master server-side control signal transmission unit 83.

  In FIG. 16, the slave servers A and B receive the display image quality information (resolution, frame / second) on the display screen in each terminal, and the video data necessary for the display image quality of each terminal is sent to each terminal. A processing configuration for transmitting a transmission signal having the same is described.

  Further, FIG. 16 shows that the slave servers A and B receive the display image quality information (resolution, frames / second) of the display screen in each terminal, and the master server M sends the video data necessary for the slave server. A processing configuration for transmitting a control signal including information (resolution, frame / second) is described.

  The maximum display image quality extraction unit 81 receives the display image quality information of the display screen at each terminal, and is necessary for each conference room (meeting room 1 / conference room 2) based on the received display image quality information of the display screen at each terminal. The maximum display image quality is extracted.

  Here, the display image quality information is information relating to the display image quality on the display screen in each terminal, and includes, for example, resolution, frame rate (frame / second), and the like. Specifically, for example, the resolution indicates whether the resolution is “CIF” or “4 CIF”, and the frame rate (frame / second) is “7.5 frames per second” and “15 frames per second”. ”,“ 30 frames per second ”, and the like.

  For example, considering “conference room 1” in “slave server A”, the terminals connected to “slave server A” are “terminal a11” and “terminal a12”, and “terminal a11” and “terminal a12”. The information on the display image quality of the display screen is as follows.

-"Display image quality" information of "terminal a11"->"CIF" and "30 frames per second"
-"Display image quality" information of "terminal a12"->"CIF" and "15 frames per second"
The maximum display image quality extraction unit 81 determines that video data of “30 frames per second” should be transmitted by “CIF” to “terminal a11” connected to “slave server A”. The data is output to the terminal side transmission signal control unit 82.

  Further, the maximum display image quality extraction unit 81 determines that it is only necessary to transmit “15 frames per second” video data with “CIF” to “terminal a12” connected to “slave server A”. Information is output to the terminal-side transmission signal control unit 82.

  Based on the determination information received from the maximum display image quality extraction unit 81, the terminal-side transmission signal control unit 82 switches to and transmits video data with the maximum display image quality to each terminal.

  For example, the terminal-side transmission signal control unit 82 switches to “terminal a11” from “4 CIF” video data of “30 frames per second” to “CIF” video data of “30 frames per second”. . Further, for example, the terminal-side transmission signal control unit 82 determines, based on the determination information received from the maximum display image quality extraction unit 81, from the video data of “4 CIF” and “30 frames per second” for “terminal a12”. “CIF” is switched to “15 frames per second” video data for transmission.

  The master server side control signal transmission unit 83 transmits a control signal for switching the quality image quality of the video data for each conference room to the master server M based on the determination information received from the maximum display image quality extraction unit 81. .

  In the case of the above example, the master server side control signal transmission unit 83 has high quality among the image quality information of “terminal a11” and “terminal a12” participating in “conference room 1” of “slave server A”. A control signal for requesting switching to the other video data is transmitted to the master server M.

  Therefore, in the case of the above example, the master server side control signal transmission unit 83 sends “4 CIF” “30 frames per second” video data to “master server M” and “CIF” “30 frames per second”. A control signal including information for switching to video data is transmitted.

(B-2) Operation of Second Embodiment Next, an operation in the multipoint video conference system 100 according to the second embodiment will be described.

  17 and 18 are explanatory diagrams for explaining the operation of changing the bandwidth allocation in the multipoint video conference system 100 according to the second embodiment.

  Each terminal enters “conference room 1” of slave server A or “conference room 2” of slave server B in the same manner as in the first embodiment. Similarly to the first embodiment, the slave servers A and B receive a request from each terminal and connect to the master server M to connect the “conference room 1” session and the “conference room 2” session. Establish.

  Note that the video conference protocol is SIP or H.264. Any of the H.323 compliant protocols can be used.

  In addition, when a plurality of conference sessions use the bandwidth at the same time, the master server M changes the bandwidth allocation for each connection line with the slave servers A and B in the conference sessions described in the first embodiment. Anyway.

  Here, it is assumed that the master server M performs the conference session of “Conference Room 1” and the conference session of “Conference Room 2” at the same time by the same processing as in the first embodiment.

  The slave server A recognizes in advance the bandwidth that can be used between the master server and the slave server A and / or between the slave server and the terminal.

  The master server M is an H.264 codec system. It is assumed that H.264 / SVC is used and the composite image for “conference room 1” is transmitted to slave server A at a resolution of 4 CIF and a frame rate (frame / second) of 30 frames per second.

  The slave server A acquires display image quality information of the display screen from each terminal (terminal a11, terminal a12) participating in “conference room 1” and each terminal (terminal a21) participating in “conference room 2”.

  Here, in the slave server A, when each terminal (terminal a11, terminal a12) makes a request to enter “conference room 1”, or each terminal (terminal a21) makes a request to enter “conference room 2”. At this time, display image quality information can be acquired as part of the performance information of each terminal.

  In the slave server A, as shown in FIG. 16, the maximum display image quality extraction unit 81 includes each terminal (terminal a11, terminal a12) that participates in “conference room 1” and each terminal (terminal that participates in “conference room 2”). When the display image quality information from a21) is acquired, the maximum display image quality information necessary for each conference room is extracted based on the display image quality information of each terminal.

  For example, the display image quality information of the terminal a11 participating in the “conference room 1” is “resolution = CIF” and “frame rate = 30 frames per second”, and the display image quality information of the terminal a12 is “resolution = CIF”, “ Assume that “frame rate = 15 frames per second”. In such a case, the maximum display image quality extraction unit 81 extracts “resolution = CIF” and “frame rate = 30 frames per second” as the maximum display image quality information. Similarly, regarding “conference room 2”, the maximum display image quality information is extracted by the same method.

  When the bandwidth between the master server M and the slave server A is insufficient, the slave server A rechanges the bandwidth allocation of some or all of the conference sessions among the plurality of conference sessions.

  Here, the bandwidth allocation of all the conference sessions among the plurality of conference sessions may be changed again, or the bandwidth allocation of some of the conference sessions among the plurality of conference sessions may be changed again. Anyway.

  When performing bandwidth allocation for some conference sessions, the bandwidth of the conference session with the largest allocated bandwidth among a plurality of conference sessions may be changed again, or the master server M may first change the bandwidth of the conference session. The bandwidth allocation of the conference session having the smaller weight may be changed again by the bandwidth allocation method described in the embodiment.

  Here, the case where the bandwidth of the conference session of “conference room 1” is changed again is illustrated.

  In the slave server A, based on the display image quality information of “terminal a11” acquired by the maximum display image quality extraction unit 81, only video data related to “resolution = CIF” and “frame rate = 30 frames per second” is transmitted to the terminal a11. Send.

  That is, as shown in FIG. 17, video data of “resolution = 4 CIF” and “frame rate = 30 frames per second” illustrated in FIG. 15F is distributed from the master server M to the slave server A. (See “91” in FIG. 17). However, the slave server A transmits video data (that is, FIG. 15C) of image quality corresponding to the display image quality information of the terminal a11 to the terminal a11 (see “92” in FIG. 17). ).

  That is, the slave server A leaves video data correlated with “code B0” and “code B0”, which are low-resolution video data (base layer), among the hierarchical codes based on time and space scalability, Video data obtained by discarding video data (enhancement layer) “code B1” and video data correlated with “code B1” is transmitted to the terminal a11 (see FIG. 11).

  Further, in the slave server A, based on the display image quality information of “terminal a11” acquired by the maximum display image quality extraction unit 81, video data related to “resolution = CIF” and “frame rate = 15 frames per second” with respect to the terminal a11. Is transmitted (see “93” in FIG. 17).

  That is, the slave server A sends from the master server M to the terminal a12 the video data of “resolution = 4CIF” and “frame rate = 30 frames per second” illustrated in FIG. Video data having an image quality corresponding to the display image quality information (that is, FIG. 15B) is transmitted.

  That is, the slave server A discards video data correlated with “code B1” and “code B1”, which are high-resolution video data (enhancement layer), among hierarchical codes based on temporal and spatial scalability. Video data having a frame rate halved is transmitted to the terminal a12 (see FIG. 11).

  As described above, the slave server A transmits the video data having the image quality corresponding to the display image quality information of the terminal a11 and the terminal a12 participating in the “conference room 1” to the terminal a11 and the terminal a12. It is possible to control to reduce the band between.

  In addition, the slave server A transmits a control signal related to the image quality of video data to be transmitted to the slave server A side to the master server M.

  That is, in the slave server A, the maximum display image quality extraction unit 81 transmits to the master server M a control signal including the maximum display image quality information among the display image quality information of the terminals a11 and a12 that participate in the “conference room 1”. .

  For example, in the above example, the slave server A transmits video data of “resolution = CIF” and “frame rate = 30 frames per second” as the maximum display image quality information among the display image quality information of the terminals a11 and a12. To request.

  When receiving the control signal from the slave server A, the master server M transmits video data of “resolution = CIF” and “frame rate = 30 frames per second” as a transmission signal on the slave server A side (FIG. 18 “91” and “94”).

  That is, the master server M leaves video data correlated with “code B0” and “code B0”, which are low-resolution video data (base layer), among the hierarchical codes based on time and space scalability. Video data obtained by discarding video data (enhancement layer) “code B1” and video data correlated with “code B1” is transmitted to the terminal a11 (see FIG. 11).

  As described above, the slave server A transmits the video data having the image quality corresponding to the maximum display image quality information necessary for the “conference room 1” to the master server M, thereby reducing the bandwidth between the master server M and the slave server A. Can be controlled.

(B-3) Effects of the Second Embodiment As described above, according to the second embodiment, in addition to the effects of the first embodiment, the By utilizing the characteristics of the H.264 / SVC encoding method, congestion between the master server and the slave server can be avoided and a conference can be performed with more stable quality.

(C) Other Embodiments Although various modified embodiments have been mentioned in the above-described embodiments, the present invention can also be applied to the following modified embodiments.

(C-1) The present invention can be applied to a combination of the first embodiment and the second embodiment. That is, according to the present invention, as described in the first embodiment, the master server allocates the bandwidth of each conference session for each conference room to each slave server using the video conference protocol, and further the video conference proceeds. When the bandwidth becomes insufficient, the slave server extracts the maximum image quality of each terminal that participates in each conference room, and requests the master server to switch the video data of each terminal connected to the slave server You may do it. Thereby, the bandwidth of each conference session can be changed dynamically.

  Further, the embodiment described in the second embodiment may be applied independently of the first embodiment.

(C-2) In the first embodiment described above, the master server exemplifies a case where the allocation of the conference session bandwidth of each conference room is changed for each slave server when a plurality of conference sessions are simultaneously performed. . However, the bandwidth allocation of the conference session for each conference room may be changed without allocating the bandwidth of the conference session for each slave server.

(C-3) In each of the above-described embodiments, the case of a one-stage cascade configuration of a master server and a slave server is exemplified, but the number of stages may be two or more.

100 ... multi-point video conferencing system, M ... master server, A and B ... slave server, 31 ... gateway, 32 ... MCU,
a11, a12, a21, b11, b12, b21... terminal, 51-1 and 51-2 ... data receiving unit, 52-1 and 52-2 ... decoding unit, 53 ... combining unit, 54-1 and 54-2 ... Encoding unit, 55-1 and 55-2 ... data transmission unit, 56 ... control unit, 65 ... bandwidth control processing unit, 61 ... conference session number equal allocation processing unit, 62 ... conference session weight allocation processing unit, 63 ... conference Session participation number allocation processing unit, 64... Conference session holding time allocation processing unit, 71. H.264 / SVC encoder, 72... MPEG2-TS generation unit, 81... Maximum display image quality extraction unit, 82... Terminal side transmission signal control unit, 83.

Claims (7)

In a conference system in which the first conference server is connected to a plurality of second conference servers by cascade connection and performs processing related to a plurality of conferences,
When the use bandwidth of the plurality of conference sessions established by the first conference server exceeds the communication bandwidth with the plurality of second conference servers, the conference information of each conference related to the plurality of conference sessions is included in the conference information. Based on a first bandwidth control means for controlling a communication bandwidth allocated to each of the plurality of conference sessions,
Each of the plurality of second conference servers includes a second bandwidth control unit configured to control a communication bandwidth with a terminal participating in each conference based on the communication bandwidth allocated by the first conference server. A conference system characterized by comprising. The first conference server transmits a conference data signal obtained by multiplexing a plurality of types of video data having different image qualities,
Each of the second conference servers is
A terminal side that processes a conference data signal from the first conference server into necessary video data in each terminal based on necessary image quality information of each terminal participating in each conference and transmits the video data to each terminal A transmission signal control means;
2. The conference system according to claim 1, further comprising: a control signal transmission unit configured to transmit a control signal instructing transmission of conference data necessary for maximum image quality information of each conference to the first conference server. . In a conference server that performs processing related to multiple conferences by cascade connection with multiple conference servers,
When the use band of the plurality of established conference sessions exceeds the communication band between the plurality of second conference servers, the plurality of conference sessions is based on the conference information of each conference related to the plurality of conference sessions. A conference server comprising bandwidth control means for controlling a communication bandwidth allocated to each of the conference servers. In the conference server that is cascade-connected to the first conference server that is the master server,
Based on the communication band assigned to each conference session, acquired from the first conference server that allocates a communication band to each of the plurality of conference sessions based on the conference information of each conference related to a plurality of conference sessions, A conference server comprising bandwidth control means for controlling a communication bandwidth between terminals participating in each conference. In the conference method in which the first conference server is connected to a plurality of second conference servers by cascade connection and performs processing related to a plurality of conferences.
When the use bandwidth of the plurality of conference sessions established by the first conference server exceeds the communication bandwidth with the plurality of second conference servers, the conference information of each conference related to the plurality of conference sessions is included in the conference information. Based on the communication bandwidth allocated to each of the plurality of conference sessions,
Each of the plurality of second conference servers controls a communication band between terminals participating in each conference based on a communication band allocated by the first conference server. . In a conference program of a conference server that performs a cascade connection with a plurality of conference servers and performs processing related to a plurality of conferences,
Computer
When the use band of the plurality of established conference sessions exceeds the communication band between the plurality of second conference servers, the plurality of conference sessions is based on the conference information of each conference related to the plurality of conference sessions. A conference program that functions as a bandwidth control unit that controls a communication bandwidth allocated to each of the conference programs. In the conference program of the conference server that cascade-connects with the first conference server as the master server,
Computer
Based on the communication band assigned to each conference session, acquired from the first conference server that allocates a communication band to each of the plurality of conference sessions based on the conference information of each conference related to a plurality of conference sessions, A conference program that functions as a bandwidth control unit that controls a communication bandwidth between terminals participating in each conference.

Images