JP2010200273A - Network control system, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To switch an MCU (Multi-point Control Unit) without disconnecting a call when MCU stop, MCU throughput reduction or communication quality reduction occurs. <P>SOLUTION: A network control system includes: a plurality of communication terminals; a first media synthesis means for synthesizing a plurality of received media data items for transmission; a first relay means disposed between the plurality of communication terminals and the first media synthesis means for transmitting to the first media synthesis means only required media data among the media data from the plurality of communication terminals; a second relay means disposed between the first media synthesis means and the plurality of communication terminals for copying synthesized media data from the first media synthesis means as many as the number of the plurality of communication terminals for distribution to the plurality of communication terminals; and a switching means for periodically acquiring state information from the first media synthesis means, the first relay means and the second relay means and switching from the first media synthesis means to the second media synthesis means as a destination of switching. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a network control system, method, and program, and can be applied to a system that performs a multipoint conference via a network, for example.

  In a conference system that performs a conference at a multipoint by a plurality of participants via a network, an MCU (Multi-point Control Unit) that exchanges voice, video, and data is generally used.

  The main functions of the MCU are call control functions, calculation processing functions that receive audio, video, and data from multiple terminals participating in the conference and decode such data as necessary, collected audio, video, and data Switching function, composition function for audio composition and spatial composition, and finally, the processed data is encoded with a specific codec, bit rate, and video size in the case of video, and audio is sent to participants. There is an encoded distribution function for distributing video and data.

  With such MCU functions, terminals that cannot directly communicate with each other participate in the same conference via the MCU, thereby enabling communication such as voice and video between each other.

  On the other hand, if the MCU stops for some reason during the conference, the above functions of the MCU are not performed, and mutual communication between terminals becomes impossible.

  As a technique for solving such a problem, there is a technique described in Patent Document 1. In Patent Literature 1, when the MCU is stopped, the terminal connected to the stopped MCU is specified, and the fact that the MCU has stopped is notified to this terminal. This is a technique for preparing an MCU to be newly connected and allowing the new MCU to establish a new connection again.

  Thereby, even if a video conference server stops for some reason during a meeting, the state of a meeting can be restored in as short a time as possible, and a meeting can be resumed.

US Patent Application Publication No. 2008/0077666

  However, the method described in Patent Document 1 described above requires that all calls are once disconnected and a call connection procedure is performed again with a new MCU when switching MCUs. Therefore, it may take a considerable amount of time to return to the state of the conference that has been held.

  In addition, the method described in Patent Document 1 described above switches the MCU only when the MCU is stopped. For this reason, for example, when the quality of audio or video deteriorates due to network congestion or the like, it may not be possible to switch to another MCU that can provide communication with higher quality than this MCU.

  Further, for example, in a conventional conference system, each terminal participating in the conference needs to transmit data such as audio and video to the MCU, and the data transmission from each terminal reduces the network load. It may increase, causing traffic interruption and quality degradation. Therefore, a connection form that can reduce the communication band of the network is strongly desired.

  Therefore, in view of the above problems, it is possible to provide a connection form that can switch to a new MCU without disconnecting a call at the time of switching, can switch to a new MCU even when the MCU is not stopped, and can reduce the communication bandwidth of the network. There is a need for a network control system, method and program capable of doing so.

  In order to solve this problem, a network control system according to a first aspect of the present invention includes: (1) a plurality of communication terminals; and (2) a first media combining unit that combines and transmits a plurality of received media data. (3) A first device that is arranged between the plurality of communication terminals and the first media synthesizing unit and transmits only necessary media data among the media data from the plurality of communication terminals to the first media synthesizing unit. Relay means, and (4) arranged between the first media composition means and the plurality of communication terminals, and copies the composite media data from the first media composition means by the number of the plurality of communication terminals, and a plurality of communications. A second relay means for delivering to the terminal; and (5) first status information is periodically acquired from the first media composition means, the first relay means, and the second relay means, and the first information is obtained based on the status information. Media composition Characterized in that it comprises a switching means for switching to a second media combining means to switch to the stage.

  The network control system according to the second aspect of the present invention includes (1) a plurality of communication terminals, (2) a first media composition means for combining and transmitting a plurality of received media data, and (3) a plurality of communication terminals. And a first relay means that is arranged between the first media composition means and transmits only necessary media data among media data from a plurality of communication terminals to the first media composition means, and (4) A second relay that is arranged between one media composition means and a plurality of communication terminals, copies the composite media data from the first media composition means by the number of the plurality of communication terminals, and distributes the data to the plurality of communication terminals. And (5) control means for instructing and determining media data to be transmitted from the first relay means to the first media composition means.

  The network control method according to the third aspect of the present invention includes (1) a plurality of communication terminals, (2) a first media composition means for combining and transmitting a plurality of received media data, and (3) a plurality of communication terminals. And a first relay means that is arranged between the first media composition means and transmits only necessary media data among media data from a plurality of communication terminals to the first media composition means, and (4) A second relay that is arranged between one media composition means and a plurality of communication terminals, copies the composite media data from the first media composition means by the number of the plurality of communication terminals, and distributes the data to the plurality of communication terminals. (5) The switching means periodically obtains status information from the first media composition means, the first relay means, and the second relay means, and based on the status information, 1 Medi And performs switching to the second media combining means to switch to the combining means.

  A network control program according to a fourth aspect of the present invention includes: (1) a plurality of communication terminals; (2) a first media composition unit that synthesizes and transmits a plurality of received media data; and (3) a plurality of communication terminals. And a first relay means that is arranged between the first media composition means and transmits only necessary media data among media data from a plurality of communication terminals to the first media composition means, and (4) A second relay that is arranged between one media composition means and a plurality of communication terminals, copies the composite media data from the first media composition means by the number of the plurality of communication terminals, and distributes the data to the plurality of communication terminals. (5) The control means instructs to determine media data to be transmitted from the first relay means to the first media composition means.

  The network control program of the fifth aspect of the present invention includes (1) a plurality of communication terminals, (2) a first media composition means for combining and transmitting a plurality of received media data, and (3) a plurality of communication terminals. And a first relay means that is arranged between the first media composition means and transmits only necessary media data among media data from a plurality of communication terminals to the first media composition means, and (4) A second relay that is arranged between one media composition means and a plurality of communication terminals, copies the composite media data from the first media composition means by the number of the plurality of communication terminals, and distributes the data to the plurality of communication terminals. The computer periodically acquires status information from the first media synthesizing unit, the first relay unit, and the second relay unit, and the first media is based on the status information. Characterized in that to function as a switching means for switching to a second media combining means to switch to the combining means.

  A network control program according to a sixth aspect of the present invention includes: (1) a plurality of communication terminals; (2) a first media combining unit that combines and transmits a plurality of received media data; and (3) a plurality of communication terminals. And a first relay means that is arranged between the first media composition means and transmits only necessary media data among media data from a plurality of communication terminals to the first media composition means, and (4) A second relay that is arranged between one media composition means and a plurality of communication terminals, copies the composite media data from the first media composition means by the number of the plurality of communication terminals, and distributes the data to the plurality of communication terminals. And a computer functioning as control means for determining and instructing media data to be transmitted from the first relay means to the first media composition means.

  According to the present invention, at the time of switching, it is possible to switch to a new media synthesizing unit without disconnecting a call, and to switch to a new media synthesizing unit even when the media synthesizing unit is not stopped, thereby reducing the network communication band. Can provide connection form

It is a block diagram which shows the structure of the conference system of 1st Embodiment. It is a functional block diagram which shows the internal structure of the media composition apparatus of 1st Embodiment. It is a functional block diagram which shows the internal structure of the uplink relay apparatus of 1st Embodiment. It is a functional block diagram which shows the internal structure of the downlink relay apparatus of 1st Embodiment. It is a functional block diagram which shows the internal structure of the control apparatus of 1st Embodiment. It is explanatory drawing explaining the screen composition layout of 1st Embodiment. It is a flowchart which shows the communication processing of the conference system of 1st Embodiment. It is a flowchart which shows the MCU switching process of 1st Embodiment.

(A) First Embodiment Hereinafter, a first embodiment of a network control system, method, and program according to the present invention will be described with reference to the drawings.

  The first embodiment will be described by exemplifying an embodiment that realizes a multipoint conference system using the present invention.

(A-1) Configuration of the First Embodiment FIG. 1 is a configuration diagram showing the overall configuration of the conference system of the first embodiment.

  In FIG. 1, the conference system 10 according to the first embodiment includes a control device 1, media synthesis devices 2-1 and 2-2, and an upstream relay device 3 via a network using IP (Internet Protocol) as a communication protocol. , The downlink relay device 4, a plurality (four in FIG. 1) of communication terminals 5-1 to 5-4, and a gatekeeper 6.

  In the conference system 10, a virtual conference room is established with the four communication terminals 5-1 to 5-4 as conference participation terminals, and the video / audio data from these communication terminals 5-1 to 5-4 is synthesized. Thus, mutual communication is realized between the communication terminals 5-1 to 5-4.

  Between the plurality of communication terminals 5-1 to 5-4 and the uplink relay device 3 or the downlink relay device 4, or between the uplink relay device 3 or the downlink relay device 4 and the media combining devices 2-1 and 2-2. Video data and audio data are exchanged by RTP (Real-time Translation Protocol) packets, and control information of RTP communication and statistical information related to communication are exchanged by RTCP (RTP Control Protocol) packets.

  Here, in the first embodiment, a case where a distributed MCU is applied is illustrated. An example in which devices that realize the functions of the MCU are not necessarily physically present in the same device, and devices that execute a part of each function are distributed over the network will be exemplified.

  In the case of the first embodiment, the control device 1, the media composition devices 2-1 and 2-2, the uplink relay device 3, and the downlink relay device 4 are constituent elements of the distributed MCU.

  In FIG. 1, one uplink relay device 3 and one downlink relay device 4 are shown, but a plurality of uplink relay devices 3 and downlink relay devices are arranged according to the connection form of the communication terminals participating in the conference. You may make it provide.

  The gatekeeper 6 performs call connection and has a function of permitting connection of the communication terminals 5-1 to 5-4 to the virtual conference room and an address conversion function. As the call connection method, various methods can be widely applied. A standardized technology conforming to H.323 and a call connection method according to SIP can be applied.

  Under the control of the control device 1, the media synthesis devices 2-1 and 2-2 decode the video / audio data received from the connected communication terminals 5-1 to 5-4, and at least one of these video and audio data. Two media are synthesized, and the synthesized data is encoded and distributed.

  Here, the media composition device 2-1 is a device that operates before switching, and the media composition device 2-2 is a device that operates after switching. Both are functionally the same.

  FIG. 2 is a functional block diagram showing an internal configuration of the media composition apparatuses 2-1 and 2-2.

  In FIG. 2, the media composition device 2 includes data reception units 21-1 to 21-4, decoding units 22-1 to 22-4, composition unit 23, encoding units 24-1 to 24-4, and data transmission unit 25. -1 and 25-2, a communication unit 26, a survival confirmation unit 27, a resource confirmation unit 28, and a communication quality measurement unit 29.

  The data receiving units 21-1 to 21-4 receive video / audio data from the communication terminals 5-1 to 5-4 through the network NT. Further, the data receiving units 21-1 to 21-4 have a received packet measuring function unit 211.

  The received packet measuring function unit 211 acquires numerical values such as packet loss, delay time, and jitter based on the received packet, and gives the result to the control device 1 through the communication unit 26.

  The decoding units 22-1 to 22-4 decode the data received by the corresponding data receiving units 21-1 to 21-4, and give the decoded data to the synthesizing unit 23.

  Here, the data reception units 21-1 to 21-4 and the decoding units 22-1 to 22-4 are generated as many as the number of communication terminals participating in the conference under the control of the control device 1.

  The synthesizing unit 23 synthesizes the decoded data from the decoding units 22-1 to 22-4 and supplies the synthesized data to the encoding units 24-1 to 24-2.

  Here, as a synthesis method by the synthesis unit 23, various synthesis methods can be widely applied. For example, FIG. 6 is a diagram illustrating an example of a screen composition layout that the composition unit 23 composes. The composition unit 23 can perform screen composition of various layouts according to the number of participants in the conference. For example, in the first embodiment, four communication terminals 5-1 to 5-4 are participants. Therefore, a method of combining the images from the communication terminals 5-1 to 5-4 into each of the four divided frames as shown in FIG. 6A can be applied.

  The encoding units 24-1 to 24-2 encode the combined data from the combining unit 23 and provide the encoded data to the data transmission units 25-1 to 25-2. The encoding units 24-1 to 24-2 are generated by an arbitrary number of at least one or more for one conference room according to the control of the control device 1. Further, the number of encoding units can be increased or decreased according to the control of the control device 1.

  The data transmission units 25-1 to 25-2 transmit the encoded data from the encoding units 24-1 to 24-2. The data transmission units 25-1 to 25-2 are also generated by an arbitrary number of at least the number according to the control of the control device 1.

  The communication unit 26 is a communication processing unit that communicates with the control device 1.

  When the survival confirmation unit 27 receives a survival confirmation signal (for example, ping or the like) from the control device 1 via the communication unit 26, the survival confirmation unit 27 returns a response signal to the response to the control device 1. Thereby, it is possible to notify the control device 1 of the existence confirmation of the media composition device.

  The resource confirmation unit 28 confirms the resource information in the media composition device (server) and periodically transmits it to the control device 1 via the communication unit 26. Here, the resource information in the server includes, for example, a CPU usage rate, a memory usage amount (memory remaining amount), a network bandwidth usage rate, and the like. Thereby, it is possible to notify the control device 1 of the processing load of the media composition device.

  The communication quality measurement unit 29 periodically measures the communication quality between the uplink relay device 3 and the downlink relay device 4 and transmits the measurement result to the control device 1 via the communication unit 26. As a method for measuring communication quality, for example, a method using information obtained from an RTCP packet, a method of periodically transmitting / receiving a test packet, and measuring a delay time, fluctuation, jitter, etc. of the test packet are applied. it can.

  The uplink relay device 3 is interposed between the plurality of communication terminals 5-1 to 5-4 and the media composition device 2-1, and the video / audio received from each of the communication terminals 5-1 to 5-4. Data is given to the media composition device 2-1.

  FIG. 3 is a functional block diagram showing the internal configuration of the uplink relay device 3.

  In FIG. 3, the uplink relay device 3 includes data receiving units 31-1 to 31-4, a sound volume measuring unit 32, a filter unit 33, data transmitting units 34-1 to 34-4, a communication unit 35, and a communication quality measuring unit 36. And at least a filter control unit 37.

  The data receiving units 31-1 to 31-4 receive video / audio data from the communication terminals 5-1 to 5-4 via the network NT on the communication terminal side. This is given to the filter unit 33. Further, the data reception units 31-1 to 31-4 have a reception packet measurement function unit 311.

  The received packet measuring function unit 311 receives packet loss and delay of the packet sequence from the corresponding communication terminals 5-1 to 5-4 based on the RTP header and RTCP header of the received RTP (Real-time Translation Protocol) data. Time, jitter, and the like are measured, and the measurement results are given to the control device 1 via the communication unit 35.

  The volume measuring unit 32 calculates the volume of the audio data received by the data receiving units 31-1 to 31-4 and transmits the measured value to the control device 1 via the communication unit 35.

  Here, the sound volume measuring method by the sound volume measuring unit 32 is not particularly limited, and various methods can be applied.

  For example, the sound volume measurement unit 32 measures the difference between the maximum value (MAX) and the minimum value (MIN) of the instantaneous sound pressure time-series data in each frame section based on the reception data of each channel, For example, a method of measuring a difference between, for example, a 90% MAX value and a 90% MIN value can be applied. For example, the sound volume measurement unit 32 can apply a method of measuring the average power value of each frame section based on the reception data of each channel.

  The filter control unit 37 receives the volume measurement value from the volume measurement unit 32, determines which channel data is to be passed based on the volume measurement value, and transmits the determination instruction to the filter unit 33. Is. The filter control unit 37 transmits the determination instruction information transmitted to the filter unit 33 to the control device 1 through the communication unit 35.

  The filter unit 33 receives the command from the control device 1 or the command from the filter control unit 37, and filters the data from the data receiving units 31-1 to 31-4. Thereby, according to the command from the control apparatus 1 or the filter control part 37, passage of data can be blocked | prevented and passage of other data can be permitted.

  The data transmission units 34-1 to 34-4 transmit the data passed from the filter unit 33 to the network NT toward the media composition devices 2-1 and 2-2.

  The communication unit 35 is a communication processing unit that communicates with the control device 1.

  The communication quality measuring unit 36 periodically measures the communication quality with the media synthesizing device 2-2 and transmits the measurement result to the control device 1 via the communication unit 35. As a method for measuring the communication quality, for example, a method of periodically transmitting / receiving a test packet and measuring the delay time, fluctuation, jitter, etc. of the test packet can be applied. As a result, the communication quality of the network with the media composition device 2-2 that is the switching destination can be measured.

  The downlink relay device 4 copies the data received from the media synthesis devices 2-1 and 2-2 by the number corresponding to the number of connected communication terminals, and the copied data is transmitted to the communication terminals 5-1 to 5-4. To send to.

  FIG. 4 is a functional block diagram showing the internal configuration of the downlink relay device 4. In FIG. 4, the downlink relay device 4 includes at least a data reception unit 41, a data distribution unit 42, data transmission units 43-1 to 43-4, and a communication unit 44.

  The data receiving unit 41 receives the synthesized data from the media synthesizing apparatuses 2-1 and 2-2 via the network NT and supplies the synthesized data to the data distributing unit 42. In addition, the data receiving unit 41 includes a received packet measuring function unit 411.

  The received packet measurement function unit 411 measures packet loss, delay time, jitter, and the like of the packet sequence based on the RTP header and RTCP header of the received RTP data, and controls the measurement result via the communication unit 44. This is given to the device 1.

  The data distribution unit 42 copies the required number of data from the data reception unit 41 and gives the copied data to the data transmission units 43-1 to 43-4.

  The data transmission units 43-1 to 43-4 transmit data from the data distribution unit 42 to the network NT toward the communication terminals 5-1 to 5-2.

  The communication unit 44 is a communication processing unit that communicates with the control device 1.

  The communication quality measuring unit 45 periodically measures the communication quality with the media synthesizing device 2-2 and transmits the measurement result to the control device 1 via the communication unit 44. As a method of measuring the communication quality, for example, there is a method of periodically transmitting / receiving a test packet and measuring a delay time, fluctuation, jitter, and the like of the test packet. Thereby, it is possible to measure the communication quality of the network with the switching destination media composition device 2-2.

  The control device 1 controls processing of the media composition devices 2-1 and 2-2, the upstream relay device 3, and the downstream relay device 4. The control device 1 performs switching control between the media composition devices 2-1 and 2-2 (MCU).

  FIG. 5 is a functional block diagram showing the internal configuration of the control device 1. In FIG. 5, the control device 1 includes an MCU switching unit 11, a communication unit 12, a survival confirmation unit 13 that periodically transmits a survival confirmation signal (such as ping) to the media composition devices 2-1 and 2-2. The network connection configuration management unit 14, the conference state data management unit 15, and the filter control unit 16 are included.

  The MCU switching unit 11 performs switching control between the media composition apparatuses 2-1 and 2-2. The MCU switching unit 11 includes at least a data storage unit 111, a state calculation unit 112, and an instruction execution unit 113.

  Here, as the scene where the MCU switching process is necessary, the following situation can be considered.

(A) When the media synthesizing apparatus 2-1 is stopped (b) When the processing performance of the media synthesizing apparatus 2-1 is increased, the processing performance is lowered, and the audio / video is deteriorated, or is deteriorated in the near future. (C) When the communication quality between the uplink relay device 3 or the downlink relay device 4 and the media composition device 2-1 is reduced, and when the quality is expected to be reduced in the near future MCU switching unit 11 However, as a method of grasping the above situation and switching the MCU, there are the following methods.

(A) A liveness confirmation signal (for example, ping) is periodically transmitted and received between the control device 1 and the media composition devices 2-1 and 2-2 to determine whether the media composition device 2-1 is operating. Inspect.

(B) Data such as CPU usage rate, memory usage amount, and network bandwidth usage rate in the media synthesis devices 2-1 and 2-2 are periodically collected and transmitted to the control device 1.

(C) Obtain numerical values such as packet loss, delay time, and jitter from the packet sequences received by the data reception units 21-1 to 21-4 of the media synthesis device 2-1 and the data reception unit 41 of the downlink relay device 4, The result is transmitted to the control device 1.

(D) The test packet is transmitted / received between the uplink relay device 3 and the downlink relay device 4 and the media composition device 2-2, the communication quality between the devices is periodically measured, and the measurement result is sent to the control device 1. Send.

  The data storage unit 111 communicates with the media synthesis devices 2-1 and 2-2, the upstream relay device 3, and the downstream relay device 4 that constitute the distributed MCU through the communication unit 12, and is information necessary for MCU switching. Is memorized.

  Based on the information for MCU switching stored in the data storage unit 111, the state calculation unit 112 determines the respective states of the media synthesis devices 2-1 and 2-2, the upstream relay device 3, and the downstream relay device 4, The state of the distributed MCU as a set of these components is quantified. The state calculation unit 112 determines whether or not MCU switching is necessary based on each digitized state.

  The instruction execution unit 113 acquires necessary data such as the configuration and connection relationship in each device from the data storage unit 111 based on the MCU switching determination of the state calculation unit 112, and sends a control command necessary for MCU switching through the communication unit 12. Is issued.

  The network connection configuration management unit 14 manages data necessary for the connection relationship of the network configuration of the conference system 10.

  For example, the connection information between the communication terminals 5-1 to 5-4 and the data receiving units 31-1 to 31-4 of the uplink relay device 3, the data transmitting units 34-1 to 34-4 of the uplink relay device 3, and Connection information between the data receivers 21-1 to 21-4 of the media synthesizer 2 and between the data transmitters 25-1 to 25-2 of the media synthesizer 2 and the data receiver 41 of the downlink relay device 4 The network connection configuration management unit 14 obtains at least all or a part of the connection information between the data transmission units 43-1 to 43-4 of the downlink relay device 4 and the communication terminals 5-1 to 5-4. to manage.

  The conference state management unit 15 manages state data related to the virtual conference room held by the media composition device 2-1. When switching the media composition device, the current virtual conference room state data is managed so that the state of the virtual conference room can be taken over as it is. At the time of switching, the MCU switching unit 11 copies this state data as the switching destination media composition apparatus. Thereby, a virtual conference room can be taken over as it is.

  The filter control unit 16 receives the volume measurement value of each channel from the uplink relay apparatus 3 and determines which channel data is allowed to pass based on the volume measurement value. Is transmitted to the uplink relay apparatus 3 via

  The communication terminals 5-1 to 5-4 are, for example, video conference dedicated terminals, personal computers (desktop PCs, notebook PCs, etc.), and mobile terminals (for example, mobile phones, PDAs, game machines, etc.) having communication functions. Can be applied.

(A-2) Operation of the First Embodiment Next, the operation of processing in the conference system 10 of the first embodiment will be described with reference to the drawings. In the following, the communication process of the conference system 10 will be described first, and then the operation of the MCU switching process will be described.

(A-2-1) Communication Process of Conference System 10 FIG. 7 is a flowchart for explaining the operation of the communication process in the conference system 10.

  First, communication terminals 5-1 to 5-4 or MCUs who wish to participate in the conference make a call and establish a connection between the virtual conference room created in the distributed MCU and the communication terminals 5-1 to 5-4. (Step S101).

  Specifically, the connection between the communication terminals 5-1 to 5-4 and the data reception units 31-1 to 31-4 of the uplink relay device 3, and the data transmission units 34-1 to 34-4 of the uplink relay device 3 4 between the data receivers 21-1 to 21-4 of the media synthesizer 2 and the data transmitters 25-1 to 25-2 of the media synthesizer 2 and the data receiver 41 of the downlink relay device 4 Connection between the data transmission units 43-1 to 43-4 of the downlink relay device 4 and the communication terminals 5-1 to 5-4, based on the address information and port number information of each connection destination Established.

  Further, the uplink relay device 3, the media synthesis device 2-1, and the downlink relay device 4 generate necessary processing units according to the control of the control device 1 (step S102).

  At this time, the uplink relay device 3 receives a notification of the number of communication terminals 5-1 to 5-4 connected from the control device 1, and receives the same number of data receiving units 31-1 to 31-4 and data as the number of communication terminals. Transmitters 34-1 to 34-4 are created.

  In addition, when receiving the notification of the number of communication terminals 5-1 to 5-4 connected from the control device 1, the media composition device 2-1 receives the same number of data receiving units 21-1 to 21-4 as the number of communication terminals. And decoding units 22-1 to 22-4 are generated, and any number of encoding units 24-1 to 24-2 and data transmission units 25-1 to 25-2 instructed by the control device 1 are generated. Is done.

  Further, in the downlink relay device, upon receiving the notification from the control device 1, a set of one data receiving unit 41, data distributing unit 42, and any number of data transmitting units 43-1 to 43-4 is combined in the media composition. It is created for the number of channels of data distributed from the device 2-1.

  After the call establishment and the processing unit in each device (media synthesis device 2-1, uplink relay device 3 and downlink relay device 4) are created, communication terminals 5-1 to 5-4 transmit audio / video data. Start is performed (step S103).

  The video / audio data transmitted from the communication terminals 5-1 to 5-4 is received by the uplink relay device 3 (step S104), and only necessary data is sent to the media composition device 2-1 in accordance with a command from the control device 1. It is transmitted (step S105).

  For example, when the media synthesizer 2-1 synthesizes the screens and sounds of a plurality of people participating in the conference, a method of determining the person to be displayed based on the magnitude of the sound can be applied.

  In this case, the volume measuring unit 32 of the uplink relay device 3 digitizes the volume based on the audio data of each channel received by the data receiving units 31-1 to 31-4, and the measurement result is represented by the control device 1. To communicate.

  The control device 1 determines the audio and video channels to be synthesized based on the measurement result of each channel measured by the uplink relay device 3 and notifies the uplink relay device 3 of this. The determination of the channel in the control device 1 can be, for example, one channel with the highest volume or a predetermined number of channels in descending order.

  Upon receiving this channel list to be combined, in the upstream relay device 3, the filter unit 33 passes only the video / audio data of the channel designated by the control device 1 or the filter control unit 37, and other video / audio data. Filter the passage of.

  Thereby, according to the command from the control apparatus 1, only the data of a required channel can be transmitted to the media composition apparatus 2-1.

  In response to the command from the control device 1, the media composition device 2-1 synthesizes the video / audio data received from the upstream relay device 3 (step S106).

  Here, in the media synthesizer 2-1, the video / audio data received by the data receiving units 21-1 to 21-4 is decoded by the corresponding decoding units 22-1 to 22-4.

  In the synthesizing unit 23, the decoded audio data is subjected to conversion processing such as amplification and attenuation in accordance with a command from the control device 1 and then added. Further, the video data is subjected to conversion processing such as reduction and enlargement in accordance with a command from the control device, and is synthesized, for example, by being arranged at a predetermined position in a plane (in a frame) illustrated in FIG.

  The synthesized video / audio data is transmitted from the media synthesis device 2-1 to the downlink relay device 4 (step S107).

  In the downlink relay device 4, a necessary number of data transmission units 43-1 to 43-4 are created according to instructions from the control device 1, and each data transmission unit 43-1 to 43-4 is connected to the communication terminal 5-. 1 to 5-4 are connected to the channel.

  The data (synthesized video / audio data) received by the data receiver 41 is copied by the data distributor 42 by the number corresponding to the number of the data transmitters 43-1 to 43-4 (step S108). This is given to each data transmission part 43-1 to 43-4, and is distributed to each communication terminal 5-1 to 5-4 (step S109).

(A-2-2) MCU Switching Process Next, the operation of the MCU switching process will be described with reference to the drawings.

  The MCU switching unit 11 of the control device 1 acquires the following information among the media synthesis device 2-1, the upstream relay device 3, and the downstream relay device 4, and performs MCU switching processing based on these information. When the MCU is switched, an instruction to that effect is transmitted to the media composition device 2-1, the upstream relay device 3, and the downstream relay device 4.

(A) A survival confirmation signal (such as ping) is periodically transmitted and received between the control device 1 and the media composition device 2-1, and it is periodically checked whether the media composition device 2-1 is operating.

(B) Data such as CPU usage rate, memory usage amount, and network bandwidth usage rate in the media synthesis device 2-1 are periodically collected and transmitted to the control device.

(C) Obtain numerical values such as packet loss, delay time, and jitter from the packet sequences received by the data reception units 21-1 to 21-4 of the media synthesis device 2-1 and the data reception unit 41 of the downlink relay device 4, The result is transmitted to the control device 1.

(D) The test packet is transmitted / received between the uplink relay device 3 and the downlink relay device 4 and the media composition device 2-2, the communication quality between the devices is periodically measured, and the measurement result is sent to the control device 1. Send.

  In the following, when the media synthesis device 2-1 is stopped or the processing performance is reduced due to an increase in processing load, or when the communication quality between the media synthesis device 2-1 and each of the relay devices 3 and 4 is lowered, A specific procedure for switching from the media composition device 2-1 to the media composition device 2-2 will be described with reference to the flowchart of FIG.

  The control device 1 receives the data related to the above (a) to (d) for determining the switching process from the media composition device 2-1, the upstream relay device 3, and the downstream relay device 4 (step S201), and stores these data in the data storage Store in the unit 111.

  Based on the data stored in the data storage unit 111, the state calculation unit 112 has a video / audio quality degradation in the virtual conference room using the media synthesis device 2-1, or transmission / reception has stopped. Or a determination is made that a problem will occur in the near future (step S202).

  At this time, the control device 1 also acquires data relating to the state of the media composition device 2-2 as an alternative media composition device. That is, data on the processing load state of the alternative media composition device 2-2 and the communication quality between the media composition device 2-2 and the upstream relay device 3 and downstream relay device 4 are also collected.

  Further, as a determination method by the state measuring unit 112 of the control device 1, the following method can be applied.

  For example, the method (a) is a method for confirming the survival of the media composition device 2-1 itself. Therefore, when a response signal to ping is not returned from the media synthesizing apparatus 2-1, the state measurement unit 112 immediately performs a switching process to the alternative media synthesizing apparatus 2-2.

  The method (b) is a method based on the resource status of the media composition device 2-1. In this case, for example, a threshold value for executing the switching process is provided in advance, and the state measurement unit 112 performs the switching process when the CPU usage rate, the memory usage amount, the network bandwidth usage rate, or the like exceeds the threshold value. Is determined.

  The method (c) is a method based on the communication quality of the network between the media composition device 2-1, the upstream relay device 3, and the downstream relay device 4. In this case, for example, a threshold value for executing the switching process is provided in advance, and the state measurement unit 112 performs data reception of the data reception units 21-1 to 21-4 of the media synthesis device 2-1 and the data of the downlink relay device 4. When numerical values such as packet loss, delay time, and jitter exceed the threshold from the packet sequence received from the reception unit 41, the state measurement unit 112 determines to perform switching processing.

  The method (d) is a method based on the communication quality of the network between the alternative media composition device 2-2, the upstream relay device 3, and the downstream relay device 4. Since the network communication quality with the alternative media composition device 2-2 can be obtained by this method, it can be compared with the network communication quality with the current media composition device 2-1 obtained by the method (c). Whether or not to perform the switching process can be determined using the comparison result. For example, when the communication quality of the switching destination network is better than the communication quality of the current network by a predetermined value or more, it can be determined that the switching process is possible.

  The state measurement unit 112 may execute the switching process when it is determined that the switching process should be performed by any one of the determination processes according to the methods (b) to (d). Further, a policy such as determining a determination method to be prioritized may be determined in advance, and the switching process may be executed according to the policy.

  In the control device 1, from the state data of the corresponding virtual conference room in the media composition device 2-1 stored in the data storage unit 111, the state / configuration data relating to the virtual conference room forming the same configuration is transmitted to the media composition device 2. A command for instructing to be created is issued (step S203).

  It should be noted that the creation of the status data related to the virtual conference room and the data related to the configuration may be performed in advance before the problem occurs, or after it is determined that the problem has actually occurred in the media composition device 2-1. You may make it do.

  At this time, it is necessary to faithfully copy the configuration, connection relationship, and the like of each unit in the media composition device, but the standby port number of the data reception unit of each device may be a different number.

  After the control device 1 makes a copy of the state / configuration data related to the virtual conference room in the media composition device 2-2, the control device 1 receives the data reception units 21-1 to 21-4 of the media composition device 2-2. Obtains the port number on which it waits (step S206), and transmits the server address information and port number of the media composition device 2-2 to the upstream relay device 3 (step S205).

  The uplink relay device 3 learns that the data transmission destination has been changed, and transmits the audio / video data that has been transmitted to the media composition device 2-1 until then to the media composition device 2-2 (step S206).

  Since the downstream relay device 4 receives the data from the media composition device at the same standby port, there is no particular change in operation, and the newly transmitted data from the media composition device 2-2 is connected. Send to communication terminal.

  This series of operations completes the switching of the media composition device.

(A-3) Effect of First Embodiment As described above, according to the first embodiment, it is possible to dynamically switch the conference connection to the optimum MCU without disconnecting or reconnecting the call. Can be realized.

  In addition, according to the first embodiment, not only when the MCU stops, but also dynamically switching the conference room to the optimum MCU according to the state that dynamically changes during the conference such as the communication quality of the network. Can be realized.

(B) Second Embodiment Next, a second embodiment of the network control system, method and program of the present invention will be described with reference to the drawings.

  A conventional conference system has a connection configuration in which a plurality of communication terminals are connected to a virtual conference room without going through an uplink relay device and a downlink relay device as shown in FIG.

  On the other hand, in the first embodiment, the data communication band between each relay device and the media composition device can be significantly reduced by passing through the uplink relay device and the downlink relay device.

  Therefore, in the second embodiment, a specific embodiment of a connection configuration for connecting to a virtual conference room via an uplink relay device and a downlink relay device will be described as an example.

(B-1) Configuration and Operation of Second Embodiment The configuration of the conference system of the second embodiment is the same as the configuration described in the first embodiment, and FIG. 1 is also used in the second embodiment. I will explain.

  The upstream relay device 3 and the downstream relay device 4 are provided, for example, at the boundary between a local network (for example, an in-house LAN) and a global network (for example, the Internet).

  The uplink relay device 3 generates the data reception units 31-1 to 31-4 and the data transmission units 34-1 to 34-4 according to the command from the control device 1, and at that time, the data reception units 31-1 to 31-1 to 31-4. The unique address information (private IP address) in the local network is assigned to 31-4, and the unique address information (global IP address) in the global network is assigned to the data transmission units 34-1 to 34-4. To do.

  By doing in this way, the uplink relay apparatus 3 is connected between the communication terminals 5-1 to 5-4 and the data receiving units 31-1 to 31-4, and the data transmitting units 34-1 to 34-4. And two sessions of connection between the media synthesizer 2-1.

  As a result, it is possible to realize a function equivalent to a proxy server in the upstream relay device 3. For example, the communication terminals 5-1 to 5-4 disposed in the local area network and the media composition device 2-2 on the global network. Two-way communication with 1 is realized.

  Similarly, when the downlink relay device 4 generates the data reception unit 41 and the data transmission units 43-1 to 43-4 according to the command from the control device 1, the data reception unit 41 has a unique address in the global network. Information (global IP address) is assigned, and unique address information (private IP address) in the local network is assigned to the data transmission units 43-1 to 43-4.

(B-2) Effect of Second Embodiment As described above, the data communication band between each relay device and the media composition device can be significantly reduced by passing through the uplink relay device and the downlink relay device. Can do.

  In addition, a function equivalent to a proxy server can be realized in the relay device, and bidirectional communication between the communication terminal arranged in the local area network and the media composition device 2-1 on the global network can be realized. .

(C) Other Embodiments In the first and second embodiments, the switching process of the media synthesizing apparatus in the video conference has been described. However, some of the communication terminals in FIG. In this case, it can be used as a highly fault-tolerant audio / video broadcasting system using a network.

1 ... Control device,
DESCRIPTION OF SYMBOLS 11 ... MCU switching part, 12 ... Communication part, 13 ... Survival confirmation part, 111 ... Data storage part,
112 ... State calculation unit, 113 ... Instruction execution unit,
2-1 and 2-2 ... Media composition device,
21-1 to 21-4 ... data receiving unit, 22-1 to 22-4 ... decoding unit, 23 ... combining unit,
24-1 to 24-2 ... encoding unit, 25-1 and 25-2 ... data transmission unit,
26 ... Communication unit, 27 ... Survival confirmation unit, 28 ... Resource confirmation unit, 29 ... Communication quality measurement unit,
211 ... Received packet measurement function unit,
3 ... Uplink relay device,
31-1 to 31-4 ... data receiving unit, 32 ... volume measuring unit, 33 ... filter unit,
34-1 to 34-4 ... data transmission unit, 35 ... communication unit, 36 ... communication quality measurement unit,
311: Received packet measurement unit,
4 ... Downlink relay device,
41... Data receiving unit, 42... Data distributing unit, 43-1 to 43-4.
44 ... Communication unit, 45 ... Communication quality measurement unit,
5-1 to 5-4 ... communication terminal, 6 ... gatekeeper, 10 ... conference system.

Claims (14)

Multiple communication terminals,
First media combining means for combining and transmitting a plurality of received media data;
The first medium is arranged between the plurality of communication terminals and the first media composition unit, and transmits only necessary media data among the media data from the plurality of communication terminals to the first media composition unit. Relay means;
It is arranged between the first media synthesizing means and the plurality of communication terminals, and copies the synthesized media data from the first media synthesizing means by the number of the plurality of communication terminals, to the plurality of communication terminals. A second relay means for delivering;
Periodic information is periodically acquired from the first media synthesizing means, the first relay means, and the second relay means, and based on the status information, the second destination of the switching destination is obtained from the first media synthesizing means. A network control system comprising: switching means for switching to the media composition means. Status information management means for managing status information of media composition processing executed by the first media composition means;
At least connection information between the first relay means and the plurality of communication terminals or the first media composition means, and the second relay means and the plurality of communication terminals or the first media composition. A connection information management means for managing connection information with the means,
Before the switching means switches to the second media composition means, the status information and the connection information of the first media composition means are transferred to the second media composition means, and then the first relay 2. The network control system according to claim 1, wherein the switching unit is notified by notifying the address information of the second media composition unit.   2. The status information is information relating to the existence confirmation of the first media composition means, and the switching means performs switching when the existence confirmation of the first media composition means cannot be performed. Or the network control system of 2.   When the status information is resource information of the first media composition unit, and the switching unit determines that there is a processing load based on the resource information of the first media composition unit, switching is performed. The network control system according to any one of claims 1 to 3, wherein   The status information is network quality information between the first relay unit and the second relay unit and the first media composition unit, and the switching unit performs switching based on the network quality information. The network control system according to claim 1, wherein the network control system is performed.   The state information is network quality information between the first relay unit and the second relay unit and the second media composition unit as a switching destination, and the switching unit includes the network quality information. 6. The network control system according to claim 1, wherein switching is performed based on the network.   7. The network control system according to claim 1, further comprising control means for determining and instructing media data to be transmitted from the first relay means to the media composition means. Multiple communication terminals,
First media combining means for combining and transmitting a plurality of received media data;
The first medium is arranged between the plurality of communication terminals and the first media composition unit, and transmits only necessary media data among the media data from the plurality of communication terminals to the first media composition unit. Relay means;
It is arranged between the first media synthesizing means and the plurality of communication terminals, and copies the synthesized media data from the first media synthesizing means by the number of the plurality of communication terminals, to the plurality of communication terminals. A second relay means for delivering;
A network control system comprising: control means for determining and instructing media data to be transmitted from the first relay means to the first media composition means. The first relay unit has a volume measuring unit that measures the volume of each channel when the received media data is audio data, and transmits the measurement result to the control unit;
9. The network control system according to claim 8, wherein the control means determines and instructs media data to be transmitted to the first media composition means based on a measurement result from the sound volume measurement unit. . The first relay means and the second relay means are:
A first communication unit assigned a global address for communicating with the first media composition means;
The network control system according to claim 1, further comprising: a second communication unit provided with a private address for communicating with the plurality of communication terminals. Multiple communication terminals,
First media combining means for combining and transmitting a plurality of received media data;
The first medium is arranged between the plurality of communication terminals and the first media composition unit, and transmits only necessary media data among the media data from the plurality of communication terminals to the first media composition unit. Relay means;
It is arranged between the first media synthesizing means and the plurality of communication terminals, and copies the synthesized media data from the first media synthesizing means by the number of the plurality of communication terminals, to the plurality of communication terminals. And a second relay means for distributing,
The switching means periodically obtains status information from the first media composition means, the first relay means, and the second relay means, and switches from the first media composition means based on the status information. A network control method comprising switching to the second media composition means. Multiple communication terminals,
First media combining means for combining and transmitting a plurality of received media data;
The first medium is arranged between the plurality of communication terminals and the first media composition unit, and transmits only necessary media data among the media data from the plurality of communication terminals to the first media composition unit. Relay means;
It is arranged between the first media synthesizing means and the plurality of communication terminals, and copies the synthesized media data from the first media synthesizing means by the number of the plurality of communication terminals, to the plurality of communication terminals. And a second relay means for distributing,
A network control method, wherein the control means instructs to determine media data to be transmitted from the first relay means to the first media composition means. Multiple communication terminals,
First media combining means for combining and transmitting a plurality of received media data;
The first medium is arranged between the plurality of communication terminals and the first media composition unit, and transmits only necessary media data among the media data from the plurality of communication terminals to the first media composition unit. Relay means;
It is arranged between the first media synthesizing means and the plurality of communication terminals, and copies the synthesized media data from the first media synthesizing means by the number of the plurality of communication terminals, to the plurality of communication terminals. And a second relay means for distributing,
Computer
Periodic information is periodically acquired from the first media synthesizing means, the first relay means, and the second relay means, and based on the status information, the second destination of the switching destination is obtained from the first media synthesizing means. A network control program which functions as a switching means for switching to a media composition means. Multiple communication terminals,
First media combining means for combining and transmitting a plurality of received media data;
The first medium is arranged between the plurality of communication terminals and the first media composition unit, and transmits only necessary media data among the media data from the plurality of communication terminals to the first media composition unit. Relay means;
It is arranged between the first media synthesizing means and the plurality of communication terminals, and copies the synthesized media data from the first media synthesizing means by the number of the plurality of communication terminals, to the plurality of communication terminals. And a second relay means for distributing,
Computer
A network control program that causes the first relay means to function as a control means for determining and instructing media data to be transmitted to the first media composition means.

Images