JPH09247643A - Multi-point video conference system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the image quality by using moving vector information in the case of conducting image synthesis with image segmentation based on coded moving image information. SOLUTION: Video conference terminal equipments 1, 19, 20 relating to the video conference system are connected to an ISDN by an ISDN line 18. Furthermore, video conference terminal equipments relating to the video conference system are not limited to the three terminal equipments. Moreover, a multi-point video conference controller 21 is connected to the ISDN by an ISDN line 31. The multi-point video conference controller 21 is provided with a means notifying a segmented area to each video conference terminal equipment. On the other hand, each video conference terminal equipment has a means to set an additional area of motion vector information based on the segmented area given from the multi-point video conference controller 21 and a means adding the motion vector information only to the set additional area and to send the result to the multi-point video conference controller 21.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multipoint video conference system involving synthesis of transmitted moving image information as encoded.

[0002]

2. Description of the Related Art In a conventional multipoint video conference control device, when cutting out and synthesizing moving image information, it is disclosed in Japanese Patent Laid-Open No.
As disclosed in Japanese Patent Laid-Open No. 63084, the moving image information received from each video conference terminal connected by a line is once decoded and then processed such as combining, re-encoded and transmitted.

However, in the above method, the multipoint video conference control device needs to be provided with the video decoders for the number of video conference terminals to be connected, resulting in an increase in the cost of the device.

Therefore, the ITU-T Recommendation Draft T. 128
As described in Section 13.4.3, the received moving image is not completely decoded (complete decoding means ITU-T Recommendation H.264).
261 indicates that all the decoding procedures shown in FIG. 261 are executed), and only the GOB number is rewritten in the encoded moving image information (this processing is not described in T.128, but H.264). This processing is necessary in consideration of the matching with the H.261).
A method of combining two FCIF moving image information has been proposed.

However, in the above method, the method disclosed in Japanese Patent Laid-Open No.
In the case where a part of an image is cut out like the multi-point video conference system disclosed in Japanese Patent Laid-Open No. 63084/1994 (however, in this multi-point video conference system, decoding is first performed and then cut-out and synthesis are performed. Therefore, it is not possible to deal with the case where the motion vector refers to the outside of the cut-out area in the received moving image information from the video conference terminal. On the side, the entire frame is coded without any motion vector information (MT specified in ITU-T Recommendation H.261).
There is no choice but to do YTRA INTRA or INTER), which causes an increase in the code amount and eventually a deterioration in image quality.

There may be a configuration in which only the image in the cutout region is encoded and transmitted on the side of the video conference terminal. In that case, the ITU-T Recommendation Draft Combined use with the SwitchKung service described in Section 13.1 of 128 (a composite image by an array processor is transmitted to one video conference terminal and another video conference terminal is not composited by a switching service. In the case of transmitting an image from a video conference terminal, for example, the video conference terminals of the speaker and the former speaker are provided with each other's images by a switching service, and the video conference terminals other than those are synthesized by the array processor. We can't accommodate you if you provide.

Further, as a method of synthesizing an image in a multipoint video conference control device, ITU-T recommended draft T. Three methods have been proposed: multiplex mode, transcoder and array processor, as described in 128, section 13.4.

Among these, in the multiplex mode and array processor, if the capacity of the communication band of the moving picture information set in the transmitting side video conference terminal is 1, the receiving side video conference terminal receives the moving picture information for the moving picture information. The premise is to set the communication band of capacity 4.

[0009] However, in a line (not a packet) that is normally exchanged by a line, the band of the line is fixed, and transmission and reception are symmetrical in the fixed band for voice,
It is usually used by dividing it for data and moving images. These procedures are described in ITU-T Recommendation H.264. 221 and 242. Recommendation H. In 221, a transmission / reception asymmetric band can be set, but it is not normally used. In addition, since the remaining part used for voice and data is allocated to the moving picture band, it is not possible to allocate exactly four times the reception band for transmission.

The ITU-T draft recommendation T. 128 of 1
In the switching service shown in Section 3.1, it is necessary to match the band on the receiving side with the band of the video selected as the source. Usually, in order to support the switching service, transmission of each video conference terminal, It is assumed that the receiving band is symmetrical and the band of each terminal is the same.

In view of the above, when considering the above two image synthesizing methods, in the multiplex mode: a) In the case where symmetrical communication bands are assigned to the video conference terminals on the transmitting side and the receiving side, there are many cases. Overflow of the transmission buffer of the point video conference control device occurs. In other words, if a symmetrical communication band is assigned, it will not be put to practical use. b) Even if asymmetric communication bands are assigned to the video conference terminals on the transmitting side and the receiving side, the communication band cannot be accurately set to 1: 4, and therefore a buffer overflow occurs (underflow also occurs). This can be avoided by inserting fill bits in the error correction frame:
ITU-T Recommendation H.264. 261). In addition, it is necessary to reset the video band castle each time the switching service and the continuous presence mode are switched, which takes time. In the array processor: a) When symmetrical communication bands are allocated to the video conference terminals on the transmitting side and the receiving side, overflow occurs in the transmission buffer of the multipoint video conference control device. That is, when the target communication band is assigned, it is not put to practical use. b) Even when asymmetric communication bands are assigned to the video conference terminals on the transmitting side and the receiving side, the communication band cannot be accurately set to 1: 4, and a buffer overflow occurs. In addition, it is necessary to reset the video band each time the switching service and the continuous presence mode are switched, which takes time. Furthermore, in the case where the video information is cut out in the multipoint video conference control device, not only the GOB number but also the address and motion vector information in each layer need to be rewritten, which increases the code amount, A transmit buffer overflow may occur.

Further, when each video conference terminal conducts a multi-point conference via the multi-point video conference control device which synthesizes images by the array processor, a VCU (video command-first-up) is issued from a certain video conference terminal. Date request: See ITU-T Recommendation H.230)
Is issued, the multipoint video conference control device issues a VCU to all the video conference terminals connected via the line, and the INTRA frame (1
A whole frame refers to a frame coded in INTRA mode: Usually in such a frame, ITU-T Recommendation H.264 is used. PTYPE-third bit (Freeze Release) defined in H.261 is turned on (= 1),
Based on the bit, it determines whether or not the frame is an INTRA frame) and combines it with the moving image information and transmits it to the video conference terminal.

However, since the response of each video conference terminal to the VCU issued from the multipoint video conference control device has a time shift, the video conference terminals of the same configuration and the transmission buffer state are the same. Even if
Since a maximum (1 / frame rate) second gap occurs,
Until the video information of the INTRA frames from all video conference terminals is available, there is a case where the combined video information cannot be transmitted as an INTRA frame to the video conference terminals.

However, since the multipoint video conference terminal waits for the video information of the INTRA frame from all the video conference terminals in the transmission buffer, the INTR of the composite video information is obtained.
Although it is possible to configure an A frame, the video information from each video conference terminal will have a large delay that varies from terminal to terminal.
Even after the transmission of the TRA frame, the moving image information from each video conference terminal forming the combined moving image information is left temporally different from each other.

Also, if the video conference terminal that issued the VCU continues to decode the received moving image data, the moving image can be restored without any problem, but normally the video conference terminal issues the VCU during communication. This is the case when an error is detected in the received data. In that case, many video conference terminals operate to freeze the received moving image, wait for an INTRA frame from the multipoint video conference control device, and release the freeze by receiving the INTRA frame. Therefore, when the INTRA frame transmitted in response to the VCU from each video conference terminal received by the multi-point video conference control device is deviated, the INTRA of the composite video information from the multi-point video conference control device to each video conference terminal is changed. Frame transmission will also be delayed, and V will be delayed by the amount of the delay.
The video conference terminal that issued the CU cannot release the freeze of the received moving image, and the moving image is stopped for a certain period of time. When the freeze is released due to the timeout defined in H.261, even if the decoding is restarted due to the timeout, the INTRA frame that should be referred to for the received INTER frame has not yet been received. The image becomes a distorted image.

On the contrary to the above case, when the multipoint video conference control device detects an error in the received moving image data from a certain video conference terminal, the multipoint video conference control device normally sends the video conference terminal to the video conference terminal. The VCU is issued in response. However, when the multi-point video conference control device is performing image composition by the array processor, it does not decode the received video data, so it is similar to the processing of the received moving image at the video conference terminal described above. However, it is impossible to freeze the received moving image for a while.

The operation of the multipoint video conference control device in such a case is as follows: a) The error frame is left as it is or only the frame in which the error is detected is discarded. b) Insert invalid data (fill frame) until there is a VCU response. Can be considered.

However, in the case of a), a decoding error occurs on the side of the video conference terminal that has received the composite moving image information including the error frame, and the image is disturbed. In the case of b), only a part of the image is stationary. Therefore, in the video conference terminal, the data of the image area for one terminal after the synthesis is lost, and the image of that part is not updated and freezes visually. Therefore, whether it is an error or is still still. Cannot be determined.

[0019]

As described above,
When the multipoint video conference control device synthesizes the moving image information received from each video conference terminal and sends the synthesized moving image information to the video conference terminal, various problems as described above may occur. was there.

The present invention has been made in view of the above circumstances, and the multi-point video conference control device synthesizes the moving image information, which has been encoded, received from each video conference terminal, and synthesizes the synthesized moving image information. It is an object of the present invention to provide a video conference system that can eliminate the problems that may occur when transmitting to a video conference terminal.

[0021]

A multipoint video conference system according to claim 1, wherein video conference terminals provided at a plurality of points are connected to a multipoint video conference control device, and each of the video conference terminals is connected to a multipoint. While transmitting the encoded moving image information to the video conference control device, the multipoint video conference control device synthesizes a cutout area obtained by cutting out a part of the video information from each of the video conference terminals, and the synthesized video In the multipoint video conference system for transmitting information to each video conference terminal, the multipoint video conference control device,
Means for notifying the respective video conference terminals of the cutout area, while each of the videoconference terminals sets an additional area of motion vector information based on the cutout area notified from the multipoint videoconference control device. And a means for adding motion vector information only to the set additional area and transmitting the motion vector information to the multipoint video conference control device.

In the multipoint video conference system according to the second aspect, the video conference terminals provided at a plurality of points are connected to the multi point video conference control device, and each of the video conference terminals codes the multi point video conference control device. In the multi-point video conference system that transmits the converted video information, the multi-point video conference control device synthesizes the video information from the video conference terminals, and sends the synthesized video information to the video conference terminals. The multipoint video conference control device,
While providing means for notifying each of the video conference terminals of the method of synthesizing the moving image information, each of the video conference terminals controls the multi-point video conference control according to the method of synthesis notified from the multi-point video conference control device. It is characterized by comprising means for adding a predetermined amount of invalid information to the encoded moving image information transmitted to the device.

According to another aspect of the present invention, there is provided a multipoint videoconference system, wherein videoconference terminals provided at a plurality of points are connected to a multipoint videoconference control device, and the multipoint videoconference control device codes from the respective videoconference terminals. In the multi-point video conference system that transmits the converted video information, the multi-point video conference control device synthesizes the video information from the video conference terminals, and sends the synthesized video information to the video conference terminals. The multipoint video conference control device,
Means for detecting an intra-frame compressed area from the encoded moving image information transmitted from each of the video conference terminals;
A means for monitoring the storage status of the video transmission buffer for temporarily storing the video information to be transmitted to each video conference terminal, and each video conference when the storage amount of the video transmission buffer exceeds a predetermined amount. A moving picture information stored in a moving picture receiving buffer for temporarily storing the coded moving picture information transmitted from the terminal, and deleting the moving picture information corresponding to the area in which the intra-frame compression is detected. It is characterized by that.

In the multipoint video conference system according to claim 4, the video conference terminals provided at a plurality of points are connected to the multi-point video conference control device, and the multi-point video conference control device codes from each of the video conference terminals. In the multi-point video conference system that transmits the converted video information, the multi-point video conference control device synthesizes the video information from the video conference terminals, and sends the synthesized video information to the video conference terminals. The multipoint video conference control device,
Means for storing image information in which a predetermined image is compressed in a frame, and image information in which the predetermined image is compressed in a frame is transmitted to each of the video conferences when a forced screen update request is made from each of the video conference terminals. Means for transmitting to the terminal.

In the multipoint video conference system according to a fifth aspect of the present invention, the video conference terminals provided at a plurality of points are connected to the multi point video conference control device, and each of the video conference terminals codes the multi point video conference control device. In the multi-point video conference system that transmits the converted video information, the multi-point video conference control device synthesizes the video information from the video conference terminals, and sends the synthesized video information to the video conference terminals. The multipoint video conference control device,
Means for detecting a transmission error of moving surface information from each of the video conference terminals, means for storing image information in which a predetermined image is compressed in a frame, and moving image information from the video conference terminal for detecting a transmission error of moving image information. Is replaced with image information in which a predetermined image is compressed in a frame, is combined with the moving image information from each of the video conference terminals, and the combined moving image information is transmitted to each of the video conference terminals. .

A multipoint video conference system according to a sixth aspect is the multipoint video conference system according to the fifth aspect, wherein the multipoint video conference control device is assumed as image information in which the predetermined image is compressed in a frame. A transmission rate, a transmission rate set between each of the video conference terminals and a unit for storing a plurality of pieces of image information having different amounts of information according to the composite number of moving image information and / or the frame rate
A unit for selecting any one of a plurality of image information having different information amounts as image information obtained by intraframe-compressing the predetermined image according to the number of combined moving image information and / or the frame rate. Is characterized by.

[0027]

BEST MODE FOR CARRYING OUT THE INVENTION A control method for a video conference system according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

FIG. 1 shows the configuration of a tele-teleconference system to which the control method of the teleconference system according to this embodiment is applied.

In the figure, reference numerals 1, 19 and 20 denote video conference terminals having the same configuration, which are related to the present invention, and are ISDs.
It is connected to the ISDN network by an N line 18. Although not shown, the video conference terminals related to the present invention are not limited to the three devices 1, 19, and 20. Reference numeral 21 is a multipoint video conference control device,
It is connected to the ISDN network by the SDN line 31.

FIG. 2 shows a block configuration of the video conference terminal 1 of the video conference terminals related to the present invention.

In FIG. 1, reference numeral 2 denotes a system control unit including a CPU, memory, timer, etc., which controls the entire system, 3 is a magnetic disk device for storing various programs and data, and 4 is a layer 1 of ISDN. The ISDN interface unit 5 which performs signal processing and D-channel layer 2 signal processing is ITU-T Recommendation H.264. 221 is a multimedia multiplexing / demultiplexing unit that multiplexes / demultiplexes data of a plurality of media, 6 is a microphone for audio input, 7 is an A / A after amplifying an input signal from the microphone 6. An audio input processing unit that performs D conversion, 8 is an audio encoding / decoding unit that performs encoding / decoding / echo cancellation of the audio signal, and 9 is an audio signal decoding unit that converts the audio signal decoded by the audio encoding / decoding unit 8 into a D / Audio output processing unit that amplifies after A conversion,
Reference numeral 10 denotes a speaker for outputting audio from the audio output processing unit 9, reference numeral 11 denotes a video camera for inputting video, and reference numeral 12 denotes NTSC decoding of a video signal from the video camera 11.
A video input processing unit 13 for performing signal processing such as A / D conversion is provided in accordance with ITU-T Recommendation H.264. Encoding of moving images conforming to H.261
A moving image encoding / decoding unit 14 for decoding is a video output for performing signal processing such as D / A conversion, NTSC encoding, and graphics synthesis on the video signal decoded by the moving image encoding / decoding device 13. A processing unit, 15 is a monitor for displaying the received moving image and graphics information, 16 is a user interface control unit for controlling a console, 1
7 is a console comprising operation keys and a display unit, 18 is I
SDN line.

FIG. 3 shows a block configuration of the multipoint video conference control device 21. In FIG. 1, reference numeral 22 denotes a system control unit that controls the entire system and includes a CPU, a memory, a timer, and the like.

Reference numeral 23 is an ISDN interface section, and 24
Is a multimedia demultiplexing / separating unit, 25 is a voice encoding / decoding unit that encodes / decodes an audio signal, and 26 is a moving image correction code generating unit (ITU) that adds a correction code to moving image data to be framed. -T Recommendation H.261), 27
Is a moving image data transmission buffer, 28 is a moving image error correction / detection unit that detects and corrects frame synchronization of received moving image data (see ITU-T Recommendation H.261), and 29 is a moving image data reception buffer. And 23
The communication channel 1 is composed of the components 29 to 29.

Although the above-mentioned configuration is the configuration of the communication channel 1, as shown in the drawing, the multipoint video conference control device 21 is provided.
Comprises 1 to n communication channels, communication channel 1
Although not shown, the other communication channels have the same configuration as the communication channel 1 and are connected to ISDN lines.

Among the connections shown in FIG. 3, the connection between each communication channel and the audio / video multiplex section 30 (voice data transmission / reception, video data transmission / reception) is described in detail below.
As shown in FIG. 4, each channel has a separate connection. Note that FIG. 4 will be described later.

The audio / video multiplex section 30 is
The person who synthesizes and delivers the audio and video data decoded in each communication channel 1 to n between the channels. 3
1 is an ISDN line connected to each communication channel.

Next, the basic operation of the video conference system will be described with reference to FIG. In the figure, when activating a video conference, it is necessary to first connect a line. This follows the normal calling procedure done through LAPD. SETUP (call setup message) is a transmission capability (B
C) is unrestricted digital, lower layer integrity (LLC) is H.264. 221, send high layer consistency (HLC) as a conference.

When the partner terminal analyzes SETUP and the communication possibility is approved, the partner terminal returns CONN (response) and the call is established. Here, in lower layer consistency, H.264 is used. Reference numeral 221 denotes the multimedia multiplexing / multiplexing in FIG.
The ITU-T Recommendation H.264 implemented in the separation unit 5 221 has been implemented.

When the call is established, the system control unit 2 controls the multimedia multiplexing / demultiplexing unit 5 to send a multiframe synchronization signal and establish multiframe synchronization.
Furthermore, the system control unit 2 uses the ITU-T Recommendation H.264. In accordance with H.242, the multimedia multiplexing / demultiplexing unit 5 is controlled to notify the capability and establish a communication mode. This is described in H. This is performed on the BAS signal on the H.221, and the necessary channel setting and bit rate allocation are performed with the common capability. In this embodiment, three channels of audio, moving image, and data (MLP) are assigned. When the communication mode is established, each channel can be treated as independent data, and the operation as a video conference starts.

By performing the above procedure between each video conference terminal and each communication channel of the multipoint video conference control device 21, a multipoint video conference can be performed via the multipoint video conference control device 21. Becomes

A dummy data channel (LSD) is established in order to set the transmission / reception of moving image data communication channels to asymmetrical 1: 4. For example, if the communication zone of the line is ISDN BRI: 128 kbps, receiving side: 3.2K (FAS / BAS) + 6.4K
(MLP) + 64K (audio) + 54.4K (video) Sending side: 3.2K (FAS / BAS) + 6.4K
(MLP) + 64K (voice) + 40K (LSD) +1
4.4K (moving image) is set, and the communication band (14.4K) of the moving image data on the transmitting side is set to the communication band (5
4.4K), set to about 1/4, and the remaining 3/4
(40k) is set as a dummy LSD.

However, as is clear from the above example, the communication band that can be taken by the LSD cannot select an arbitrary capacity, and there is no choice but to select from a predetermined band (ITU-T Recommendation H). .221), it is not possible to accurately set the communication band of 1: 4 in the moving image data to which the remaining communication band obtained by subtracting the communication capacity allocated to other data from the communication band is allocated.

When the video conference is activated, the system control unit 2 activates the voice encoding / decoding unit 8 and the moving image encoding / decoding unit 13, and bidirectional communication of voice, moving image and data becomes possible. .

On the data channel (MLP), the ITU
-T Recommendation Draft Various data related to the conference operation specified in the 120 series are exchanged. The data on the data channel is sent to the multimedia multiplexer / separator 5
(24 on the multipoint video conference control device 21 side, the same below)
Is separated and synthesized with audio and video data. The system control unit 2 (or 22) reads / writes data from / to the multimedia multiplexing / demultiplexing unit 5 (or 24), and each protocol shown in the above draft recommendation is on the system control unit 2 (or 22). To be executed.

At the end of the video conference, the system control unit 2
Are the audio encoding / decoding unit 8 and the video encoding / decoding unit 13
And the ISDN interface unit 4 is controlled and the call is released according to the procedure shown in FIG.

The user uses the operations (calling,
The start of “end of meeting” is performed by operating the console 17.
The input operation data is notified to the system controller 2 via the user interface controller 16. The system control unit 2 analyzes the operation data, starts or stops an operation according to the operation content, creates display data of guidance to a user, and sends the guidance display data to the console 17 via the user interface control unit 16. Display.

On the multipoint videoconference control device 21 side, one videoconference terminal is connected for each communication channel and a videoconference between multipoints is operated by the procedure described above.
In addition, in the above-mentioned example, an example of connecting by calling from the video conference terminal side has been described, but a call is made from the multipoint video conference control device 21 side to a video conference terminal set at a predetermined time, You can also connect.

Next, the process of multiplexing audio and moving images will be described. FIG. 4 shows the configuration of the audio / video multiplex unit 30 of the multipoint video conference control device 21. In the figure, 101 is a speaker detection unit that monitors the volume level of decoded audio data from each communication channel and detects from which channel (video conference communication terminal) the volume level is maximum, and 102 is An audio mixing unit that weights and mixes audio data from each communication channel, 103 is a matrix switch, and distributes the audio data from each communication channel and the mixing data from the audio mixing unit 102 to each communication channel. The audio switching unit 104 is an array for synthesizing moving image data from each communication channel (not decoded, code data as it is) by a method as described in Section 13.4.3 of ITU-T draft draft T128. Processor part,
A matrix switch 105 is a moving image switching unit that distributes the moving image data from each communication channel and the combined data from the array processor unit 104 to each communication channel.

The system control unit 22 uses the ITU-T recommended draft T. According to 120 series protocol, or /
Also, by the adaptive control based on the parameters set in advance in the system control unit 22, the form of synthesizing voice and moving image (weighting of each channel in the voice mixing unit 102, synthesizing position, shape, and voice of moving image in the array processor unit 104). The distribution form in the switching unit 103 and the distribution form in the moving image switching unit 105) are determined and set in each unit. When it is necessary to identify the speaker as a determinant of the above-mentioned combination form, the channel number (terminal number) determined to be the speaker is read from the speaker detection unit 101 and used as a factor.

As an example, when nine video conference terminals are connected and a conference is held, the terminal number 2 as a speaker is
FIG. 6 shows an example of output data to each channel when the terminal number 3 is detected as the last speaker. Further, an example of the synthesizing form (ratio) in the audio mixing unit 102 at this time is shown in FIG. 7, and the synthesizing form in the array processor unit 104 is shown in FIG.

FIG. 8 also shows an example of processing in the array processor section 104 accompanied by image cutout. In the array processor, an image of 1/4 of the image format to be transmitted (horizontally and horizontally) is received and combined, but in this example, a part of the received image is further cut out and combined. There is. Here, an example is shown in which FCIF format is used for transmission and QCIF format is used for reception. "MB" in the figure
A macro block indicates a macro block, which is cut out and combined with this MB as a minimum unit.

FIG. 8 as it is as coded data (without decoding)
In order to synthesize the image as shown in FIG. It is necessary to rewrite the address information in each layer in the frame structure in 261. In addition, the value of the quantization step size is also adjusted as necessary (this value is not necessarily added in the MB layer, so the value added in the GOB layer or the value previously changed in the MB layer is managed and combined. Compare to the GOB layer at the position or the previous value, and add as necessary). A motion vector cannot be added because there is no way to refer to it when the area outside the cutout area in the image is pointed out (all frames are coded on the video conference terminal side without a motion vector).

Next, some operation procedures in the video conference system according to the present invention will be described separately for each embodiment.

First, the first embodiment will be described. In the first embodiment, the multipoint video conference control device notifies the video conference terminal of the cutout area when the array processor performs the combining process, and the videoconference terminal is based on the notified cutout area. Then, the motion vector addition area is set to encode the moving image.

The cutout area is notified when the array processor section 104 is activated and when the composition mode is changed in the section.
H. Notification is made using the BAS command (MBE) on 221 (notification is also performed when the entire one frame that is not cut out is used). Therefore, in the first embodiment, the capability exchange is performed with the MBE capability when performing the capability exchange shown in FIG. (As another method, a protocol on MLP may be used.)

The procedure of the process in which the video conference terminal sets the motion vector addition region and encodes the moving image in response to the notification of the cut-out region from the multipoint video conference control device 21 will be described with reference to FIG. . Note that only the video conference terminal 1 will be described as a representative of the video conference terminals connected to the multipoint video conference control device via a line, but the same applies to the other video conference terminals (the embodiments described below). Is also the same).

In FIG. 9, when the video conference terminal 1 receives a BAS from the multipoint video conference control device 21, the system control unit 2 sends it to the multimedia multiplexing / demultiplexing unit 5.
It is read from, and it is checked whether or not it is the notification of the cutout area (decision 1001).

FIG. 10 shows an example of the BAS command in that case. In the figure, the first data is H.264 indicating the start of MBE. 221 data (command: 0xF
9), followed by the number of bytes of data (5 bytes) and an identifier (0x1) indicating that the data is a cut-out area notification.
D) is data indicating the cutout area. The data indicating the cutout area is data for designating the x-coordinate and the y-coordinate in macroblock (MB) units with one of the four vertices of the quadrangle (top left vertices) as the cutout start position and the quadrangle. Of the macro block (MB) in the x-coordinate and y-coordinate directions.

Now, in the procedure of FIG. 9, the received BA
S is a notification of the cutout area (Yes in determination 1001)
s), the system control unit 2 determines the additional area of the motion vector information (process 1002).

The additional area to be set is (1) The area inside the 1 MB image from the cut-out area is the motion vector additional area. (2) However, when the cutout area is in contact with the edge surface of the image area, its side is the same as the cutout area. Judgment according to the two-point rule.

FIG. 11 shows the relationship between the image cutout area and the motion vector addition area. Since the macroblocks at the edges of the cutout area may refer to the outside of the cutout area in order to obtain the motion vector, in order to guarantee the decoding of the motion vector in the multipoint video conference control device 21, FIG. As shown in (a), the area to which the motion vector is added is an area excluding the edge of one macroblock width of the cutout area. As a result, in the multipoint video conference control device 21, the motion vector added by the video conference terminal can be reliably decoded (in the case of (1) above).

Further, in the case of the above-mentioned (2), the examples of (b), (c) and (d) in the same figure correspond to any side of the cutout area and any side of the original image. In this case, since the motion vector is not originally added to the contacting macroblock, the contacting macroblock is not set as the motion vector addition area.

By the way, in the procedure of FIG.
The system control unit 2, which has determined the motion vector addition area as described above, sets the determined motion vector addition area in the moving image coding / decoding unit 13 (process 1003).

As a result, the image quality can be improved by using the motion vector information even when the array processor performs image combination accompanied by image cutout.

Next, the second embodiment will be described. In the present embodiment, the method of image synthesis performed by the multipoint video conference control device 21 is notified to the video conference terminal, and the video conference terminal invalidates a predetermined amount of the video data encoded based on the notified system. Add information and send.

The notification of the cut-out area is sent by the H.264 standard at the start of communication and at the time of changing the composition method in the audio / video multiplex unit 30. Notification is made using the BAS command (MBE) on the H. 221 (notification is performed even when composition is not performed). Therefore, in the second embodiment, the capability exchange is performed assuming that the MBE capability is present during the capability exchange shown in FIG. (As another method, a protocol on MLP may be used.)

FIG. 12 shows the processing procedure of the second embodiment.
This will be described with reference to FIG. In the figure, when the video conference terminal 1 receives a BAS, the system control unit 2 reads it from the multimedia multiplexing / separating unit 5 and checks whether it is a notification of a combining method (decision 200).
1).

FIG. 13 shows an example of the BAS command in that case. In the figure, the first data is H.264 indicating the start of MBE. The data (command) defined in 221 is followed by the number of bytes of data, an identifier indicating that the data is a combination method notification, and data indicating the combination method.

Now, in the procedure shown in FIG. 12, the received BAS is the notification of the combining method (Y in judgment 2001).
es), the system control unit 2 determines the evaluation value Q stored in advance as shown in FIG. 14 and corresponding to the combining method (process 2003). Then, the system control unit 2
Is the invalid information amount Ii to be added from the evaluation value obtained by the determination.
nv is calculated (process 2003).

For the invalid information amount Iinv, the transmission transmission band of the moving image is Btx, the reception transmission band is Brx, and the evaluation value is Q.
It is expressed by the following Equation 1. Iinv = Btx− (Brx × Q) / 100− (Formula 1)

In FIG. 14, the evaluation value of 80 in the transcoder is that the frame rate increases as the high frequency component of the image is reduced or reduced during transcoding and / or cut out. To prevent it from dropping, it is assumed that the operation will reduce the amount of information when transmitting from the terminal in advance. The evaluation value in multiplex assumes that four source images are processed in multiplex.

The system control unit 2 sets the calculated invalid information amount Iinv in the moving picture coding / decoding unit 13 (process 2004). As a method of adding invalid information in the moving picture encoding / decoding unit 13, ITU-T Recommendation H.264 is used. 261
Two methods (macro block fill /
Either or both of (fill bit insertion) are used.

According to the above-mentioned procedure, according to the image synthesizing method in the multipoint video conference control device 21, the original video image is displayed on the video conference terminal 1 side so that the video transmission buffer 27 of the multipoint video conference control device 21 does not overflow. Since the invalid information added to the data is adaptively increased or decreased, the overflow of the moving image transmission buffer in the multipoint video conference control device 21 can be avoided.

Next, a third embodiment will be described. In the present embodiment, when the image synthesis is being performed by the array processor (or multiplex), the processing of the procedure shown in FIG. 15 is performed in order to avoid overflow of the moving image transmission buffer.

In the figure, when image synthesis is started by the array processor (or multiplex) during communication, the system control unit 2 of the multipoint video conference control device.
2 monitors the accumulated amount of the moving image transmission buffer 27 of each communication channel (process 3001, No loop of determination 3002).

When the accumulated amount equal to or more than the predetermined amount is detected (Yes in determination 3002), the system control unit 22 searches the moving image reception buffer 29 of the communication channel which is the source of image synthesis (process 3003). When a macro block (MB) encoded in INTRA mode (encoding processing within a frame) is detected (Yes in determination 3004)
s), the system control unit 22 obtains the image position after the MB is combined, and deletes the moving image data at the same image position from the moving image transmission buffer 27 detected in process 3001 (process 3005).

The system control unit 22 executes the above processing 300.
The processes from 3 to 3005 are repeated for all the communication channels that are the sources (No loop of determination 3006). Furthermore, the system control unit 22 makes a determination 3002.
The accumulated amount of the moving image transmission buffer 27 detected in (1) is confirmed (decision 3007), and if it is not improved (No in the determined 3007) (if the accumulated amount is still more than the predetermined amount), the error of the conventional buffer overflow Processing (for example, clearing the moving image transmission buffer 27 and issuing a VCU command to all communication channels (video conference terminals) that are sources) is performed (processing 3008).

Thus, the multipoint video conference control device 2
When the image is synthesized by the array processor in 1 and the accumulated amount in the transmission buffer becomes equal to or more than a predetermined amount, the INTRA-MB data is searched from the reception buffer, and the data at the same position on the synthesized image is deleted from the transmission buffer. Therefore, overflow of the moving image transmission buffer 27 can be avoided.

Next, a fourth embodiment will be described. In this embodiment, when the VCU command is received from the video conference terminal while the image synthesis is being performed by the array processor, the process of the procedure shown in FIG. 16 is performed as a response.

In the figure, when image composition is started by the array processor during communication, the system control unit 22 of the multipoint video conference control device 21 makes the multi-channel of each communication channel transmitting the image-composed moving image. The C & I code of the media demultiplexing unit 24 (see ITU-T Recommendation H.230: VCU command is one of the C & I codes) is read and monitored (No in processing 4001 and determination 4002).
loop).

When a VCU command is detected (decision 40)
02), the system control unit 22 issues a VCU command to all communication channels (video conference terminals) that are sources of image synthesis (VCU command to the multimedia multiplexing / separation unit 24 of all communication channels). C &
Write the I code) (process 4003).

Further, the system control unit 22 instructs the array processor unit 104 to transmit the stored image data, and the array processor unit 104 pre-stores the image stored in the nonvolatile memory (ROM etc.) in the unit. Data (This data is the image data for one frame.
(Encoded in A mode) is output, and the image combining process is stopped (process 4004). After that, the system control unit 22 monitors the moving image data of all the communication channels that have been the source of the image synthesis (process 4005, determination 4).
When the INTRA frame is detected (Yes in determination 4006), the image synthesis of the communication channel is restarted (process 4007).

The above processes 4005 to 4007 are repeated until the image composition of all communication channels, which has been the source of the image composition, is restarted (No in judgment 4008).
loop).

FIG. 17 shows changes in the image on the video conference terminal side during the above processing. In the figure,
(A) is the process 400 in the procedure shown in FIG. 16 described above.
4 is an image, and (B) is an image in which image synthesis of some communication channels is restarted in processes 4005 to 4007,
(C) is an image for which all image composition has been resumed (EN
D) is shown.

Thus, the multipoint video conference control device 2
1 receives a VCU command from a video conference terminal, issues a VCU command to another video conference terminal and transmits image data of an INTRA frame that has been stored in advance. It is possible to avoid the occurrence of a delay difference between the images combined in 21. Moreover, it is possible to avoid image locking or image distortion in the video conference terminal.

Next, a fifth embodiment will be described. In the present embodiment, when a transmission error is detected in the communication channel that is the source of the image synthesis performed by the array processor, the response is as shown in FIG.
The process of the procedure shown in 8 is performed.

In the figure, when image synthesis is started by the array processor during communication, the system control unit 22 of the multipoint video conference controller 21 corrects the moving image error of each communication channel which is the source of image synthesis. The error information of the detection unit 28 is read and monitored (process 500
1, No loop of judgment 5002). When an uncorrectable transmission error is detected (Yes in determination 5002), the system control unit 22 issues a VCU command to the communication channel (video conference terminal) in which the error was detected (communication channel multimedia multiplexing, The C & I code of the VCU command is written in the separation unit 24) (process 5003).

Further, the system control unit 22 instructs the array processor unit 104 to synthesize the stored image data, and the array processor unit 104 pre-stores an image stored in a non-volatile memory (ROM or the like) in the unit. Data (This data is the image data for one frame.
Image synthesis is performed using (encoded in A mode) as image data of the communication channel in which an error is detected (process 50).
04). After that, the system control unit 22 monitors the moving image data of the communication channel in which the error is detected (process 5005,
If the INTRA frame is detected (No in decision 5005) (Yes in decision 5006), image composition based on the data received from the communication channel is restarted (process 5).
007).

FIG. 19 shows changes in the image on the video conference terminal side during the above processing. In the figure,
(A) shows an image in the processing 5004 described above, and (B) shows an image (END) in which the image synthesis by the data received in the processing 5007 is restarted.

Thus, the multipoint video conference control device 2
When 1 detects a transmission error in the video data from the video conference terminal, the VCU command is issued to the video conference terminal and the INTR stored in advance is stored.
Since the image data of the A frame is combined, it is possible to avoid image distortion in the video conference terminal.
The video conference terminal can clearly indicate to the user that the image is being updated.

The fifth embodiment has the advantages described above, but when an error occurs in the received moving image data from a certain video conference terminal, the multipoint video conference control device 21 stores the encoded image stored in advance. When data is transmitted instead of the received moving image data in which an error is detected, if the size of the image data is constant (it is assigned to one video conference terminal (bandwidth / frame rate)
In the above case, the transmission buffer overflows (because the array processor cannot reduce the frame rate (reduce information by frame skipping)).

A sixth embodiment for solving the problem will be described below. In the present embodiment, a plurality of image data having different data lengths are provided as the image data used in the process 5004 in the process procedure shown in FIG. Select and use.

FIG. 20 shows an example of the data length of the stored image data. In the array processor unit 104 of the multipoint video conference control device 21, image data having a plurality of data lengths as shown in FIG. 20 are stored in advance in a nonvolatile memory (ROM or the like) in the unit corresponding to the image data number. Remembered
It should be noted that these image data have the same contents and have different compression rates in encoding (different degrees of definition).

When instructing the array processor unit 104 to synthesize the stored image data in the process 5004 shown in FIG. 18, the system control unit 22 (bandwidth allocated to moving image transmission / number of synthesis (source of image synthesis) Number)) is calculated, and based on it, the image data number to be used is determined by referring to the table shown in FIG.
Notify 4. The array processor unit 104 synthesizes and outputs the image data according to the notified image data number.

In the sixth embodiment, the case in which the frame rate is fixed at 15 fps has been described, but since the frame rate also causes an overflow, it is effective to have a plurality of image data adapted to this. Although an example in which a plurality of image data having the same content but different compression rates are stored has been shown, the image content itself may be different, and data having the same compression rate but different data length may be stored.

[0096]

According to the first aspect of the present invention, when the moving image information which is still encoded in the multipoint video conference control device is subjected to image composition with image cutting, the image cutout area is used. To each video conference terminal, each video conference terminal sets a motion vector addition area based on the notified cut-out area of the image, and is coded in order to add moving vector information only to the addition area. It is possible to improve the image quality by using the motion vector information even when the moving image information as it is is combined with the image cutting. Further, there is an advantage that the so-called array processor can be used in combination with the image synthesis accompanied by the image cutout and the switching service.

According to the second aspect of the invention, when the multipoint video conference control device performs image synthesis, each video conference terminal is notified of the synthesis method, and each video conference terminal notifies the notified synthesis method. According to the above, since a predetermined amount of invalid information is added to the moving image data, overflow of the moving image transmission buffer in the multipoint video conference control device can be avoided. Further, there is an advantage that the image composition and the switching mode can be smoothly switched in the multipoint video conference control device.

According to the third aspect of the present invention, when the so-called array processor performs image composition in the multipoint video conference control device, if the accumulated amount in the moving image transmission buffer becomes a predetermined amount or more, the moving image reception buffer moves within the frame. Since it is configured to search the compressed area and delete the data of the corresponding area on the composite image from the video transmission buffer, the overflow of the video transmission buffer in the multipoint video conference control device can be avoided. .

According to the invention of claim 4, when the multipoint video conference control device issues a forced screen update request from a video conference terminal, the forced screen update request is issued to another video conference terminal, and Since the previously stored in-frame-compressed image information is transmitted to each video conference terminal, the multi-point video conference control device responds to the forced screen update request by each video conference communication terminal. It is possible to avoid the adverse effect caused by the delay difference between the images transmitted from the. Further, in each video conference terminal, there is an advantage that it is possible to avoid locking of the moving image information received from the multipoint video conference control device or image disturbance.

According to the invention of claim 5, when the multipoint video conference control device detects a transmission error in the video data from each video conference terminal, the video conference terminal in which the transmission error occurs in the video data. In addition to issuing a compulsory screen update request to, replace the previously stored in-frame-compressed image information with the video information from the video conference terminal in which the transmission error occurred in the video data,
Since it is configured to be combined with moving image information from another video conference terminal, it is possible to avoid image distortion in each video conference terminal. In addition, each video conference terminal has an advantage of being able to clearly indicate to the user that the image is being updated.

According to the invention of claim 6, in the video conference system according to claim 5, the multipoint video conference control device has an assumed transmission rate as image information obtained by intra-frame compression of the predetermined image, A plurality of pieces of image information having different amounts of information according to the number of combined video information and / or frame rate are stored, and the transmission rate, the number of combined video information and / or frame rate set with each video conference terminal are stored. According to the configuration, the image information having an appropriate data amount is selected and combined, so that the overflow of the transmission buffer in the multipoint video conference control device can be avoided.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a video conference system according to an embodiment of the present invention.

FIG. 2 is a diagram showing a block configuration of a video conference terminal according to the embodiment of the present invention.

FIG. 3 is a diagram showing a block configuration of a multipoint video conference control device according to an embodiment of the present invention.

FIG. 4 shows a configuration of an audio / video multiplex unit 30 of the multipoint video conference control device according to the embodiment of the present invention.

FIG. 5 is a diagram showing a basic operation of the video conference system according to the embodiment of the present invention.

FIG. 6 is a diagram showing an example of audio and video data output to each communication channel.

FIG. 7 is a diagram showing an example of a combination form (ratio) of weighting for each communication channel in the audio mixing unit.

FIG. 8 is a diagram showing an example of an image synthesizing form in the array processor unit and an example of processing in the array processor unit with image clipping.

FIG. 9 is a first view of the video conference system according to the present invention.
It is a flow chart which shows a processing procedure of an embodiment.

FIG. 10 is a BAS in the processing procedure according to the first embodiment.
It is a figure which shows an example of a command.

FIG. 11 is a diagram showing a relationship between an image cutout area and a motion vector addition area according to the first embodiment.

FIG. 12 is a flowchart showing a processing procedure of the second embodiment in the video conference system according to the present invention.

FIG. 13 is a BAS in the processing procedure according to the second embodiment.
It is a figure which shows an example of a command.

FIG. 14 is a diagram showing an example of correspondence between a synthesis method and an evaluation value in the processing procedure according to the second embodiment.

FIG. 15 is a flowchart showing a processing procedure of the third embodiment in the video conference system according to the present invention.

FIG. 16 is a flowchart showing a processing procedure of the fourth embodiment in the video conference system according to the present invention.

FIG. 17 is a diagram showing a change in image on the video conference terminal side in the processing procedure according to the fourth embodiment.

FIG. 18 is a flowchart showing a processing procedure of the fifth embodiment in the video conference system according to the present invention.

FIG. 19 is a diagram showing a change in image on the video conference terminal side in the processing procedure according to the fourth embodiment.

FIG. 20 is a diagram showing an example of a data length of stored image data according to the fifth embodiment.

[Explanation of symbols]

 1, 19 and 20 Video conference terminal 2 System control unit 3 Magnetic disk device 4 ISDN interface unit 5 Multimedia multiplexing / separating unit 6 Microphone 7 Audio input processing unit 8 Audio encoding / decoding unit 9 Audio output processing unit 10 Speaker 11 Video Camera 12 Video input processing unit 13 Video encoding / decoding unit 14 Video output processing unit 15 Monitor 16 User interface control unit 17 Console 18, 31 ISDN line 22 System control unit 23 ISDN interface unit 24 Multimedia multiplexing / separating unit 25 Audio Encoding / decoding section 26 Video correction code generation section 27 Video transmission buffer 28 Video error correction / detection section 29 Video data reception buffer 30 Audio / video multiplexing section 101 Speaker detection section 102 Audio mixing section 103 Audio switching section 04 array processor unit 105 video switching unit

Claims (6)

[Claims] 1. Video conferencing terminals provided at a plurality of points are connected to a multipoint videoconference control device, and while each videoconference terminal transmits encoded video information to the multipoint videoconference control device, In the multipoint videoconference system in which the multipoint videoconference control device combines the cutout areas obtained by cutting out a part of the video information from each of the videoconference terminals, and transmits the combined video information to each videoconference terminal, The multipoint video conference control device comprises means for notifying each of the video conference terminals of the cutout region, while each of the videoconference terminals is based on the cutout region notified from the multipoint video conference control device. Means for setting an additional area of motion vector information, and the multipoint television by adding motion vector information only to the set additional area. Multipoint video conference system, characterized in that a means for transmitting to the discussions controller. 2. Video conferencing terminals provided at a plurality of points are connected to a multipoint videoconference control device, and the encoded video information is transmitted from each of the videoconference terminals to the multipoint videoconference control device. The multipoint video conference control device synthesizes video information from each of the video conference terminals,
In the multipoint video conference system for transmitting the synthesized video information to each video conference terminal, the multi-point video conference control device includes means for notifying each video conference terminal of a method of synthesizing the video information. The video conference terminal comprises means for adding a predetermined amount of invalid information to the encoded moving image information to be transmitted to the multipoint videoconference control device, in accordance with the combining method notified from the multipoint videoconference control device. A multipoint video conferencing system characterized by 3. Video conferencing terminals provided at a plurality of points are connected to a multipoint videoconference control device, and the encoded video information is transmitted from each of the videoconference terminals to the multipoint videoconference control device. The multipoint video conference control device synthesizes video information from each of the video conference terminals,
In the multipoint video conference system for transmitting the composite video information to each video conference terminal, the multi-point video conference control device is an area compressed in a frame from the encoded video information transmitted from each video conference terminal. Detecting means, a means for monitoring the storage status of the video transmission buffer for temporarily storing the video information to be transmitted to each of the video conference terminals, and when the storage amount of the video transmission buffer exceeds a predetermined amount. Of the moving picture information stored in the moving picture receiving buffer for temporarily storing the coded moving picture information transmitted from each video conference terminal, moving picture information corresponding to the area in which the intra-frame compression is detected. A multipoint video conference system comprising means for deleting. 4. Video conferencing terminals provided at a plurality of points are connected to a multipoint videoconference control device, and the encoded video information is transmitted from each of the videoconference terminals to the multipoint videoconference control device. The multipoint video conference control device synthesizes video information from each of the video conference terminals,
In a multipoint videoconference system for transmitting the composite moving image information to each videoconference terminal, the multipoint videoconference control device stores means for storing image information in which a predetermined image is compressed in a frame, and from each of the videoconference terminals. A multipoint video conference system comprising: means for transmitting image information in which the predetermined image is compressed in a frame to each of the video conference terminals when a forced screen update is requested. 5. A video conference terminal provided at a plurality of points is connected to a multi-point video conference control device, and while each video conference terminal transmits encoded video information to the multi-point video conference control device, The multipoint video conference control device synthesizes video information from each of the video conference terminals,
In the multipoint video conference system for transmitting the composite moving image information to each video conference terminal, the multipoint video conference control device detects a transmission error of moving surface information from each of the video conference terminals, and a predetermined image. Means for storing the image information compressed in the frame, and the moving image information from the video conference terminal that has detected the transmission error of the moving image information is replaced with the image information in which a predetermined image is compressed in the frame, A multipoint video conference system comprising: means for synthesizing video information and transmitting the synthesized video information to each video conference terminal. 6. The multipoint video conference control device has different information amounts according to an assumed transmission rate, a composite number of moving image information and / or a frame rate as image information in which the predetermined image is compressed in a frame. According to the transmission rate set between the means for storing a plurality of pieces of image information and each video conference terminal, the number of composites of moving image information and / or the frame rate, the predetermined image is intraframe-compressed as image information. 6. The multipoint video conference system according to claim 5, further comprising means for selecting one of a plurality of pieces of image information having different amounts of information.