JPH09261615A - Multi-point video conference system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a multi-point video conference system in which resolution of an image is not deteriorated even when total number of terminal equipments taking part in a conference is increased, an image update request is easily controlled and it is not required to increase the number of channels among multi-point controllers. SOLUTION: After receiving an image code signal from a terminal equipment, image decoding sections 54-1-54-M of a multi-point controller decode the signal to reproduce an image signal and image changeover sections 58-1-58-M select an image signal to be sent to each terminal equipment on request from each terminal equipment, image coding section 57-1-57-M interleave number of frames of the image signal received for a unit time (e.g. 1 second) by one frame each per N-frames to form N sets of sub frames whose motion information is equivalent to 1/N and send the image code signal obtained by coding seqtuentially each sub frame to each terminal equipment.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video conference system, and more particularly to a remote multipoint system in a multimedia video conference system configured by using media such as voice, image, data (graphics information such as conference material). The present invention relates to a multipoint control device for connecting videoconferencing terminals of the present invention by a communication line to simultaneously implement a videoconference between multiple points.

[0002]

2. Description of the Related Art As shown in FIG. 8, a conventional video conference system has a multipoint control unit 82 (Multipoint) at the center of a network in which a plurality of video conference terminals (hereinafter referred to as terminals) 81 are connected in a star shape. Cont
(roll unit) is installed to perform connection / disconnection control of lines between terminals, audio signal addition (mixing) processing, image signal switching / synthesis processing, data signal exchange, and the like.

Image signal processing has an automatic switching mode and a manual switching mode. The former is a mode in which the transmission source of the image is determined based on the voice level of the speaker and this is automatically switched. The latter is a mode in which the switching state can be specified by an instruction from the terminal.
It has its own room transmission function and chairperson control function that the chairperson bureau collectively decides the broadcast image during the conference. In addition, there are multi-point control devices that switch image signals at a single point, those that combine image signals at a plurality of points, or those that have both functions.

By the way, in the image coding method disclosed in Japanese Patent Laid-Open No. 4-179793 and the multipoint video conference method using the same, the image signal processing is performed by the number of pixels per frame of the image signal on the transmission side of the terminal. Is thinned out by one pixel for every N pixels to create N subframes which are reduced screens of the original image signal, an encoder for sequentially encoding the subframes, and an encoder for encoding by the encoder. The multipoint control device for switching the image code signal in units of subframes, the image decoder that sequentially decodes the image code signal to obtain N subframes at the receiving side of the terminal, and the N subframes are combined. Then, a frame memory for reproducing an image signal of one frame is performed.

Each terminal thins out the image signal of one frame by one pixel for every N pixels on the transmission side and outputs 1 / N of the original image signal.
N sub-frames corresponding to the reduced image are created, sequentially encoded, and output to the multipoint control device.

When the terminal is set to the automatic switching mode for each terminal, the multipoint control device selects the image code signal from the terminal determined to be the speaker based on the voice level, and the terminal receives the image code signal. When the manual switching mode such as selection and chair control is set, the designated image code signal is selected, this is switched in sub-frame units, and is sent to the terminal.

After receiving the image code signal from the multipoint control device, each terminal sequentially decodes the image code signal to obtain N subframes and reproduces an image signal of a single point or an image signal of a plurality of points. Depending on whether or not to synthesize and reproduce, the composition method of N sub-frames is changed, and an image signal of one frame is reproduced.

By the above operation, the terminal receives an image signal having a resolution close to the original image when receiving an image signal at a single point, and an image signal at a plurality of points when receiving an image signal at a plurality of points. It is possible to reproduce an image signal obtained by combining N screens.

[0009]

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention
In the image coding method shown in No. 993 and the multipoint video conference method using the same, only one multipoint control device is described, and a conference between terminals is performed via a plurality of multipoint control devices. It does not describe a multipoint video conferencing system that does. Further, in the embodiment, when the N-split screen is applied, it is described that N + 1 terminals can participate in the conference in the multipoint control device.

As shown in FIG. 9, such an image coding system can be realized by installing two multipoint control devices 91 and 92 at two points and connecting them to each other through a communication path 100. Think The multipoint control devices 91 and 92 include video conference terminals (hereinafter, referred to as terminals) 101 and 1 respectively.
02 and 201, 202, 203 are connected.

Since the conventional multipoint control device only switches the image code signal from the terminal in subframe units, the subframe creation units handled by the multipoint control devices 91 and 92 must be the same as each other. I won't. Inevitably, the terminal 10 connected to the multipoint control devices 91, 92
The units of creation of subframes created by 1, 102 and 201, 202, 203 are also the same. Therefore, assuming that the total number of terminals participating in the conference in the multipoint control devices 91 and 92 is S, the (S-1) split screen is applied to all terminals, and in FIG. 9, the total number of terminals is Therefore, the 4-split screen shown in FIG. 10 is applied.

By the way, since there is one line between the multipoint control units 91 and 92, the multipoint control unit 91 transmits the image code signals of the subframes of the terminals 201, 202 and 203 from the multipoint control unit 92, that is, the terminals. 201, 202, 2
Select any four of the image code signals of the reduced image of 03 to receive, or divide the image signal of any one of the terminals 201, 202, 203 into four (subframe)
It will receive four image code signals.

In the former case, the terminal 101 and the terminal 102 request reception of image signals of different terminals connected to the multipoint control unit 92, or one of the terminals requests reception of a combined image signal of terminals other than the own station. In the latter case, the terminal 101 and the terminal 202 request reception of the image signal of the same terminal of the multipoint control device 92, or one of them does not request the image signal from the multipoint control device 92. .

Further, if the terminal 101 requests the reception of the image signal of the terminal 202, the terminal 101 cannot receive all four of the image code signals of the four subframes of the terminal 202. Therefore, the terminal 101 reproduces a 4-split screen including unnecessary image signals or decodes a receivable image code signal of one sub-frame and converts one pixel into four pixels in the frame memory (enlargement processing). ) Is performed to reproduce the image signal of one frame. The same applies to the terminal 102.

According to this operation, the multipoint control device 91,
If the total number of terminals participating in the conference in 92 is small, it is acceptable, but as the total number of participation increases, the split screen reproduced by the terminal becomes smaller, and the resolution of the enlarged image significantly deteriorates. There is a problem that it becomes difficult to see.

When high-efficiency coding is applied, generally, at the receiving side of a terminal, when decoding of an image is started or when decoding is interrupted due to a transmission error or the like, an initial value is given to a decoder. Need to give. This is conventionally realized by the operation in which the multipoint control device issues an image update request to the encoder of the transmission source terminal, but in the above configuration, this operation is performed by the two multipoint control devices 91, 92. However, there is a problem that the control becomes very complicated because it is involved in both.

Further, the number of lines between the multipoint control devices is not limited to one line, and a plurality of communication paths are provided between the multipoint control devices so that the images of all terminals can be arbitrarily switched. However, the communication cost will increase in proportion to the number of terminals.

Considering that the multipoint control devices are connected by a single communication path as described above, the conventional technique has the following problems.

First, the image signal reproduced by the terminal can be tolerated when the total number of terminals participating in the conference between the multipoint control devices is small, but the split screen reproduced by the terminal each time the total number of participation increases. Becomes smaller, and the resolution of the enlarged image is significantly degraded, making it very hard to see.

Second, when high efficiency coding is applied,
Generally, on the receiving side of the terminal, when starting the decoding of the image, it is necessary to give an initial value to the decoder. However, in the above-mentioned configuration, this operation involves both of the two multipoint control devices, so that the control becomes very complicated.

Thirdly, the number of lines between the multipoint control devices is not limited to one line, and a plurality of communication paths are provided between the multipoint control devices so that images of all terminals can be arbitrarily switched. Is possible, but communication costs will increase in proportion to the number of terminals.

Therefore, the object of the present invention is to prevent the image resolution from deteriorating even if the total number of terminals participating in the conference increases.
An object of the present invention is to provide a multipoint video conference system in which an image update request can be easily controlled and the number of lines between multipoint control devices need not be increased.

[0023]

According to the present invention, a plurality of multipoint control devices connected to each other by a communication line and a plurality of terminals connected to each of these multipoint control devices are provided. In a multipoint video conference system for conducting a conference between the terminals via the multipoint control device, the multipoint control device receives an image code signal from the terminal and then decodes it to reproduce an image signal. After selecting an image signal to be sent to another terminal in response to a request from another terminal, the number of frames of this image signal input in a unit time is thinned out by one frame for every N frames to obtain N subframes. The multipoint control unit also sequentially decodes the received image code signal by constructing and sending the image code signal obtained by sequentially coding each of these N subframes to the terminal that made the request. Give N subframes Te, the N
A multipoint video conference system is obtained which is characterized in that N subframes are interpolated to reproduce N image signals.

The multipoint control device thins out the number of frames of the image signal input in the unit time by one frame for every N frames to form N subframes, and a subframe multiplexer. An image encoder that sequentially encodes each of the N subframes multiplexed by the frame multiplexer, an image decoder that sequentially decodes the encoded image signal to obtain a subframe, and an image decoder And a subframe separator that reproduces an image signal by interpolating the obtained subframes.

Further, the multipoint control device further includes an image provisional encoder for executing provisional encoding of the image signal input in the unit time before encoding by the image encoder, and the image provisional encoding. An image information amount counter for measuring the information amount of the image coded signal generated by the encoder, and controlling the subframe multiplexer by the output information of the image information amount counter,
The thinning rate of the image signal may be changed.

[0026]

When the prior art and the present invention are simply compared, in the prior art, a method of creating a subframe by thinning out the number of pixels per frame of an image signal by one pixel for every N pixels is used. In the present invention, the number of frames of the image signal input per unit time is thinned out every N frames to form a subframe.

Further, in the prior art, the multipoint control device only switches the image code signal from the terminal in subframe units. However, in the present invention, after receiving the image code signal from the terminal, it is decoded. After reproducing the image signal and selecting the image signal to be sent to each terminal in response to the request from the terminal, the number of frames of this image signal input in a unit time (for example, 1 second) is set to 1 frame per N frames. By thinning out, N subframes having motion information corresponding to 1 / N are configured, and image code signals obtained by sequentially encoding each subframe are transmitted to the terminal.

Further, according to the present invention, on the transmitting side of the multipoint control device, temporary encoding is executed before the input image signal is formally encoded, and the amount of the image encoded signal generated is measured. The amount of motion information is obtained, the decimation rate of the image signal is varied with reference to this result, and the subframes obtained by this decimation processing are sequentially encoded in subframe units.

As described above, according to the present invention, it is not the method of constructing the sub-frame of the image signal that emphasizes the movement of the image signal as realized by the method shown in the prior art, but the image resolution that emphasizes the image. We adopted the method to construct the sub-frame of the signal. In recent years, it can be seen that there are application examples in which an image processing board or the like is incorporated in a personal computer or the like to realize a desktop conference or the like, but in these cases, the movement of the image signal is not so important and the resolution of the image is emphasized. Because there are many.

However, in the present invention, it is useless to give the amount of information by dividing the image signal having a small amount of motion into equal parts, so that the thinning ratio is large for the image signal having a small coding amount by the above function. The thinning ratio can be reduced for an image signal having a large coding amount.

[0031]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. 1 and 2 are block diagrams for explaining an image coding method in the multipoint video conference system of the present invention.

The image coding unit 11 is composed of a sub-frame multiplexer 12 and an image coder 13. Here, one frame of an image signal input in a unit time (for example, one second) is set to four frames. Extract by thinning each one,
It is assumed that four subframes are configured and these are coded.

FIG. 1 shows the case where a single original image signal A is handled, and FIG. 2 shows the case where four separate original image signals B, C, D and E are handled. Subframes A1, A2, A3, and A4 in FIG. 1 are obtained by thinning out the original image signal A every four frames, and subframes B1, C1, D1, and E1 in FIG. 2 are original image signals B and C. , D, and E are thinned out every four frames and extracted. The motion information of each sub-frame corresponds to 1/4 of the original image signal. The image encoder 13 sequentially encodes the subframes in units of subframes, adds subframe identification signals, and outputs the signals. The image code signal is obtained from the image encoder 13 by A1 ′ in FIG.
It is output in the serial signal format of A2 ', A3', A4 ', and in the serial signal format of B1', C1 ', D1', E1 'in FIG.

3 and 4 are block diagrams for explaining the image decoding method in the multipoint video conference system of the present invention. The image decoding unit 21 includes an image decoder 22 and a subframe separator 23. FIG. 3 shows A1 of FIG.
The case where a single image signal A is reproduced by decoding the image code signal input in the format of the serial signal of ', A2', A3 'and A4' is shown. On the other hand, FIG. 4 shows B1 ′, C1 ′,
The case where an image code signal input in the serial signal format of D1 'and E1' is decoded and four image signals B, C, D and E are reproduced is shown.

The subframe separator 23 refers to the subframe identification signal, and if the image code signal is a single original image signal (A1 ', A2', A3 ', A4'), it is decoded. One image signal A is reproduced by interpolating the subframes.
Image code signal of another original image signal (B1 ', C1', D1
', E1'), the number of frames of the image signal per unit time becomes 1/4 of the original image signal when these are decoded, so the number of sub-frames not transmitted on the image encoding side (here The image signals (B, C, D, E) are separately reproduced by processing such as repeating subframes reproduced only three times.

FIG. 5 is a block diagram showing an example of the configuration of the image processing unit of the multipoint control device used in the multipoint video conference system of the present invention. The image processing unit 51 of the multipoint control device of the present invention includes an image decoder 52 and a subframe separator 5
M image decoding units 54-1 to 54-M composed of 3 units, M composed of a subframe multiplexer 55 and an image encoder 56.
The image coding units 57-1 to 57-M, the M image switching units 58-1 to 58-M, and the image switching control unit 59.

Each of the image decoding units 54-1 to 54-M receives the image code signal 6 received via the communication interface unit 60.
1 is decoded by the image decoder 52 in units of subframes, the subframe separator 53 refers to the subframe identification signal, and in the case of a single original image signal, one image signal is converted into another original image signal. In the case of the above, they are reproduced as separate image signals and are output to the image switching unit 58.

Each of the image switching units 58-1 to 58-M switches the image signal reproduced by the image decoding unit according to the control signal from the image switching control unit 59 and outputs it to the image coding unit.
Each of the image coding units 57-1 to 57-M receives the image signal from the image switching unit, codes the image signal, and outputs the coded signal to the communication interface unit 61. The sub-frame multiplexer 55 sequentially switches the four sub-frames created from the original image signal when encoding a single image signal, and outputs the plurality of original image signals when encoding a plurality of image signals. The created subframes are sequentially switched. The image encoder 56 encodes these in subframe units, adds a subframe identification signal, and outputs the result.

The image switching control unit 59 controls the image switching units 58-1 to 58-M as described above, and also encodes a single image by each of the image encoding units 57-1 to 57-M. Or whether to encode a plurality of image signals. This control, for example, when the multipoint control device operates in the automatic switching mode, so as to transmit a single image of the speaker terminal based on the voice level of the speaker, and when operating in the manual switching mode, The switching state is changed from the image switching control unit 59 to the image switching units 58-1 to 58-M according to an instruction from the terminal according to the reception selection function, the room transmission function, the chair control function, and the like.
And the image coding units 57-1 to 57-M.

Here, a mode in which a plurality of users exchange images by a configuration in which multipoint control devices 71 and 72 are connected to each other through a communication path 100 as shown in FIG. 6 will be described. The multipoint control devices 71 and 72 respectively include video conference terminals (hereinafter referred to as terminals) 101, 102 and 2.
01, 202 and 203 are connected. Since the multipoint control devices 71 and 72 have the configuration shown in FIG. 3,
The operation will be described with reference to FIG.

The operation of the image processing unit of the multipoint control unit 71 is such that the image decoding unit 54-1 and the image encoding unit 57-1 send and receive the image signal to and from the terminal 101 via the communication interface unit 60, and the image decoding unit 54. -2 and the image coding unit 57-2 send and receive image signals to and from the terminal 102 via the communication interface unit 60, and the image decoding unit 54-M and the image coding unit 57-M control multipoints via the communication interface unit 60. The case where an image signal is transmitted and received to and from the device 72 is shown. The image code signals transmitted from the terminals 101 and 102 and the multipoint control device 72 are decoded by the image decoding units 54-1, 54-2 and 54-M, respectively, and the image signals A, B and C, D and E are decoded. Is supplied to the image switching units 58-1, 58-2, 58-M. Since the image signals A and B are image signals decoded from a single image, the amount of motion information is almost the original image, and the amount of motion information of the image signals C, D, and E is about 1/3. However,
The resolution of each image remains the original image. Terminal 10
1, 102 and the image signal to be sent to the multipoint control unit 72 are selected by the image switching units 58-1, 58-2, 58-M, and then the image coding units 57-1, 57-2, 57-. It is encoded with M.

Image switching units 58-1, 58-2, 58-M
Since the image signals of all terminals are input to
Any image signal can be selected for the 1 and 102 and the multipoint control device 72, and either a single image or a plurality of images can be transmitted.

In the terminal, when a plurality of images are received from the multipoint control device, one of the plurality of decoded image signals is selected and displayed, or optionally combined and displayed.
However, since this operation falls outside the scope of the present invention, description thereof will be omitted here.

As described above, when the image coding method and the image decoding method of the present invention are applied to the multipoint control device, the multipoint control devices 71 and 72 are reproduced by the terminal each time the total number of participants increases. Although the amount of motion information in the image signal is small,
The resolution is characterized by the original image. Furthermore,
Since the image decoding unit and the image encoding unit of the multipoint control device have a one-to-one correspondence with the image encoding unit and the image decoding unit of the opposite terminal, the image may be applied even when high efficiency encoding is applied. This is a method suitable for an application example in which the update request can be easily controlled and a multipoint control system is constructed by connecting a plurality of multipoint control devices.

FIG. 7 is a block diagram showing another embodiment of the present invention in which the image coding system is improved. This image encoding system has a configuration in which an image provisional encoder 75 and a code information amount counter 76 are added to the image encoding unit 11 shown in FIG. 1, and the image signal of the multipoint control device of FIG. Processing unit 5
The image processing unit can be configured by substituting 1 for the image processing unit.

This operation will be described below. The image provisional encoder 75 provisionally encodes a plurality of input image signals before encoding them formal, and outputs an image encoded signal. The amount of the generated image code signal is measured by the code information amount counter 76, whereby the image encoder 13 obtains the amount of motion information. The image encoder 13 refers to this result to instruct the subframe multiplexer 12 about the thinning ratio of each image signal, and also encodes the subframes input in a variable cycle in subframe units. For example, image signals B, C, D,
Assuming that the ratio of the code amount output from the code information amount counter 76 to E is 2: 2: 1: 1 per unit time, the subframe multiplexer 12 outputs the image signals B,
2/6, 2/6, 1/6, 1 from C, D, E frames
Subframes are created at a rate of / 6 frames, and the image encoder 13 sequentially encodes the subframes.

By the above operation, the present invention avoids equally dividing the image signal having a small amount of motion to give the information amount, and increases the thinning rate for the image signal input having a small image code amount to perform the encoding. By allocating a small number of subframes to be used, conversely reducing the thinning ratio for image signals with a large image coding amount and allocating a large number of subframes to be encoded, the line can be used efficiently. did.

[0048]

According to the present invention described above, even if the total number of terminals participating in a conference increases, the resolution of the image does not deteriorate, the image update request can be easily controlled, and the multipoint control devices can be controlled easily. It is possible to obtain a multipoint video conference system that does not need to increase the number of lines.

[Brief description of drawings]

FIG. 1 is a block diagram for explaining an image coding method in a multipoint video conference system of the present invention.

FIG. 2 is a block diagram for explaining an image encoding system in the multipoint video conference system of the present invention.

FIG. 3 is a block diagram for explaining an image decoding method in the multipoint video conference system of the present invention.

FIG. 4 is a block diagram for explaining an image decoding method in the multipoint video conference system of the present invention.

FIG. 5 is a block diagram showing a configuration example of an image processing unit of a multipoint control device used in the multipoint video conference system of the present invention.

FIG. 6 is a block diagram showing an overall configuration of a multipoint video conference system of the present invention.

FIG. 7 is a block diagram for explaining another image encoding method in the multipoint video conference system of the present invention.

FIG. 8 is a block diagram showing an overall configuration of a conventional multipoint video conference system.

FIG. 9 is a block diagram showing an overall configuration of another conventional multipoint video conference system.

FIG. 10 is a diagram for explaining the operation of the conventional multipoint video conference system shown in FIG. 9.

[Explanation of symbols]

11 Image Coding Unit 12 Subframe Multiplexer 13 Image Encoder 21 Image Decoding Unit 22 Image Decoder 23 Subframe Separator 51 Image Processing Unit 52 Image Decoder 53 Subframe Separator 54-1 to 54-M Image Decoding unit 55 Sub-frame multiplexer 56 Image encoder 57-1 to 57-M Image encoding unit 58-1 to 58-M Image switching unit 71, 72 Multipoint control device 100 Communication path 101, 102, 201, 202, 203 Video conference terminals

Claims (3)

[Claims] 1. A plurality of multipoint control devices connected to each other by a communication line, and a plurality of terminals connected to each of these multipoint control devices, the terminal being provided via the multipoint control device. In the multipoint video conference system for conducting a conference between the users, the multipoint control device receives the image code signal from the terminal, decodes the image code signal, reproduces the image signal, and responds to a request from another terminal. After selecting the image signal to be sent to the terminal, the number of frames of this image signal input per unit time is thinned out by one frame for every N frames to form N subframes, and each of these N subframes. To the terminal that made the request, and the multipoint control unit also sequentially decodes the received image code signal to obtain N subframes. A multipoint video conference system characterized by reproducing N image signals by interpolating N subframes. 2. The multipoint control device thins out the number of frames of the image signal input in the unit time by one frame for every N frames to form N subframes, and a subframe multiplexer. An image encoder that sequentially encodes each of the N subframes multiplexed by the frame multiplexer, an image decoder that sequentially decodes the encoded image signal to obtain a subframe, and an image decoder And a sub-frame separator that interpolates the obtained sub-frames and reproduces an image signal.
The multipoint video conference system described. 3. The multipoint control device further includes an image provisional encoder that performs provisional encoding of the image signal input in the unit time before encoding by the image encoder, and the image provisional encoding. Image information amount counter for measuring the information amount of the image coded signal generated by the encoder, and controlling the sub-frame multiplexer by the output information of the image information amount counter to change the thinning rate of the image signal. The multipoint video conference system according to claim 2, characterized in that.