JP4361430B2 - Bidirectional image communication apparatus, processing method thereof, client apparatus, and program - Google Patents

Description

  The present invention relates to a technology related to a bidirectional image communication system such as a server / client video conference system, and more particularly to a technology that enables transmission / reception of encoded data suitable as much as possible to a transmission condition and a reception condition in a client.

  For example, Quality Meeting (http://avs.kddlabs.co.jp/qmeet/body.html) commercialized by KDDI is known as a prior art of a bidirectional image communication system such as a video conference system. In Quality Meeting, the bandwidth that can be communicated on the client side is monitored, the bit rate of the encoder is changed on the client side, and communication according to the bandwidth is performed (see, for example, Patent Document 1). However, the following problems occur in the prior art.

  For example, in the configuration in which the client 1 and the client 2 are connected to the server, when the client 1 transmits the encoded data to the client 2 via the server, the above-described conventional technology considers only the bandwidth between the client 1 and the server. Therefore, when the bandwidth between the client 2 and the server is narrower than the bandwidth between the client 1 and the server, the client 2 may not be able to receive the video of the client 1. Therefore, the bit rate of the encoder of the client 1 must be set according to the narrower band.

  Therefore, when three or more clients are connected to the server, the encoder bit rate must be set to the line with the narrowest bandwidth in order to realize bidirectional communication between the three clients. Therefore, there is a problem that the bandwidth between the server and each client cannot be effectively used. In order to solve this problem, it is conceivable that the encoded data is decoded at the server and encoded at an appropriate bit rate for each client. However, this method requires complicated processing on the server side, and the load on the server is reduced. There is a problem that the bandwidth between the server and the client on the transmission side is wasted as the size increases.

In addition, the reception environment (image size, frame rate, bit rate, etc. in the client on the receiving side, and each value is an example of the reception condition in the present specification and claims) is generally different for each client. . Therefore, a problem similar to the problem in the above band occurs. That is, the encoding parameters (image size, frame rate, bit rate, etc.) matched to the receiving client having the lowest receiving condition, and each value is an example of the transmitting condition in this specification and claims. If the image has to be encoded at the same time, another receiving client can receive a higher quality image but cannot do so, and the server decodes the encoded data and the client When a method of encoding with an appropriate encoding parameter is adopted every time, the burden on the server increases, and the amount of encoding processing in the client on the transmission side may be unnecessarily large compared to the reception condition on the reception side. There is a problem.
JP 2003-116133 A "Check Point H.323 / MPEG4 Textbook", IE Institute, 2001

  The present invention has been made in view of the above points, and satisfies the transmission conditions of the transmission side client and the reception conditions of each client on the reception side as much as possible, and uses a necessary and sufficient bandwidth and image processing amount for bidirectional images. An object of the present invention is to provide a technology that enables communication.

  The above problem is to copy and distribute video with at least one client device that encodes video using a video compression scheme that generates intra-frame compressed key frames and inter-frame compressed interpolated frames. A bi-directional image communication apparatus used in a bi-directional image communication system having a bi-directional image communication apparatus, which receives a reception condition from each client apparatus and holds the reception condition; Receiving condition transmitting means for transmitting the receiving condition of the client apparatus; storage means for receiving encoded video data from the transmission source client apparatus; copying the received encoded data and storing it in a buffer; Depending on the receiving conditions of the destination client device, the encoded data of the key frame received from the source client device Alternatively, the problem can be solved by using a bidirectional image communication apparatus characterized by comprising a selective transmission means for transmitting the encoded data of the key frame and the encoded data of the interpolated frame to the transmission destination client apparatus. .

  According to the present invention, since the reception condition received from each client apparatus is transmitted to each client apparatus, each client can compare the own transmission condition with the reception condition, thereby obtaining necessary and sufficient bandwidth and encoding processing. A coding parameter suitable for the quantity can be determined. Furthermore, according to the present invention, encoded data can be appropriately transmitted even to a client on the receiving side that does not satisfy the conditions of the transmitted encoded data.

  The selective transmission unit compares a predetermined encoding parameter value of the encoded data with a reception condition in the transmission destination client device, and the predetermined encoding parameter value is equal to or larger than a reception condition value in the transmission destination client device. In some cases, only the encoded data of the key frame received from the transmission source client apparatus is transmitted to the transmission destination client apparatus, and the predetermined encoding parameter value is less than the value of the reception condition in the transmission destination client apparatus. Transmits the encoded data of the key frame and the encoded data of the interpolated frame to the transmission destination client device.

  The present invention also includes at least one client device that encodes video using a video compression method having a scalability function, and a bidirectional image communication device that duplicates and distributes video received from the client device. A bidirectional image communication apparatus used in a bidirectional image communication system, which receives and holds reception conditions from each client apparatus, and transmits reception conditions of other client apparatuses to each client apparatus. A receiving condition transmitting unit; a storage unit that receives encoded data of a plurality of layers in a scalability function from a transmission source client device; and copies and stores the received encoded data of a plurality of layers in a buffer; and a transmission source client device Of the received encoded data of multiple layers, the destination client device Selection transmission means for selecting encoded data of one or a plurality of layers to be transmitted according to reception conditions of the transmission destination client device, and transmitting the selected encoded data of one or a plurality of layers to the transmission destination client device; It can also be configured as a bidirectional image communication apparatus characterized by having

  The selective transmission means compares the layer of the encoded data received from the transmission source client device with the layer as the reception condition of the transmission destination client device, and the layer of the encoded data received from the transmission source client device is the transmission destination. When included in the layer as the reception condition of the client device, the entire encoded data received from the transmission source client device is transmitted to the transmission destination client device, and the encoded data layer received from the transmission source client device is the transmission destination. If the layer exceeds the reception condition of the client apparatus, the encoded data received from the transmission source client apparatus is selected from the encoded data of the reception condition layer of the transmission destination client apparatus to the transmission destination client apparatus. Send.

  The above-described problem is a client device used in a bidirectional image communication system having at least one client device that encodes and transmits a video, and a bidirectional image communication device that duplicates and distributes the video. Means for holding reception conditions and transmission conditions, reception condition transmission means for transmitting reception conditions to the bidirectional image communication apparatus, and reception conditions of other client devices in the bidirectional image communication system from the bidirectional image communication apparatus. The reception condition holding means for receiving and holding, the transmission condition, and the maximum one of the reception conditions of other client devices are compared. If they are equal, they are used as encoding parameters, and if they are not equal, the smaller one is encoded. Encoding parameter determination means as a parameter, and video is encoded using the encoding parameter and sent to the bidirectional image communication apparatus. It is solved by using the client device; and a coding transmission means for.

  According to the present invention, the transmission condition is compared with the maximum one of the reception conditions of the other client devices, and if it is equal, it is the encoding parameter, and if it is not equal, the smaller one is the encoding parameter. There is no useless band use or encoding processing on the transmission side, and the reception side can receive the best video that can be received on the reception side and transmitted on the transmission side.

  Each of the reception condition and the transmission condition includes one or more parameters of a bit rate, an image size, and a frame rate. When the reception condition and the transmission condition include a plurality of parameters, the encoding parameter determination unit The encoding parameters are determined by comparing the transmission conditions and the reception conditions.

  The video encoding may be performed using a video compression method having a scalability function, and each of the reception condition and the transmission condition may be layer information of encoded data.

  Each means in the present invention is realized by cooperation between a program for executing processing corresponding to each means and hardware resources of the apparatus. Therefore, in the description of the embodiments, each means of the present invention is described as a process corresponding to each means.

  As described above, according to the present invention, there is no useless bandwidth use or encoding processing on the transmission side, and the reception side can receive on the reception side and can transmit on the transmission side. Video can be received. Therefore, it is possible to provide a technique that enables bidirectional image communication using necessary and sufficient bandwidth and image processing amount.

  Embodiments of the present invention will be described below with reference to the drawings. In each embodiment described below, an example in which the present invention is applied to a video conference system will be described. However, the present invention can be applied to a system other than a video conference system.

(First embodiment)
FIG. 1 shows a configuration of a video conference system according to the first embodiment of the present invention. As shown in FIG. 1, the video conference system according to the first embodiment of the present invention includes a server 3, a client A (4), a client B (5), and a client C (6). B is connected to the server 3 via the network 1, and the client C is connected to the server 3 via the network 2. A camera is connected to each client. The camera connected to each client is, for example, a USB camera or an NTSC TV camera connected via a capture board. The server 3 is also referred to as a bidirectional image communication device.

  The server 3 has a general function as a server in a server / client video conference system, and also has a function according to the present invention described later. The server 3 has a hardware configuration of a general computer including a CPU, a memory, a hard disk, a network communication device, and the like, and each unit according to the present invention is executed by executing a program mounted on the server 3. Realized. The program can be installed on the server from a recording medium such as a CD-ROM, or can be downloaded and installed on the server via a network. Each client also has a general computer hardware configuration including a CPU, a memory, a hard disk, a network communication device, etc., and each means according to the present invention executes a program installed in the client. It is realized by.

  FIG. 2 is a flowchart showing the operation of the video conference system according to the first embodiment shown in FIG. With reference to FIG. 2, the operation of the video conference system according to the first embodiment will be described.

  First, transmission conditions and reception conditions are set in each client (step 1). The transmission condition is one or more of bit rate, image size, and frame rate, but other parameters may be used as the transmission condition. The reception condition is any one or more of a bit rate, an image size, and a frame rate, but other parameters may be used as the reception condition. The transmission condition is the maximum value of the encoding parameter determined according to the client performance, load, line bandwidth, etc. The reception condition is determined according to the client performance, load, line bandwidth, etc. Is the maximum value of the parameter. Note that setting the transmission condition and the reception condition means storing the transmission condition and the reception condition in a storage device such as a memory.

  In this embodiment, the bit rate of the reception condition of client A is αbps (bits per second), the image size is VGA (640 × 480 dots), the frame rate is βfps (frame per second), and the bit rate of transmission condition of client A is Assume that 2αbps, the image size is VGA, and the frame rate is βfps. Similarly, the bit rate of the reception condition of client B is αbps, the image size is CIF (352 × 288 dots), the frame rate is β / 2 fps, the bit rate of the transmission condition of client B is αbps, the image size is CIF, and the frame rate is β / 2fps. Also, the client C reception condition bit rate is α / 2 bps, the image size is QCIF (176 × 144 dots), the frame rate is β / 4 fps, the transmission condition bit rate of client C is α bps, the image size is QCIF, and the frame rate is Let β / 2fps. This setting may be performed by the user of the client, or may be dynamically performed automatically by the client based on conditions such as CPU load.

  Next, each client transmits its reception condition to the server 3 (step 2). The transmission of the reception condition may be performed at regular time intervals or may be performed every time the reception condition changes. The server 3 receives the reception conditions, and stores the received reception conditions in the storage device in association with the client.

  Then, the server 3 transmits reception conditions of clients other than the client to each client. That is, the reception conditions of the client B and the client C are transmitted to the client A, the reception conditions of the client A and the client C are transmitted to the client B, and the client A and the client B are transmitted to the client C. Send reception conditions. As a result, each client receives the reception conditions of the clients other than itself from the server 3, and stores them in the storage device in association with the clients (step 3).

  Each client may acquire the reception conditions of other clients by accessing the server at regular time intervals, for example. The server 3 may be configured to transmit the reception condition of the client to a client other than the client when the reception condition of the client changes. The processing of step 1 to step 3 is a pre-processing of a series of processes for obtaining encoded video data. However, when the reception conditions of each client are dynamically changed, the encoded video data is acquired and transmitted. Among these, the control of the present invention can be dynamically performed by performing the processing of Steps 1 to 3.

  After the pre-processing in Step 1 to Step 3, the camera video is acquired in each client (Step 4). Each client compares the transmission conditions of its own client with the reception conditions of other clients, and determines and sets an encoding parameter for encoding the acquired video (step 5). In the present embodiment, for each of the bit rate, the image size, and the frame rate, the transmission condition of the own client is compared with the maximum value in the reception condition of the other client. The smaller one is set as the encoding parameter value. As a result, in this embodiment, the bit rate in the encoding parameters of the clients A and B is α bps, the image size is CIF, the frame rate is β / 2 fps, the bit rate in the encoding parameters of the client C is α bps, and the image size is The QCIF and the frame rate are β / 2 fps.

  The above comparison process will be described in more detail with reference to FIG. In FIG. 3, description will be made by paying attention to the column indicated by 10. Client A receives the client B reception conditions (bit rate αbps, image size CIF, frame rate β / 2 fps), client C reception conditions (bit rate α / 2 bps, image size QCIF, frame rate β / 4 fps), and itself Transmission conditions (bit rate 2αbps, image size VGA, frame rate βfps) are stored in the storage device. Then, the client A compares the bit rate αbps which is the larger of the bit rate α / 2 bps of the client B and the bit rate α / 2 bps of the client C with the bit rate 2αbps of the transmission condition of the client A. A certain bit rate αbps is determined as an encoding parameter of the client A. The same comparison is made for each of the image size and the frame rate, and the encoding parameters are determined as shown in FIG. The same applies to client B and client C. In this way, wasteful bandwidth usage that can occur when the transmission condition of the sending client is directly used as the encoding parameter by setting the value below the maximum value in the reception condition of the other client as the encoding parameter of the sending client. And processing can be avoided. Further, since the maximum value in the reception conditions of other clients is compared with the transmission conditions, the client on the transmission side can select the best image among the images that can be received by other clients and can be transmitted by the transmission client. Can be sent.

  In the flowchart of FIG. 2, each client performs a camera video encoding process (step 6). Here, the encoding process is performed using the encoding parameter determined in step 5. The encoding uses a video compression format that generates an intra-frame compressed frame (referred to herein as a key frame) and an inter-frame compressed frame (referred to herein as an interpolated frame). Specifically, for example, MPEG4 and H.264. Encoding is performed using an encoding method such as H.263 (see Non-Patent Document 1).

  Next, the encoded video data is transmitted from each client to the server 3 (step 7). Here, for example, RTP / UDP or TCP obtained by encapsulating RTP with HTTP is used as a communication protocol, and the encoded data is transmitted as a video packet divided in frame units.

  Next, the server 3 copies and stores the received encoded data for each client (step 8). Then, the server 3 selects encoded data based on the reception condition of each client, and transmits the selected encoded data to each client (step 9). Thereafter, the client that has received the encoded data performs a process of decoding and displaying the received data (step 10).

  Hereinafter, the processing in the server 3 from duplication and storage of video packets in the above steps 8 and 9 to transmission will be described in more detail with reference to FIGS. This processing is realized by a program according to the present invention.

  FIG. 4 is a diagram illustrating an example of processing up to duplication, accumulation, and transmission of a video packet, and illustrates a case where a video packet received from the client C is transmitted to the client A and the client B. In FIG. 4, I indicates a video packet for an I picture (key frame compressed in a frame), P indicates a P picture (an interpolation frame for forward prediction), and B indicates a video packet for a B picture (an interpolation frame for bidirectional prediction). . The server 3 holds the reception conditions of each client in the storage device.

  When the server 3 receives a video packet from the client C, the server 3 duplicates the video packet by the number of connected clients in the order in which the video packet is received, and stores it in a buffer prepared for each client. The buffer is realized as a predetermined area in the memory of the server 3, for example.

  The buffer amount prepared here is set so as to include at least one I-picture video packet. For example, in the case of FIG. 4, a buffer amount that can store I, B, P, and B is set.

  In the case shown in FIG. 4, each receiving condition of the receiving client (client A, client B) is greater than or equal to each corresponding encoding parameter of the sending client (client C). The encoded data encoded on the transmission side can be received as it is without thinning out the data. Accordingly, the following processing is performed.

  First, the video packet of the I picture stored in the buffer is transmitted to each client. Then, the buffer is cleared and the video packet of the next B picture is waited. When the video packet of the B picture from the client C arrives, it is stored in the buffer and the video packet of the B picture is sent to each client in the same manner as the video packet of the I picture. Send. By repeating this, a video packet is transmitted to each client. Thus, since the encoded data of the client C satisfies all the reception conditions of the client A and the client B, the encoded data of the client C is received by the clients A and B as they are.

  FIG. 5 shows a case where the encoded data received from the client A is transmitted to the client C as an example in which the reception condition of the client on the receiving side does not satisfy the encoding parameter of the encoded data on the transmitting side. Here, as shown in FIG. 3, the encoding parameters of the client A are the bit rate αbps, the image size CIF, and the frame rate β / 2 fps, whereas the reception conditions of the client C are the bit rate α / 2 bps, The image size is QCIF and the frame rate is β / 4 fps. Therefore, the server 3 compares the bit rate of the reception condition of the client C with the bit rate of the encoded data, and determines that the bit rate of the reception condition of the client C is smaller than the bit rate of the encoded data. In this case, since all the encoded data cannot be transmitted to the client C, only the I picture compressed in the frame is transmitted to the client C as shown in FIG. Note that the server 3 may determine to transmit only the I picture based on the comparison of the frame rates.

  First, as in the case of FIG. 4, the video packet of the I picture is received from the client A, copied and stored in the buffer, and the video packet of the I picture stored in the buffer is transmitted to the client C. Then, without clearing the buffer, the B picture video packet is held in the buffer following the I picture video packet. Similarly, the subsequent P packet and B packet video packets are also held in the buffer, then the buffer is cleared, and the next received I picture is transmitted to the client C. Then, by repeating this process, only the I picture is transmitted to the client C.

  Thereby, although the frame rate is smaller than the reception condition of the client C, it is possible to reliably transmit the video to the client C. Here, only the I picture is transmitted. For example, when the reception condition indicates 90% or more of the encoding parameter, the I picture and the P picture may be transmitted.

  In the processes shown in FIGS. 4 and 5, the encoding parameter in the transmission side client can be automatically determined when the server 3 receives the encoded data. Alternatively, the transmission client may transmit the encoding parameter to the server 3 in advance, and the server 3 may refer to the encoding parameter.

  Since the processing of FIG. 4 or FIG. 5 is performed, the encoded data received by the client is the same encoded data as that of the transmission source or encoded data including only the I frame. Therefore, whichever encoded data arrives, the encoded data can be decoded and displayed without the need for separate processing. If the image size in the encoded data is larger than the reception condition, the image data may be reduced and displayed after being decoded by the reception client.

  In the present embodiment, a method for setting the encoding parameter before starting the video acquisition has been described. However, the client itself changes the transmission condition or the reception condition or both according to the processing capability of the client during the video transmission. It may be. As an example, FIG. 6 shows a transmission condition based on the CPU usage rate of the client and the number of received screens. 6A is a threshold table for CPU (A), FIG. 6B is a threshold table for CPU (B), and FIG. 6C is a threshold table for CPU (C). It is. The CPU capability increases from CPU (A) to CPU (C). For example, in FIG. 6A, if the number of received images (corresponding to the number of other clients) is 1 and the CPU usage rate is 0% or more and less than 40%, the transmission condition parameter is the image size. QCIF and the frame rate is 5 fps. In FIG. 6A, when the number of images is 5 or more or when the CPU load is high, the load is not allocated to the encoder (encoding) and only the image is received. In the example of FIG. 6, the bit rate is constant. An encoding parameter is determined based on transmission conditions determined as needed based on such a table. Further, the reception conditions may be changed as needed based on the CPU load or the like.

  As described above, by dynamically changing the transmission condition or the reception condition, it is possible to dynamically control the transmission / reception of the encoded data in the present invention.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. The system configuration in the second embodiment is the same as that in the first embodiment, as shown in FIG. In the second embodiment, for example, MPEG4 or H.264 is used. Using a compression coding technique having a scalability function in H.263 (see Non-Patent Document 1) or the like, reception conditions and transmission conditions are defined for each coding layer. As a layer structure of scalability, an intra-frame compressed key frame is generated as the basic layer 1, an intra-frame compressed key frame and an inter-frame compressed interpolated frame are generated as the basic layer 2, and the extended layer group 1 As described above, a high-frequency difference information frame for the base layer 1 and the base layer 2 is generated, and an interpolation frame for smoothing motion is generated for the enhancement layer group 1 as the enhancement layer group 2. Such a function of generating encoded data hierarchically is called scalability. Here, the hierarchical coding having the base layer 1, the base layer 2, the enhancement layer group 1, and the enhancement layer group 2 as described above is taken as an example, but the number of layers and the contents of each layer are It is not restricted to the example of embodiment. Note that the base layer in the present embodiment corresponds to the encoded data in the first embodiment.

  A processing flow in the second embodiment will be described with reference to the flowchart of FIG.

  First, transmission conditions and reception conditions are set in each client (step 21). In the present embodiment, the above layer information is used as a transmission condition and a reception condition. Setting transmission conditions and reception conditions means storing layer information in a storage device. The transmission condition in this embodiment is the maximum layer for encoding determined according to the client performance, load, line bandwidth, etc. The reception condition depends on the client performance, load, line bandwidth, etc. This is the maximum layer for decoding. In the present specification and claims, a large layer means a large layer range. For example, the layers of the basic layers 1 and 2 and the extended layer groups 1 and 2 are larger than the layers of the basic layers 1 and 2.

  Here, both the reception conditions and transmission conditions of client A are assumed to be basic layers 1 and 2 and enhancement layer groups 1 and 2. Further, the reception conditions of the client B are the basic layers 1 and 2 and the enhancement layer group 1, and the transmission conditions of the client B are the basic layers 1 and 2 and the enhancement layer groups 1 and 2. Furthermore, both the reception conditions and transmission conditions of client C are assumed to be basic layers 1 and 2. This setting may be performed by the user of the client, or may be dynamically performed automatically by the client based on conditions such as CPU load.

  Next, each client transmits its reception condition to the server 3 (step 22). The transmission of the reception condition may be performed at regular time intervals or may be performed every time the reception condition changes. The server 3 receives the reception conditions, and stores the received reception conditions in the storage device in association with the client. Then, the server 3 transmits reception conditions of clients other than the client to each client. Each client receives the reception conditions of clients other than itself from the server 3, and stores them in the storage device in association with the clients (step 23).

  Each client may acquire the reception conditions of other clients by accessing the server at regular time intervals, for example. The server 3 may be configured to transmit the reception condition of the client to a client other than the client when the reception condition of the client changes. The processes in steps 21 to 23 are pre-processes for a series of processes for obtaining encoded video data. However, when the reception conditions of each client fluctuate at any time, the encoded video data is acquired and transmitted. Among them, the processing of steps 21 to 23 may be performed. Thereby, dynamic control becomes possible.

  After the pre-processing in steps 21 to 23, the camera video is acquired in each client (step 24). Subsequently, each client compares the transmission conditions of its own client with the reception conditions of other clients, and determines and sets an encoding parameter for encoding the acquired video (step 25). In this embodiment, each client compares the encoding layer of its own transmission condition with the maximum encoding layer in the encoding layer of the reception conditions of other clients, and encodes the lower encoding layer. Set as a parameter. As a result, in the present embodiment, the encoding parameters of the client A are the basic layers 1 and 2 and the enhancement layer group 1, and the encoding parameters of the client B are the basic layers 1 and 2, and the enhancement layer group 1 and 2 and the encoding parameters of client C are base layers 1 and 2.

  The above comparison process will be described in more detail with reference to FIG. In FIG. 8, description will be made by paying attention to the column indicated by 20. The client A receives the reception condition of the client B (basic layers 1 and 2 and the enhancement layer group 1), the reception condition of the client C (basic layers 1 and 2), and its transmission condition (the basic layers 1 and 2 and the extension layer) Layer groups 1 and 2) are retained. The client A is the larger of the reception condition of the client B and the reception condition of the client C (basic layers 1 and 2 and the enhancement layer group 1), and the transmission condition of the client A (basic layers 1 and 2). And the enhancement layer groups 1 and 2), and the smaller basic layers 1 and 2 and enhancement layer group 1 are determined as the encoding parameters of the client A. The same applies to client B and client C.

  In the flowchart of FIG. 7, each client performs a camera video encoding process (step 26). Here, the hierarchical encoding process is performed using the encoding parameters determined in the step process.

  Then, the hierarchically encoded data of the video is transmitted from each client to the server 3 (step 27). Here, as a communication protocol, for example, RTP / UDP or TCP obtained by encapsulating RTP / UDP using HTTP is transmitted as a video packet obtained by dividing hierarchically encoded data in units of frames.

  Next, the server 3 copies and stores the received encoded data for each client (step 28). Then, hierarchically encoded data is selected based on the reception condition of each client and transmitted to each client (step 29). Thereafter, the client receives the encoded data (step 30). The data transmission process from the server 3 and the data reception process by the client will be described later in detail.

  After receiving the data, the client decrypts and displays the received data. That is, when receiving up to basic layer 1 or basic layer 2, the encoded data is decoded and displayed. In addition, when receiving up to enhancement layer group 1 or enhancement layer group 2, as shown in step 31 of FIG. 7, the base layer and enhancement layer group 1 are combined and decoded. If there is no data of the enhancement layer group 2, display processing is performed (step 32). If there is data of the enhancement layer group 2, the enhancement layer group 2 is decoded to generate an interpolation frame. Inserted between frames created based on layer 2 and enhancement layer group 1 (step 33). Then, display processing is performed (step 32).

  Next, the processing in the server up to the duplication, accumulation and transmission of the video packet in the above steps 28 and 29 will be described in more detail with reference to FIG. This processing is realized by a program according to the present invention.

  FIG. 9 is a diagram for describing an example of processing up to duplication and accumulation of video packets, layer control, and transmission in the second embodiment.

  The encoded data transmitted from the client and received by the server is duplicated in the order of reception for the number of connected clients and stored in a buffer prepared for each destination client. The buffer amount prepared here is set so as to include at least one base layer 1 encoded data. In the present embodiment, the buffer amount that can store the encoded data of base layer 1, base layer 2, enhancement layer group 1, and enhancement layer group 2 is set. The buffer is realized as a predetermined area in the memory in the server. Further, as shown in FIG. 9, the server holds reception conditions in each client.

  A case will be described in which encoded data is transmitted from the client A to the client B and the client C under the encoding parameter on the transmitting side and the receiving condition on the receiving side as shown in FIG.

  First, the server 3 analyzes the contents of the hierarchically encoded data stored in the buffer. In this case, since the encoding parameter of the client A is “up to extension 1”, the base layer 1, the base layer 2, and the extension layer group 1 are transmitted from the client A to the server 3. Therefore, the server 3 determines that the base layer 1, the base layer 2, and the enhancement layer group 1 are accumulated in the buffer. The server 3 holds the encoding parameter “up to extension 1” of the client A, and the contents of the hierarchically encoded data stored in the buffer can be determined by referring to the held data.

  Next, the reception conditions of the client B and the client C are determined from the table shown in FIG. As a result of the determination, since the reception condition of the client B is a reception condition that can be received up to the enhancement layer group 1, the server 3 receives from the client A and accumulates in the buffer corresponding to the client B. All the encoded data of the layer group 1 is transmitted to the client B. In addition, since the reception condition of the client C is a reception condition that can be received up to the basic layer 2, the basic layer 1 among the encoded data of the basic layer 1 to the extended layer group 1 accumulated in the buffer corresponding to the client B 2 and 2 are selected and transmitted to the client C.

  Similarly, the encoded data up to the enhancement layer group 2 transmitted from the client B and the encoded data up to the base layer 2 transmitted from the client C respectively meet the reception conditions of each client as shown in FIG. The encoded data is selected and transmitted to each client. Since client C transmits only up to basic layer 2 to server 3, for example, even if client A can receive up to enhancement layer group 2, only client 2 transmits up to basic layer 2 to client A. It will be.

  Next, the encoded data reception process at the client will be described.

  FIG. 10 shows the order of transmission of encoded data transmitted from the server 3 to the client. As shown in FIG. 10A, when the client receives up to the base layer 2, the encoded data of the base layers 1 and 2 are sequentially transmitted. As shown in FIG. 10B, when receiving up to the enhancement layer group 1, the codes are encoded in the order of the enhancement layer group 1, the base layers 1 and 2, the enhancement layer group 1, the base layers 1 and 2,. Data is sent. As shown in FIG. 10C, when receiving up to the enhancement layer group 2, the enhancement layer group 1, the basic layers 1 and 2, the enhancement layer group 2, the enhancement layer group 1, the base layers 1 and 2, and the enhancement layer group The encoded data is transmitted in the order of 2.

  Since the encoded data is transmitted as described above, the client receives the encoded data according to the procedure shown in FIG.

  First, the client is waiting to receive the encoded data of the enhancement layer group 1 (step 41). Next, the encoded data of base layers 1 and 2 or enhancement layer group 1 is received. When the encoded data of the enhancement layer group 1 is received while waiting for reception of the encoded data of the enhancement layer group 1 (step 42), the reception of the encoded data of the basic layers 1 and 2 is awaited (step 43). When the encoded data of the basic layers 1 and 2 is received in this state (step 44), the reception of the encoded data of the enhancement layer group 1 is awaited (step 45). As shown in FIG. 10, the next received data is the encoded data of the enhancement layer group 1 or the enhancement layer group 2, and when the encoded data of the enhancement layer group 2 is received (step 46), the process proceeds to step 41. Return. When the encoded data of the enhancement layer group 1 is received in the state of step 45 (step 47), the base layer 1 and 2 in the reception waiting state of step 43 are entered. On the other hand, when the encoded data of the basic layers 1 and 2 are received in the encoded data reception waiting state of the enhancement layer group 1 in step 41 (step 48), the process returns to step 41 again.

  In the procedure of FIG. 11, the repetition of step 41 and step 48 corresponds to the case of FIG. Moreover, repetition of step 41, step 42, step 43, step 44, step 45, step 47, step 43,... Corresponds to FIG. Further, the repetition of step 41 to step 46 corresponds to FIG. Therefore, the reception client can receive the encoded data appropriately selected from the server 3 by the procedure shown in FIG.

  Similar to the first embodiment, in this embodiment, the client may be configured to change the transmission condition or the reception condition according to the processing capability of the client during video transmission. As an example, FIG. 12 shows a transmission condition based on the CPU usage rate of the client and the number of received screens. Each table shown in FIG. 12 is the same as that shown in FIG. 6 except that the types of encoding parameters are different from those shown in FIG.

  Note that the control method in the first embodiment can be applied to transmission / reception of base layer encoded data in the second embodiment.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

It is a figure which shows the structure of the video conference system in embodiment of this invention. It is a flowchart showing operation | movement of the video conference system in the 1st Embodiment of this invention. It is a table | surface which shows the relationship between the transmission conditions in the 1st Embodiment of this invention, reception conditions, and an encoding parameter. It is a figure for demonstrating operation | movement of the server in the case of transmitting the encoding data received from the client C to the client A and the client B in 1st Embodiment. It is a figure for demonstrating operation | movement of the server in the case of transmitting the video packet received from the client A to the client C in 1st Embodiment. It is a table | surface which shows the transmission conditions based on the CPU usage rate of a client and the number of received screens in 1st Embodiment. It is a flowchart showing operation | movement of the video conference system in the 2nd Embodiment of this invention. It is a table | surface which shows the relationship between the transmission conditions in the 2nd Embodiment of this invention, reception conditions, and an encoding parameter. It is a figure for demonstrating operation | movement of the server in 2nd Embodiment. It is a figure which shows the transmission order of the encoding data from a server to a client. It is a figure which shows the procedure which receives encoded data in a client. It is a table | surface which shows the transmission conditions based on the CPU usage rate of a client and the number of received screens in 2nd Embodiment.

Explanation of symbols

1, 2 Network 3 Server 4 Client A
5 Client B
6 Client C

Claims (16)

At least one client device that encodes video using a video compression method that generates an intra-frame compressed key frame and an inter-frame compressed interpolated frame, and a bidirectional image communication device that replicates and distributes video A bidirectional image communication apparatus used in a bidirectional image communication system,
Receiving condition holding means for receiving and holding receiving conditions from each client device;
Receiving condition transmitting means for transmitting the receiving conditions of other client devices to each client device;
Storage means for receiving encoded data of a video from a transmission source client device, copying the received encoded data and storing it in a buffer;
Only the encoded data of the key frame received from the transmission source client device or the encoded data of the key frame and the encoded data of the interpolated frame is transmitted according to the reception condition of the transmission destination client device of the encoded data. have a selection transmitting unit that transmits device,
In the client device that has received the reception condition of the other client device, the transmission condition of the client device is compared with the maximum one of the reception conditions of the other client device. A bi-directional image communication apparatus which is used as an encoding parameter for encoding video, and a smaller one if not equal is used as an encoding parameter for encoding video in the client apparatus.   The selective transmission unit compares a predetermined encoding parameter value of the encoded data with a reception condition in the transmission destination client device, and the predetermined encoding parameter value is equal to or larger than a reception condition value in the transmission destination client device. In some cases, only the encoded data of the key frame received from the transmission source client apparatus is transmitted to the transmission destination client apparatus, and the predetermined encoding parameter value is less than the value of the reception condition in the transmission destination client apparatus. The bidirectional image communication apparatus according to claim 1, wherein the encoded data of the key frame and the encoded data of the interpolation frame are transmitted to the transmission destination client apparatus.   The bidirectional image communication apparatus according to claim 1, wherein the reception condition is any one of a bit rate, an image size, and a frame rate. Used in a bidirectional image communication system having at least one client device that encodes a video using a video compression method having a scalability function and a bidirectional image communication device that duplicates and distributes a video received from the client device A bidirectional image communication device,
Receiving condition holding means for receiving and holding receiving conditions from each client device;
Receiving condition transmitting means for transmitting the receiving conditions of other client devices to each client device;
Storage means for receiving encoded data of a plurality of layers in a scalability function from a transmission source client device, replicating the received encoded data of the plurality of layers and storing the data in a buffer;
Of the encoded data of multiple layers received from the transmission source client device, one or more layers of encoded data to be transmitted to the transmission destination client device are selected according to the reception conditions of the transmission destination client device, and selected. Selective transmission means for transmitting encoded data of one or a plurality of layers to a transmission destination client device,
In the client device that has received the reception condition of the other client device, the transmission condition of the client device is compared with the maximum one of the reception conditions of the other client device. A bi-directional image communication apparatus which is used as an encoding parameter for encoding video, and a smaller one if not equal is used as an encoding parameter for encoding video in the client apparatus. The selective transmission means compares the layer of the encoded data received from the transmission source client device with the layer as the reception condition of the transmission destination client device,
When the layer of the encoded data received from the transmission source client device is included in the layer as the reception condition of the transmission destination client device, the entire encoded data received from the transmission source client device is transmitted to the transmission destination client device,
When the layer of the encoded data received from the transmission source client device exceeds the layer as the reception condition of the transmission destination client device, the reception condition of the transmission destination client device out of the encoded data received from the transmission source client device. The bidirectional image communication apparatus according to claim 4, wherein encoded data for a layer is selected and transmitted to a transmission destination client apparatus.   The plurality of layers in the scalability function includes a base layer 1, a base layer 2, an enhancement layer group 1, and an enhancement layer group 2. The encoded data of the basic layer 1 is encoded data of key frames compressed in a frame. Yes, the encoded data of the basic layer 2 is encoded data of an intra-frame compressed key frame and an inter-frame compressed interpolated frame, and the encoded data of the enhancement layer group 1 includes the basic layer 1 and the basic layer 2 5. The encoded data of the difference information frame of the high-frequency part with respect to, and the encoded data of the enhancement layer group 2 is the encoded data of the interpolation frame for smoothing the motion with respect to the enhancement layer group 1. Or the bidirectional image communication apparatus according to 5. A client device used in a bidirectional image communication system having at least one client device that encodes and transmits a video and a bidirectional image communication device that duplicates and distributes the video,
Means for holding reception conditions and transmission conditions;
Receiving condition transmitting means for transmitting the receiving condition to the bidirectional image communication device;
Receiving condition holding means for receiving and holding reception conditions of other client apparatuses in the bidirectional image communication system from the bidirectional image communication apparatus;
An encoding parameter determining means that compares the transmission condition with the maximum of the reception conditions of other client devices, and if it is equal, sets it as an encoding parameter;
An encoding transmission unit that encodes video using the encoding parameter and transmits the encoded image to a bidirectional image communication device ;
In the bidirectional image communication apparatus that has received the reception condition of the client apparatus, the type of encoded data to be transmitted to the client apparatus is determined based on the reception condition .   Each of the reception condition and the transmission condition includes one or more parameters of a bit rate, an image size, and a frame rate. When the reception condition and the transmission condition include a plurality of parameters, the encoding parameter determination unit The client apparatus according to claim 7, wherein the encoding parameter is determined by comparing the transmission condition and the reception condition.   The client apparatus according to claim 7, wherein the video is encoded using a video compression method having a scalability function, and each of the reception condition and transmission condition is layer information of encoded data.   The plurality of layers in the scalability function includes a base layer 1, a base layer 2, an enhancement layer group 1, and an enhancement layer group 2. The encoded data of the basic layer 1 is encoded data of key frames compressed in a frame. Yes, the encoded data of the basic layer 2 is encoded data of an intra-frame compressed key frame and an inter-frame compressed interpolated frame, and the encoded data of the enhancement layer group 1 includes the basic layer 1 and the basic layer 2 10. The encoded data of the difference information frame of the high-frequency part with respect to, and the encoded data of the enhancement layer group 2 is the encoded data of the interpolation frame for smoothing the motion with respect to the enhancement layer group 1 The client device described in 1.   The client device according to claim 7, further comprising means for changing a transmission condition or a reception condition according to a processing capability of the client device. At least one client device that encodes video using a video compression method that generates an intra-frame compressed key frame and an inter-frame compressed interpolated frame, and a bidirectional image communication device that replicates and distributes video A bidirectional image communication apparatus used in a bidirectional image communication system having:
Receiving the reception condition from each client device and holding it in the storage means of the bidirectional image communication device;
Transmitting the reception conditions of other client devices to each client device;
Receiving encoded video data from a source client device, copying the received encoded data and storing it in a buffer;
Only the encoded data of the key frame received from the transmission source client device or the encoded data of the key frame and the encoded data of the interpolated frame is transmitted according to the reception condition of the transmission destination client device of the encoded data. It has a transmitting device,
In the client device that has received the reception condition of the other client device, the transmission condition of the client device is compared with the maximum one of the reception conditions of the other client device. A method characterized in that it is used as an encoding parameter for encoding video, and the smaller one is used as an encoding parameter for encoding video in the client device . Used in a bidirectional image communication system having at least one client device that encodes a video using a video compression method having a scalability function and a bidirectional image communication device that duplicates and distributes a video received from the client device A method of processing executed by the bidirectional image communication apparatus,
Receiving the reception condition from each client device and holding it in the storage means of the bidirectional image communication device;
Transmitting the reception conditions of other client devices to each client device;
Receiving encoded data of a plurality of layers in a scalability function from a transmission source client device, copying the received encoded data of a plurality of layers and storing them in a buffer;
Of the encoded data of multiple layers received from the transmission source client device, one or more layers of encoded data to be transmitted to the transmission destination client device are selected according to the reception conditions of the transmission destination client device, and selected. Transmitting encoded data of one or more layers to a destination client device,
In the client device that has received the reception condition of the other client device, the transmission condition of the client device is compared with the maximum one of the reception conditions of the other client device. A method characterized in that it is used as an encoding parameter for encoding video, and the smaller one is used as an encoding parameter for encoding video in the client device . At least one client device that encodes video using a video compression method that generates an intra-frame compressed key frame and an inter-frame compressed interpolated frame, and a bidirectional image communication device that replicates and distributes video A program for causing a bidirectional image communication apparatus used in a bidirectional image communication system to execute processing,
A procedure for receiving the reception condition from each client device and holding it in the storage means of the bidirectional image communication device;
A procedure for transmitting reception conditions of other client devices to each client device;
A procedure for receiving encoded data of a video from a transmission source client device, replicating the received encoded data and storing it in a buffer;
Only the encoded data of the key frame received from the transmission source client device or the encoded data of the key frame and the encoded data of the interpolated frame is transmitted according to the reception condition of the transmission destination client device of the encoded data. A program for causing a bidirectional image communication apparatus to execute a procedure of transmitting to the apparatus ,
In the client device that has received the reception condition of the other client device, the transmission condition of the client device is compared with the maximum one of the reception conditions of the other client device. A program characterized in that it is used as an encoding parameter for encoding video, and a smaller one is used as an encoding parameter for encoding video in the client device . Used in a bidirectional image communication system having at least one client device that encodes a video using a video compression method having a scalability function and a bidirectional image communication device that duplicates and distributes a video received from the client device A program for causing a bidirectional image communication apparatus to execute processing,
A procedure for receiving the reception condition from each client device and holding it in the storage means of the bidirectional image communication device;
Transmitting the reception conditions of other client devices to each client device;
A procedure for receiving encoded data of a plurality of layers in a scalability function from a transmission source client device, copying the received encoded data of a plurality of layers and storing them in a buffer;
Of the encoded data of multiple layers received from the transmission source client device, one or more layers of encoded data to be transmitted to the transmission destination client device are selected according to the reception conditions of the transmission destination client device, and selected. A program for causing a bidirectional image communication apparatus to execute a procedure for transmitting encoded data of one or a plurality of layers to a destination client apparatus ,
In the client device that has received the reception condition of the other client device, the transmission condition of the client device is compared with the maximum one of the reception conditions of the other client device. A program characterized in that it is used as an encoding parameter for encoding video, and a smaller one is used as an encoding parameter for encoding video in the client device . A program for causing a client device used in a bidirectional image communication system having at least one client device that encodes and transmits a video and a bidirectional image communication device that duplicates and distributes the video to execute processing,
A procedure for reading the reception condition from the storage means and transmitting the reception condition to the bidirectional image communication apparatus;
A procedure for receiving the reception conditions of other client devices in the bidirectional image communication system from the bidirectional image communication device and holding them in the storage means;
A procedure for comparing the transmission condition read from the storage means with the maximum one of the reception conditions of other client devices, and setting it as an encoding parameter if they are equal, and setting the smaller one as an encoding parameter if they are not equal; ,
A program for causing a client device to execute a procedure of encoding video using the encoding parameter and transmitting the video to a bidirectional image communication device ;
In the bidirectional image communication apparatus that has received the reception condition of the client apparatus, a type of encoded data to be transmitted to the client apparatus is determined based on the reception condition .

Images