JP2008005349A - Video encoder, video transmission apparatus, video encoding method, and video transmission method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video encoder and a video encoding method capable of effectively encoding video data while keeping the quality of a person video that is an object to be noticed in video in television conferences and speeches of lectures or the like unchanged, and provide video transmission apparatus and a video transmission method capable of effectively transmitting video data in response to the state of a communication rate or the like of a transmission line. <P>SOLUTION: A remote conference device 1 encodes conference video data [10] in which conference scenes are photographed and transmits it to a partner apparatus when a scene is changed, and it encodes only person video data [11] to [14] and transmits them to the partner apparatus when the scene is not changed. Hereby, a video of a person who is an object to be noticed in conference video is transmitted at all times even when a communication rate on a transmission line is lowered and the feature of the video is changed. Consequently, the video data [11] to [14] of the person are synthesized with the conference video data [10] in the partner apparatus, whereby the video of the conference scene can be updated with a less data transmission amount. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a video encoding device and video encoding method for encoding video data including a person such as a teleconference or speech, and a video transmission device for transmitting the video data between different points via a network or the like. And a video transmission method.

Conventionally, there has been a remote conference system used for a conference between remote locations such as a head office and a sales office. As such a remote conference system, for example, in Patent Document 1, one microphone is installed for each conference participant to collect the voice of each speaker, and a zoom-up camera is used. A video conference system for photographing a speaker is disclosed.
JP-A-2-202275

  In a video conference system as shown in Patent Document 1, audio data and video data are exchanged via a dedicated line or the like. Also, in recent video conference systems, MPEG1, 2 and ITU-T H.264 have been adopted so that a video conference can be performed via a network using a low bit rate communication line such as an ISDN line or an analog telephone network. Digital video data compressed using a video compression method defined by H.263 is exchanged.

  MPEG1, 2 and ITU-TH. When video is encoded using H.263 or the like, bit allocation is performed using a constant and uniform algorithm from the motion information and the spatial information amount of the video. For example, such a method is used for contents such as television broadcast sports programs and movie DVDs.

  On the other hand, the conventional video conferencing system is designed so that video conferencing can be performed even with a low bit rate communication line, but the communication line traffic increases due to an increase in the number of simultaneously connected users. If the rate drops, the quality of the whole video will drop, so you will not be able to send the necessary information.For example, a moving person will look like a mosaic picture, or a moving person's image will be missing. There was a problem to do.

  Therefore, the present invention provides a video encoding apparatus and a video encoding method capable of efficiently encoding video data while maintaining the quality of a video of a person who is a target of attention in a video such as a video conference or speech. It is another object of the present invention to provide a video transmission apparatus and a video transmission method capable of efficiently transmitting image data in accordance with the communication rate of the transmission path and the ratio of the amount of human information to the entire video.

  The present invention has the following configuration as means for solving the above problems.

(1) imaging means for photographing a person and its surroundings and outputting the image;
A person detecting means for detecting a person from the video;
Encoding for periodically extracting the detected person's video from the video, encoding the human video data at a low compression rate, and encoding the video data other than the person at a high compression rate Means,
It is provided with.

  In this configuration, the video encoding device extracts a video of a person from a person and surrounding video, encodes this video data at a low compression rate, and encodes video data other than the person at a high compression rate. To do. Therefore, in video such as video conferences and speeches, there is almost no movement, and attention is focused on the human area. Encode well.

(2) imaging means for photographing a person and its surroundings and outputting the image;
A person detecting means for detecting a person from the video;
Encoding means for encoding the entire area of the video imaged by the imaging means, and thereafter encoding the person video data that is a rectangular area including a person in the video periodically;
It is provided with.

  In this configuration, the video transmission apparatus encodes the entire area of the person and the surrounding video, and thereafter encodes only the video data of the person. Therefore, in video such as video conferences and speeches, there is almost no movement, and it is the person's area that is noticed. Therefore, the video of the person who is the subject of attention should be encoded periodically. Thus, at the time of decoding, only the person area needs to be updated in the video of the entire area, so that the video data can be encoded more efficiently.

(3) a scene change detecting means for detecting a change in configuration in the entire area of the video imaged by the imaging means;
The encoding means encodes the entire area of the video at a timing when the scene change detection means detects a change in configuration in the entire area of the video.

  In this configuration, the entire area of the video is encoded at the timing when the scene change detection means detects the change in the configuration in the entire area of the video. Therefore, when a scene change occurs in a video such as a video conference or speech, the entire video area is set (updated), so that the video can be encoded without any contradiction during decoding. .

(4) The video encoding device according to any one of (1) to (3),
Communication means for transmitting video data encoded by the video encoding device to a counterpart device;
It is provided with.

  In this configuration, the video transmission device transmits the encoded video data to the counterpart device. Accordingly, since video such as a video conference or lecture speech is efficiently encoded, the video data can be reliably transmitted to the partner apparatus regardless of the communication rate of the transmission path.

(5) imaging means for photographing a person and its surroundings and outputting the image;
A person detecting means for detecting a person from the video;
A calculation means for performing a predetermined calculation based on the state of the communication rate of the transmission path and the ratio of the information amount of the person in the entire video;
If the calculation result of the calculation means is equal to or greater than a threshold, the detected person's video is periodically extracted from the video taken by the imaging means, and the video data is encoded at a low compression rate. Video data at a high compression rate,
When the calculation result of the calculation means is less than the threshold value, the entire area of the video imaged by the imaging means is encoded, and thereafter the person video data that is a rectangular area including the person in the video is encoded periodically. Encoding means for
Communication means for transmitting the video data encoded by the encoding means to a counterpart device;
It is provided with.

  In this configuration, when the result of a predetermined calculation performed based on the state of the communication rate and the ratio of the information amount of the person occupying the entire video is equal to or greater than a certain value, the video transmission apparatus converts the video of the person from the video. The person video data is extracted and encoded at a low compression rate, the video data other than the person is encoded at a high compression rate, and both video data are transmitted to the partner apparatus. In addition, when the result of the above calculation is less than a certain value, the video transmission apparatus encodes the entire area of the video data, transmits the video data to the partner apparatus, and thereafter extracts the person into a rectangle. The video data is encoded, and the video data is transmitted to the partner apparatus. Therefore, depending on the state of the communication rate on the transmission path and the ratio of the amount of human information to the entire video, the information amount in the video is reassigned and unnecessary information is deleted, so that video conferences, speeches, etc. In the video, it is possible to reliably transmit the video of the person who is the target of attention to the partner apparatus.

(6) a procedure for periodically detecting a person from an image of the person and the surroundings;
Extracting the detected person's video data and encoding it at a low compression rate, and encoding the video data other than the person at a high compression rate,
It is provided with.

  In this configuration, the same effect as (1) can be obtained.

(7) a procedure for periodically detecting a person from an image of the person and its surroundings;
A procedure for encoding the entire area of the person and the surrounding video, and encoding the person video data which is a rectangular area including the person in the video periodically thereafter;
It is provided with.

  In this configuration, the same effect as (2) can be obtained.

(8) A procedure for detecting a configuration change of the entire area in the person and the surrounding image,
A procedure for encoding the entire area of the video data at a timing when a change in the configuration of the entire area in the video is detected;
It is provided with.

  In this configuration, the same effect as (3) can be obtained.

(9) After performing the video encoding method according to any one of (6) to (8),
A procedure for transmitting the encoded video data to a counterpart device;
It is provided with.

  In this configuration, the same effect as (4) can be obtained.

(10) A procedure for performing a predetermined calculation based on the state of the communication rate of the transmission path and the ratio of the information amount of the person in the entire video,
When the calculation result is equal to or greater than a threshold value,
A procedure for periodically detecting a person from an image of the person and its surroundings,
Extracting the detected person's video data and encoding it at a low compression rate, and encoding the video data other than the person at a high compression rate,
A procedure for transmitting the encoded person video data and video data other than the person to the counterpart device;
When the calculation result is less than the threshold value,
A procedure for periodically detecting a person from an image of the person and its surroundings,
A procedure for encoding the entire area of the video, and thereafter encoding the person video data which is a rectangular area including the person in the video periodically;
A procedure for transmitting the encoded video data to a counterpart device;
It is provided with.

  In this configuration, the same effect as (5) can be obtained.

  In video images such as video conferences and speeches, there is little movement, and there is a feature that information about objects other than the subject of interest such as people is almost unnecessary. When the predetermined calculation result based on the state of the communication rate of the transmission path and the proportion of the information amount of the person in the entire video is less than the threshold by reallocating the information amount or deleting unnecessary information , Only the target video of interest is encoded and transmitted, and if the calculation result is equal to or greater than the threshold, the target video of interest is encoded with a low compression rate and the background video is encoded with a high compression rate and transmitted. Therefore, even if the communication rate decreases according to the line status and video characteristics, it is possible to reliably send the target video. In addition, it is possible to improve the image quality of the video at the same transfer rate, and to reduce the bit rate if the quality is the same.

  In the following description, a remote conference apparatus having the functions of the video encoding apparatus and the video transmission apparatus of the present invention will be described as an example. Of course, the encoding device and the video transmission device of the present invention may have a single configuration.

  FIG. 1 is a perspective view of a remote conference apparatus according to an embodiment of the present invention, and a diagram showing two remote conference apparatuses connected to a network. As shown in FIG. 1A, the remote conference apparatus 1 has a rectangular parallelepiped shape that is long in one direction, and the eight speaker units SP1 to SP8 that constitute the speaker array 26 and the microphone array 30 that constitute the microphone array 30. The microphone units MIC <b> 1 to MIC <b> 14 are arranged on a straight line at regular intervals on the front side. Although not shown, an input / output connector 64 is provided on the back side of the remote conference device 1.

  The number of speaker units constituting the speaker array 26 and the number of microphone units constituting the microphone array 30 are not limited to the above quantities.

  A video camera 50 and a monitor 52 can be connected to the remote conference apparatus 1. When these are connected, the remote conference apparatus 1 is placed on the upper surface of the monitor 52 as shown in FIG. The video camera 50 may be mounted on the upper surface of the remote conference device 1 for use. The remote conference apparatus 1 can also be used by placing it on the end or table of a conference desk regardless of whether the video camera 50 and the monitor 52 are connected.

  As shown in FIG. 1B, a remote conference system can be constructed by connecting two remote conference apparatuses 1a and 1b to a network 8 such as the Internet or a LAN. The remote conference apparatus 1a exchanges an audio signal picked up by the microphone array 30 and a video signal picked up by the video camera 50 with another remote conference apparatus 1b connected to the network 8 using a protocol such as SIP ( Send and receive). By using the remote conference apparatuses 1a and 1b, the video cameras 50a and 50b, and the monitors 52a and 52b, the user can perform a video conference using audio and video between the different points a and b. Although not shown, it is possible to hold a video conference between a plurality of points by connecting a plurality of remote conference apparatuses 1 to the network 8.

  Next, a specific configuration of the remote conference apparatus 1 will be described. FIG. 2 is a functional block diagram of the remote conference apparatus according to the embodiment of the present invention. The remote conference apparatus 1 includes a remote controller (remote controller) 10, a signal receiving unit 12, a control unit 14, a memory 16, a voice processing unit 20, a speaker array 26 including eight speaker units SP1 to SP8, and 14 microphone units MIC1. To the MIC 14, a video camera 50, a monitor 52, a video encoding unit 54, a video decoding unit 56, a multiplexing / separating unit 60, a communication unit 62, and an input / output connector 64.

  In the remote conference apparatus 1, the ITU-T H.264 261, H.H. 262, H.C. 263, H.M. H.264, Motion JPEG, and other video encoding schemes.

  Here, the present invention relates to the video processing function of the video transmission apparatus, and the audio processing function is not the subject of the present invention. Therefore, in the following description, the description of the audio processing system of the remote conference apparatus 1 will be simplified.

  The remote controller 10 includes an operation unit 101 for performing various settings of the remote conference apparatus 1. The remote controller 10 outputs a signal corresponding to a user operation as infrared light.

  The signal receiving unit 12 receives (receives) a signal (infrared light) output from the remote controller 10 and outputs it to the control unit 14.

  The control unit 14 controls the audio processing unit 20, the video camera 50, the video encoding unit 54, the video decoding unit 56, the communication unit 62, and the like. The control unit 14 controls each unit based on the signal transmitted from the signal receiving unit 12, reads a program from the memory 16, and writes data to the memory 16.

  The memory 16 stores a program executed by the control unit 14, data written by the control unit 14, and the like.

  The audio processing unit 20 compresses and encodes the audio signal picked up by the microphone array 30 and outputs it to the multiplexing / separating unit 60, or the encoded audio signal is sent from the multiplexing / separating unit 60. When it comes, it decodes and outputs it to the speaker array 26. In addition, the sound processing unit 20 performs an echo cancellation process on the sound signal picked up by the microphone array 30 and the sound signal output to the speaker array 26, thereby performing appropriate echo removal. Only the speaker voice of the own device is transmitted to the network as an output voice signal.

  The speaker units SP <b> 1 to SP <b> 8 are composed of omnidirectional speakers, which convert sound output signals individually given from the sound processing unit 20 into sound and emit the sound to the outside.

  Each microphone unit MIC <b> 1 to MIC <b> 14 constituting the microphone array 30 collects sound from outside the remote conference device 1 and converts it into an electrical signal, and outputs the collected sound signal to the sound processing unit 20.

  The video camera 50 captures the video of the conference scenery and conference participants (persons) and outputs the video data to the video encoding unit 54 at regular intervals.

  The monitor 52 displays video data sent from another remote conference apparatus and decoded by the video decoding unit 56.

  The video encoding unit 54 encodes (encodes) video data output from the video camera 50 at a constant period and outputs the encoded video data to the multiplexing / separating unit 60. In addition, the video encoding unit 54 outputs the video block information amount and the human weight coefficient to the control unit 14.

  The video decoding unit 56 decodes (decodes) the encoded video data output from the multiplexing / demultiplexing unit 60 and outputs the decoded video data to the monitor 52.

  The multiplexing / separating unit 60 multiplexes the encoded audio data output from the audio processing unit 20 and the encoded video signal output from the video encoding unit 54, and transmits the stream data to the communication unit. To 62. The multiplexing / separating unit 60 separates the stream data transmitted from the communication unit 62 into encoded audio data and encoded video data, and outputs an audio signal to the audio processing unit 20. The video signal is output to the video decoding unit 56.

  The communication unit 62 converts the stream data output from the multiplexing / demultiplexing unit 60 into a data format (protocol) corresponding to the communication method in the network 8, and transmits another remote data via the input / output connector 64 and the network 8. It transmits to the conference apparatus 1. In addition, the communication unit 62 converts the stream data sent from another remote conference apparatus 1 input via the network (LAN) 8 connected to the input / output connector 64 into a data format (protocol corresponding to the network 8). ) And output to the multiplexing / separating unit 60. Further, the communication unit 62 monitors the communication state (loss) in the network (transmission path) 8 and transmits the information to the control unit 14 as needed.

  Next, a detailed configuration of the video encoding unit 54 will be described. FIG. 3 is a block diagram illustrating a configuration of the video encoding unit.

  The video encoding unit 54 includes a face / person detector 70, a screen change detector 72, an encoding controller 74, a first changeover switch 76, a discrete cosine transformer (DCT) 78, a quantizer (Q) 80, a code A quantizer 81, an inverse quantizer (IQ) 82, an inverse discrete cosine transformer (IDCT) 84, an adder 86, a video memory 88 with motion compensation, a second changeover switch 90, and a subtractor 92.

  When performing the intra coding, the video encoding unit 54 switches the first changeover switch 76 to the intra terminal 76A side and the second changeover switch 90 to the intra terminal 90A side. Further, in the case of performing inter coding, the video coding unit 54 switches the first changeover switch 76 to the inter terminal 76B side and the second changeover switch 90 to the inter terminal 90B side.

  Video data input from the video camera 50 is sent to the face / person detector 70, the intra terminal 76A of the first changeover switch 76, the video memory 88 with motion compensation, and the subtractor 92.

  When the face / person detector 70 determines whether or not there is an image having the characteristics of a face or a person in the transmitted image data, and detects an image having the characteristics of a face or a person, the case of a face or a person is detected. Are output to the encoding controller 74. The face / person detector 70 detects the face using the skin color, the screen outline configuration, and the like. That is, the face / person detector 70 detects a face portion using features such as skin tone, face parts (eyes, glasses, nose, mouth, etc.), and luminance change of the image.

  In addition, the face / person detector 70 acquires a global boundary / motion detection signal from the screen switching detector 72, and extracts a person in combination with the face detection signal. Specifically, the face / person detector 70 performs a degeneration / expansion process on a region where a face is present and performs a near edge detection to detect a person from a global region. Also, since conference participants often move their mouths and hands, parameters such as weak movements are used to improve human detection accuracy.

  Note that the above processing need not be performed for each frame of a coding unit. Further, according to the mode set in the remote conference apparatus 1, the detection result is used as a sensitivity coefficient for raising the priority of bit priority assignment, or used as information for determining a limited area. For example, the weight (sensitivity) of the face is increased, and the person (body) that is the periphery of the face is set as the subsequent weight coefficient.

  When performing intra coding, the video data input from the video camera 50 is sent to the discrete cosine transformer 78 via the first changeover switch 76, and the discrete cosine transformer 78 performs a discrete cosine transformation process. , And sent to the quantizer 80. The quantizer 80 quantizes the video data sent from the discrete cosine transformer 78 based on the quantization parameter sent from the coding controller 74 and outputs the quantized video data to the encoder 81. The video data quantized by the quantizer 80 is inversely quantized by the inverse quantizer 82, further subjected to inverse discrete cosine transform processing by the inverse discrete cosine transformer 84, and sent to the adder 86. .

  When the intra coding is performed, the adder 86 outputs the video data output from the inverse discrete cosine transformer 84 to the video memory 88 with motion compensation as it is. In addition, the adder 86 outputs the video data with motion compensation sent from the video memory with motion compensation 88 via the second changeover switch 90 and the inverse discrete cosine transformer 84 when performing inter coding. The video data thus added is added to the video memory 88 with motion compensation.

  The motion compensated video memory 88 outputs the motion compensated video generated using the video data output from the adder 86 and the video data from the video camera 50 to the subtracter 92. The motion compensated video memory 88 outputs the motion vector to the encoder 81. Further, when performing inter coding, the motion compensated video memory 88 outputs the motion compensated video to the adder 86 via the second changeover switch 90.

  The subtracter 92 subtracts the video data of the video camera 50 from the motion compensated video data output from the motion compensated video memory 88 and outputs difference data between frames to the screen switching detector 72. In addition, when performing inter coding, the subtractor 92 outputs the inter-frame difference data to the discrete cosine transformer 78 via the first changeover switch 76.

  The difference data between the frames is subjected to a discrete cosine transform process by a discrete cosine transformer 78 and sent to a quantizer 80. The quantizer 80 quantizes the difference data between frames sent from the discrete cosine transformer 78 based on the quantization parameter sent from the coding controller 74, and converts the pixel coefficient into the encoder 81. And output to the inverse quantizer 82.

  The screen switching detector 72 detects screen switching based on the difference data between frames output from the subtracter 92. When the screen switching detector 72 detects that the screen has been switched, such as when the brightness or saturation has changed greatly, as well as when the scene has changed, or when the camera has performed pan, zoom, or tilt operations, A signal to that effect is output to the control controller 74.

  The encoding controller 74 outputs a switching signal to the first changeover switch 76 and the second changeover switch 90 depending on whether intra-encoding or inter-encoding is to be performed, and at the same time, to the encoder 81. Output flag or interflag. In addition, the encoding controller 74 outputs an encoding flag or a non-encoding flag to the encoder 81. Further, the encoding controller 74 outputs the quantization parameter to the quantizer 80 and the encoder 81.

  Next, the operation of the remote conference apparatus 1 will be described. FIG. 4 is a diagram illustrating a conference scene using the remote conference device and a normal mode image captured by the video camera.

FIG. 5 is a diagram showing a video shot in the bit priority assignment mode. FIG. 6 is a diagram illustrating an image captured in the limited area transmission mode. FIG. 7 is a diagram illustrating an entire image of the conference landscape before zooming and an entire image of the conference landscape after zooming.

  As shown in FIG. 4A, by installing the monitor 52, the remote conference device 1, and the video camera 50 in front of the conference table 41, the video of the conference and the voices spoken by the conference participants are separated from each other. It is possible to communicate with the conference apparatus via the network (transmission path) 8.

  The conference scene shot by the video camera 50 is sent to another remote conference device as an image as shown in FIG.

  In the remote conference apparatus 1. Normal mode, 2. 2. bit priority allocation mode, and Any one of the limited area transmission modes can be set. However, even in the limited area transmission mode, the state may be automatically changed to the bit priority assignment mode or the normal mode state, and the bit priority assignment mode may be automatically changed to the normal mode state.

  1. The normal mode is a mode in which bits are allocated using a constant and uniform algorithm from motion information and the amount of spatial information of the video when encoding the video. This mode is used when the transmission path has a sufficient margin for the required bit rate or when the entire screen has to be updated.

  2. In the bit priority allocation mode, bit allocation is performed using motion information, the amount of time difference information, and a person weighting factor. In other words, the setting is made so that more code bits are allocated to a person who is an important area in the video and particularly to a face portion. In this mode, the person area and the face area are extracted. For example, as shown in FIG. 5, the remote conference apparatus 1 detects persons A to C and their face portions in the conference scene video by using the face / person detector 70 of the video encoding unit 54. The encoding controller 74 of the video encoding unit 54 uses the human weighting coefficient to determine whether to stop encoding a block (for example, a macro block). Also, the encoding controller 74 uses the human weighting coefficient to set so that more code bits are allocated in conjunction with the quantization parameter output to the quantizer 80. Accordingly, in the bit priority allocation mode, the face areas [1], [3], and [5] of the persons A to C in the conference scenery video shown in FIG. The body areas [2], [4], and [6] of A to C are high-definition images with a low compression rate next to the face areas of the persons A to C, and the background areas [7] that are other areas. ] Is a high-compression, low-quality video that is sent to another teleconferencing device.

  In addition, the remote conference apparatus 1 usually divides the screen into a band-like structure and transmits the unit in order from the upper end of the screen to the lower end, but in this mode, it is possible to prevent a code from being assigned at the end of transmission. In order to avoid this, send important areas on the screen first. That is, in the video of a conference scene, the person area is usually located from the center to the bottom of the screen. However, in this mode, the remote conference device 1 has a band-like structure as shown in FIG. When the person in the video of the meeting scene is detected by dividing, the area including the person (area [3] including the top of person B in the figure) is transmitted from the upper end to the lower end in order, and then sequentially from the upper end. Send up to just before the area containing the person.

  3. The limited area transmission mode is applied to video conferencing and speech modes at a low bit rate such as ISDN, GSTN (analog telephone network), and a high congestion and low bandwidth IP network. When the transmission path has a low bit rate, the header overhead becomes relatively large even in an area where there is no motion in the video. Therefore, in this mode, as shown in FIG. 6 (A), when the transmission of video data is started and when the scene changes, the video of the entire area [10] of the conference scene is transmitted. 6B, as shown in FIG. 6B, only the necessary areas in the video, that is, the face and person areas [11] to [14] are transmitted, and the other areas are block blocks. The attribute information is also set not to be transmitted.

  In this mode, in order to reduce the amount of data to be transmitted, the face and person areas are set as follows. That is, in order to facilitate the implementation, the block shape of the face or person area is set to a rectangle, and this rectangle area is determined using a person weighting factor. In addition, the setting is made so that the number of persons included in the video of the meeting scene is not divided into very fine blocks. Further, in this mode, the above block transmits the address of the first block and transmits image data of successive blocks to another remote conference apparatus 1.

  Further, as another encoding method, as shown in the upper part of FIG. 6 (C), for an area where data is not transmitted, that is, an area other than the face or person area (solid black area in the figure) It is possible to send only the information specifying whether to skip the image, and to send a block address and video data for the face and person areas (person A, B, and C areas in the figure). In this method, it is not dependent on the number of persons, and even if the shape is not limited to a rectangle, a region to be encoded (region to transmit) and a region not to be encoded (region to not transmit) are determined. Video data can be easily transmitted.

  Also, in this mode, the background video is not sent, so when the video data starts or when the scene changes, the brightness or saturation changes greatly, or when the video camera 50 is pan / zoom / tilt. When the global area of the video is changed by the operation, for example, when the video of FIG. 7A is changed to the video of FIG. 7B by the zoom operation, the normal mode is switched to refresh the video. Need to do.

  The remote conference apparatus 1 can set the three modes as described above.

  In the remote conference device 1, it is also possible to set an automatic switching mode for automatically switching the above three modes. In the remote conference apparatus 1, the communication unit 62 monitors the communication state (loss) of the network (transmission path) 8. Therefore, when the automatic switching mode is set, the communication rate and the communication state in the network 8 are set. Further, the above three modes are automatically switched according to the ratio of the amount of information of the person in the entire video.

  FIG. 8 is a block diagram illustrating a specific configuration of the control unit. As shown in FIG. 8, in the control unit 14, the block information amount calculator 32 obtains the product of the block information amount obtained from the video encoding unit 54 and the human weight coefficient (α), and the person region integrator 33 These are integrated to obtain the information amount Rp of the person area occupying the screen. Further, the total area accumulator 34 accumulates the block information amount to obtain the information amount Ra of the entire screen. Then, Rp / Ra is calculated by the divider 35 to obtain the person activity Ap. Here, the person activity Ap is the ratio of the person area to the screen.

  The mode decision evaluator 36 performs a predetermined function calculation using the bit rate and transmission loss rate acquired from the communication unit 62 and the person activity Ap obtained as described above. Here, the bit rate is a set value of the line communication rate. The transmission loss rate is an actual data rate at which transmission has failed.

  The mode decision evaluator 36 selects the normal mode if the calculation result is equal to or greater than the first threshold A. The mode decision evaluator 36 selects the bit priority allocation mode when the calculation result is the first threshold A to the second threshold B. Further, the mode determination evaluator 36 selects the limited region transmission mode when the calculation result is less than the second threshold value B.

  In the remote conference apparatus 1, for example, when the person activity Ap >> 0, the limited area transmission mode is selected. Further, when the line communication rate is low, the bit priority mode is selected, and when the line communication rate is lower, the limited area transmission mode is selected. Further, when the transmission loss rate is high, the bit priority assignment mode or the limited area transmission mode is selected. However, when the transmission rate is low and loss is large, the limited mode is selected without performing automatic switching.

  The mode decision evaluator 36 may select a mode corresponding to the calculation result with reference to the table 37. The function used for the calculation performed by the mode decision evaluator 36 is set in advance by experiments or the like.

  Further, in the remote conference apparatus 1, it is possible to set the automatic switching mode so as to automatically switch the bit priority allocation mode and the limited area transmission mode among the above three modes. In this case, in the teleconference device 1, the bit priority allocation mode is obtained when the calculation result of the mode decision evaluator 36 is equal to or higher than the second threshold B, and the limited area transmission mode is obtained when the calculation result of the mode decision evaluator 36 is less than the second threshold B. Switch to.

  In the remote conference system having the configuration shown in FIG. 1B, when stream data including video data in each mode is output from the remote conference device 1a installed at the point a as described above, the remote conference device 1a is installed at the point b. The received 1b is received by the communication unit 62 via the input / output connector 64 as stream data. Then, the video data encoded by the multiplexing / separating unit 60 is separated, and the video decoding unit 56 combines the video data and outputs it to the monitor.

  As shown in FIG. 4B, the video data sent from the remote conference apparatus 1a in the normal mode is displayed on the monitor 52b as a video of a constant quality as a whole.

  Further, the video data sent from the remote conference apparatus 1a in the bit priority assignment mode is the face areas [1], [3], [5] of the persons A to C as shown in FIG. Is a high-definition image with a low compression rate, and the body areas [2], [4], and [6] of the persons A to C are high-definition images with a low compression ratio next to the face areas of the persons A to C. The background area [7], which is the other area, is a high-compression and low-quality image that is displayed on the monitor 52b.

  In addition, as shown in FIG. 6, the video data sent from the remote conference apparatus 1a in the limited area transmission mode is video of the entire area [10] of the conference scene shown in FIG. After that, unless a scene change or the like occurs, as shown in FIG. 6 (B), the persons A to C and the surrounding areas [11] to [14] are transmitted as fixed quality images. It will be. When receiving the stream data including the video data of the entire area [10] of the conference scene, the remote conference device 1b displays the video shown in FIG. 6A on the monitor 52b. When the remote conference device 1b receives the stream data including the video data of only the persons A to C and the surrounding area, the area shown in FIG. 6B of the video shown in FIG. [11] to [14] are replaced and displayed on the monitor 52b.

  Here, the background area where the video data is not updated and the person area where the video data is updated may be discontinuous videos. For this reason, the remote conference device 1b includes a low-pass filter (not shown) in the video decoding unit 56. When the person video is synthesized with the entire video of the conference scene, the remote conference device 1b applies to a region near the boundary of the person video. In order to make it difficult to detect the block boundary, a low-pass filter is inserted.

  Next, operation | movement of the remote conference apparatus 1 is demonstrated based on a flowchart. FIG. 9 is a flowchart for explaining the operation of the remote conference apparatus.

  The control unit 14 reads the stored contents of the memory 16 at the time of activation and confirms which mode is the video data transmission mode (s1). When the normal mode is set (s2), the control unit 14 performs video transmission in the normal mode (s3). When the bit priority allocation mode is set (s4), the control unit 14 transmits video in the bit priority allocation mode (s5). When the limited area transmission mode is set (s6), the control unit 14 transmits the video in the limited area transmission mode (s7).

  On the other hand, when the automatic switching mode is set (s8), the control unit 14 performs a calculation based on the current communication state of the network 8 and the ratio of the information amount of the person in the entire video, and the result The mode to be executed is set based on (S9). When the calculation result is greater than or equal to the first threshold A (s10), the control unit 14 transmits the video in the normal mode (s11). When the calculation result is the first threshold value A to the second threshold value B (s12), the control unit 14 performs video transmission in the bit priority assignment mode (s13). In step s12, when the calculation result is less than the second threshold value B, the control unit 14 transmits the video in the limited area transmission mode (s14).

  The control unit 14 performs calculation based on the current state of the network 8 and the characteristics of the video to be transmitted (s15, s17, s19) at regular intervals, that is, every 3 seconds as an example. If there is no change (s16, s18, s20), video transmission is continued in the currently set mode.

  On the other hand, when the calculation result has changed (s16, s18, s20), the process after step s10 is performed.

  As described above, in the remote conference device of the present invention, even if the communication rate decreases or the ratio of the information amount of the person occupying the entire video changes according to the setting or the network state, It is possible to reliably send the target video in the entire image of the meeting scene. In addition, it is possible to improve the image quality of the video at the same transfer rate, and to reduce the bit rate if the quality is the same.

  In the above description, the case where the video of the conference scene is transmitted has been described as an example, but the present invention is not limited to this. Other images may be used.

1 is a perspective overview of a remote conference device according to an embodiment of the present invention, and a diagram showing two remote conference devices connected to a network. FIG. It is a functional block diagram of the remote conference apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of a video coding part. It is a figure which shows the image | video of the normal mode image | photographed with the meeting scenery and video camera which used the remote conference apparatus. It is a figure which shows the image | video image | photographed in the bit priority allocation mode. It is a figure which shows the image | video image | photographed in limited area transmission mode. It is a figure which shows the whole image of the meeting scenery before zooming, and the whole image of the meeting scenery after zooming. It is a block diagram which shows the specific structure of a control part. It is a flowchart for demonstrating operation | movement of a remote conference apparatus.

Explanation of symbols

  1, 1a, 1b-Remote conference device 8-Network 10-Remote control 12-Signal reception unit 14-Control unit 16-Memory 20-Speaker signal processing unit 20-Audio processing unit 26-Speaker array 30-Microphone array 32-Block information Quantity calculator 33-Person area integrator 34-Whole area integrator 35-Divider 36-Mode decision evaluator 37-Table 41-Conference desk 50, 50a, 50b-Video camera 52, 52a, 52b-Monitor 54-Video Encoding unit 56-Video decoding unit 62-Communication unit 64-Input / output connector 72-Screen switching detector 74-Coding controller 76-First changeover switch 78-Discrete cosine transformer 80-Quantizer 81-Coding 82-inverse quantizer 84-inverse discrete cosine transformer 86-additive Calculator 88-Video memory 90-Second changeover switch 92-Subtractor 101-Operation section

Claims (10)

Imaging means for photographing a person and its surroundings and outputting the image;
A person detecting means for detecting a person from the video;
Encoding for periodically extracting the detected person's video from the video, encoding the human video data at a low compression rate, and encoding the video data other than the person at a high compression rate Means,
A video encoding device comprising: Imaging means for photographing a person and its surroundings and outputting the image;
A person detecting means for detecting a person from the video;
Encoding means for encoding the entire area of the video imaged by the imaging means, and thereafter encoding the person video data that is a rectangular area including a person in the video periodically;
A video encoding device comprising: A scene change detecting means for detecting a change in configuration in the entire area of the video imaged by the imaging means;
The video encoding device according to claim 2, wherein the encoding means encodes the entire area of the video at a timing when the scene change detection means detects a change in configuration in the entire area of the video. A video encoding device according to any one of claims 1 to 3,
Communication means for transmitting video data encoded by the video encoding device to a counterpart device;
A video transmission apparatus comprising: Imaging means for photographing a person and its surroundings and outputting the image;
A person detecting means for detecting a person from the video;
A calculation means for performing a predetermined calculation based on the state of the communication rate of the transmission path and the ratio of the information amount of the person in the entire video;
If the calculation result of the calculation means is equal to or greater than a threshold, the detected person's video is periodically extracted from the video taken by the imaging means, and the video data is encoded at a low compression rate. Video data at a high compression rate,
When the calculation result of the calculation means is less than the threshold value, the entire area of the video imaged by the imaging means is encoded, and thereafter the person video data that is a rectangular area including the person in the video is encoded periodically. Encoding means for
Communication means for transmitting the video data encoded by the encoding means to a counterpart device;
A video transmission apparatus comprising: A procedure for periodically detecting a person from an image of the person and its surroundings,
Extracting the detected person's video data and encoding it at a low compression rate, and encoding the video data other than the person at a high compression rate,
A video encoding method comprising: A procedure for periodically detecting a person from an image of the person and its surroundings,
A procedure for encoding the entire area of the person and the surrounding video, and encoding the person video data which is a rectangular area including the person in the video periodically thereafter;
A video encoding method comprising: A procedure for detecting a composition change of the entire area in the person and the surrounding image;
A procedure for encoding the entire area of the video data at a timing when a change in the configuration of the entire area in the video is detected;
The video encoding method according to claim 7, further comprising: After performing the video encoding method according to claim 6,
A procedure for transmitting the encoded video data to a counterpart device;
A video transmission method comprising: A procedure for performing a predetermined calculation based on the state of the communication rate of the transmission path and the ratio of the information amount of the person in the entire video,
When the calculation result is equal to or greater than a threshold value,
A procedure for periodically detecting a person from an image of the person and its surroundings,
Extracting the detected person's video data and encoding it at a low compression rate, and encoding the video data other than the person at a high compression rate,
A procedure for transmitting the encoded person video data and video data other than the person to the counterpart device;
When the calculation result is less than the threshold value,
A procedure for periodically detecting a person from an image of the person and its surroundings,
A procedure for encoding the entire area of the video, and thereafter encoding the person video data which is a rectangular area including the person in the video periodically;
A procedure for transmitting the encoded video data to a counterpart device;
A video transmission method comprising:

Images