JP4222557B2 - Video composition device - Google Patents

Description

  The present invention relates to a video synthesizing apparatus, and more particularly to a video synthesizing apparatus capable of synthesizing a plurality of moving images into one image at high speed and with high accuracy on an encoding region.

  There are two types of video conference systems: a mode in which information such as an image is directly exchanged between terminals without going through a server, and a mode in which information such as an image is exchanged through a server. In a form in which no server is interposed between terminals, it is necessary to connect a line to all other users whenever the number of users increases.

  On the other hand, in the form in which the server exists, even if the number of users increases regardless of the number of users, the system can be configured only by connecting the terminals to the server. Therefore, in general, a video conference system that connects multiple points adopts a form in which a server is interposed.

  In Patent Documents 1 to 3, a transmitted encoded image is once decoded to a pixel area, and the decoded pixel is manipulated and rearranged on one screen, and an image obtained thereby is encoded again. A video conference system is described in which a plurality of encoded images are combined and distributed by sending them out.

Patent Documents 4 to 6 describe elemental technologies of a server that is not a video conference system but does not decode an encoded image up to a pixel area and performs processing including approximation processing on the encoded area. ing.
JP 2003-102009 A JP 2003-163903 A JP 2003-219309 A JP 2000-350207 A JP 2001-103482 A JP 2003-348598 A

  A video conference system in which no server is interposed between terminals is easy to set up the system, but it is necessary to connect a line between a new terminal and all other users' terminals as the number of users increases. Therefore, it is not realistic as a system that connects multiple points of one to one or more.

  In a video conference system in which a server is interposed between terminals, the server not only has a function of redistributing images transmitted from multiple points, but also processes each transmitted image according to the connection environment. It is required to have a function for distributing images.

  For example, smooth video conference progress can be expected by combining images transmitted from multiple points and reconstructing them as a single image. Also, the overhead of control information can be reduced, and the bit rate and frame rate can be adapted. Can also be converted. Furthermore, when combining images transmitted from multiple points into a single image, it is also desirable to enlarge only the speaker or encode only the speaker with high image quality in the combined screen. .

  In the techniques described in Patent Documents 1 to 3, the encoded image that has been transmitted is once decoded up to the pixel area, and is re-arranged on one screen in the pixel area and the reconstructed image is encoded. Therefore, there is a problem that a large delay occurs due to the processing load for decoding the encoded image to the pixel region and the processing load for re-encoding the image synthesized on the pixel region.

  In a video conference system that requires strict real-time performance, a delay of only 0.3 seconds per one way is allowed in order to use the system comfortably. Considering that the delay in the transmission path is unavoidable to some extent, the delay in the server processing must be suppressed as much as possible, and a large delay in the server becomes a fatal problem.

  By applying the techniques described in Patent Documents 4 to 6, it is possible to omit the decoding process and re-encoding process of each encoded image, and to reduce the processing load on the server. However, since the processing using the encoded information is an approximation process here, the image quality obtained is inferior to the image quality obtained by the conventional method of once decoding and processing to the pixel area. Occurs.

  The object of the present invention is to solve the above-mentioned problems of the prior art, and to synthesize a plurality of transform-coded moving images at high speed and with high precision, and to generate one transform-coded image. To provide an apparatus.

  In order to solve the above-described problem, the present invention provides a video synthesizing apparatus that reconstructs a transform-coded image, partially decoding coding information of the transform-coded moving image, and performing quantized prediction. A code information extraction unit for extracting error information and motion prediction information;

The inverse quantization unit that generates the prediction error information by dequantizing the quantized prediction error information extracted by the encoding information extraction unit, the prediction error information generated by the inverse quantization unit, and the code information extraction As a result, the processing equivalent to the inverse motion compensation on the pixel region is collectively executed in the motion compensation unit on the coding region using the motion prediction information extracted by the unit. An inverse motion compensation unit that reconstructs transform coding information by performing computation with a reduced amount of calculation due to product symmetry, and a quantized prediction error by quantizing the prediction error information generated by the motion compensation unit A quantization unit that generates information, and an encoding unit that encodes and outputs the quantized prediction error information generated by the quantization unit and the motion prediction information generated by the motion compensation unit, All of multiple animated images Or it has as its basic features the point that outputs a part thereof as a single transform coded image.

According to the present invention, it is possible to reconstruct a single encoded image by synthesizing it on the encoding region without decoding a plurality of encoded images to the pixel region. As a result, the decoding process up to the pixel region and the re-encoding process can be omitted, so that the processing load can be greatly reduced as compared with the method of synthesizing by manipulating the pixels. Moreover, the synthesis processing of the process, in particular, enables high-speed processing with a reduced amount of calculation by devising the arithmetic processing in the inverse motion compensation and motion compensation. Furthermore, the image quality obtained in comparison with the method of processing information on the coding region as described in Patent Documents 3 to 6 is high image quality, and an image quality equivalent to the method of manipulating pixels is obtained. Can do.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of a video composition apparatus according to the present invention. The video composition apparatus according to the present embodiment includes a code information extraction unit 1, an inverse quantization unit 2, an inverse motion compensation unit 3, a synthesis unit 4, a motion compensation unit 5, a quantization unit 6, and an encoding unit 7.

  First, the code information extraction unit 1 partially decodes coding information of a plurality of moving images compressed by transform coding, thereby predicting motion prediction information (motion vector) and quantized prediction error information (difference DCT coefficient). To extract.

  Note that the extraction process here does not have to be performed on all of the plurality of input moving images, and may be performed on the moving images included in the output image. In addition, it is not necessary to perform extraction processing for all the input moving images. For example, only the portion is included in an image to be output by performing extraction processing on a part of the moving image such as a face portion. Can also be included.

  The motion vector extracted by the code information extraction unit 1 is output to the inverse motion compensation unit 3 and the motion compensation unit 5, and the quantized differential DCT coefficient is output to the inverse quantization unit 2.

  The inverse quantization unit 2 performs inverse quantization on the quantized differential DCT coefficient obtained by the code information extraction unit 1. The differential DCT coefficient including the quantization error generated by the inverse quantization is output to the inverse motion compensation unit 3. The inverse quantization unit 2 can be provided with a function of selecting and outputting a differential DCT coefficient necessary for an output image.

The inverse motion compensation unit 3 includes a frame memory, and the difference between the reference DCT coefficient one frame before obtained from the frame memory, the motion vector extracted by the code information extraction unit 1, and the processing target frame obtained by the inverse quantization unit 2 Inverse motion compensation is performed on the DCT coefficient region using the DCT coefficient (8 × 8 DCT coefficient) to derive a complete DCT coefficient of the processing target frame.

  DCT coefficients for a plurality of images derived by the inverse motion compensation unit 3 are input to the synthesis unit 4. The synthesizer 4 synthesizes the input DCT coefficients as one frame. Note that DCT coefficients necessary for the output image can be selected and combined here.

  The motion compensation unit 5 reuses the motion vector obtained by the code information extraction unit 1 as a motion vector of an output image and performs motion compensation on the DCT coefficient obtained by the synthesis unit 4. The motion compensation on the DCT coefficient region here is an operation reverse to the processing in the inverse motion compensation unit 3. The motion vector obtained by the motion compensation unit 5 is output to the encoding unit 7, and the motion compensated DCT coefficient is output to the quantization unit 6.

  The quantization unit 6 quantizes the DCT coefficient obtained by the motion compensation unit 5 and outputs the quantized DCT coefficient to the encoding unit 7. The encoding unit 7 encodes the quantized DCT coefficient obtained by the quantization unit 6 and the motion vector obtained by the motion compensation unit 5 and outputs the result.

Next, a specific example of processing in the inverse motion compensation unit 3 will be described. FIG. 2 shows a relationship between four sets of difference 8 × 8 DCT coefficients generated by inverse quantization in the inverse quantization unit 2, that is, macroblocks, and reference DCT coefficients of a previous frame referred based on a motion vector. . R00 to R22 are blocks of 8 x 8 DCT coefficients.

When reverse motion compensation is performed, four sets of difference 8 x 8 DCT coefficients for one macroblock may straddle up to nine sets of 8 x 8 DCT coefficients R00 to R22 in the previous frame. An example will be described above in which four sets of differential 8 x 8 DCT coefficients are inversely motion compensated from a motion vector and nine sets of reference 8 x 8 DCT coefficients.

  First, when the horizontal and vertical components of the motion vector are multiples of 8 including 0, one block (DCT coefficient block) overlaps with the block of the previous frame and does not span a plurality of blocks. The reference DCT coefficient of the corresponding place is acquired, and the process ends. In other cases, since it extends over a plurality of blocks, it is necessary to obtain and reconstruct DCT coefficients from those blocks.

If both the horizontal and vertical components of the motion vector are not a multiple of 8, nine sets of reference 8 x 8 DCT coefficients Rm, n including the reference region pointed to by the motion vector are set as one 24 x 24 matrix R as shown in equation (1). To express.

Here, when inverse motion compensation on the pixel region (baseband) for the four sets of differences 8 × 8DCT coefficients D is expressed in matrix form, the 8 × 8DCT coefficient matrix Y for the motion vector (u, v) is It is expressed by equation (2). Note that “0” represents a matrix of all zeros.

Here, Tn represents an n-th order DCT transformation matrix, and the operator t represents a transposition operation. In addition, i and j respectively represent the remainders obtained by dividing the motion vector components u and v by the order of the DCT coefficient. Since the degree of the expression (2) is 8, i and j can take 1 to 7, respectively. The matrices Vi and Hj are 16 x 24 and 24 x 16 shift matrices that change according to the input motion vector to extract 16 x 16 elements. When the motion vector is an integer pixel unit, It can be expressed by equations 3) and (4). Note that using the matrices Vi and Hj individually corresponds to extracting a region of the reference DCT coefficient for each motion vector component.

Here, En represents an n × n unit matrix. Also, 0i and 0j represent a 16 x i 0 matrix and a j x 16 0 matrix, respectively. That is, the 16 × 16 unit matrix E16 in the 16 × 24 matrix Vi is preceded and followed by a remainder of 8 of the horizontal component of the motion vector. Similarly, the 16 x 16 unit matrix E16 in the 24 x 16 matrix Hj goes up and down with a remainder of 8 of the vertical component of the motion vector. In any case, since there are seven ways in which the remainder is not 0, it is possible to calculate and hold in advance.

When the motion vector is a half pixel unit, the matrices V _j and H _i can be expressed by equations (5) and (6), respectively, and when the motion vector is a 1/4 pixel unit, the motion vector It can be expanded according to accuracy.

Here, the half-pixel generation matrix composed of 0, 1/2, and 1 in the equation (5) is a 16 × 24 matrix, and the half-pixel generation matrix composed of 0, 1/2, and 1 in the equation (6) is 24 × 16 matrix. In any case, all but the DCT coefficient matrix R in the equation (2) are constant matrices, and the number of operations can be reduced by calculating and storing them in advance.

Furthermore, when the product of the constant matrix of equation (2) is obtained, symmetry appears as in equation (7). Considering this symmetry, the amount of calculation in the backward motion compensation can be reduced. Here, Ai, A′i, Aj, and A′j represent an 8 × 8 matrix. Ai and A′i can be calculated and stored in advance as horizontal component conversion tables and Aj and A′j as vertical component conversion tables. These conversion tables are stored in accordance with the accuracy of the motion vector. change.

  When the above equation (7) is applied to the above equation (2), the DCT coefficient matrix Y subjected to inverse motion compensation in the DCT coefficient region is expressed by the following equation (8).

When equation (1) is applied to R and equation (8) is expanded, equation (9) is derived. The formulas in parentheses in formula (9) have a common part represented by formulas (10) and (11), and are calculated by precalculating and holding formulas (10) and (11). The amount can be reduced. Further, since the resolution conversion matrix is only the matrix A _i , A ′ _i , A _j , A ′ _j , the amount of memory required for the storage can be reduced.

In general, there is a close relationship between the sub-matrices A and A ′ (A _i and A ′ _i or A _j and A ′ _j ), and the calculation is performed by considering the symmetry of the sub-elements of both. The amount can be reduced. For example, when i = 1 and i = 2, the relationship between A ₁ and A ′ ₁ and A ₂ and A ′ ₂ is shown in FIGS. In the figure, “+” is a place where the elements completely match between A _i and A ′ _i , “−” is a place where the sign is reversed, “” (no mark) is a completely different place, and “0” is the place where the element is 0 0 ”.

Here, 'a matrix obtained by extracting only the elements that match exactly the _i A' A _i and A _i, although the absolute value equal to define a matrix sign is extracted by an inverse element A _"i and, A _i the remaining elements consist of matrix 'a _i, a' matrix consists remaining elements of _i "when defined as a _i, the matrix a _i and the matrix a _'i can be rewritten as (12). Equation (12) calculates A ′ _i , A ” _i , 'A _i , and“ A _i by decomposing the matrix A _i and the matrix A ′ _i into a plurality according to the symmetry of A _i. it can.

  Substituting (12) into (10) yields (13).

At first glance, equation (13) shows that the matrix product was originally increased from 2 to 4, but A _i ', A " _i ,' A _i, and 'A _i are all sparse matrices (zero elements). Since this is a matrix containing a large number of For example, when i = 1, there are 12 non-zero elements in the matrix A ′ _i composed only of elements that completely match A _i and A ′ _i , and the remaining 52 are 0. Similarly, the matrix A " _i , 'A _i ," A _i has only non-zero elements of 21, 31, and 31, respectively. At this time, the number of multiplications necessary for the calculation of the expression (10) is 1024, whereas the number of multiplications required for the calculation of the expression (13) is 760, and the calculation amount can be reduced by about 25%. In the above description, the calculation example of the expression (10) has been described, but the calculation amount can be reduced in the same way for the calculation of the expression (9) and the expression (11). Thus, according to the equation (13), the calculation of the common terms by using the symmetry of A ′ _i , A ” _i , 'A _i ,“ A _i between the matrix A _i and the matrix A ′ _i. Can be reduced and the amount of calculation can be reduced.

  Further, as shown in FIGS. 3A and 3B, the distribution of symmetry differs depending on i, and i has an even number more than an odd number. Reduction effect is great. Therefore, when i and j have different numerical values, the entire calculation amount can be reduced by performing extraction processing from components having a large factor of 2.

  Although the amount of calculation can be reduced by the above process, the above process is equivalent to the inverse motion compensation on the pixel area, and as a result, it is an equivalent process. The image quality equivalent to the compensation method can be maintained. On the contrary, in the inverse motion compensation on the pixel area, it is necessary to store the result rounded to an integer for each processing such as inverse DCT transform and inverse motion compensation. Since processing is performed in a lump and there are few rounding operations, image quality degradation is minimized.

Further, when one of the horizontal component and the vertical component of the motion vector is a multiple of 8 and the other is not a multiple of 8, the processing can be omitted for the multiple of 8 components. For example, when the horizontal component is a multiple of 8, the matrix R is set to 24 × 16 matrix, and conversely, when the vertical component is a multiple of 8, the matrix R is set to 16 × 24 matrix to reduce the amount of calculation. it can.

  The processing in the motion compensation unit 5, that is, motion compensation on the DCT coefficient region, can be realized by using a conversion table or the like and performing an operation reverse to the above-described inverse motion compensation in accordance with the output image.

As mentioned above, although embodiment was described, this invention is not limited to the said embodiment, It can change variously. For example, when the composition unit 4 performs composition, the low-frequency component of the 8 × 8 DCT coefficient is extracted and reduced according to the frame size of the output image, the composition ratio of the frame size with respect to a plurality of input images, or the like. It is also possible to enlarge by inserting 0 in the high frequency component of the 8 × 8 DCT coefficient.

  In addition, when any enlargement / reduction processing is performed on the frame size of the input image as described above, the motion vector and the DCT coefficient must be matched with each other. For example, when reduction processing is performed, one motion vector is estimated from a plurality of motion vectors for each coding unit, and motion compensation is performed on the DCT coefficient region for low-order DCT coefficients. In this case, motion vector estimation is based on the weighted average weighted by the median value of the motion vector in the area occupied by the coding unit, the activity of the coding unit, or the area occupied by the coding unit. This can be achieved by calculating the average.

Further, since only the low-order DCT coefficients are extracted when the reduction process is performed, the order n of the DCT transformation matrix T _n used for the process in the motion compensation unit 5 is set to the order corresponding to the reduction rate. It is necessary to replace with. For example, when the frame size is reduced to ½, a fourth-order DCT transformation matrix T ₄ is used. At this time, it is sufficient to hold only three types of shift matrices of 1, 2, and 3 in the vertical and horizontal directions. Also, the DCT coefficient after motion compensation is converted from a low-order DCT coefficient to a DCT coefficient of the order used for the output image. For this conversion, the base conversion technique described in the previous application (Japanese Patent Application No. 2003-372223) can be used.

  The present invention is not limited to DCT coefficients and can be similarly applied to transform coding information transformed by other transform coding schemes, and can synthesize an image at high speed and with high accuracy on a coding region. Therefore, the present invention can be effectively applied to a system that processes and distributes a plurality of images such as a multipoint video conference system.

1 is a block diagram showing an embodiment of a video composition device according to the present invention. It is explanatory drawing which shows the relationship between the difference DCT coefficient used for a reverse motion compensation process, and a reference DCT coefficient. It is explanatory drawing of the partial matrix in a reverse motion compensation process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Encoding information extraction part, 2 ... Inverse quantization part, 3 ... Inverse motion compensation part, 4 ... Synthesis | combination part, 5 ... Motion compensation part, 6 ... Quantization part , 7 ... Encoding unit

Claims (17)

In a video composition device for reconstructing a transform-coded image,
A code information extraction unit that partially decodes coding information of the transform-coded moving image and extracts quantized prediction error information and motion prediction information;
A dequantization unit that dequantizes the quantized prediction error information extracted by the encoded information extraction unit to generate prediction error information;
Using the prediction error information generated by the inverse quantization unit and the motion prediction information extracted by the code information extraction unit, a process equivalent to inverse motion compensation on the pixel region is performed as a result on the encoding region. An inverse motion compensator that performs batch execution in motion compensation units, performs calculation with a calculation amount reduced by symmetry of a matrix product that becomes a constant matrix at the time of execution, and reconstructs transform coding information;
A combining unit that combines a plurality of transform coding information reconstructed by the inverse motion compensation unit into one transform coding information on a coding region;
Motion compensation for generating prediction error information and motion prediction information by performing motion compensation on a coding region using the transform coding information synthesized by the synthesis unit and the motion prediction information extracted by the coding information extraction unit And
A quantization unit that quantizes the prediction error information generated by the motion compensation unit to generate quantized prediction error information;
An encoding unit that encodes and outputs the quantized prediction error information generated by the quantization unit and the motion prediction information generated by the motion compensation unit;
A video synthesizing apparatus that outputs all or part of a plurality of transform-coded moving images as one transform-encoded image. The code information extraction unit partially decodes all or a part of the encoded information according to the output image from the transform-coded moving image, and obtains the quantized prediction error information and the motion prediction information. The video composition device according to claim 1, wherein the video composition device is extracted. The video synthesizing apparatus according to claim 1, wherein the inverse quantization unit performs inverse quantization on prediction error information necessary for an output image. The video synthesizing apparatus according to claim 1, wherein the inverse motion compensation unit reconstructs transform coding information on a coding area from motion prediction information, prediction error information, and transform coding information to be referred to. 5. The video composition apparatus according to claim 4, wherein the inverse motion compensation unit uses a different conversion table for each component of motion prediction information and for each numerical value that motion prediction information can take. The video synthesizing apparatus according to claim 5 , wherein the inverse motion compensation unit extracts a region of transform coding information to be referred to for each component of motion prediction information. The inverse motion compensation section, according to claim 6, characterized in that upon extraction of the region of the transform coding information to be referred to each component of the motion prediction information, the extraction from the components of a factor of 2 is large motion prediction information Video synthesizer. 6. The video composition apparatus according to claim 5 , wherein the conversion table used in the inverse motion compensation unit can be expanded according to the accuracy of motion prediction information. The video composition apparatus according to claim 5 , wherein the conversion table used in the inverse motion compensation unit is calculated and stored in advance. 6. The video according to claim 5 , wherein the inverse motion compensation unit is configured to reduce an overall calculation amount in the inverse motion compensation by calculating in advance a common term in a calculation process. Synthesizer. The inverse motion compensation section, according to claim 5, characterized in that the calculation amount of the entire is configured to be reduced by calculating in consideration of the local symmetry in subelements conversion table Video synthesizer. 12. The video composition apparatus according to claim 11 , wherein the inverse motion compensation unit decomposes and uses the conversion table according to local symmetry in the subelements of the conversion table. The inverse motion compensation unit is configured to reduce the number of multiplications by using a local symmetry in a subelement of a conversion table and calculating a common term collectively. 12. The video composition device according to 12 . The video according to claim 1, wherein the motion compensation unit performs motion compensation on a coding region from motion prediction information, transform coding information, and reference transform coding information to generate prediction error information. Synthesizer. The video synthesis apparatus according to claim 14 , wherein the motion compensation unit performs inverse operation of inverse motion compensation. The video synthesizing apparatus according to claim 14 , wherein the motion compensation unit uses a conversion table for motion compensation processing and reconstructs motion prediction information according to an output image. The video synthesizing apparatus according to claim 1, wherein the synthesizing unit synthesizes prediction error information necessary for an output image.

Images